infiplot/infiplot-web
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

feat(tts): StepFun voice selection via CharacterDesigner + provider-aware beat-audio

Browse Source 
Make homepage cards and live sessions produce sound when the server is
configured for StepFun TTS, instead of silently failing (the prebaked
Xiaomi voice was useless on a StepFun server, and wasted ~220KB/beat in
Fast Origin Transfer).

Three coordinated changes:

1. CharacterDesigner now picks a StepFun preset voice id directly from the
   32-entry catalog in the SAME LLM call that designs the character — zero
   extra latency, LLM-grade match quality. The Xiaomi prompt path is
   byte-identical to history (verified programmatically) so cache hit rate
   and voice quality are preserved. pickStepfunVoiceId (keyword scorer)
   remains the fallback for orphan speakers / invalid LLM picks.

2. The 32-preset catalog moves to lib/tts-client/stepfun-voices.json as the
   single source of truth, shared by the scorer, the CharacterDesigner
   prompt, /api/tts-provider, and the offline enrich script.

3. A new GET /api/tts-provider endpoint lets the client probe the server's
   TTS provider at /play mount. fetchBeatAudio then shapes its request body:
   on a StepFun server it sends the lightweight stepfunVoiceId /
   voiceDescription and omits the ~220KB Xiaomi reference audio (FOT saving
   ~13MB per protagonist per session on prebaked cards). requestBeatAudio
   re-provisions on a provider mismatch before synth, so audio never goes
   silent on a cross-provider replay or mid-session provider flip.

New type fields are all optional and backward-compatible: Character.stepfunVoiceId,
BeatAudioRequest.voiceDescription/characterName/stepfunVoiceId, voice made
optional. AGENTS.md updated for the new route, type fields, dependency map,
and StepFun voice-selection flow.
This commit is contained in:
yuanzonghao
2026-06-15 12:49:25 +08:00
parent da191dd7a2
commit ca73a41a0b
15 changed files with 754 additions and 90 deletions
Download Patch File Download Diff File Expand all files Collapse all files
app/api/beat-audio/route.ts
+11 -10
View File
@@ -17,18 +17,19 @@ export async function POST(req: Request) {
return NextResponse.json({ error: "Invalid JSON" }, { status: 400 });
}
// Accept either provider's voice shape — xiaomi carries referenceAudioBase64,
// stepfun carries voiceId. We only check the discriminator + the line text;
// shape-specific validation lives in each provider's synth function.
// Voice is now optional — when the server runs StepFun, the client omits
// the ~220KB Xiaomi reference audio and sends stepfunVoiceId /
// voiceDescription instead (saves Fast Origin Transfer bandwidth). The
// engine's resolveVoice re-provisions on a provider mismatch. We only
// require the beat text + SOMETHING to synthesize from.
const VALID_TTS_PROVIDERS = ["xiaomi", "stepfun"];
if (
!body.beat?.id ||
!body.beat?.line ||
!body.voice?.provider ||
!VALID_TTS_PROVIDERS.includes(body.voice.provider)
) {
const hasVoice =
!!body.voice?.provider && VALID_TTS_PROVIDERS.includes(body.voice.provider);
const hasFallback =
!!body.stepfunVoiceId || !!body.voiceDescription;
if (!body.beat?.id || !body.beat?.line || (!hasVoice && !hasFallback)) {
return NextResponse.json(
{ error: "beat.id, beat.line and voice.provider (xiaomi|stepfun) are required" },
{ error: "beat.id and beat.line are required, plus either voice.provider (xiaomi|stepfun) or stepfunVoiceId/voiceDescription" },
{ status: 400 },
);
}