feat(tts): StepFun voice selection via CharacterDesigner + provider-aware beat-audio

Make homepage cards and live sessions produce sound when the server is
configured for StepFun TTS, instead of silently failing (the prebaked
Xiaomi voice was useless on a StepFun server, and wasted ~220KB/beat in
Fast Origin Transfer).

Three coordinated changes:

1. CharacterDesigner now picks a StepFun preset voice id directly from the
   32-entry catalog in the SAME LLM call that designs the character — zero
   extra latency, LLM-grade match quality. The Xiaomi prompt path is
   byte-identical to history (verified programmatically) so cache hit rate
   and voice quality are preserved. pickStepfunVoiceId (keyword scorer)
   remains the fallback for orphan speakers / invalid LLM picks.

2. The 32-preset catalog moves to lib/tts-client/stepfun-voices.json as the
   single source of truth, shared by the scorer, the CharacterDesigner
   prompt, /api/tts-provider, and the offline enrich script.

3. A new GET /api/tts-provider endpoint lets the client probe the server's
   TTS provider at /play mount. fetchBeatAudio then shapes its request body:
   on a StepFun server it sends the lightweight stepfunVoiceId /
   voiceDescription and omits the ~220KB Xiaomi reference audio (FOT saving
   ~13MB per protagonist per session on prebaked cards). requestBeatAudio
   re-provisions on a provider mismatch before synth, so audio never goes
   silent on a cross-provider replay or mid-session provider flip.

New type fields are all optional and backward-compatible: Character.stepfunVoiceId,
BeatAudioRequest.voiceDescription/characterName/stepfunVoiceId, voice made
optional. AGENTS.md updated for the new route, type fields, dependency map,
and StepFun voice-selection flow.
This commit is contained in:
yuanzonghao
2026-06-15 12:49:25 +08:00
parent da191dd7a2
commit ca73a41a0b
15 changed files with 754 additions and 90 deletions
+52 -2
View File
@@ -7,6 +7,7 @@ import type {
StoryState,
WriterPlan,
} from "@infiplot/types";
import { formatStepfunCatalogForPrompt } from "@infiplot/tts-client";
// ══════════════════════════════════════════════════════════════════════
// Multi-agent scene generation pipeline:
@@ -599,7 +600,14 @@ function collectPriorSceneKeys(session: Session): string[] {
// (e.g., gentle-looking character with energetic voice).
// ──────────────────────────────────────────────────────────────────────
export const CHARACTER_DESIGNER_SYSTEM = `你是视觉小说的「角色设定师」。给你一个**新登场角色的名字**,你要为这个角色同时设计两份卡片:
// CHARACTER_DESIGNER_SYSTEM is split into a provider-agnostic CORE (visual +
// voice-text rules) and a provider-specific TAIL (the JSON contract). When the
// server runs StepFun, the tail additionally asks the model to pick a preset
// voice id from the 32-entry catalog — so the SAME LLM call that designs the
// character also selects its voice, at zero extra latency. When StepFun is
// off (Xiaomi / no TTS), the tail is byte-identical to the historical prompt
// (Xiaomi path is cache- and behavior-preserving).
const CHARACTER_DESIGNER_SYSTEM_CORE = `你是视觉小说的「角色设定师」。给你一个**新登场角色的名字**,你要为这个角色同时设计两份卡片:
1. **视觉设定卡(英文)**——给生图模型 FLUX 用,遵循 prompt engineering 风格
2. **音色设定卡(中文)**——给小米 MiMo 配音设计用
@@ -652,7 +660,12 @@ export const CHARACTER_DESIGNER_SYSTEM = `你是视觉小说的「角色设定
- 随后描述:年龄段(如「约17岁少女」「30 出头男性」)、音色质感、性格情绪基调、语速节奏、人设腔调、口音方言
- 用中文,整段连续描述,不分段
- 长度:5080 个中文字为宜
- 例:"女性,约17岁少女,音色清亮带点稚嫩甜美,性格开朗外向但容易害羞,语速偏快,标准普通话"
- 例:"女性,约17岁少女,音色清亮带点稚嫩甜美,性格开朗外向但容易害羞,语速偏快,标准普通话"`;
// JSON-contract tail for the NON-stepfun path (Xiaomi voicedesign / no TTS).
// Byte-identical to the historical prompt so the Xiaomi path keeps its cache
// hit rate and voice quality unchanged.
const CHARACTER_DESIGNER_TAIL_DEFAULT = `
必须输出严格 JSON
{
@@ -662,6 +675,43 @@ export const CHARACTER_DESIGNER_SYSTEM = `你是视觉小说的「角色设定
不要输出 JSON 以外的任何文本。`;
// JSON-contract tail for the StepFun path. Same core output, plus the model
// picks a preset voice id from the catalog. The id must match the SAME person
// the voiceDescription describes (gender / age / vibe) — designed together so
// appearance and voice stay coherent (the same invariant the CORE enforces).
const CHARACTER_DESIGNER_TAIL_STEPFUN = `
**StepFun 预设音色选择(必做):**
除 voiceDescription 外,你还必须从下列 StepFun 预设音色清单中,为本角色挑选一个与 voiceDescription 描绘的「同一个人」(性别 / 年龄段 / 气质都要一致)最贴合的预设,并把它的 id 填入 stepfunVoiceId。清单:
${formatStepfunCatalogForPrompt()}
挑选原则:
- stepfunVoiceId 必须是上表里某个 id,原样复制(拼写、大小写、连字符都不能变)。
- 必须与 voiceDescription 的性别一致(男声选 male 行,女声选 female 行)。
- 年龄段尽量一致;拿不准时优先气质匹配(例如“冷艳御姐”选 lengyanyujie、“软萌萝莉”选 ruanmengnvsheng)。
- 不允许编造清单外的 id,也不允许留空。
必须输出严格 JSON
{
"visualDescription": "English visual card, comma-separated tags...",
"voiceDescription": "中文音色卡,以性别开头...",
"stepfunVoiceId": "清单内某个 id"
}
不要输出 JSON 以外的任何文本。`;
/** Build the CharacterDesigner system prompt, provider-aware.
* - stepfun:false → identical to the historical Xiaomi/no-TTS prompt.
* - stepfun:true → additionally asks the model to pick a StepFun preset
* voice id from the 32-entry catalog (see formatStepfunCatalogForPrompt). */
export function buildCharacterDesignerSystem(opts: {
stepfun: boolean;
}): string {
return opts.stepfun
? CHARACTER_DESIGNER_SYSTEM_CORE + CHARACTER_DESIGNER_TAIL_STEPFUN
: CHARACTER_DESIGNER_SYSTEM_CORE + CHARACTER_DESIGNER_TAIL_DEFAULT;
}
export function buildCharacterDesignerUserMessage(
charName: string,
session: Session,