feat(tts): StepFun voice selection via CharacterDesigner + provider-aware beat-audio
Make homepage cards and live sessions produce sound when the server is configured for StepFun TTS, instead of silently failing (the prebaked Xiaomi voice was useless on a StepFun server, and wasted ~220KB/beat in Fast Origin Transfer). Three coordinated changes: 1. CharacterDesigner now picks a StepFun preset voice id directly from the 32-entry catalog in the SAME LLM call that designs the character — zero extra latency, LLM-grade match quality. The Xiaomi prompt path is byte-identical to history (verified programmatically) so cache hit rate and voice quality are preserved. pickStepfunVoiceId (keyword scorer) remains the fallback for orphan speakers / invalid LLM picks. 2. The 32-preset catalog moves to lib/tts-client/stepfun-voices.json as the single source of truth, shared by the scorer, the CharacterDesigner prompt, /api/tts-provider, and the offline enrich script. 3. A new GET /api/tts-provider endpoint lets the client probe the server's TTS provider at /play mount. fetchBeatAudio then shapes its request body: on a StepFun server it sends the lightweight stepfunVoiceId / voiceDescription and omits the ~220KB Xiaomi reference audio (FOT saving ~13MB per protagonist per session on prebaked cards). requestBeatAudio re-provisions on a provider mismatch before synth, so audio never goes silent on a cross-provider replay or mid-session provider flip. New type fields are all optional and backward-compatible: Character.stepfunVoiceId, BeatAudioRequest.voiceDescription/characterName/stepfunVoiceId, voice made optional. AGENTS.md updated for the new route, type fields, dependency map, and StepFun voice-selection flow.
This commit is contained in:
+52
-2
@@ -7,6 +7,7 @@ import type {
|
||||
StoryState,
|
||||
WriterPlan,
|
||||
} from "@infiplot/types";
|
||||
import { formatStepfunCatalogForPrompt } from "@infiplot/tts-client";
|
||||
|
||||
// ══════════════════════════════════════════════════════════════════════
|
||||
// Multi-agent scene generation pipeline:
|
||||
@@ -599,7 +600,14 @@ function collectPriorSceneKeys(session: Session): string[] {
|
||||
// (e.g., gentle-looking character with energetic voice).
|
||||
// ──────────────────────────────────────────────────────────────────────
|
||||
|
||||
export const CHARACTER_DESIGNER_SYSTEM = `你是视觉小说的「角色设定师」。给你一个**新登场角色的名字**,你要为这个角色同时设计两份卡片:
|
||||
// CHARACTER_DESIGNER_SYSTEM is split into a provider-agnostic CORE (visual +
|
||||
// voice-text rules) and a provider-specific TAIL (the JSON contract). When the
|
||||
// server runs StepFun, the tail additionally asks the model to pick a preset
|
||||
// voice id from the 32-entry catalog — so the SAME LLM call that designs the
|
||||
// character also selects its voice, at zero extra latency. When StepFun is
|
||||
// off (Xiaomi / no TTS), the tail is byte-identical to the historical prompt
|
||||
// (Xiaomi path is cache- and behavior-preserving).
|
||||
const CHARACTER_DESIGNER_SYSTEM_CORE = `你是视觉小说的「角色设定师」。给你一个**新登场角色的名字**,你要为这个角色同时设计两份卡片:
|
||||
1. **视觉设定卡(英文)**——给生图模型 FLUX 用,遵循 prompt engineering 风格
|
||||
2. **音色设定卡(中文)**——给小米 MiMo 配音设计用
|
||||
|
||||
@@ -652,7 +660,12 @@ export const CHARACTER_DESIGNER_SYSTEM = `你是视觉小说的「角色设定
|
||||
- 随后描述:年龄段(如「约17岁少女」「30 出头男性」)、音色质感、性格情绪基调、语速节奏、人设腔调、口音方言
|
||||
- 用中文,整段连续描述,不分段
|
||||
- 长度:50–80 个中文字为宜
|
||||
- 例:"女性,约17岁少女,音色清亮带点稚嫩甜美,性格开朗外向但容易害羞,语速偏快,标准普通话"
|
||||
- 例:"女性,约17岁少女,音色清亮带点稚嫩甜美,性格开朗外向但容易害羞,语速偏快,标准普通话"`;
|
||||
|
||||
// JSON-contract tail for the NON-stepfun path (Xiaomi voicedesign / no TTS).
|
||||
// Byte-identical to the historical prompt so the Xiaomi path keeps its cache
|
||||
// hit rate and voice quality unchanged.
|
||||
const CHARACTER_DESIGNER_TAIL_DEFAULT = `
|
||||
|
||||
必须输出严格 JSON:
|
||||
{
|
||||
@@ -662,6 +675,43 @@ export const CHARACTER_DESIGNER_SYSTEM = `你是视觉小说的「角色设定
|
||||
|
||||
不要输出 JSON 以外的任何文本。`;
|
||||
|
||||
// JSON-contract tail for the StepFun path. Same core output, plus the model
|
||||
// picks a preset voice id from the catalog. The id must match the SAME person
|
||||
// the voiceDescription describes (gender / age / vibe) — designed together so
|
||||
// appearance and voice stay coherent (the same invariant the CORE enforces).
|
||||
const CHARACTER_DESIGNER_TAIL_STEPFUN = `
|
||||
|
||||
**StepFun 预设音色选择(必做):**
|
||||
除 voiceDescription 外,你还必须从下列 StepFun 预设音色清单中,为本角色挑选一个与 voiceDescription 描绘的「同一个人」(性别 / 年龄段 / 气质都要一致)最贴合的预设,并把它的 id 填入 stepfunVoiceId。清单:
|
||||
${formatStepfunCatalogForPrompt()}
|
||||
|
||||
挑选原则:
|
||||
- stepfunVoiceId 必须是上表里某个 id,原样复制(拼写、大小写、连字符都不能变)。
|
||||
- 必须与 voiceDescription 的性别一致(男声选 male 行,女声选 female 行)。
|
||||
- 年龄段尽量一致;拿不准时优先气质匹配(例如“冷艳御姐”选 lengyanyujie、“软萌萝莉”选 ruanmengnvsheng)。
|
||||
- 不允许编造清单外的 id,也不允许留空。
|
||||
|
||||
必须输出严格 JSON:
|
||||
{
|
||||
"visualDescription": "English visual card, comma-separated tags...",
|
||||
"voiceDescription": "中文音色卡,以性别开头...",
|
||||
"stepfunVoiceId": "清单内某个 id"
|
||||
}
|
||||
|
||||
不要输出 JSON 以外的任何文本。`;
|
||||
|
||||
/** Build the CharacterDesigner system prompt, provider-aware.
|
||||
* - stepfun:false → identical to the historical Xiaomi/no-TTS prompt.
|
||||
* - stepfun:true → additionally asks the model to pick a StepFun preset
|
||||
* voice id from the 32-entry catalog (see formatStepfunCatalogForPrompt). */
|
||||
export function buildCharacterDesignerSystem(opts: {
|
||||
stepfun: boolean;
|
||||
}): string {
|
||||
return opts.stepfun
|
||||
? CHARACTER_DESIGNER_SYSTEM_CORE + CHARACTER_DESIGNER_TAIL_STEPFUN
|
||||
: CHARACTER_DESIGNER_SYSTEM_CORE + CHARACTER_DESIGNER_TAIL_DEFAULT;
|
||||
}
|
||||
|
||||
export function buildCharacterDesignerUserMessage(
|
||||
charName: string,
|
||||
session: Session,
|
||||
|
||||
Reference in New Issue
Block a user