feat: prefetch, vision split, provider adapter, UI polish
Engine - Split /api/vision out from /api/interact so client can drive prefetch + cache lookup independently of click interpretation - Image client switched to chat-completions+modalities API (OpenRouter/ provider style), supporting markdown image URL responses - annotateClick now resizes to 768w before composite to keep vision payloads small and avoid CDN timeouts - Prompts updated to mention "JSON" in user messages (required by Gemini's strict JSON mode) - Shared fetchWithRetry helper: 2 retries for chat/image, 0 for vision (with 60s hard timeout) Client - Parallel prefetch of all three choice branches on each new frame - Effect deliberately excludes phase from deps so user-click doesn't abort in-flight prefetches - Cache hit/miss/free-form fallback handled in handleClick - PlayCanvas reads img naturalWidth/Height and adapts container to whatever aspect AI returns (no more cropped third choice) - max-width raised to 560px, max-height calc(100dvh - 200px) Misc - README env-path corrected to apps/web/.env.local - users.md: BGM/TTS idea note - .env.example moved into apps/web alongside next config Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This commit is contained in:
@@ -60,12 +60,22 @@ export type StartResponse = {
|
||||
imageBase64: string;
|
||||
};
|
||||
|
||||
export type InteractRequest = {
|
||||
export type VisionRequest = {
|
||||
session: Session;
|
||||
prevImageBase64: string;
|
||||
click: { x: number; y: number };
|
||||
};
|
||||
|
||||
export type VisionResponse = {
|
||||
intent: ClickIntent;
|
||||
};
|
||||
|
||||
export type InteractRequest = {
|
||||
session: Session;
|
||||
intent: ClickIntent;
|
||||
click?: { x: number; y: number };
|
||||
};
|
||||
|
||||
export type InteractResponse = {
|
||||
session: Session;
|
||||
frame: StoryFrame;
|
||||
|
||||
Reference in New Issue
Block a user