Voice, Video & Multimodal Frontiers

Realtime voice models changed what a system prompt has to carry. When the model speaks and listens live — OpenAI's Realtime voices, Gemini Live, and their peers — latency and turn-taking become part of the prompt's job. You specify persona and speaking style (pace, warmth, brevity), how to handle interruptions and barge-in, when to pause versus keep talking, and how to pronounce names or jargon. You also design for being cut off: an instruction like 'answer in one or two sentences unless asked for more' matters far more in voice than in text, because nobody wants to hear a model read three paragraphs aloud. Voice prompting is prompting for the ear and the clock, not the page.

Video understanding is now a routine input. Gemini and other long-context multimodal models ingest video and let you prompt over it — at what timestamp does the speaker mention pricing, summarize each scene with its time range, flag every frame showing a safety violation. The prompting moves that work: ask for timestamps so answers are checkable, request per-segment structure rather than one blob, and tell the model to say when something isn't shown rather than inferring. Treat frames and transcript as grounded evidence the way you treat retrieved chunks — the same cite-what-you-see and admit-what-you-can't discipline from the vision lesson carries straight over.

Video generation prompts are their own craft, and 2026's tools — Google's Veo 3, OpenAI's Sora 2, and others — reward describing a shot like a director, not a still. Beyond subject and style you specify camera movement (slow dolly-in, handheld pan), timing and pacing across the clip's seconds, what changes over time, and increasingly audio and dialogue, since Veo generates synchronized sound. A useful structure: set the scene, then the action beat by beat, then the camera and lens, then the mood and audio. Vague motion prompts produce drifting, incoherent clips; specific, sequenced direction produces something that holds together as a real shot.

The real frontier is cross-modal chains, where the output of one modality becomes the input to the next. You might transcribe and diarize a meeting recording, summarize it to action items as structured JSON, then generate a spoken recap in a chosen voice — three models, one pipeline. Or turn a product photo into a description, then into a short promo-video prompt. Each hop is a place errors compound, so the same practices carry through: contracts between stages, grounding at every step, validation at the boundaries, and a test set for the chain end to end. Multimodal doesn't retire the fundamentals — it multiplies where they apply.