Audio and speech

Dialogue, TTS, voice, cleanup, separation, translation and localization

• articleHow do I add a YouTube audio description track, and what must already exist before I can upload one?
  Problem: delivering a narration track that describes on-screen action for blind or low-vision viewers on YouTube. Conditions: Advanced features access; YouTube Studio on computer > Languages > video > language. Method: the language's original or dubbed audio track must be uploaded first; then, next to Descriptive audio, Add > Select file > Publish. The file must be a supported audio-only format and roughly the same length as the video. To replace a track, delete the descriptive audio for that language and upload again. YouTube describes the track as narration placed in natural pauses in the audio. Failure modes: no descriptive slot appears if the base-language audio track is missing; an audio file that is much shorter or longer than the video. Limitations: this page gives no writing guidance (tense, density, extended description), which earlier candidates cover from ACB/DCMP/Netflix/Ofcom; supported file types are not listed inline; availability is gated. Version/date: help page rea...
• articleWhy is dialogue usually mixed to the centre even when the actor is off to one side, and what does that imply for placing generated dialogue and effects in stereo or surround?
  Problem: deciding where to pan AI-voiced or dubbed dialogue versus effects and music. Conditions: stereo or surround mixes with picture. Method (Murch's observation): in almost all stereo films of the previous 25 years dialogue is placed in the centre regardless of the actor's screen position, yet audiences and mixers still believe the voices come from the actors; effects such as car pass-bys get full stereo or multichannel treatment. He explains this by saying that while the brain is busy decoding language it lets the image 'steer' the apparent source, whereas the less linguistic a sound is, the more its spatial location matters. General rule he states: the warmer (more embodied) the sound the more it gets a stereo/multitrack treatment; the cooler (more encoded) the more it is placed in the centre. Early 1950s experiments moving dialogue around the screen were abandoned as 'artificial'. Use (inference): keep AI dialogue and dub lines centred (or head-locked in 360/VR, see the spati...
• articleWhat are Walter Murch's 'encoded' and 'embodied' sound, and what does the distinction imply for how dialogue, music and effects should be layered?
  Problem: choose how to balance speech, music and effects without a muddy mix. Conditions: any narrative sound design, including AI-generated beds. Method (Murch, 2005 lecture essay): imagine a sound spectrum whose ends are not frequencies but kinds of meaning. Encoded sound is at one end, with speech as the clearest example, because language is a code the listener must decode. Embodied sound is at the other end, with music as the clearest example, because its meaning is carried directly by the sound without a code. Effects sit in between and are 'warmer' (more embodied) or 'cooler' (more encoded) depending on how code-like they are. Use (inference): when planning layers, classify each element (dialogue, narration, signage-like sounds as encoded; score and abstract drones as embodied; footsteps, doors, ambiences between) and avoid stacking several elements from the same end of the spectrum at once. Failure modes: treating the spectrum as literal frequency; using it as a hard rule (Mu...
• articleWhat is Murch's Law of Two-and-a-Half and how many simultaneous sound layers can a mix carry?
  Problem: AI sound-design tools make it easy to pile on layers, which turns a scene into noise. Conditions: a mix with several simultaneous elements. Method (Murch): the mix problem he met on Apocalypse Now was that six well-justified premixes collapsed into a 'big ball of noise' when played together. His rule from THX 1138 footsteps: one robot's steps must be in sync, two robots' too, but with three the ear hears a group (a new singularity). The law of two-and-a-half applies to sounds of the same 'colour', i.e. from the same part of the encoded-embodied spectrum: you can follow two conversations plus a half (a third, background one, as in the Godfather scene he cites), or two pieces of music with care. Spread the layers across the spectrum (dialogue, music, effects) and he reports you can manage about five layers rather than two-and-a-half. If layers pile up in one region the limit shrinks back to two-and-a-half, and if you want every word understood in multiple dialogue layers, eli...
• articleHow do I add captions to a YouTube video (file upload, auto-sync, typing) and what limits does YouTube state for auto-sync and automatic captions?
  Problem: getting accurate, timed captions on a published video. Conditions: YouTube Studio > Subtitles, per language. Method: add the language, then choose Upload file (with timing or without timing), Auto-sync (paste or upload the transcript and YouTube aligns it), Type manually, or rely on automatic speech-recognition captions. The page tells creators to include sound cues such as [applause] so viewers know what is happening. Auto-sync needs a transcript in the same language as the speech and in a language supported by YouTube's speech recognition; transcripts are not recommended for videos over an hour or with poor audio. Automatic captions are created only in the video's default language. Failure modes: a transcript in a different language from the audio; poor audio quality undermines alignment. Limitations: this page does NOT give a word-error rate or an accuracy caveat for automatic captions, and does not list supported file types inline (a link is referenced but its list was ...
• articleHow does masking work between dialogue/vocal and music or effects, and what does Sound On Sound's technical editor recommend before reaching for EQ?
  Problem: a generated music bed or effect buries the speech. Conditions: simultaneous sounds with similar frequency spectrum. Method (Hugh Robjohns, SOS Technical Editor): the louder of two sounds with a similar spectrum tends to mask the quieter one. Best first step is to minimise the problem at source by fine-tuning orchestration and arrangement of parts. Next, reduce the dynamic range of the dominant signal so that the important signal is always the strongest (vocals are usually heavily compressed). Where that is insufficient, use EQ to de-emphasise frequency ranges of the backing where the clash occurs; conventional EQ affects an octave or more so the cut centre rarely needs to track the music, and a dynamic EQ (cut applied only when the vocal is present, optionally via side-chain) can keep the backing's full tone when the vocal is absent. Use for film (inference): pick or generate music with space in the speech range, duck/automate under dialogue, then use broad gentle cuts or a...
• articleWhat can Demucs (HTDemucs) separate, what can it not, and what are the practical artefact and licence considerations if I try to build an M&E or dialogue stem from a mixed track?
  Corrected 2026-10-23 - see C3-B709-01; the original statements below are kept for history and are superseded where the correction says so. Problem: a dub or remix needs a music-and-effects (or dialogue) stem but only a final mix exists. Conditions: single mixed audio file. Method: the Demucs README describes a MUSIC source separation model that outputs four stereo 44.1 kHz stems: drums, bass, other, vocals (a --two-stems option gives e.g. vocals versus accompaniment); an experimental 6-source model adds guitar and piano. v4 is Hybrid Transformer Demucs, trained on MUSDB HQ plus an extra 800-song set, reporting 9.00 dB SDR on the MUSDB HQ test set. What it does not offer (inference from the stem list and training data): no dialogue / effects / ambience classes, so 'vocals' is a singing-voice target, not a speech-dialogue target, and film effects land in 'other' or leak into any stem; the README itself reports 'a lot of bleeding and artifacts' for the experimental piano source. Artefa...
• articleHow do I run a small, honest listening test with hidden reference and anchors (ITU-R BS.1534 MUSHRA), and what does that method not cover?
  Problem: compare several audio versions (codec, denoiser, TTS, separation) with a defensible test. Conditions: audio differences are in the small-to-medium impairment range (BS.1534 is for intermediate quality; BS.1116 is for small impairments). Method (BS.1534-3): present listeners with a known reference, several systems under test, plus hidden reference and anchors in one trial; the standard anchor is the original low-pass filtered at 3.5 kHz and the mid-quality anchor at 7 kHz; keep a trial to no more than 12 signals; use experienced listeners; post-screen assessors by excluding those who give a very high grade to a heavily impaired anchor or frequently grade the hidden reference as impaired (the Recommendation gives numeric thresholds of 15% of items, e.g. reference below 90 or mid anchor above 90); train listeners on the range of impairments first; analyse with the statistics in the annexes (confidence intervals; non-parametric options). The Recommendation also notes the name M...
• articleIn spatial/360 audio, what changes when the listener's head moves: which parts rotate with the head, which stay fixed, and how many channels do the orders need?
  Problem: planning audio for 360/VR/head-tracked playback (e.g. a need about head movement). Conditions: ambisonic (scene-based) delivery decoded to binaural headphones. Method: ambisonic scenes are decoded and rendered binaurally, and the whole sound field is rotated by the head/camera orientation; Omnitone, Google's Web Audio decoder, applies a 3x3 or 4x4 rotation matrix each animation frame (it can be wired to a three.js camera or device orientation) to rotate the entire sound field. Channel counts by order are 4 (first order), 9 (second) and 16 (third) channels in that library. YouTube's spatial audio spec uses AmbiX (ACN ordering, SN3D normalisation), first order only, W,Y,Z,X at 48 kHz in one audio track, and it also supports a 6-channel FOA plus head-locked stereo (L,R) layout, where the stereo pair does not change when the viewer moves their head. Production consequences: put narration, dialogue-over-picture, music beds and UI sounds in the head-locked pair; place only source...
• articleWhich SSML tags does Amazon Polly support for neural, long-form and generative voices (is emphasis available)?
  Problem: SSML written for one voice type silently loses effect on another. Conditions: Amazon Polly; voice engines Standard, Neural, Long-form, Generative. Method: Polly's page says all tags except <amazon:domain name="news"> are supported for Standard voices, then tabulates the others by voice type. From the table as fetched: <break>, <lang>, <p>, <s>, <sub>, <w>, <speak> are fully available for neural, long-form and generative voices; <emphasis> is not available for any of the three; <prosody> is only partially available for all three (and <prosody amazon:max-duration> is not available); <mark> and <phoneme> are partial for generative voices; <amazon:auto-breaths> and <amazon:effect phonation=soft> are not available for the three newer engines; the dynamic-range-compression effect works for neural and long-form but not generative. Use (inference): when moving from Standard to neural/generative voices, replace emphasis with wording/punctuation, use <break> and <p>/<s> for rhythm, a...
• articleWhat adaptation techniques do dubbing translators use to reach lip-sync versus isochrony, and what do they give up in fidelity?
  Problem: a literal translation rarely fits the mouth or the clock. Conditions: any dubbed dialogue line. Method: Chaume's techniques as reported: for lip-sync, repetition (cognates), word-order shift so the marked sound lands on a similar sound, substitution with a synonym/antonym/hypernym, reduction or amplification, and omission or addition. For isochrony, amplification (repetition, gloss, paraphrase, longer synonyms) or reduction (ellipsis of modal verbs, interjections, phatic markers, social formulae, vocatives and proper names; omission of redundancy with the image; deictics and all-purpose words). Fidelity trade-off: such omissions and additions would count as errors in written translation but are permitted in dubbing. Example (measured, 10 segments of a Persian dub of The Green Mile): repetition, word-order shift and substitution dominated lip-sync, while isochrony used reduction 58 percent and amplification 42 percent; amplification was also used to disambiguate. Koveriene f...
• articleHow do I measure and fix a constant audio offset in an edit with FFmpeg (-itsoffset, adelay, aresample async), and what does each tool actually change?
  Problem: separately generated or dubbed audio lands early or late against picture. Conditions: a CONSTANT offset has been measured (clap/flash marker, or a transient in both streams); drift is a different problem. Method (all statements from the FFmpeg docs unless marked): -itsoffset is an input option; the offset is added to the input's timestamps and a positive value delays that input's streams, so place it before the -i of the audio file to delay audio (a negative value to advance it is my inference from 'added to the timestamps'; test it). The adelay filter delays selected channels and fills the gap with silence; delays are in milliseconds per channel separated by '|', with an S suffix for exact samples and 'all=1' to apply the last delay to all remaining channels, e.g. adelay=delays=64S:all=1. aresample with async can stretch/squeeze samples to follow timestamps (docs example async=1000, maximum 1000 samples per second of compensation), which is a drift-handling tool, not an of...
• articleWhat do Chaume's three dubbing synchronies (lip, kinesic, isochrony) each require, according to translation-studies sources?
  Problem: dubbing adaptation is judged on several kinds of match at once, but briefs often say only 'sync'. Conditions: dubbing (not subtitles or voice-over) of dialogue that is on screen. Method: Chaume's typology, as reported in the open-access sources, has three parts. Lip synchrony adapts the translation to the on-screen character's articulatory movements, above all bilabial and labio-dental consonants and open vowels. Kinesic (kinetic) synchrony keeps the words logically consistent with the characters' gestures and body movements (for example a nod must not accompany 'no'). Isochrony matches the duration of the target speech phrases and pauses to the source; its absence is called the most noticeable sign of a poorly dubbed product, and the Persian study says the viewer notices when lips have closed but speech continues, or lips move in silence. Chaume's 2012 book defines synchronisation as the process of matching the translation to actors' body and articulatory movements in a st...
• articleHow exact must phonemes match for lip-sync in dubbing, and what do open-vowel, bilabial and viseme substitutions allow?
  Problem: judging whether a dubbed word 'fits' the mouth. Conditions: visible mouth, usually close-up. Method: the guidance targets visually distinct sounds rather than identical ones: bilabials (open/closed lips), labio-dentals and open vowels are the most visible. Substitutions are allowed within a class: open vowels can be swapped, and a /p/ can be replaced by /m/, /b/, or the labiodentals /f/ or /v/. The viseme theory formalises this by grouping phonemes by lip shape; rounded and protruded vowels form the 'fourth viseme'. Example data (one Lithuanian dub of an English animated film, 476 stressed rounded/protruded vowel cases, 1 h 35 min): the translator used the fourth viseme in 46 percent of cases, the third in 32 percent, the second in 12 percent and the first in 10 percent; stress matched in 68 percent of cases. When a rounded vowel could not be matched, first- and second-viseme sounds fitted better than third-viseme sounds. Failure modes: matching spelling or phoneme identity...
• articleIn an open TTS evaluation, what are WER and speaker-similarity (SIM) computed with, and why can a lower WER mean worse speaker similarity?
  Problem: judge a TTS or voice-cloning output with objective numbers. Conditions: zero-shot TTS evaluated on a fixed test set. Method: the Seed-TTS evaluation (repo and paper) computes WER by transcribing the generated audio with an ASR (Whisper-large-v3 for English, Paraformer-zh for Mandarin) and comparing with the intended text, and SIM as the cosine similarity between speaker embeddings from a WavLM-large model fine-tuned for speaker verification, comparing each synthesized utterance with its reference clip. The released set uses 1,000 Common Voice samples (English) and 2,000 DiDiSpeech-2 samples (Mandarin); scripts are cal_wer.sh and cal_sim.sh. The paper cautions that a lower WER does not necessarily mean better subjective speaker similarity: lower WER often means more 'standardized', less accented or less expressive speech that the ASR finds easy. Use (inference): report WER and SIM together with a listening test; name the ASR and embedding models and versions, because scores ...
• articleWhat do break, emphasis and prosody mean in W3C SSML 1.1, and how much is a TTS engine actually obliged to do with them?
  Problem: control pause, stress, rate and pitch of synthetic speech portably. Conditions: engines that accept SSML. Method: <break/> is an empty element; strength takes none, x-weak, weak, medium (default), strong or x-strong, and time takes a duration (e.g. 250ms or 3s); if neither attribute is given the processor produces a break stronger than it otherwise would; if both are given the time sets the duration and the strength shapes other prosodic changes. <emphasis> requests prominence, but the engine decides how to render it because emphasis differs between languages, dialects and voices. <prosody> controls pitch, contour, range, rate, duration and volume; rate may be a percentage multiplier (100% = unchanged), volume a signed dB value (-6 dB roughly halves amplitude) or a label, and it is an error to use prosody with no attributes. The spec says all prosodic attribute values are indicative (the engine may clamp or substitute) and that authors are encouraged not to casually mix the...
• articleHow do Azure AI Speech and Google Cloud Text-to-Speech document emphasis, prosody and style control, and what are the caveats for HD or newer voices?
  Problem: choose markup that the target voice honours. Conditions: Azure Speech SSML and Google Cloud TTS SSML as documented. Method (Azure): <prosody> adjusts pitch, contour, range, rate and volume (contour takes percent/pitch pairs); <emphasis> word-level stress tuning is documented only for en-US-GuyNeural, en-US-DavisNeural and en-US-JaneNeural, and the page warns low-pitch, short words may not be raised enough to notice. For HD voices (DragonHD family) expressive control moves to styles: <mstts:express-as style=...> or inline style markers such as [whispering]; a marker applies to all later sentences until reset with [Neutral], and \r\n, <p> and automatic sentence boundaries reset to default style while <break> does not. Style degree 0.01-2 is ignored if unsupported for the voice. (Google): the page states that not all elements of the W3C spec are supported; <emphasis> levels are strong, moderate, none, reduced; it advises using <emphasis> and <prosody> only around full sentence...
• articleWhat lip-sync (sound/picture) timing tolerance does EBU R37 set for a broadcast programme, and what is the stage-by-stage versus end-to-end figure?
  Problem: an edit or AI-dubbed/AI-generated-audio deliverable needs a defensible number for 'is the audio in sync'. Conditions: broadcast or broadcast-style delivery; R37 is a 2007 EBU recommendation measured under EBU R28 viewing conditions. Method: R37 gives two figures. (1) Accuracy at each stage of the chain: audio from 5 ms early (sound before picture) to 15 ms late (sound after picture). (2) At any output intended for emission, the overall end-to-end difference should lie within Table 1: sound before picture up to 40 ms, sound after picture up to 60 ms. It says the values come from subjective tests of when lip/speech asynchrony becomes perceptible to 50% of observers, that perceptibility depends on content and viewing distance, and that HD needs particular care. It also says audio and video should be kept coincident in the file and not advanced or retarded to compensate for downstream delays, and that concurrent flash/blip (1 kHz tone, one frame) header markers are the measurem...
• articleWhat does MMAudio take as input, what duration does it work at, what hardware does it need, and where does it fail (speech, unfamiliar sounds)?
  Problem: choose or set up a video-to-audio (foley) generator for a silent AI video clip. Conditions: a short clip, optionally a text prompt. Method (repo README): demo.py takes --video and --prompt (omit --video for text-to-audio) with --duration=8; the default output and training duration is 8 seconds and a large deviation from it may lower quality, so longer clips need segmenting and crossfading (inference). Inference is reported at about 6 GB GPU memory in 16-bit mode; the authors tested only on Ubuntu. Known limitations listed by the authors: sometimes unintelligible speech-like sounds, sometimes background music, struggles with unfamiliar concepts (can generate gunfire but not an RPG). The paper adds that it generates 'unintelligible mumbles' when asked for human speech from mouth movement or text because it targets general audio effects (Foley). Implication (inference): use it for foley/ambience beds under a separate dialogue track, never for lines; audit the output for stray ...
• articleWhat exact numbers does Netflix set for subtitle in-times, out-times, shot changes, minimum duration and gaps (at 24 fps and other frame rates)?
  Problem: timing rules for subtitles in frames, not just characters per second. Conditions: Netflix timed text at any frame rate; the page writes rules for 24 fps and scales the half-second rule (12 frames at 24 fps, 15 at 30 fps, 30 at 60 fps) while the minimum gap stays 2 frames. Method (source statements): in-time on the first frame of audio, within 1-2 frames acceptable; out-time ideally at least half a second after audio ends when no subtitle follows; shot-change rules: dialogue starting on or within half a second past the cut sets the in-time to the first frame of the shot; an out-time within half a second of the last frame before a cut extends to the cut, respecting a 2-frame gap; subtitles may cross a shot change only if the dialogue does; minimum gap between subtitles 2 frames, and gaps of 3-11 frames at 24 fps must be closed to 2 frames ('chaining'), so gaps are either 2 frames or at least half a second. Minimum duration is 20 frames (stated as 4/5 s on this page, which doe...
• articleWhat do ITU-R BT.1359-1 detectability and acceptability thresholds say about audio leading versus lagging picture, and which sign convention does it use?
  Problem: viewers tolerate audio lag and audio lead very differently, so a single symmetric +/- figure misleads. Conditions: broadcast television; thresholds are averages from subjective tests, original (1998) Recommendation. Method: BT.1359-1 states detectability thresholds of about +45 ms to -125 ms and acceptability thresholds of about +90 ms to -185 ms on average, where a POSITIVE value means sound is advanced (early) relative to vision. It recommends an overall sound/picture tolerance of +90 / -185 ms, +25 / -100 ms between image source and the zero reference, and +22.5 / -30 ms from final source selection to the transmitter input, with no downstream segment outside the broadcaster's control adding more than +/-2 ms. Its appendix explains the 22.5/30 ms limits as half a grade point above the detectability threshold on the five-point scale (one grade is about 60 ms for sound delay, about 45 ms for sound lead). Practical reading (inference): audio that lags picture is detected lat...
• articleHow is a CMOS (comparative MOS) test set up in a recent TTS paper, and what threshold is treated as no difference?
  Problem: decide whether two voice outputs differ audibly. Conditions: listeners compare a synthesized clip against a human or other-system clip for the same text. Method (as described by Seed-TTS): for each test sample evaluators first hear a reference clip of the target speaker; then hear the synthesized output and the ground-truth human recording in random order; they rate which has higher speaker similarity and expressiveness on a scale from -2 to +2 (the extremes mean least and strongest preference for the first sample); results are rearranged into one direction (system versus human) and averaged over evaluators and sentences. The authors state that an absolute CMOS below 0.1 is empirically considered insignificant between systems, and report CMOS of -0.07 (English) and -0.08 (Mandarin) versus human. Use (inference): randomise order and sample-to-listener assignment, hide system identity, include the human reference as a control, fix the reference clip, and report n listeners, n...
• articleWhat input requirements, failure modes and options does a hosted lip-sync API (sync.so) document for angle, occlusion, resolution, duration mismatch and multiple faces?
  Problem. Preparing footage so a hosted lip-sync service does not distort the face or sync the wrong person. Conditions. sync.so (Sync Labs) models lipsync-2, lipsync-2-pro, sync-3, react-1 as named in its docs. Method. (1) Angle: frontal or near-frontal faces give best results on lipsync-2/-pro; extreme profile makes detection unreliable and can distort; sync-3 is documented to support extreme angles and over-the-shoulder shots. (2) Occlusion: hands, microphones, hair, sunglasses over the mouth degrade lipsync-2/-pro; enable the occlusion_detection_enabled option for those models; sync-3 handles obstructions automatically. (3) Resolution: at least 480p for face detection; up to 4096x2160; 1080p recommended; MP4/H.264; if compositing back, export SDR BT.709 with explicit colour tags, H.264 yuv444p when colour consistency matters. (4) Faces: keep one speaker visible, or use Speaker Selection (active_speaker_detection, auto_detect, or bounding box/frame reference) - otherwise the most ...
• articleWhat are Wav2Lip's practical flags (pads, nosmooth, resize_factor), its training sanity thresholds, and why can't it be used commercially?
  Problem. Older GAN lip-sync still appears in many pipelines; its failure modes and licence are often missed. Conditions. Wav2Lip open-source release trained on LRS2. Method. inference.py takes --checkpoint_path, --face (video or image) and --audio (any FFmpeg-readable file or a video whose audio is extracted). Tips from the README: adjust --pads (e.g. --pads 0 20 0 0 adds bottom padding to include the chin); if the mouth is dislocated or two mouths appear, over-smoothing of face detections is the cause - use --nosmooth; use --resize_factor to lower resolution because models were trained on lower-resolution faces, and 720p often looks better than 1080p; the non-GAN model needs more tuning but can be better; Wav2Lip+GAN trades slightly inferior lip-sync for better visual quality. Training sanity: expert-discriminator eval loss about 0.25 and Wav2Lip eval sync loss about 0.2 are said to be needed for good results; fine-tuning on a few minutes of one speaker is stated not to work. Right...
• articleHow does MuseTalk define its face region and mouth-openness control, what fps does it expect, what limits does it state, and what does its MIT licence not cover?
  Problem. Real-time or near-real-time dubbing of existing footage. Conditions. MuseTalk 1.0/1.5 (Tencent Music), latent-space inpainting of the lower face with a 256x256 face region, Whisper audio features. Method. (1) Use 25 fps input: the README says this matches training fps; convert lower-fps video with interpolation or ffmpeg. (2) The face-region centre matters a lot: in v1.0 use bbox_shift (positive moves the mask's upper bound toward the lower half and increases mouth openness; negative decreases it), e.g. --bbox_shift -7. (3) In the Gradio demo, generate only the first frame to tune parameters before running the full video. (4) For higher resolution, the README suggests adding a super-resolution model such as GFPGAN. Failure modes stated. Identity details such as moustache, lip shape and lip colour are not well preserved; resolution bound of the 256x256 face region. Paper abstract frames the trade-off: GAN-based real-time methods sacrifice lip-sync accuracy or dental detail. ...
• articleWhat is Dolby Atmos 'binaural render mode' metadata (Off/Near/Mid/Far) and how does it relate to distance and room feel on headphones?
  Problem: control how each bed channel or object sounds on headphones (perceived distance/room). Conditions: Dolby Atmos masters with Dolby AC-4 immersive stereo; documented in the Dolby Atmos Renderer Guide v3.0 (2018, older software). Method: the Renderer supports binaural render mode metadata, a binaural renderer distance model, designed for content encoded as Dolby AC-4 immersive stereo. Per input (bed channel or object) you choose Off (no binaural virtualization), Near, Mid or Far. The metadata is written globally for the whole master, applies to headphone output during monitoring, recording and playback, and is not included with speaker processing. With multiple beds there is only one setting per bed channel (all left channels share a setting). Backwards-compatibility mode deletes bed settings and writes object-only metadata; full edit mode allows per-bed-channel settings but is not backwards compatible for non-AC-4 encodes. Use (inference): choose Near for dialogue-like, close...
• articleHow does head tracking work when monitoring a Dolby Atmos mix on AirPods in Logic Pro, and why does a bounced binaural file not keep it?
  Problem: judge a spatial mix with head movement, then deliver a headphone version. Conditions: Logic Pro with Dolby Atmos plug-in on a Mac; head-tracking-capable headphones (AirPods Pro/Max). Method: the Dolby Renderer and Apple Renderer monitoring formats each have a Head Tracking button; turning it on makes the sound field of the three-dimensional mix remain fixed in place when you turn your head. The button appears only when head-tracking-capable headphones are selected as the output device. Apple Renderer (Head Tracking) also requires a Mac with Apple silicon. For delivery to other devices you bounce a two-channel binaural file; Apple states that if you used an Apple Renderer (Head Tracking) monitoring format you will not hear head tracking in the bounced file, and that on AirPods the Spatial Audio / Spatialize Stereo setting should be turned off to play a binaural file (otherwise the headphones process it again). An ADM BWF file is the required format for submitting a Dolby Atm...
• articleWhat does ElevenLabs say about emotion and pace for its newer TTS models, and what happens to emotional cue text?
  Problem: get emotion and pacing without SSML on ElevenLabs. Conditions: ElevenLabs best-practices page (fetched 23 Oct 2026); prior finding covers the SSML-break limitation. Method: convey emotion through narrative context or explicit dialogue tags (e.g. 'she asked, her voice trembling with sadness'); explicit tags give more predictable results than context alone, but the model will still speak the emotional delivery guides aloud, which can be removed in post-production. Pace is strongly influenced by the audio used to create the voice (use longer, continuous samples to avoid unnaturally fast speech); the speed setting ranges from 0.7 to 1.2 (default 1.0) and extreme values may affect quality; pacing can also be steered by writing in a natural narrative style, with ellipses for hesitation. The page also notes Director's Mode as upcoming. Use (inference): when using narrative tags, generate with the guide text and cut it out in editing, or record the line in two takes; keep speed wit...
• articleHow does Hume's Octave TTS take acting instructions (description, speed, trailing silence), and what limits does Hume state?
  Problem: get a specific emotional performance and pacing from TTS. Conditions: Hume Octave TTS API. Method: per utterance, set (1) description: natural-language acting instructions (emotional tone, delivery style such as whispering or rushed, performance context such as speaking to a crowd); (2) speed: relative rate from 0.5 to 2.0 on a non-linear scale, 1.0 normal, with 2.0 not exactly twice as fast; (3) trailing_silence: seconds of silence added after the utterance; plus [pause] / [long pause] inside text. Hume's best practices: keep instructions concise, aim for no more than 100 characters (e.g. 'frightened, rushed' instead of a long sentence), use precise emotions (melancholy or frustrated rather than sad), and combine emotion with delivery style (e.g. excited but whispering). With no voice specified, the description acts as a voice prompt that creates a voice; without any instructions Octave infers delivery from the voice description and text. Limitation as fetched: description...
• articleHow do I direct delivery with OpenAI's gpt-4o-mini-tts 'instructions', and what does the page now say about its future?
  Problem: control emotion and style of TTS speech by prompt. Conditions: the OpenAI Speech API (/v1/audio/speech) with gpt-4o-mini-tts. Method: pass an instructions string with the input text and a voice; the guide's example is 'Speak in a cheerful and positive tone.' The guide says the model lets you use prompts to control accent, emotional range, intonation, impressions, speed of speech, tone and whispering. The page presents prompt control as a gpt-4o-mini-tts feature; tts-1 and tts-1-hd are described only as lower-latency and higher-quality options with a smaller voice set. Voices are optimised for English. Usage policies require clear disclosure to end users that the voice is AI-generated. Important limitation as fetched: a banner says tts-1, tts-1-hd and gpt-4o-mini-tts are deprecated and will shut down on 6 January 2027, with migration to a realtime model via the Realtime API (a different API workflow, not supported through /v1/audio/speech). Use (inference): write short, conc...
• articleI am joining AI-generated video with a separately generated audio track: what are the first checks for lip-sync drift and frame-rate offsets, and what do the sources actually support?
  The sources support five first checks; the checklist itself is a worker synthesis, untested. (1) Fix one sign convention: three inspected sources use different ones. (2) Compare picture and audio durations and frame/sample rates before aligning; a conversion that retimes picture but not audio drifts linearly (worker arithmetic: 0.1% = about 60 ms per minute; 4% = about 2.4 s per minute), so measure offset at the start and the end. (3) Measure at utterance onsets and mid-utterance for speech. (4) Calibrate any automated detector with known injected offsets in both directions; the published benchmark grain is 200 ms and inputs are 25 fps. (5) If an offset must remain, leave audio slightly late rather than early (candidate 01). No inspected source measures AI-generator offsets.
• articleCan an automatic audio-visual sync detector or lip-sync metric (Synchformer offset classes, SyncNet-style LSE-D/LSE-C) certify that generated video and separately generated audio are in sync, and at what resolution?
  Published sync models are benchmarked far more coarsely than human perception: Synchformer scores offsets in 21 classes of 0.2 s and counts a prediction correct within +/-0.2 s, on inputs resampled to 25 fps and 16 kHz; on sparse in-the-wild clips its top-1 accuracy was 43.7% (60.2% within one class). Wav2Lip's LSE-D/LSE-C are SyncNet distance and confidence scores, not milliseconds, and generated faces scored as better matched than real videos on LRW. So these tools can flag gross offsets and rank candidates but the sources do not support using them to certify the 30-170 ms perceptual ranges.
• articleWhen captions, subtitles and audio description are added to an IMF or DCP, which profile or format carries each, what do the open validators check, and what is lost when the same text is moved between formats or into an archive container?
  Timed text crosses a profile boundary at each handoff: IMSC 1.3 is a text-only TTML profile; the Photon IMF validator requires UTF-8 timed text and accepts PNG image and OpenType font resources, checking encoding and resource types rather than rendering; DCP subtitles follow ST 428-7; and platform profiles can be narrower (Netflix accepted IMSC 1.1 only for Japanese at observation). libdcp's verifier contains no hearing-impaired or audio-description checks. Correction: ST 428-7:2014 is publicly readable from pub.smpte.org (read in CA-B249); only ST 2067-2 remains unread here.
• articleAt loudness QC, which measurements must be repeated after every re-render, and which values (numbers and metadata) do EBU R 128, Netflix v1.4 and BWF carry to the delivery package?
  R 128 V5 and Netflix v1.4 differ in target (-23.0 LUFS vs -27 LKFS), gating basis (programme vs dialog-gated) and peak limit (-1 dBTP vs -2 dBTP); one measured value cannot demonstrate both.
• articleWhere do public delivery specs disagree on forced narratives, textless elements and supplemental IMF packages (Netflix, Sky, Apple, SMPTE), and what does that imply for a multi-destination delivery?
  Delivery platforms handle forced narrative (FN) subtitles and textless material in incompatible ways. Sky UK (spec dated Feb 2026) wants FN burnt into the picture and textless elements appended mute after the programme in the same file. In IMF, Sky refuses subtitle tracks (because 'Forced' tagging is used inconsistently), supplementary packages and OPLs. Netflix v5.2 forbids burned-in subtitled dialogue in the texted master, may request a separate textless IMF after approval, and takes supplemental IMPs for redeliveries. Apple's 2019 Compressor guide delivers forced subtitles as a separate .itt file. SMPTE ST 2067-2:2020 defines a ForcedNarrativeSequence and illustrates incomplete 'Version File' IMPs (in an informative annex), so a standard-conformant IMF package can still be refused by Sky.
• articleWhat is a forced narrative, how does it differ from a full subtitle track and from 'textless' picture, and how do public specs from Netflix, SMPTE, Apple and Sky each treat it?
  Partially verified: the curator re-read S12, S7; S10, S14, S15, S6, S8 were not re-read by the curator and rest on the producer's pinned reading. Netflix and SMPTE ST 2067-2 define forced narrative the same way: timed text shown when no other subtitles or captions are enabled. Netflix does not burn it into picture and hides it when full subtitles or CC are on, so FN events must also be inside every full subtitle file (consistent with CA-B196 and CA-B260).
• articleHow do WebVTT and TTML2/IMSC 1.3 decide text direction for mixed right-to-left and left-to-right caption lines, and what must an author add to fix the cases the default gets wrong?
  Partially verified: the curator re-read S1, S3; S11, S15, S16, S2 were not re-read by the curator and rest on the producer's pinned reading. WebVTT sets each line's base direction from its first strong character (CSS unicode-bidi: plaintext), leaving fixes to Unicode marks/isolates in the cue text, while IMSC 1.3 permits explicit bidi/direction features (bidi-version-2 only partially) and horizontal-lr/rl and vertical writing modes. Both rely on the Unicode Bidirectional Algorithm.
• articleWhat do XAG 107 and the Game Accessibility Guidelines require for input remapping?
  XAG 107 asks games to offer in-game remapping of all controls even where the platform remaps, including analog/digital controls and per-stick axis inversion, sensitivity adjustable by at least 50% of default, full operability with digital input, and attention to long holds (2-3 s or more) and simultaneous presses; the Game Accessibility Guidelines list remappable controls as a Basic guideline and say system-level remapping is a safety net, not a substitute.
• articleWhat do the public SMPTE ST 428-10, ST 429-12 and ST 430-10 documents actually define for closed captions in a DCP, and how do they relate?
  SMPTE ST 428-10 (free PDF) is a constraint profile of the ST 428-7 D-Cinema subtitle XML for closed captions and subtitles: no images, at most three Text elements per Subtitle, and non-overlapping time windows; ST 429-12 adds optional ClosedCaption, ClosedSubtitle and MainCaption elements to the CPL. Both were downloadable without login on 2026-10-22.
• articleWhat is lost, or has to be decided, when converting between IMSC/TTML, WebVTT and SCC (CEA-608)?
  Converting captions to SCC with pycaption forces them onto the CEA-608 grid: vertical positions become rows 1-15 (bottom-up from 15 without layout), horizontal positions become columns 0-31 snapped to 4-column bases plus tab offsets, long pop-on cues are split by codeword count, and timing is quantised to 30x1000/1001 frames in both drop-frame and non-drop modes. Styling and positions beyond the grid are therefore lost or approximated; a full loss matrix needs CTA-608-E and an executed round trip.
• articleFor speech and video datasets, how often does the dataset licence disagree with the terms of the underlying sources, and where can I see the per-dataset records?
  Corrected 2026-10-23 - see C3-B643-05; the original statements below are kept for history and are superseded where the correction says so. The Data Provenance Initiative's 2025 audit (arXiv 2412.17847v2) reports, for 3916 datasets, that 25%, 33% and 32% of text, speech and video datasets carry non-commercial licences but that most of their source content carries non-commercial restrictions through source terms (author-reported); its released AMI record lists the dataset licence as CC BY 4.0 while the GitHub licence inferred from a converter repository (AMICorpusXML) is MIT, and Papers with Code reports Unspecified.
• articleWhat viseme vocabularies and timing outputs do Oculus Lipsync, Azure Speech, Amazon Polly and Rhubarb produce?
  Corrected 2026-10-23 - see C3-B642-01 and C3-B642-02; the original statements below are kept for history and are superseded where the correction says so. Four public viseme outputs differ in vocabulary and time unit: Oculus Lipsync 15 named visemes (docs marked no longer updated and subject to removal); Azure 22 integer IDs (0 = silence) with offsets in 100-ns ticks, optional SVG (en-US only) or 60 fps blend-shape frames of 55 values; Amazon Polly speech marks with millisecond times and per-language viseme symbols (archived awsdocs guide); Rhubarb 6 basic + 3 optional shapes (A-F, G, H, X) with seconds-based TSV and a Preston Blair alias only for dat export.
• articleDo over 80 percent of speech content carry source restrictions in the Data Provenance Initiative multimodal audit? Corrected: the body reports 78 percent for speech (99.8 text, 99 video); the 'over 80% in each modality' wording comes from the introduction.
  Problem. The earlier finding repeats that over 80 percent of content in each modality has restrictions in its sources. Conditions. arXiv 2412.17847v2 (19 Feb 2025). Method. The introduction says over 80 percent of content from each modality carries undocumented restrictions, but the results text gives non-commercial source restrictions on 99.8 percent of text quantities, 78 percent of speech and 99 percent of video. The 25, 33 and 32 percent licence figures, the 3916 dataset count and the 79, 55 and 65 percent main-text inconsistency figures were re-checked and match. Failure modes. Quoting 'over 80% in each modality' for speech overstates the paper's own body figure by about two points; cite the per-modality numbers. Limitations. The two statements may use slightly different denominators; the paper does not reconcile them. Version and date. v2 fetched 2026-10-23.
• articleIs the Azure-to-Oculus viseme mapping many-to-one only? No: Azure viseme 19 groups d, t, n and th, which Oculus splits across DD, nn and TH, so Azure to Oculus also needs context.
  Problem. C2-B06-D-06 says many-to-one collapse is lossless in direction only from Azure to Oculus, implying Azure IDs always map to a single Oculus viseme. Conditions. Compare the Azure ID table with the Oculus viseme table. Method. Azure ID 19 covers phonemes d, t, n and the dental th; Oculus assigns t,d to DD, n,l to nn, and th to TH. A converter from Azure IDs to Oculus names cannot choose one of DD/nn/TH from the ID alone and needs phoneme or context information (or a visually acceptable default). Failure modes. A naive 22-to-15 lookup table will mislabel n/th frames as DD. Limitations. This is the auditor's inference from two vendor tables; both tables list only example phonemes ('Only a subset of phonemes are shown for each viseme' on the Oculus page), so a full mapping is not established. Version/date. Pages fetched 2026-10-23.
• articleIs Oculus Lipsync documentation marked end-of-life by Meta? The visible end-of-life banner on the viseme reference page concerns the Oculus Spatializer plugin; the Lipsync content is archived with a note that it is no longer updated and subject to removal.
  Problem. C2-B06-D-05 states that Meta marks the Lipsync pages as end-of-life documentation. Conditions. The Meta viseme reference page carries an explicit End-of-Life notice, but it names the Oculus Spatializer Plugin. Method. Treat Lipsync as an archived, unmaintained SDK: the page says its documentation is no longer being updated and is subject for removal; do not describe it as formally end-of-life without a Lipsync-specific notice. Failure modes. Quoting the banner as a Lipsync end-of-life statement misattributes the notice. Limitations. Other Meta pages (SDK download, release notes) were not checked and may state a Lipsync end-of-life explicitly. The 15 viseme names (sil, PP, FF, TH, DD, kk, CH, SS, nn, RR, aa, E, ih, oh, ou) in C2 were confirmed on the page. Version/date. Page fetched 2026-10-23; the identical text was served from developer.oculus.com.
• articleDoes Azure Speech return viseme IDs, SVG and blend shapes for every voice and language? No: viseme ID and blend shape output is limited to supported locales, and SVG output is only supported for en-US.
  Problem. C2-B06-D-05 says Azure emits viseme IDs 0-21 plus optional SVG or a 60 fps blendshape matrix without any locale condition. Conditions. The Azure viseme page restricts the outputs by locale. Method. Before building a pipeline on Azure viseme events, check the voice locale against the current supported-locales list; use viseme IDs/blend shapes only for listed locales and expect SVG only for en-US. Failure modes. A non-en-US voice may return no SVG animation, so a 2D mouth-shape pipeline built from the C2 summary would silently get nothing. Limitations. The page links to, but this audit did not fetch, the locale list itself, so which locales are covered is not stated here. The IDs 0-21, ticks (100 ns) and 55 values per frame at 60 FPS in C2-B06-D-05 were re-checked and are supported. Version/date. Page fetched 2026-10-23; no version stamp on the page.
• articleWhat legal effect do the Commission's Article 50 guidelines claim, and what does the signatory table show for audio-capable providers?
  Partially verified: the curator re-read S8; S10, S9 were not re-read by the curator and rest on the producer's pinned reading. The Commission's Article 50 guidelines (2026-07-20) are non-binding and treat an adequate code of practice as a predictable way to demonstrate compliance and a possible fine-mitigating factor, not a substitute for the Act.
• articleHow does spatial audio for XR/immersive delivery (ASAF/APAC, IAMF, MPEG-H, OpenXR) differ in role and scope from the cinema/broadcast Audio Definition Model (ITU-R BS.2076-3)?
  Partially verified: the curator re-read S14, S18, S2; S13, S15 were not re-read by the curator and rest on the producer's pinned reading. ADM (ITU-R BS.2076-3, 2025) describes audio rather than carrying it: channels are typed DirectSpeakers, Matrix, Objects, HOA or Binaural, with head-locked and headphone-virtualise parameters, and the audio travels in BW64 with rendering left to a separate renderer. Apple's ASAF, by contrast, packages metadata with linear PCM in Broadcast Wave and depends on Apple's built-in renderer, which adapts to object and listener position and orientation; it is streamed as APAC. The Apple talk does not mention ADM, and no inspected source documents converting an ADM master to ASAF. IAMF combines audio elements and mix presentations for device-specific rendering.
• articleFor ambisonic (HOA) audio in immersive delivery, what orders, channel ordering/normalisation, bitrates and listener-motion behaviour do Apple APAC, IAMF, OpenXR's BD extension, glTF MPEG_audio_spatial and ADM specify?
  Partially verified: the curator re-read S11, S14, S3; S13, S15, S18, S2 were not re-read by the curator and rest on the producer's pinned reading. Ambisonic conventions differ by delivery format: Apple's built-in APAC encoder takes 1st-3rd order (384-768 kbps recommended) without naming a normalisation in the session; IAMF fixes AmbiX (ACN order, SN3D) with (n+1)^2 channels up to order 14 in mono or projection mode; glTF MPEG_audio_spatial renders HOA sources at the listener with head rotation only (3DoF). State order, ordering and normalisation explicitly when moving material between them.
• articleWhich spatial-audio formats and APIs do public XR and immersive-delivery documents actually define (Apple ASAF/APAC, IAMF, MPEG-H, glTF MPEG_audio_spatial, OpenXR, WebXR, Web Audio), and where is there nothing to read?
  Partially verified: the curator re-read S2; S11, S13, S14, S15, S16, S17 were not re-read by the curator and rest on the producer's pinned reading. Spatial audio is defined at different layers: Apple uses ASAF for production and the APAC codec for delivery (APAC required for Apple Immersive Video); AOMedia's IAMF is an open model and container for channel, ambisonic and object audio with binaural rendering signals; WebXR defines no audio of its own (web apps rely on Web Audio). No single open format spans production, delivery and XR playback in the documents read.

More