Burned-in captions stopped being a nice-to-have the moment short-form video became a muted medium. Platform auto-captions exist, but they are positioned by the platform, styled by the platform, inconsistent between platforms, and gone the moment you re-upload the file somewhere else. If captions are part of your creative, they need to be part of the file. That means rendering them into the pixels, and the standard way to do that is an ASS subtitle document handed to libass through ffmpeg.
This post covers the whole path: getting word-level timings, building the subtitle document that produces the word-by-word reveal, and burning it in without wrecking quality. Everything here works on any video, generated or filmed.
Captions, already handled
siriusly.ai transcribes each generated reel, matches the caption style, burns word-by-word captions in, and gives you an editor to move or restyle them afterwards.
Try it freeWhy ASS and Not SRT
SRT can hold text and timings and nothing else. No font, no color, no position, no per-word styling. Advanced SubStation Alpha (ASS) carries all of it: a style table, absolute positioning, outlines, shadows, background boxes, and inline override tags that let you restyle a single word inside a line. libass, the renderer ffmpeg uses, is the same engine that has rendered anime fansubs for two decades, so it is fast, well tested and available everywhere ffmpeg is.
The choice is really between ASS and compositing captions frame by frame with a video editor's API. ASS wins on effort by a wide margin for anything a subtitle renderer can express, which includes the entire CapCut-style vocabulary.
Step 1: Word-Level Timings
The reveal-per-word effect needs a start and end time for every individual word, which rules out sentence-level transcripts. Whisper returns word timestamps directly when asked for them:
Python
result = client.audio.transcriptions.create(
model="whisper-1",
file=audio_file,
response_format="verbose_json",
timestamp_granularities=["word", "segment"],
)
words = result.words # [{"word": "actually", "start": 1.34, "end": 1.71}, ...]Two adjustments matter before those timings are usable.
Pull every reveal slightly forward
Whisper's word starts land a fraction after the word is actually audible, and a caption that appears on the syllable already reads as late - the eye wants the word slightly before the ear gets it. Subtracting a small constant from every reveal fixes it:
Python
WORD_LEAD = 0.12 # seconds
reveals = [max(w["start"] - WORD_LEAD, 0.0) for w in group]0.12 seconds is a good default. Much beyond 0.15 and the reveal starts running ahead of the voice, landing on the previous word's tail, which looks worse than being late.
Drop words that were never spoken
Whisper hallucinates on silence. A clip that ends with two seconds of room tone will frequently come back with a confident "thank you" or a stray "bye" attached to it. Filter on the segment-level signals it already gives you - no_speech_prob above roughly 0.6 combined with a low avg_logprob - and additionally drop any word whose midpoint falls inside a silent range you detect from the audio itself. A hallucinated word burned into the video is permanent.
Step 2: Group Words into Lines
Words are not revealed one at a time on an empty screen; they accumulate into a short line, then the line clears and the next one starts. Two or three words visible at once is the standard look, and the grouping rules are worth getting right because this is what makes captions readable at a glance:
- Cap the group at 2 to 4 words for vertical video. More than four and the line wraps, which breaks the fixed-position illusion.
- Break on sentence-ending punctuation so a group never spans a full stop.
- Break on a long pause between words - roughly 0.5 seconds or more - since that is where the speaker themselves broke.
- Never let a group's hold time overlap the next group's first reveal. Two caption blocks at the same screen position drawn at the same time render on top of each other.
That last one is the single most common rendering bug in hand-rolled caption pipelines. The finished line wants to hold on screen for a moment after the last word, but that hold has to be clamped:
Python
group_end = group[-1]["end"] + 0.15
if next_group:
group_end = min(
group_end,
max(next_group[0]["start"] - WORD_LEAD, reveals[-1] + 0.05),
)Step 3: The ASS Document
An ASS file is three sections: script info, a style table, and a list of timed events. Here is a complete minimal one for a 1080x1920 vertical video:
captions.ass
[Script Info]
ScriptType: v4.00+
PlayResX: 1080
PlayResY: 1920
WrapStyle: 0
[V4+ Styles]
Format: Name, Fontname, Fontsize, PrimaryColour, SecondaryColour,
OutlineColour, BackColour, Bold, Italic, Underline, StrikeOut,
ScaleX, ScaleY, Spacing, Angle, BorderStyle, Outline, Shadow,
Alignment, MarginL, MarginR, MarginV, Encoding
Style: Caption,Montserrat ExtraBold,96,&H00FFFFFF,&H00FFFFFF,
&H00000000,&H80000000,-1,0,0,0,100,100,0,0,1,6,2,5,75,75,0,1
[Events]
Format: Layer, Start, End, Style, Name, MarginL, MarginR,
MarginV, Effect, Text
Dialogue: 0,0:00:01.22,0:00:01.59,Caption,,0,0,0,,{\an5\pos(540,1344)}I
Dialogue: 0,0:00:01.59,0:00:01.94,Caption,,0,0,0,,{\an5\pos(540,1344)}I ACTUALLY
Dialogue: 0,0:00:01.94,0:00:02.40,Caption,,0,0,0,,{\an5\pos(540,1344)}I ACTUALLY BOUGHTThe Format lines are wrapped here for readability - in a real file each must be a single line.
Notice how the reveal works: there is no animation primitive doing it. Each event redraws the whole visible group with one more word than the last, so word three appearing is really the line being redrawn with three words instead of two. It costs nothing and it is exactly what the effect looks like.
The four things that go wrong here
PlayResX and PlayResY must match the video. Every size and position in the file is expressed in this coordinate space, and libass scales it to the real frame. Leave them at some default while rendering onto 1080x1920 and your 96px font arrives at a completely different visual size. Set them from the actual video dimensions, always.
ASS colors are BGR, not RGB, with alpha first. The format is &HAABBGGRR, and alpha is inverted: 00 is fully opaque, FF is fully transparent. So #FFD700 gold becomes &H0000D7FF. Writing RGB straight in is the most common ASS bug there is, and it fails silently by producing a plausible but wrong color:
Python
def ass_color(hex_color, alpha=0):
"""'#RRGGBB' -> '&HAABBGGRR'"""
rgb = hex_color.lstrip("#")
return f"&H{alpha:02X}{rgb[4:6]}{rgb[2:4]}{rgb[0:2]}"Timestamps are centiseconds, not milliseconds. ASS uses h:mm:ss.cc with exactly two decimal places, and the hour field has no leading zero. Emit three decimals and libass will misparse the line.
Braces and backslashes in the text are override syntax. A transcript containing a literal brace will silently eat the rest of the line. Strip {, }, \ and newlines from every word before writing it into an event.
Positioning and alignment
{\an5\pos(540,1344)} at the start of the text does the placement: \an5 sets the alignment origin to the middle-center of the text block, and \pos(x,y) puts that origin at an absolute point. Together they mean "center this line on that pixel", which is what keeps a growing line visually centered as words are added rather than expanding to the right.
The y value is where the caption block sits vertically, and it is the number worth tuning per platform. Expressed as a fraction of frame height, roughly 0.68 to 0.75 keeps captions below the speaker's chin and above the platform's bottom UI on a 9:16 frame.
Highlighting the active word
For a karaoke-style highlight where the whole line is visible and the current word is colored, wrap just that word in an inline color override and restore the fill immediately after:
ASS event text
{\an5\pos(540,1344)}I {\c&H0000D7FF&}ACTUALLY{\c&H00FFFFFF&} BOUGHTASS also has real karaoke tags (\k, \kf) that time the highlight within a single event. They are elegant but harder to control precisely against externally-derived word timings, and they cannot express the accumulate-then-clear pattern at all. Explicit per-word events are more verbose and much easier to reason about.
Step 4: Safe Zones
Every platform draws its own interface over the video, and captions placed underneath it are invisible in the only place they matter. The exact overlays change with app versions, so treat these as starting margins to verify against a real device rather than as fixed truth:
| Region | Keep clear | Why |
|---|---|---|
| Bottom | ~20% of frame height | Caption, handle, music ticker, CTA button |
| Right edge | ~15% of frame width | Like, comment, share and profile column |
| Top | ~10% of frame height | Status bar and in-app tabs |
Horizontal margins in the style row keep wrapped lines off the frame edge. Setting MarginL and MarginR to about 7% of frame width each is a reasonable default, and with WrapStyle: 0 libass will balance a wrapped line across those bounds rather than leaving a single orphan word.
Step 5: Burn It In
One ffmpeg call, using the ass filter rather than subtitles - both route through libass, but ass takes the file directly and skips the format conversion layer:
Shell
ffmpeg -i input.mp4 \
-vf "ass=captions.ass" \
-c:v libx264 -crf 18 -preset veryfast \
-c:a copy \
-movflags +faststart \
-y output.mp4Point by point:
-vf "ass=captions.ass"renders the subtitle document onto the video. Burning in requires re-encoding the video stream, so there is no stream-copy shortcut here.-crf 18is visually near-lossless for x264. Captions are high-contrast hard edges, which is exactly what a low bitrate destroys first, so this is not the place to economize - and the platform is going to re-encode your upload anyway, on top of whatever you hand it.-c:a copyleaves the audio untouched. Nothing about captioning affects the audio stream, and re-encoding it is pure loss.-movflags +faststartmoves the file index to the front so the result begins playing before it has fully downloaded.
Fonts
libass resolves font names through fontconfig, using whatever is installed on the machine doing the render. A font named in the style row but missing from the system is silently substituted, which usually means your carefully chosen heavy sans renders as the default serif - a difference you will not notice until you look at the output. On a container image, install the fonts explicitly and verify with fc-match "Montserrat ExtraBold". If you would rather not depend on system fonts at all, -vf "ass=captions.ass:fontsdir=/path/to/fonts" points libass at a directory you control.
Glyph coverage is the same problem in a harsher form for non-Latin scripts. A font without Arabic, Devanagari, Hangul or CJK glyphs renders missing characters as boxes, and those boxes are burned in permanently. Check coverage per language before rendering, not after.
Common Mistakes
PlayResX/PlayResY left at a default that does not match the video. Every font size and position silently scales wrong.
Fix: probe the real dimensions with ffprobe and write them into the script info section for every render.
Writing RGB into an ASS color field. Blue and red swap. It looks intentional, so it survives review.
Fix: convert through a helper that reorders to BGR and prepends inverted alpha, and never hand-write a color literal.
Trusting Whisper's output on silent tails. Hallucinated sign-offs get burned into the last two seconds of the video.
Fix: detect silent ranges from the audio and drop any word whose midpoint falls inside one, on top of Whisper's own no-speech signals.
Re-encoding the audio along with the video. Costs quality and time for nothing.
Fix: -c:a copy. The caption burn touches video only.
Reusing one language's word timings for a translated caption track. The captions drift against the audio almost immediately.
Fix: transcribe each localized audio track separately. See the localization post.
FAQs
Can I do this without re-encoding the video?
No. Burning in means changing pixels, which means the video stream is re-encoded. The alternative is a soft subtitle track muxed into the container (-c:s mov_text), which stays toggleable and lossless but is ignored by every social platform and cannot carry the styling.
What if my video is not 9:16?
Nothing changes structurally - set PlayResX and PlayResY to the real dimensions and derive font size and vertical position as fractions of frame height rather than as fixed pixel counts. That way one style definition works across every aspect ratio you render.
How large should the font be?
About 4 to 6 percent of frame height for vertical video, so 77 to 115 pixels on a 1920-tall frame. Below 4 percent it stops being glanceable on a phone; above 6 percent, three words no longer fit on a line.
Why do my captions look softer than the rest of the video?
Almost always the CRF. Hard-edged white text on a moving background is the worst case for a video codec, and the ringing artifacts show up on the text before they show up anywhere else. Drop to CRF 18 or lower and it clears up.
How does siriusly.ai do this?
Broadly as described: each generated chunk is transcribed with word-level timestamps, non-speech words are filtered out using both Whisper's own signals and detected silent ranges, words are grouped into short lines with clamped hold times, and an ASS document is built at the video's real resolution and burned in with libass at CRF 18. The caption style itself is read off the source video by a vision model, so the captions match what the origin footage was already doing, and the whole thing stays editable afterwards in the captions editor.