Getting text out of video and audio: extraction vs. transcription

Someone on the team says “we need to pull text out of our video library.” The ticket gets written as if that’s one job. It’s two. The first is reading text that’s already sitting in the file: a caption track, a sidecar, an ID3 tag. The second is speech recognition. It listens to audio and guesses at words that were never written down anywhere. Mix the two up, and a one-day integration turns into a week of confused debugging. The two failure modes are completely different. Neither one tells you which kind of failure you’re looking at.

Four places text can hide in a media file

A sidecar caption file (.srt, .vtt, .ttml) is the easy case: a plain text file that happens to sit next to a video. Open it and the words are right there. They’re wrapped in cue numbers and timestamps you’ll want to strip before handing the text downstream.

An embedded caption track lives inside the container itself, a tx3g track in an .mp4, an S_TEXT track in a .mkv. The words exist, but a tool that only reads container tags (title, artist, duration) never looks at this track at all. It sits muxed in alongside the video and audio streams. Reading it means actually parsing the container’s structure, not just skimming its metadata block.

A container tag is metadata the encoder wrote, not a transcript of anything: a title, an artist, sometimes an album. It’s genuinely text, and it genuinely comes back from a text extractor. But it has nothing to do with what’s said in the recording. An .mp3 with title: "Q3 all-hands" hands back exactly that string, nothing about what anyone actually said in the meeting.

Burned-in text is different: captions painted into the video frame as pixels. Audio with no caption track and no useful tags at all has no text to extract either. Full stop. That’s the case speech-to-text exists for, and no amount of clever parsing turns pixels or raw waveform samples back into a text file.

You need to know which of these four categories a file falls into. That’s most of the battle. Guessing wrong looks like an extractor reporting “success” with a one-line title string. What you actually wanted was the caption track, sitting two boxes deeper in the same file.

Extraction reads; transcription writes

Extraction takes text that already exists in a file and returns it unmodified, wherever it happened to be typed, tagged, or muxed in. Transcription takes sound that was never written down and produces new text by guessing at words from audio waveforms. txtfetch does the first job. It reads a caption sidecar, an embedded track, a container tag, whatever text is genuinely in the file, and returns it as-is. It does not listen to audio, and it never will. That’s a different, much heavier problem: acoustic models, language models, speaker diarization. It belongs to a dedicated ASR tool, not a document-extraction API.

If a media file’s actual content only exists as sound, no caption track, no transcript file anywhere, extraction can’t produce what isn’t there. That’s not a bug to work around. It’s the honest boundary of what “extract the text from this file” can mean.

Getting a caption track out with ffmpeg or yt-dlp

If a video already has an embedded caption or subtitle track and you just need it as a standalone file:

ffmpeg -i meeting-recording.mp4 -map 0:s:0 meeting-recording.srt

-map 0:s:0 selects the first subtitle stream in the container. Run ffprobe meeting-recording.mp4 first if a file carries more than one and you need to check the stream index.

For a video hosted somewhere with its own auto-generated or uploader-supplied captions, yt-dlp pulls those down without touching the video itself:

yt-dlp --write-auto-sub --skip-download --sub-lang en "https://example.com/watch?v=..."

Neither of these transcribes anything. Both just extract a caption track that already existed, the same distinction this whole post has been making. If a file genuinely has no caption track and no useful tags, a real speech-to-text pass has to run first. That could be Whisper, a hosted ASR API, or whatever your stack already uses. Once you have the resulting .srt or plain text file, it goes through txtfetch exactly like any other document.

From transcript to RAG pipeline

Say you’ve got a .srt, a .vtt, or a plain-text transcript. It might come from a caption track or from ASR. Whatever the source, the rest of the pipeline doesn’t care that it started life as a video. If your file is still in caption form, use the subtitle-to-text tool to strip cue numbers and timestamps into readable prose first. Then treat the result like any other document: chunk it, embed it, index it.

Chunk previewer shows exactly where your chunk boundaries land before you commit to a chunk size. That matters more for transcripts than for prose. A chunker tuned for paragraphs will happily cut a sentence in half at a caption-cue boundary if you don’t check first. The chunking strategies guide covers the fixed-size/recursive/structure-aware tradeoff in more depth if you’re picking a strategy from scratch.

Want the exact behavior for a specific container format, what .mp4, .mp3, .wav, or .mkv actually return? See the full breakdown on /extract/captions, checked against a real extraction run rather than assumed from the format’s documentation.