What the machine got wrong

01Why write this down

TL;DROne long video in, ten shorts out. The failures behind that bet are the real story.

Every AI engineer job posting I've read lately asks for the same thing: evaluation. Most postings can't even name the tooling for it — they just know they need it. I've been living inside that problem for a year, so here is the honest version: a system that kept failing, the receipts that caught it, and the exact changes that moved the numbers.

It started simpler than that. I knew I needed shorts going out every day, and I wanted one point of leverage: me, making a video. One longform in, ten shorts out. A single hour of talking becomes ten days of daily content, exposure, consistency, and practice. And it finds the hidden gems, because nobody finishes a ten-minute video, but they will finish a twenty-second short.

The system: feed it a longform talking-head video and it returns short vertical reels — exact spoken words, word-aligned to the source audio, filler removed, dead air compacted, captions and boards burned in, every step backed by a JSON receipt. It's called the EDL Composer (an EDL is an edit decision list — the cut plan; the machine writes them, a renderer films them), and it went through about a dozen major versions. The version numbers aren't vanity. Each one exists because a specific failure mode got found and got fixed.

Built in Python on top of ffmpeg, with WhisperX forced alignment as the source of truth and a 16 kHz waveform-level QA pass on every render. The architecture — provenance gates, multi-layer QA, a human as the terminal gate — transfers to any AI system where the output has to be verifiably right. The video is just the first place I proved it.

02The machine

TL;DRThe machine cuts, cleans and checks. My ears decide what ships.

The chain, in one line:

Everything below the line is machine. The last step is a human. That ordering is the single most important decision in the whole project, and it was learned the hard way.

Two doctrines carried the whole thing:

Hard QA proves the clip survived. Gold QA proves the lesson landed.

Keep the lesson. Keep the words. Cut the dead air.

And one more, from the editor side: a clean reel is acceptable. A causal reel is better. A causal reel with a memorable payoff is the target.

03The failure taxonomy

TL;DRFour failure families: audio, structure, honesty and taste. Taste cost me the most.

After hundreds of receipts, the machine's failures cluster into four families. These are the gates that exist today — each one is a failure that actually happened.

That last family is the one most people skip, and it's the one that cost the most.

If I'm honest about which ones got under my skin the most: the double takes, the dead air, and the ums and ahs. A machine that can hear the lesson but still ships "guinea pig and ask for a guinea pig" hasn't listened yet.

04Incident 1 — the gate that checked existence, not trust

TL;DRA shortcut transcript slipped past a gate that only checked existence. Now the source must be trusted.

One afternoon the pipeline produced seven EDL drafts from a source transcript that was never meant for the gold lane. It was a convenience repair run: faster-whisper, the small model, proportional word timings. The pipeline validated that a word-timestamp file existed, and that was it. The drafts looked fine on the surface: 1080×1920, post-render heard-text checks passing. But the timings were estimates, not forced alignments, and every cut built on them inherited that lie.

The reason this stung is we'd already done the work. We tested engines and models until we found the one that produced the best transcripts. WhisperX won on two counts: it was the most accurate, and it heard the filler — the ums and ahs — which is exactly what you want when you're cutting against it. The plan was simple: create the transcript once, then run the whole machine off that single source. A faster-whisper convenience file was a shortcut that broke the trust the entire system was built on.

Here is the actual diff that fixed it. The whole incident reduced to two lines:

From the WhisperX provenance gate incident note, 9 July 2026.

Behind that diff: a hard gate (SOURCE_NOT_WHISPERX_ALIGNED) that requires the transcript to carry a WhisperX alignment receipt — engine, alignment success, word count, unique durations all verified — before anything else runs. A regression fixture was built from the exact offending file, so the mistake is now a permanent test: BAD_09_FAST_WHISPER_TSV_NOT_GOLD. The smoke suite blocks it with the expected code, every time. Fourteen smoke steps, nine bad plans, all blocked.

Numbers from the incident: source video 946 seconds, 2,432 word rows from the wrong engine, 7 drafts generated before the block existed, 0 drafts possible after it.

05Incident 2 — the repair loop that took three renders

TL;DROne clip failed twice before passing: dead silence, a chopped word, fixed by 340 milliseconds.

This is the one I want to show every engineer who thinks QA gates are theatre. One clip failed. Then it failed again. Then it passed. The receipts tell the story better than I can.

Real receipt evidence, too_dark_2_v76c_fresh3_edls. The fix was 340 milliseconds of source video.

First render: the machine cut around two dead-silence gaps that made the reel feel broken. We removed the gappy opener. Second render: the machine shaved one word too many off the top — the cut started on "tell" but the consonant got clipped, so the ASR heard "your flash" and the first word of the lesson vanished. Third render: the start was moved back to the conjunction before the phrase, and the gate passed: hard audio 3/3, taste audit 3/3 gold, zero repairs, zero failures.

The actual receipt from rendered_v2. The machine heard what the cut did to the first word.

Most of the early EDLs didn't land at all, and it wasn't the cuts. The machine just didn't have enough context, or enough fine-tuning on what to look out for. It could find a clean clip. It couldn't find the lesson. The repair loop was where that education happened — one failed render at a time.

The rule that came out of it: do not weaken QA to make outputs pass. Repair the source range and rerender. That rule is why the receipts still mean something.

06Incident 3 — the machine passed. My ears overruled it.

TL;DRA clip passed every machine check but taught nothing. My ears caught it, so ears are the last gate.

The hardest lesson. A clip of 4.783 seconds ("TTL TOO DARK? / BUMP COMPENSATION") passed hard audio QA. Waveform clean, ASR matched, no filler, no gaps. The machine stamped it ready for review.

It was not ready. It opened with "bump it up" — no object, no context, nothing to tell the viewer what to bump. The title was doing all the teaching. It was too short to carry the rule. A ruthless editor would have cut it. A planner would have shipped it.

This exact clip is now a permanent regression fixture: the clearest proof that technical pass is not gold pass.

That incident created the taste layer: the "gold QA" that sits above hard QA and asks the question the waveform can't. Does the lesson land? Its tests are blunt and human: a 13-year-old should understand what to do, what changes, and why. The compact test is do X → get Y → because Z. If the recording doesn't say the mechanism or the payoff, the board may not invent it. And the phone-face-down test: the audio alone must teach, because that's how the viewer actually watches.

The same week, my own ears demoted a batch the machine had passed. Seven clips, checked, five failed polish: filler words I could hear, double takes I could hear, a point cut off mid-sentence ("brightening effect has, okay"). The status changed by hand:

The demotion receipt, 9 July 2026. This status exists because the machine's word lists missed what a listener hears instantly.

From that day, the codebase has a rule the machine cannot override: no code path may set final_pass without a recorded human-listen evidence path. The pipeline can recommend. My ears decide.

07Incident 4 — the "Okay" that cost 140 milliseconds

TL;DRAny um or ah at the open is a fail. The first half-second decides if viewers stay.

The smallest failure on this list, and the one that taught me the most about standards. A 57-reel rerender run: every reel needed to start on the first real word, zero lead-in, because dead air at the open of a short is a death sentence. Audit result: 57/57 first-speech onset pass, 57/57 first and last words survived, 57 unique MP4s, maximum effective lead-in across the whole pack: 0.134 seconds.

Except one reel. The final ASR began "Okay, we're gonna jump back…" instead of "We're going…". One filler word, 0.14 seconds of nothing, sitting in front of the lesson. The repair receipt:

NFB57-005B repair receipt, 11 July 2026. The repaired file is byte-identical to the clip in the final pack (hash-verified).

Nobody would have heard it and complained. The system can now be trusted to catch it, because "good enough" was never the spec.

0.14 seconds of me with no content and the viewer is gone. The first second — the first half-second — is everything. So the rule became cutthroat: any um or ah at the open is a fail, full stop. When you're about to scale something, you seal the cracks first.

08The boring changes that moved the numbers

TL;DRBoring fixes moved the numbers: match the decibels, tune the silence floor, never cut a word.

Not every fix is a story. Some are a constant. The gap-compactor was cutting silence between words. But how much silence is too much, and what counts as silence? The machine measured waveform activity at 16 kHz, and the floor for what it called a "hidden vocalization" needed tuning. One decimal place:

A 40-millisecond change that stopped breaths and mouth sounds being mistaken for speech — which changed which gaps were safe to cut.

The other boring fix that mattered: you couldn't hear the cuts. When two clips merge, the decibels have to match, or the join lands on the viewer like a door slamming. Matching levels isn't glamorous. Get it wrong and every join makes the reel flinch.

The compaction layer itself is deliberately conservative. Its guard is brutal in the best way: the word list of the compacted edit must equal the original, byte for byte. It can cut air. It can never cut a word.

304 gaps mapped in the first series, 46 cuttable, only 14 accepted after waveform verification. The machine leaves silence alone unless it can prove the silence.

The seven-reel benchmark runs start to finish in about five minutes — 284.9 seconds wall clock, two render workers — with every number above written to JSON receipts as it goes.

09What the receipts taught me

TL;DRWrite the spec before the code, and give the agents one source of truth.

Write the test before you build the thing. The eval set is the spec: twenty-eight gold examples, written before the next version shipped — then the failures tell you where the spec was wrong.

Give the agents one source of truth before they build anything. A markdown file the AI can always go back to. Without it, the agents just keep creating files and files, and you lose count, you lose measure, you lose the root. One manifest you can always return to beats twenty inferred copies.

Label failures by hand, in your own words. The machine's word lists found fillers I'd never have listed ("kind of", "you know", "moving on"). I listened to every rejected clip, wrote down what was wrong in plain English, and clustered the notes. The four families in section 03 are those clusters. They were not the ones I predicted.

Change one thing. Score. Keep only what moved the number. Each version bumped one layer, not five. When the number didn't move, the change was reverted or rethought. No rewrites of the whole system out of enthusiasm.

The human is the terminal gate, and the machine knows it. READY_TO_POST is a status the automation is not allowed to emit. It requires a recorded human approval. That single rule is why every green receipt in this article means something.

10The honest tail

TL;DRNothing nags me now. The machine passes and I still pause. That pause is the job.

Nothing nags me about it now. It's pretty much done — for the moment. This is not a success story with a tidy ending. I still listen to every clip twice before it ships. One reel in an early seven-pack carries a warning I wrote myself: its ASR word-order agreement collapsed to 0.379 while the gate still passed it — I flagged it for my own ears and moved on. The machine passes, and I still pause. That pause is the job.

And it doesn't depend on me: the fixtures, the gates and the receipts mean another reviewer can run the same suite tomorrow and argue with the machine using the same evidence.

Here's what I actually built, if you strip the romance: a system that finishes a process end to end against real data, unattended, with receipts for every decision it makes — and a human ear as the last gate. The failures were the real work.