Jason Peterson Every shot in a film begins or ends with a slate: a clapperboard chalked with the shot name and take number. A dumb slate is just the board. A smart slate adds a running timecode display, jam-synced to the sound recorder. Either kind labels the take, and when the sticks snap shut it gives picture and sound a shared reference. Sound is recorded on a separate device from the camera, so the two have to be paired and aligned afterward. Both files carry timecode, so you match a take to its audio by timecode and get them roughly lined up. But timecode drifts, from wireless timecode boxes, long takes, and the like, so it rarely lands frame-accurate on its own. That’s what the clap is for: the instant the sticks snap shut is one unambiguous event, a single frame in the picture and a sharp spike in the sound, so lining that frame up with that spike corrects the drift to the exact frame. Lose the slate, lose that anchor.
Finding the slate isn’t hard. Finding it across thousands of clips per project, every project, every week, is the part that adds up. Slates show up anywhere in the clip: deep into the head, at the tail, sometimes inverted on tail slates. Dailies operators call it “slate hunting.” Minutes per scene, scaled across an entire production.
I built a tool that does the whole job: it finds the slate in the picture, pinpoints the exact frame the clapper snaps shut, pairs the shot with its production sound by timecode, and slides that audio so the recorded bang lands on the clap frame, correcting the drift. On a smart slate it also reads the displayed timecode off the picture for logging. The interesting parts are the constraint that shaped the design, the change-point detector that replaced a pile of heuristics, the audio side, and where the training data came from (it didn’t exist).
The constraint that shaped everything
Footage for unreleased films is confidential. Studios won’t put dailies through a cloud service for inference. The detector had to run on-device, on the dailies operator’s existing machine, sharing the GPU with the color grading platform they were already running.
That single requirement ruled out most of the obvious modern choices. No transformer, no zero-shot vision-language model, no API call. Inference had to fit alongside the host application without taking the GPU it was using, and per-frame latency had to be low enough that scanning a clip didn’t feel like a wait.
I designed for the constraint: small models, on-device, sub-50ms per frame, under a gigabyte of VRAM each.
Finding the clap
Three small specialists beat one big network here, and the job splits cleanly.
A MobileNet V2 does the seeing. Per frame it finds the slate and classifies its state: clapper open (raised) or shut (snapped closed). It’s a tiny, fast model, which matters because of the constraint above.
The clap is the moment the sticks meet, the single frame the sound team needs. My first version sampled a sparse set of frames and applied a stack of hand-tuned rules to decide which one was the clap. It worked, until it didn’t: every new edge case bought another threshold, and the thing started to smell.
So I threw the heuristics out. With parallel inference I can now run the state model densely, every frame across the head of the clip, cheaply enough to be practical (about 18ms per frame across seven workers). That turns “open vs shut” into a clean time series, and finding the clap becomes a solved problem: CUSUM, a change-point algorithm that’s been around since the 1950s. It learns the shot’s own baseline, then keeps a running tally of how hard the signal is leaning from open toward closed. The moment that evidence piles past a confident threshold, that’s the clap. A single flickery misread can’t trip it. Only a real, sustained flip does. And the size of the accumulated jump doubles as a confidence score. One principled rule replaced the whole pile of thresholds.
On a smart slate, a second MobileNet V2 reads the displayed timecode off that clap frame, useful for logging and for confirming the take. A dumb slate carries no timecode on its face, but that doesn’t matter for sync: the camera file and the sound file carry their own embedded timecode, and the clap frame is the anchor the audio gets locked to.
Locking the sound
Timecode gets the right sound file onto the right shot and roughly in place. The clap makes it frame-accurate. The picture side hands me the exact frame the sticks close, and the job is to find that same event in the paired audio and pin the two together.
Three steps:
Find the right file. Production sound arrives as a folder of WAVs, each carrying its own timecode. I match a take to its audio by searching that folder for the WAV whose timecode overlaps the shot, no naming convention required. One wrinkle: two shoot days can share the same time-of-day timecode, so the search is first scoped to the shot’s shoot day (read from the clip’s date) to keep those from colliding. A free sanity signal falls out of this: on a dual-camera shoot the left and right cameras are jam-synced to the same timecode, so both resolve to the same sound take. If they didn’t, something would be off.
Find the clap in the audio. A clap is one physical event that hits every microphone in the same instant, so I look for the broadband onset that fires coincidentally across all channels, gated to the neighborhood the picture predicts. That beats averaging to mono, which on a busy multitrack can lock onto a handling bump instead of the slate. Detection is sub-frame.
Lock it to the frame. I set the audio offset so the bang lands exactly on the detected clap frame. The alignment is anchored to the physical clap, not to rough timecode, so it survives the drift that creeps in from wireless sound carts. Picture is frame-quantized, so the audio inherits that ±half-frame, and to the eye it sits dead on.
Where the training data came from (it didn’t exist)
Here’s the part I learned the most from. There is no public dataset of film slates at the scale a detector needs. A couple of small Creative Commons sets exist, there’s one on Roboflow, but nowhere near enough to train on alone. And real slates vary enormously: hand-chalked dumb slates, smart slates with digital readouts, different fonts, layouts, colors, motion blur, glare, lens characteristics. Hand-labeling thousands more examples wasn’t viable as a side project.
So I synthesized the bulk of it.
The pipeline ran in two stages.
Blender generated each slate as a 3D object. Different slate types (a plain dumb slate, a smart slate with a timecode readout), different fonts, randomized values for shot name, take number, timecode. Procedural variation across hundreds of combinations of layout, color, font, type style, and crucially, both clapper states. Output: clean rendered slates with perfect ground-truth labels: known bounding boxes, known timecode strings, known clapper state.
ComfyUI then ran each clean render through a diffusion pipeline that added the texture of real footage: motion blur, lens grain, lighting variation, sensor noise, slightly off-axis angles, partial occlusion. The diffusion model knew nothing about slates as a concept. It just knew what film footage looks like, and it applied that texture to the Blender renders.
The result was thousands of training images that looked like real captured footage, every one of them with ground-truth labels attached automatically. Those, alongside the handful of small public sets, trained the detector. The client’s own production footage was held out for evaluation only, never for training.
The detector generalizes to real slates it has never seen. The OCR model reads timecodes off them. That generalization is the whole point of doing it this way.
The system in action
Three dumb-slate takes on a dailies timeline, no production audio attached to start. I set the detection preferences, select the shots, and run. A clap mark lands on each take. Scrub onto one and the sticks are shut on the exact marked frame. The matching production WAV has been found and synced, so on playback the clap sits right under the spike in the waveform. The debug view on the last shot shows a clean open-to-closed state signal with the clap flagged at the transition, confidence high at the boundary and fading to near-transparent everywhere else.
The numbers
- Inference: ~18ms per frame across seven CPU workers, down from ~60ms single-threaded, run densely across the head of every clip.
- Debayer: ~64ms per frame, cached once. A cold shot is about a minute end to end, mostly the debayer. A cached re-run is instant.
- Footprint: under a gigabyte of VRAM, alongside the host color grading application without contention.
- Accuracy: on a real dailies set of 60-plus shots across two shoot days and two cameras, the detector found the clap on every shot, including MOS takes correctly flagged as no-clap, and synced the audio on all but one, with zero-frame error on the hits.
- Training data: mostly synthetic, plus a few small public sets. Client footage: evaluation only.
How it runs
Everything happens on the operator’s finishing workstation, inside the color grading application they already use, so detection and sync land in the same place as the grade instead of a separate app to context-switch into. Early versions ran the model frame by frame inside the host’s ML plugin. To iterate toward the accuracy above I moved inference into my own Python layer, which lets me run it in parallel, manage my own cache, and infer densely. The host still does the heavy debayer and hands me frames through its API. Everything downstream, the dense inference, the change-point detector, the audio sync, is mine. It’s a proof of concept, and for a model this small it may well fold back into the host plugin once that path is tuned for speed.
What changed for the operator
I sent v0.1.0 to a dailies operator at a post house for testing. The note that came back:
I have been running slate detect v0.1.0 through its paces and I have to say I am extremely impressed. As is, I can see this tool offering tremendous value to a Dailies operator, eliminating much of the “slate hunting” that has to be done. This is one of the first things I’ve worked on in a while that feels really bleeding edge.
“Slate hunting” went from minutes per scene to seconds across a whole shot group.
What’s next
A CoreML port is in progress. Apple Silicon’s neural engine should drop inference latency further and free the discrete GPU entirely for the host. Timecode OCR error correction lands soon after, and I’m testing a small vision-language model for the smart-slate read: feed it the clap frame plus a couple on either side, and it can recover the frozen timecode even when my detected frame is off by one, error-correcting the clap placement in the same pass.
The lesson I’m taking forward: when the data doesn’t exist, build it. When the constraint says small, design for it. Several specialists with clean job boundaries usually beat one generalist with a fuzzy one.