I had hours of first-person video of people cooking, washing dishes and folding clothes. I needed every frame labelled with the action happening in it: pick up, pour, fold. Labelling it by hand would have taken weeks.
A vision-language model can do this job. But which one? Public leaderboards rank models on general benchmarks, and none of them told me which model would find the exact frame where "pick up the plate" ends and "rinse" begins in my footage.
So I built egovlm, a small tool whose real job is answering one question: which VLM should produce this dataset?
The stack at a glance
| Layer | What I used |
|---|---|
| Language | Python |
| Models | Multiple VLMs through OpenRouter, one identical prompt |
| Video | OpenCV frame extraction and filtering |
| Validation | Pydantic schemas with retry on invalid JSON |
| Interface | CLI: benchmark, annotate, export |
The workflow
clip ──▶ [ VLM ] ──▶ episodes ──▶ [ rasterise ] ──▶ frame labels
▲ (start, end, task) (frame.jpg, "pick up")
│
which model?
Only one arrow in that diagram is a real decision, and it decides the dataset's quality. So the process is:
- Hand-label two or three clips. You can't skip this. Without ground truth, a benchmark can only tell you which models are broken, not which one is right.
- Benchmark. Run every candidate model over those clips with the same prompt, then score and rank them.
- Annotate. Run the winner over the full corpus.
- Export. Turn the episodes into labelled frames.
What leaderboards don't measure
When I scored the models against my ground truth, the differences were large, and they showed up on measures no general benchmark reports:
- Boundary precision. Does the model know when an action starts and stops, or only roughly what happened?
- Over-segmentation. Some models split one smooth action into five tiny ones. That's technically "detailed" and useless for training.
- Schema discipline. How often does the model return JSON that doesn't match the schema? At corpus scale, a few percent bad output is a lot of retries.
- Latency and price. A model that's slightly better and ten times the cost may not be the right choice for a million frames.
The best model on paper wasn't the best model for this footage.
Making video cheap enough for VLMs
The same thinking drove an industrial-video pipeline I built that turns site footage into a timeline of work progress. Sending every frame to a VLM is slow and expensive, and most frames are near-duplicates. So frames get filtered first using histogram similarity, SSIM, blur detection, dark-frame detection and motion thresholds. Only the frames that carry new information reach the model, as sliding windows that give it temporal context.
What I took away
- Your data is the benchmark. Two hand-labelled clips told me more than any leaderboard.
- Define "good" before comparing models. Boundary precision and schema discipline mattered more than raw accuracy.
- Filter before you infer. The cheapest VLM call is the one you never make.