Robots need training data showing how people actually use their hands: how a plate gets picked up, how a shirt gets folded, how a pan gets tilted. One of the most scalable ways to collect it is to put cameras on people's heads and wrists and let them do ordinary chores.
Recording is the easy part. Turning those recordings into something a vision-language-action model or a world model can learn from is where most of the work is. This post covers the pipeline I built for that.
Raw recordings are messy
A single session produces several video streams and IMU (motion sensor) readings, each running on its own clock. Before any AI runs, you hit plain engineering problems:
- Clocks drift. A head camera and a wrist IMU that are 80 ms apart will teach a model that the hand moves before the person decides to move it.
- Sensors disagree about the world. Each one has its own coordinate frame, and all of them need to share one.
- Formats vary. Every downstream team wants a different output.
The stack at a glance
| Stage family | What I used |
|---|---|
| Pipeline | Python, config-driven stages (YAML), CLI |
| Hands | MediaPipe hand landmarks, MANO hand meshes (PyTorch, SMPL-X) |
| Body and objects | YOLO11 pose and detection |
| Camera motion | Visual(-inertial) odometry, with ORB-SLAM3 as an optional backend |
| Smoothing | SciPy temporal filtering |
| Annotation formats | Datumaro / CVAT |
Fourteen stages, each swappable
The pipeline is split into fourteen independent stages: ingestion, synchronization, calibration, coordinate alignment, visual odometry, trajectory, scene reconstruction, hands, body, objects, tracking, interaction, actions and export.
Two rules shaped the design:
- Every stage runs on its own. Rerunning hand tracking shouldn't mean reprocessing a whole session from scratch. You can run any subset of stages on any slice of frames.
- Every stage's backend comes from config. Hand tracking can use a fast landmark model or a full 3D mesh model. Odometry can run natively on Windows or use a more accurate SLAM backend under WSL2. Swapping one is a single line in a YAML file, not a code change.
The second rule matters because the "best" model for a stage changes every few months. A pipeline welded to one model goes stale fast.
Measure the pipeline, not just the models
Each run writes a report with timing, memory and frames per second for every stage. That made it obvious where the time actually went, which was often not where I expected. It also let me compare backends on the dimension clients care about: how long until the dataset is ready.
The other habit that paid off was testing against real recordings from the start, never synthetic fixtures. Real data has dropped frames, motion blur and a hand that briefly leaves the frame. Synthetic data has none of that, and it hides exactly the bugs you need to find.
What I took away
- Time sync is the foundation. Every model downstream inherits its errors.
- Design for swapping backends. Embodied AI models change too quickly to hard-code any of them.
- Per-stage telemetry turns optimisation into arithmetic.
If you want to know which model should label this kind of footage, I wrote about that in benchmarking VLMs on your own data.
All posts