Reading EgoVista's Annotation Pipeline Through a Robotics Lens
A first-person video of someone handling an object contains useful evidence about motion and contact, but it is still just video. To study hand-object interaction or train from human demonstrations, we usually need more: hand pose, object masks, contact times, camera information, and some account of what the person is doing. EgoVista offers an automated pipeline that adds these annotations and exports the result in LeRobot or RLDS format.
I have not run the pipeline or evaluated its output. This note is based on EgoVista’s public documentation, especially its Technical Annex v1.3 and format description. I am interested in what the system makes available to a researcher, and in the measurements I would want before treating its labels as evidence of 3D hand-object motion.
Nine outputs, not nine sequential steps
EgoVista lists nine annotation layers: 2D hand pose, 3D hand pose, depth, hand-object segmentation, contact timing, action labels, face blurring, camera intrinsics, and metadata enrichment. Frame extraction happens before annotation; it is not one of the nine layers. The Technical Annex describes a dependency graph in which hand pose, depth, and segmentation can run after frame extraction without waiting for one another. The action-labeling stage depends on face blurring and video re-encoding because it sends video to an external model.
That distinction matters. A diagram that turns the nine layers into a single chain would misrepresent both the computation and the privacy boundary. The relevant privacy rule, as described by EgoVista, is that the video sent to its external action-labeling service has first had detected faces blurred. Its GDPR page correctly calls this pseudonymisation, since a blurred video can still contain personal information.
What the labels say—and what they do not
The LeRobot feature description is unusually helpful because it names the exported fields. It describes 21 hand keypoints per hand in 2D, 3D keypoints lifted using the estimated depth map, a five-class hand/object segmentation, contact transitions inferred from mask overlap, and natural-language action labels. Those are useful signals for indexing and filtering a large video collection. They also have different failure modes.
3D hand pose and depth. The Technical Annex names MediaPipe Hand Landmarker and the standard Depth Anything V2 Small checkpoint. The Depth Anything V2 project provides separate models for metric depth. EgoVista’s format page says its depth is metric when camera intrinsics are available, but the public material I found does not explain the scale-recovery procedure or report a metric-depth benchmark. I would therefore check the units and compare depth and 3D joints against measured 3D references before using either for geometric evaluation.
Contact timing. EgoVista derives contact starts and ends from overlap between hand and object masks. That is a reasonable way to generate candidate events. It is not a direct measurement of physical contact: a hand can occlude an object without touching it, and real contact may be hidden from the camera. For manipulation research, I would inspect transition timing, false contacts, and missed contacts separately from segmentation quality.
Camera information. The site lists per-clip focal length, principal point, and distortion. These fields deserve a direct calibration check, particularly when a video has been cropped, stabilized, or captured with a wide-angle camera. Camera metadata alone does not establish a world-space hand trajectory or an object pose.
Action labels. Timestamped language can make a collection searchable, but a description such as “pick up a cup” is not a robot control action. The exported action.contact_event and action.label fields should not be mistaken for joint commands, end-effector trajectories, or forces.
Useful as annotations; different from ground truth
EgoVista’s own site calls its labels “predictions, not ground truth” and says it provides uncertainty metadata and QA reports. That is the right distinction. A dataset can be well packaged, traceable, and useful for training while still needing independent checks of pose, depth, segmentation, and contact accuracy. Exporting to LeRobot or RLDS solves a data organization problem; it does not validate every annotation in the exported file.
If I were evaluating a sample for hand-object interaction, I would start with a small set of clips that have reliable reference measurements and inspect:
- 3D hand-joint error in a clearly defined coordinate frame.
- Segmentation quality for each hand and the manipulated object, including occluded frames.
- Contact precision, recall, and the timing error of contact transitions.
- Depth scale and drift across a sequence, using measured distances.
- Camera calibration and synchronization after any cropping or re-encoding.
- Processing time per hour of input video, together with failure and retry rates.
The most useful feature of the pipeline may be its modularity: each label has a named source, and each can be checked on its own. For research on hand-object interaction, I would use such output as a set of hypotheses about a video, then decide which signals are accurate enough for the particular experiment.