Hand-Object Interaction

When a person handles an object, each can hide parts of the other. I study how to reconstruct both through time: the hand pose, the object’s shape and 6D pose, and their relative motion. This extends my earlier work on 3D shape reconstruction and provides the human demonstrations I use in dexterous manipulation.

Progress log

2026 — A smaller capture setup

We are testing a smaller version of the HO-Cap capture system with three external cameras and one egocentric OAK-D camera. The published setup used eight external RealSense cameras and a HoloLens. We are also comparing existing pose-estimation methods on recordings from the new rig.

2026 — Comparing hand pose estimates in everyday video

We recorded a short hand-object interaction outside the calibrated rig and ran WiLoR, HaMeR, SAM3D-Body, HaWoR, and Dyn-HaMR on the same frames. The side-by-side view lets us inspect how each estimate follows the moving hands. I use it as a visual comparison, without ranking the methods by accuracy.

2025 — HO-Cap dataset and benchmark

We released the capture setup, annotation tools, dataset, and evaluation baselines with the NeurIPS Datasets and Benchmarks paper. The dataset covers pick-and-place, handovers, and object use, with hand and object pose annotations alongside third-person and egocentric video.

June 2024 — First HO-Cap paper

The first HO-Cap paper described our multi-camera system, object reconstruction, and semi-automatic hand-object pose annotation. We wanted usable 3D interaction trajectories without motion-capture markers or a library of pre-scanned objects.

Capture and reconstruction

My main published work in this direction is HO-Cap. Its original capture rig used eight calibrated RealSense RGB-D cameras and a HoloLens headset to record synchronized external and first-person views. People performed pick-and-place, handover, and object-use tasks. We used a separate Azure Kinect recording to reconstruct a textured 3D mesh for each object.

The newer four-camera setup

The newer rig uses three external cameras and an OAK-D camera for the first-person view. In the video, the 3D scene is on the left; the three external feeds and OAK-D feed are on the right. This setup is ongoing work and differs from the published HO-Cap rig above.

Tracking the interaction

For objects, the annotation pipeline starts with SAM 2 masks and meshes reconstructed with BundleSDF. FoundationPose initializes the object poses, which we then refine against the RGB-D point clouds.

For hands, we combine 2D landmarks from several camera views, triangulate 3D joints, and fit MANO models. A final optimization refines hand and object poses together to reduce implausible intersections.

Pose estimates on the new rig

We ran Sapiens 2, ViTPose, HaMeR, and Fast 3D Body on the same short sequence from the newer setup. The clips show each method’s output across the external views. I use them for visual inspection; they do not establish an accuracy ranking.

Hand pose beyond the capture rig

I also compare hand-pose methods on video recorded outside the calibrated rig. Each panel below shows the same frames with a different estimate overlaid. The panel labeled “3DB” is SAM3D-Body; HaWoR appears in two configurations.

From capture to manipulation

The published HO-Cap dataset contains 64 interaction sequences from nine participants using 64 objects, with approximately 656,000 RGB-D frames. The annotations let us study pose estimation under occlusion. The tracked motion and object models also serve as demonstrations for my dexterous hand manipulation work. Testing vision outside the calibrated rig brings this direction closer to my robot perception research.

Selected work: “HO-Cap: A Capture System and Dataset for 3D Reconstruction and Pose Tracking of Hand-Object Interaction,” NeurIPS Datasets and Benchmarks, 2025. Paper · Project page and dataset · Code