Dexterous Hand Manipulation

I want dexterous robot hands to learn from the way people use objects. I start with recorded hand and object motion, replay it with simulated hands, and study how to map it across different hand models. The longer-term goal is to turn those motions into manipulation skills that work beyond a single recording.

This work follows directly from HO-Cap, which provides tracked hand motion and reconstructed objects. I use Isaac Lab to test motion following and task-specific policies, and Isaac Sim to prepare objects for those experiments.

Progress log

October 2026 — Studying everyday hand-object interaction

I am looking beyond scripted lab recordings to everyday hand-object interactions. The short clip below is an example of the human motion I study before trying to reproduce it with a dexterous hand.

March 2026 — MANO-Finger control

I moved to a 28-degree-of-freedom hand with six virtual wrist joints and 22 finger joints. I set up motion-following experiments using recorded 3D hand joints as targets and the simulated hand’s joint state and velocity as observations. The HO-Cap replay below uses this model.

July 2025 — HO-Cap replay in simulation

I replayed recorded HO-Cap sequences in Isaac Lab and updated the setup to Isaac Lab 2.0.2. Contact-rich interactions were initially unstable, so I adjusted the simulation’s contact settings before continuing with manipulation experiments.

May 2025 — Building the simulated hand

I tuned the MANO-based hand’s joint effort, stiffness, and damping, and tested the effect of self-collision. I compared these settings with the Shadow and Allegro hands while building a model that could follow human hand motion directly.

March–April 2025 — Defining the learning problem

I began with a concrete goal: learn a policy that can reproduce demonstrated hand-object interactions. That led to the current combination of motion targets from HO-Cap and reinforcement learning in simulation.

A MANO-based hand in simulation

I built a MANO-based hand with separate visual and collision models for simulation. The earlier prototype shown below used 25 degrees of freedom. A later MANO-Finger version has 28 controllable degrees of freedom: six for the wrist and 22 for the fingers. In these MANO-based tests, I can follow recorded human poses directly. Mapping the same motion to robot hands is a separate problem I am now exploring.

Reconstructed objects in simulation

We use SAM3D-Object to reconstruct assembly parts and everyday objects. An automated script then converts the results into USD assets for Isaac Sim. The images show those assets in simulation.

Learning from demonstrations

In Isaac Lab, HO-Cap motions serve as targets for a policy controlling the hand’s wrist and fingers. The policy observes hand and object states, and its rewards compare both with the recorded motion. The object has to follow the demonstration too; matching the hand pose alone is not enough.

I have tested pick-and-place and in-hand object reorientation in simulation. These policies are still task-specific and need retraining when the task changes. Broader imitation and transfer across hand models remain open questions.

The videos below show motion rendering, replay, and task-specific policies at different stages of this work.

Demonstrations

Retargeting across dexterous hands

The same recorded hand-object motion is rendered across several hand models. One column uses UHAS, a published Unified Hand Action Space method developed by colleagues at IRVL. The clip shows motion mapping across hands; it does not show a learned manipulation policy.

HO-Cap motion replay

A 28-degree-of-freedom MANO-Finger hand follows a recorded HO-Cap hand-object sequence in simulation.

Pick-and-place policy

A task-specific policy performing a pick-and-place motion in simulation.

In-hand manipulation

Parallel simulation of in-hand object reorientation. This recording is shown at 2× speed.