Stereo Hand Pose Estimation: Methods, Datasets, and Open Questions

Created on October 26, 2026 by Jikai Wang

2026

A second camera can help with the two things that make monocular hand pose estimation difficult: depth ambiguity and occlusion. But “stereo” is used loosely in this literature. A method evaluated on the Stereo Hand Pose Tracking Benchmark (STB) may still take only one image as input. For this note, I count a method as stereo or multi-view only when it uses synchronized views at inference time.

This is a reading note, not an exhaustive survey. I am mainly interested in egocentric hand tracking: estimating 3D joints or an articulated hand model from two or more head-mounted views, preferably over a video sequence. That setting is useful for studying hand-object interaction, where one camera often loses sight of the fingers at the moment of contact.

Where the views meet

Recent methods make different choices about when to combine camera views:

Approach Representative work What the second view contributes
Adapt a monocular predictor S2DHand (CVPR 2024) Cross-view agreement supplies supervision for adaptation, even without multi-view pose labels or known camera parameters.
Fuse features before pose prediction HandMvNet (VISAPP 2025) Joint-aligned features from multiple views are combined with attention before predicting hand joints and shape.
Build a compact stereo predictor Mobile-StereoHPE (ICME 2025) Stereo grayscale features support absolute 3D joint prediction under mobile compute constraints.
Track across cameras and time ECCV 2024 multiview hand-tracking challenge solution Camera-aware multi-view prediction is followed by sequence-level refinement.

These papers solve related but distinct problems. S2DHand asks how to make a single-view estimator useful with two cameras. HandMvNet and Mobile-StereoHPE fuse visual evidence inside the model, with different output and runtime goals. The challenge solution also treats temporal consistency as part of tracking. Comparing their reported errors directly would be misleading unless the camera setup, annotations, and evaluation protocol match.

Choosing a dataset

The dataset choice determines what “stereo hand pose” means in practice. I would start with these public resources:

Dataset Capture setting Useful for Main caveat
STB Real stereo hand sequences A simple stereo geometry and 3D joint benchmark Limited subject and interaction diversity; it does not represent current egocentric hand-object video.
UmeTrack Head-mounted, synchronized fisheye views Egocentric multi-view tracking and camera-aware models Its fisheye setup differs from a conventional pair of RGB cameras.
HOT3D Egocentric hand-object video from Project Aria and Quest 3 Hand pose, hand-object interaction, calibrated multi-view sequences, and MANO annotations The available camera streams are not a standard left-RGB/right-RGB pair.
MVHand Multi-view RGB-D capture Testing stereo fusion and absolute 3D joint estimation Its capture setting differs from head-mounted, in-the-wild interaction.

HOT3D is particularly relevant to my work because the hand and object are both present in the scene. UmeTrack is a strong reference for head-mounted multi-view tracking. STB remains useful for a controlled check, but success on STB alone says little about contact-heavy interaction in the wild.

What I would check in a paper

When reading a new method, I ask four questions:

  1. What enters the model at test time? Two synchronized images, multiple views, or just one image from a stereo dataset?
  2. Where are the views combined? In image features, pose features, 2D joints, or a later geometric optimization?
  3. Which geometry is assumed? Known calibration, triangulation, reprojection, or camera-independent cross-view agreement?
  4. Is time used? A frame-level prediction and smoothed output are different from a tracker that uses past frames during inference.

For hand-object interaction, I would add a fifth: does the evaluation include occlusion and contact? Those are the moments when a second view should matter most, yet they are easy to hide in an average joint error.

The useful direction, to me, is not merely to attach a second camera to a strong monocular estimator. It is to understand when the two views provide genuinely complementary evidence, and whether that evidence survives changes in camera placement and scene conditions. This note is a starting point for that comparison; I will update it as I read and test more methods.

Papers and resources