Joint 2D–3D generation · Monocular video

Video-Conditioned Generative Joint
2D–3D Hand Motion Recovery

Chen Xu¹Yunqi Li¹Binbin Huang¹Brent Yi²Shenghua Gao¹Yi Ma¹

¹The University of Hong Kong ²UC Berkeley

Paper · arXiv Watch results Code · GitHub ↗Code & weights coming soon

Accurate, smooth hand motion.
Generated directly from video.

See it in motion

2D and 3D. One motion.

Explore aligned joint trajectories and mesh recovery
across static and moving camera views.

HOT3D · Sequence 01

Moving camera

2D hand motion Prediction & ground truth3D hand motion Recovered mesh

Choose a sequence01 / 04

Both panels show the same frames. The left panel overlays 2D joints; the right panel places the recovered mesh in the camera view.

Interactive reconstruction

Hand motion, in world space.

Rotate the view. Explore the trajectory.
A moving hand, in a reconstructed HOT3D scene.

World coordinates · metres
RGB scene point cloud

Load sequencePlayback

Scene context uses recorded RGB, saved SLAM depths, and dataset camera poses.

Load the interactive mesh to rotate, zoom, and play the sequence.

Selected HOT3D sequence with visible hand movement.

Scene points come from recorded RGB and saved SLAM depths, registered with dataset cameras. They provide partial scene context, not a JoHan output. Hand predictions retain the evaluation’s first-aligned world coordinates. This selected example is independent of the videos above.

01 / Overview

Joint generation, from video.

View full-resolution figure ↗

JoHan directly generates joint local 2D–3D motion before global recovery. Qualitative comparisons with Dyn-HaMR on HO3D and HaWoR on HOT3D show recovered mesh trajectories alongside ground truth. — Joint generation of local motion, followed by geometric global recovery. Qualitative comparisons cover both static-camera HO3D and moving-camera HOT3D videos.

Abstract

Occlusion and motion blur make hand reconstruction ambiguous and temporally inconsistent. JoHan learns a joint motion prior directly from video: it generates both 2D and 3D local trajectories, using the evolving 2D geometry to guide 3D reconstruction. Their correspondence then enables geometric recovery of the hand’s position and orientation relative to the camera.

02 / The core idea

Generate together.
Recover through alignment.

Learn local hand dynamics.
Recover global placement geometrically.

A joint motion prior

Two modality-specific generators learn 2D and 3D local motion within one flow-matching process. Both motion states evolve along the same generation timeline, conditioned on full-frame video features.

2D-Guided Fusion

At every flow step, the predicted clean 2D joints provide visibility-weighted guidance to their corresponding 3D joints. This lets the 3D generator use the evolving 2D geometry throughout reconstruction.

Geometric global recovery

After restoring the image-plane location and scale of the 2D poses, a PnP solver uses the 2D–3D correspondences and camera intrinsics to recover global orientation and translation.

Model overview

Video → aligned local motion → camera-space recovery

JoHan pipeline: full-frame visual encoder and hand masks condition two motion generators; 2D-Guided Fusion links the 2D and 3D branches, followed by 2D denormalization and PnP global motion recovery. — Full-frame video features and hand masks condition joint motion generation. The generated correspondence connects local motion modeling to global recovery.

Inside the spatiotemporal generator

Visual cross-attention introduces image evidence, while spatial and temporal self-attention model joint relationships and motion across frames. Timestep and hand-side information condition the generators through adaptive layer normalization.

Conditioned generator and expanded spatiotemporal block with visual cross-attention, spatial attention, temporal attention, and adaptive layer normalization.

Reference

Citation

If you find this work useful, please cite our preprint.

@misc{xu2026johan,
  title  = {Video-Conditioned Generative Joint
            2D-3D Hand Motion Recovery},
  author = {Xu, Chen and Li, Yunqi and Huang, Binbin
            and Yi, Brent and Gao, Shenghua and Ma, Yi},
  year   = {2026},
  eprint = {2610.10512},
  archivePrefix = {arXiv},
  primaryClass = {cs.CV},
  url    = {https://arxiv.org/abs/2610.10512}
}

Video-Conditioned Generative Joint 2D–3D Hand Motion Recovery