EasyMocap: Turning Video into 3D Human Motion

Highlights and Results

EasyMocap is ZJU3DV’s toolbox for markerless human motion capture from RGB video. It combines pose detectors, camera geometry, tracking, and parametric human models to recover 3D skeletons and body motion. It supports several capture settings, including calibrated multi-camera recordings, monocular Internet videos, and mirror-assisted capture. For animation and vision researchers, its value is an inspectable pipeline whose intermediate detections, reconstruction, and fitting results can be checked separately. Official repository.

The repository demonstrates body, hand, and face fitting with SMPL, SMPL+H, SMPL-X, and MANO, alongside multi-person reconstruction and related novel-view-synthesis projects. Its quick-start dataset contains 800 frames from 23 synchronized, calibrated cameras. These are concrete demonstrations rather than a single accuracy guarantee for every supported mode. Quick start.

One research foundation, Fast and Robust Multi-Person 3D Pose Estimation from Multiple Views, reports 96.3% PCP on Campus and 96.9% on Shelf. PCP measures correctly estimated body parts under those benchmark protocols. These are results of that paper’s specific method, not measurements of the present EasyMocap checkout or its monocular path.

How It Works

The standard multi-view route proceeds from synchronized images and camera calibration to 2D keypoints, cross-view association when several people are present, triangulated 3D skeletons, temporal tracking, and staged body-model fitting. The main orchestration entry selects configuration files for reconstruction and fitting; this is a modular estimation system, with different routes for different capture conditions. Entry point.

For a single person, detected joints carry confidence scores. Camera projection matrices turn matching 2D observations into 3D joint estimates. The multi-view demo reprojects those estimates into the cameras, rejects observations exceeding its reprojection-error threshold, and triangulates again. This feedback makes detection errors visible before a body mesh is fitted. Calibration and synchronization are therefore inputs that determine reconstruction quality, not optional polishing steps. Single-person reconstruction.

Multiple people introduce an identity-association problem: detections in different cameras must belong to the same person. The research paper combines appearance and epipolar evidence with cycle-consistent multi-way matching. The inspected EasyMocap demo is more specific: its default configuration uses ray-based geometric affinity and singular-value-thresholding matching, followed by association and geometric acceptance checks. It requires at least three supporting views in that configuration. Thus the paper’s full appearance-based method should not be silently attributed to every repository preset. Default configuration; Affinity implementation.

Fitting converts skeleton measurements into a coherent human surface. The inspected pipeline estimates shared body shape from limb lengths, optimizes global rotation and translation, then body pose, followed by hands or facial expression when the selected model supports them. An optional 2D stage further fits reprojection evidence. Weighted joint losses, regularization, and temporal terms constrain the result. These are explicit human-model parameters, rather than a diffusion latent being denoised into motion. Staged fitting.

The monocular route instead combines 2D observations with CNN initialization and body-model priors. It has no simultaneous camera views to triangulate, so its depth ambiguity differs from multi-view capture. Outputs include per-frame 3D-joint JSON and body parameters such as poses, shape, global rotation Rh, and translation Th. The documented representation uses V = R X(θ, β) + T; Rh should not be copied blindly into SMPL-X’s global_orient field. Output contract.

Applications, Availability, and Future Directions

EasyMocap provides a useful research starting point for motion analysis, avatar animation, and human reconstruction. Its capture output is not automatically a textured avatar or an animation retargeted to any character. BVH export is documented as a separate step using Blender and SMPL assets. At the inspected commit, the repository’s Project Registration License permits research, educational, evaluation, and personal use without registration, while organizational project use requires registration; body-model dependencies have their own terms. Repository license.

The README still labels joint analysis of multiple Internet videos showing a specific action, and one challenging-motion view-synthesis example, as “coming soon.” The generic TODO supplies no detailed schedule, so these remain documented unfinished items rather than dated delivery promises. This review reads commit e6006fd and the linked research; it does not report a new capture experiment. Release status.