AI motion tracking pipeline — 3D
body mesh, hand tracking, pose estimation, and depth reconstruction from monocular video.
GVHMR — 3D Body Mesh
World-Grounded Human Mesh
Recovery (SIGGRAPH Asia 2024). Recovers full 3D SMPL body meshes with world-space positioning
and ground plane attachment. Includes automatic WiLoR 3D hand tracking (MANO finger pose,
attached to the body), Savitzky-Golay temporal smoothing, and foot-contact ground pinning.
Outputs: animation.fbx (rigged skeleton +
finger bones) · meshes.zip (OBJ per frame, hands attached) · joints_3d.zip · smpl_params.zip ·
gvhmr_camera.zip (the body's own camera for DCCs) · body-over-plate overlay
Post-processing: Savitzky-Golay smoothing
(body + hands) · foot-contact ground pinning · cm-scale OBJ + FBX
Depth Point Cloud
DepthAnything2 monocular
depth estimation, masked by a SAM3 subject matte and back-projected into a 3D point cloud (.ply)
of just the isolated subject — the rest of the scene is discarded (depth + matte fusion). Includes
a grayscale depth preview video. Useful for volumetric subject capture and spatial analysis.
Outputs: pointcloud.zip · depth_video.mp4
DWPose — 2D Body
Skeleton
YOLOv8-Pose for real-time 2D
skeleton extraction. Detects 17 body keypoints (COCO format) per person per frame with
confidence scoring. Includes temporal smoothing (Savitzky-Golay) to reduce jitter between
frames.
Outputs: pose_overlay.mp4 · skeletons.zip
(JSON per frame) · pose_summary.json
Wholebody —
133-Point
Full COCO-WholeBody keypoint
detection: 17 body + 68 face + 42 hand (21 per hand) + 6 foot keypoints. Uses the DWPose model
for comprehensive whole-body motion capture including finger articulation and facial landmark
tracking.
Outputs: wholebody_overlay.mp4 ·
wholebody.zip (JSON per frame)
Hand Track (WiLoR)
WiLoR 3D hand mesh
recovery. Per-hand MANO pose (15 finger joints, axis-angle) recovered from a single image,
robust on in-the-wild hands. When a body track (GVHMR) is present the hand pose merges into
the SMPL-X body; otherwise it runs standalone. Tracks both hands.
Outputs: hands_overlay.mp4 ·
hand_keypoints.zip (2D keypoints + MANO pose per frame)
Camera Track
Solves the shot camera from
the plate — locked-off (static) or moving (MegaSaM SLAM + metric depth). Auto-detect picks the
mode from measured camera motion. Focal length is estimated automatically (UniDepth) — no
sensor or lens input needed.
Outputs: camera_solve.zip — a ready-to-import
camera for Nuke (.chan) · Blender · Maya · Houdini · 3ds Max · After Effects, plus camera.json
(raw per-frame intrinsics + world→camera) and a solve-markers overlay.
🎯
Pipeline
Select
one or more methods per job. Each runs independently, routed to a free GPU (Thor or linuxdev):
- GVHMR — 3D body + hands (WiLoR) → FBX
skeleton + OBJ meshes
- Point Cloud — DepthAnything2 depth
→ 3D .ply export + depth preview
- DWPose — YOLOv8-Pose 17-point body
skeleton with smoothing
- Wholebody — 133-point body + hands + face
keypoints
- Hand Track — WiLoR 3D MANO hand mesh
+ finger pose
- Camera Track — static / MegaSaM moving
solve → Nuke · Blender · Maya · Houdini · 3ds Max · After Effects
⚡
Features
- Test Frame — Process a single frame for
quick validation before committing to full video
- Ground Pinning — Automatic foot-contact
detection pins the skeleton to the floor plane
- Temporal Smoothing — Savitzky-Golay
filter removes jitter from body pose, translation, and 2D keypoints
- FBX Export — Animated skeleton with body
+ finger rig, cm-scale, ready for Blender/Maya/3ds Max import
- 3D Viewer — Preview meshes and skeletons
in-browser via Three.js with playback controls
- Dedup Upload — Files are hashed and
cached — re-submitting the same file skips the upload