Datasets

Video sets for SFT. Query sets for RL.

Collect, import, or generate robot-ready data from human video, with every clip gated by a quality grader. Each record carries the metadata, video, instruction and action sequence a training stack needs.

Generate from human videoCollectImportGenerate from text

Datasets

Dataset	Type	Source	Robot	Scene	Size	Scope
ds_kitchen_h2r_v2	video set · SFT	egocentric human → robot	Bimanual YAM	kitchen_counter_v3	14.2 h	task-specific
ds_kitchen_real_v1	video set · SFT	teleop (collected)	Humanoid-G1	kitchen_counter_v3	3.1 h	task-specific
droid_subset_tabletop	video set · SFT	import · DROID	Franka	mixed	41 h	generic
kitchen_manip_v1	query set · RL	authored + generated	any	kitchen_counter_v3	4 tasks	task-specific

Record schema

{ meta: { name, source, robot?, scene?, license, collected_at }, video: [rgb, wrist?, depth?], instruction: "load the mugs into the top rack", actions: list<action>, # per-frame # ee pose + gripper quality: { score, reasons[] } }

Stored in LeRobot dataset format so it loads directly into openpi, Isaac-GR00T and LeRobot training.

Sources

Collect

Egocentric human video (head / wrist cams) and teleoperation in the Artha Labs robotics lab or on partner rigs.

Import

Owned and partner sources, open sets (Open X-Embodiment, DROID, EgoDex), including multi-agent datasets. Normalized to one schema.

Generate from human video

Human demos converted for a specific robot: hand extraction, robot overlay, action mapping (below).

Generate from text

Source matching clips from the web or synthesize them with a world model, then run the same pipeline and quality gate.

Human → robot pipeline

Hand pose

3D hand keypoints per frame (MANO-style).

→

Hand extraction

Segment + inpaint the human arm out.

→

Robot overlay

Render the target robot at the estimated end-effector pose.

→

Action mapping

Retarget to robot joints; IK feasibility check.

→

Quality grader

Score, reject, or send to human review.

clip_00412 · "place mug in top rack"

source · human

hand extracted

robot overlay · YAM

0.91

quality score

0.94

hand-pose confidence

100%

frames IK-feasible

keep

decision

Quality gate · last 10K clips illustrative

Kept71%

Hand occluded / low pose confidence11%

Kinematically infeasible for robot8%

Inpainting / overlay artifacts6%

Sent to human review4%

Artha Labs adds this gate plus an agentic retry loop (re-estimate pose, re-render) on top of the published pipelines.

Measured on our own video (Phantom pipeline)

77 / 424

frames valid (5 cm check) on uncalibrated RGB

0.46–0.73 px

reprojection error after phone ChArUco calibration

3 / 4

calibrated clips with no rejected frames; clip 4 rejected 217 / 360. Logs can't say why, which is the gap a quality grader fills

Prior art this builds on

Phantom

up to 92%

success, zero-shot on real hardware, trained on human video only

Inpaint the hand, render the robot at the estimated hand pose. Tasks: deformable manipulation, multi-object sweeping, insertion. arXiv 2503.00779

Masquerade

5–6×

over baselines on 3 long-horizon bimanual kitchen tasks

675K edited frames from in-the-wild video + 50 robot demos per task; performance scales log-linearly with edited video. arXiv 2508.09976

Mock only — illustrative data, except the cited prior-art results. Next: Rollouts →