Evaluation

Grade rollouts you can actually train on.

Task sets with rubrics, a Grader Store with video-grounded, reward-hack-resistant success detection, and human judges in the loop. Every score is auditable and every sim number is paired with a real-world eval.

Graders are the hard part, and the bar is rising. LLM-as-physical-grader is promising but easy to reward-hack, and credible zero-cost baselines now ship in LeRobot (Robometer, TOPReward). Artha Labs' graders are multi-signal (geometric + VLM judge + physics consistency), adversarially tested, and calibrated by human judges. Reconstruction is becoming commodity; reliable graders are not.

Task sets & rubrics

Task set · kitchen_manip_v1 · scene kitchen_counter_v3 × Humanoid-G1

Task	Rubric	Graders	Status	Mean score
load_dishwasher	0–5 progress	4	scored	3.6
open_drawer	0–5 progress	3	scored	4.4
wipe_counter	0–5 progress	3	in review	—
stack_plates	0–5 progress	2	draft	—

Lifecycle: create (from the task description + scene) → review (human judge signs off the rubric) → score (graders run on rollouts). Graded partial progress instead of binary success, the same style RoboWorld uses to reproduce real-world policy rankings.

Rubric · load_dishwasher

No item grasped

Item grasped, not moved to rack

1–2 items placed

3–5 items placed

All 6 placed, one not upright

All 6 upright, no breakage, door closed

Grader Store

Graders · task: load_dishwasher

📐

geometric_success

all 6 items inside rack bounding volume, upright (privileged sim state)

primary

🧠

vlm_judge

Cosmos-Reason: "is this correctly & safely loaded?" (video only, so it also runs on real rollouts)

cross-check

⚖️

physics_consistency

no clipping / interpenetration / impossible forces

anti-hack

🚫

breakage_penalty

no glass shattered, no collision > force threshold

shaping

reward = 1.0·geometric_success + 0.3·vlm_judge.agree − 0.5·(1 − physics_consistency) # kills sim-exploit policies − 0.2·breakage_penalty

Hack detector

⚠ Reward hack flagged
Batch #5512: 12% of "successful" rollouts placed items by clipping through the rack wall (physics_consistency = 0.0). Excluded from training; grader auto-tightened.

Without the physics-consistency term, these would have trained a policy that exploits the simulator and fails on real hardware.

Baseline check: beat the free graders

Grader	Method	False-positive rate on hacking policy
Robometer	Qwen3-VL-4B + progress / success heads, trained on RBM-1M	31%
TOPReward	Zero-shot: Qwen3-VL-8B log-prob that the task is complete	38%
Artha Labs multi-signal	geometric + VLM + physics consistency, judge-calibrated	4%

Illustrative numbers. The protocol is real: this table is the release gate for every grader.

artha eval.adversarial( grader = "load_dishwasher@v4", suite = "hack_suite_v1", # degenerate policies: # clip-through, hover-near, # knock-over-then-count baselines= ["robometer", "topreward"], metric = "false_positive_rate", )

Each task gets a deliberately degenerate policy that should score 0. A grader ships only if it beats both LeRobot baselines on that suite.

Human judges in the loop

Review queue

scene

kitchen_counter_v3 · drawer travel inferred 0.48 m, confidence low

approveedit

rollout

#5512 · 37 rollouts where vlm_judge and geometric_success disagree

label

dataset

ds_kitchen_h2r_v2 · 112 clips with hand-overlay artifacts

keepdrop

rubric

wipe_counter · rubric draft awaiting sign-off

approveedit

Only disagreements and low-confidence items reach humans. Their labels recalibrate the graders. Judges are an internal team plus partner orgs.

Calibration illustrative

94%

judge–grader agreement on spot checks (n = 500)

of rollouts routed to humans

Real-world eval: the verdict

Batch #5512 on real hardware illustrative

65% ± 5

real success, 95% CI (RoboArena-style, n = 350)

sim success71% ± 1.4 (n = 4,096)

sim-to-real gap−6 ± 5 pts

Why n matters

95% CI half-width = 1.96 · √( p(1−p) / n ) p = 0.65, n = 40 → ±15 pts # a 6-pt gap is noise p = 0.65, n = 350 → ± 5 pts # a 6-pt gap is measurable p = 0.50, n = 2,401 → ± 2 pts # each rollout needs a manual reset

Real-robot eval doesn't parallelize, so the sample size is reported with every number. Sim and neural-sim evals are used to rank policies cheaply. Real eval is used to confirm the ranking.

Measured in our lab

94.0%

π0.5 on LIBERO libero_10, 100 eps (published 92.4 / 96.0)

100 → 20%

π0.5 success when the wrist camera moves 10.5–56 mm (external: 80% at 269 mm)

1 misgrade

Arena grader scored a cube that bounced off the bowl as failure → grader audits

Mock only — illustrative data. Next: Training →