FIRST PRINCIPLE
Do not call a literature result an OpenRGD measurement.
A paper may evaluate a policy on one robot, another number of demonstrations, a different camera setup or a different task. Those values are useful background, not a head-to-head comparison for this lab.
An OpenRGD claim about ACT versus SmolVLA requires the same task success definition, same physical embodiment, matched evaluation starts and controlled data/compute conditions—or a transparent explanation of differences.
TASK PROTOCOL
Freeze the physical task before collecting success statistics.
The first proposed task is a bounded object transfer inside a marked, safe, reachable workspace. The success condition is explicit: a lightweight object is released entirely inside the target zone before the predeclared timeout, without human intervention or a safety abort.
Object properties, target geometry, camera angles, starting conditions and reset procedure must be fixed. The protocol intentionally leaves the timeout unset until the experiment is actually configured; no fabricated threshold is published.
COMPARISON CONDITIONS
Use the same evidence protocol across policy variants.
The protocol defines four future conditions: human teleoperation reference, ACT autonomous policy, SmolVLA synchronous inference and the same SmolVLA checkpoint with asynchronous inference. Conditions remain marked as planned until their actual checkpoints, dependencies and observed trials are archived.
Twenty matched trials per condition is an operational floor for a first comparative report, not a statistical guarantee. Every abort, loss of tracking and operator rescue must be counted under an explicit failure category.
Human-driven reference
Measures task execution and physical setup; does not represent autonomous policy ability.
Imitation policy
Requires a pinned trained checkpoint, dataset revision and evaluation configuration.
Synchronous action chunks
Same task protocol with explicit inference and execution timing.
Asynchronous action chunks
Same SmolVLA checkpoint; log queue starvation, action age and correction behavior.
FAILURE TAXONOMY
Failure type matters as much as aggregate success.
Use a fixed list including grasp miss, object drop, off-target release, perception/occlusion, limit/workspace, action staleness, controller mismatch, timing/dropped frames, operator abort, safety abort and timeout.
If a safety check stops a trial, this is not a hidden exclusion. Report it both in the denominator treatment and as its own event; the protocol must be frozen before comparing conditions.
EVIDENCE RELEASE
What a publishable comparison must contain.
At minimum: exact robot configuration, task video and numeric state/action traces where permitted, data card, calibration and camera hashes, policy checkpoint revision, trial-level CSV with outcomes, predeclared exclusions, run timestamps and checksums. Publish raw counts and uncertainty instead of a single heroic success percentage.
The public protocol and header-only CSV template are available from the SO-101 lab; they contain no invented experiment rows.