An operator describes a plant-floor incident in plain language. A fine-tuned retriever finds the matching procedure among 1,012 SOPs, a planner turns it into a sequence of skills, and a simulated Unitree G1 runs that sequence in one continuous simulation: walk, pick, carry, press, lever. A verifier checks each step against measured simulator state and reports any step the physics does not confirm as failed.
The system has two parts joined by a typed contract: a brain that turns language into a sequence of skills, and a body, a Unitree G1 with an added gripper in MuJoCo, that executes it. A verifier checks every precondition and postcondition from simulator state rather than from the policy’s own report. It measures the outcome whether a controller or a learned policy produced the motion.
The benchmark has 1,012 SOPs across 46 equipment families with within-family hard negatives, and 2,338 incidents written in operator language that avoids the SOP’s title words. On the held-out test, the fine-tuned MiniLM bi-encoder scores 0.584 R@1 against 0.301 for keyword search. Two negative results also held: a larger from-scratch bert-base retriever did not beat TF-IDF, and an off-the-shelf cross-encoder reranker lowered R@1 from 0.584 to 0.442, so it is not used.
Full 1,012-SOP index, 269-incident held-out test, reproduced locally. The reranker lowered R@1 from 0.584 to 0.442 and is not used.
The split follows from the results in section 05. For geometric tasks such as walking to a pose or servoing a gripper, a controller is used. For contact-dominated tasks such as a spring-loaded button or a latched lever, RL is used.
The latch engages only when measured pad forces confirm contact on both sides, so it cannot grab across a gap. 11 of 12 held lifts in validation.
A learned reach of about 19 cm from the rest pose, with no IK seed. The press reaches 29 to 40 mm and holds, deterministic across 8 of 8 starts. Pressing once is enforced by ending the episode on success.
The motion is jerky and reliability is about 50%. The cause is a per-episode arm-reach bias the policy cannot observe. The fix is identified but not yet retrained.
A plain-language ticket goes through the trained 1,012-SOP retriever, the planner produces the sequence, and the robot executes it. Steps the robot cannot perform, such as wait, read, or notify, are shown as captions.
An SOP written for this demo is retrieved at rank 1 over the full corpus (0.897) and its five-action plan runs in one take. The recording below is take 3 of 3, which is the weakest of the three. Pick and the 2.8 m carry pass. The place drifts, the press misses, and the lever springs back, and the verifier reports each of these as failed.
RL grasping did not work and was replaced by a controller. Across about 10 reward variants, deterministic verified success was 0. Training metrics reported 30 to 60% “lift”, but the videos showed the box being squeezed out of the grip and moving upward, with the height check triggering while the box was airborne. The replacement is an IK plus force-gated-latch controller, which held 11 of 12 lifts. RL is used for the press and lever instead. Each of the exploits below was found by watching the rendered rollout, not from a metric.
Each of these was found by watching the rendered rollout, not by a metric.
The full write-up covering the system, the retrieval benchmark, the RL results, and the complete list of reward hacks is in the repo. All numbers in it were measured by us.