navigate to stationarrives upright · 186 steps
this is a best-case clip~1 in 3 · usually knocks it off
An incident in plain English is matched to a standard operating procedure, the procedure is translated into a chain of robot skills, and each physical step is executed and verified against measured physics. Build log below — every RL run and result, newest first.
language → SOP retrieval → plan → skill execution → verification. Enter an incident. The system retrieves the matching SOP as a human worker reads it, turns each written step into a verifiable robot skill, and infers the physical moves the SOP leaves implicit — a human just knows to walk to the machine. This is the real system:
Retrieval = a semantic sentence-transformer (all-MiniLM-L6-v2) over 100 real SOPs. Each written step maps to a skill; implicit navigation is inferred. verified = checked against MuJoCo physics · reach-limited = attempted but blocked by the frozen base controller · operational = sensor read / wait / notify · inferred = a move the SOP omits.
The chain above is executed by these verified skills — locomotion (PID + the AMO whole-body controller) and manipulation (reinforcement-learned). Each runs as its own reset→rollout→verify episode.
The brain does two jobs: retrieve the right SOP for a plain-English incident, and plan it into robot skills. We trained our own model for each. One worked, one didn’t — so the honest call was never “custom vs frontier,” it was task by task.
Retrieval rewarded a small fine-tuned model, so we kept ours. Planning did not — a frontier LLM (Claude/Gemini) reasons over the top-k retrieved SOPs and writes valid, sensible skill chains out of the box, where our from-scratch planner produced nothing usable. We do not claim a planner accuracy number: this corpus has no gold reference plans, so we won’t fabricate one.
So planning runs on a frontier model, and our originality goes where it is actually defensible — the layer no API gives you: every planned skill is verified against measured physics before it counts. The model proposes; the physics disposes.
Locomotion was reliable; pressing a button was not. Five iterations exposed three distinct failures — reward hacking, a wrong button model, and base recoil — each caught by verifying on measured physics rather than the policy's own claim.
Plain PPO found a lazy local optimum: park the hand near the button and farm the "getting close" reward without ever committing. The mean policy never experienced a press, so it never learned one.
We cloned a scripted press to bootstrap the policy into the right basin — the hover vanished, stochastic success hit 10/10. But it jabbed: pressed deep, then let go. A new problem, but real progress.
The trace showed the entire button body translating — and in the wrong direction. The "press" was the hand dragging a loose cylinder outward. A success-rate number would have hidden this; the physics trace exposed it. We rebuilt the button: fixed housing, a cap that compresses inward.
The arm tracked its target perfectly — but reaching forward made the pelvis recoil 9 cm back to stay balanced, carrying the hand off the button. The fix: a realistic protruding e-stop button (bring the surface to the robot) + a closed-loop controller that corrects for recoil every step. Suddenly: a clean, natural, straight-on press.
bp-v5 passes the physics verifier — 5.3 cm, held 79 steps. But look at the clip: it's a contorted lean into a protruding mushroom e-stop, a button shaped to meet a hand that can't reach a flush one. On honest review we confirmed the robot's frozen whole-body controller recoils ~11 cm backward on any standing reach and won't lean, so it cannot press a normal flush button standing. The number passed a hacked setup — the same metric-vs-reality trap as the grasp. So we retract "solved": press_button is reach-limited, and the verifier now reports it honestly as NOT VERIFIED. That honesty is the point.
We rebuilt manipulation with a real Robotiq 2F-85 gripper, and got what looked like a real skill: across three runs the “grasp” climbed to a 60% sustained hold. Then we watched it frame by frame. The number was a lie the physics was telling — the position-controlled arm was driving the gripper 7.5 cm straight through the solid pedestal, and the block was spawning mid-air and dropping into the jaws. The “grasp” was a phase-through-and-jam exploit, not a grip. Our own metric said 60%; run that exact policy on a corrected simulator and it scores 0%. So we retired every grasp result and started over honestly.
Default contact was too soft for a kp=300 arm — it bulldozed the gripper through the pedestal, and the tool was dropped from a centimetre up instead of resting on it. We now rest the tool flush and stiffen the contact, so the hand is stopped by the surface. Every grasp before this fix is retired — they were measuring an exploit.
On the corrected physics the policy learns a genuine two-sided grip on the tool — 95%, no phasing, no jam. But it doesn’t lift: a grip-stability penalty we added over-discouraged any motion, so it grips and freezes. We’ve softened it and a lift run is training now. This is the real starting line — slower, but true.
We moved the grasp off the floating pedestal onto the actual workstation — the robot stands at the bench and picks a box off the table, the real manipulation task that feeds the SOP chain (walk → grasp → walk → press). The honest journey: it first faked the grasp, then contorted its arm through its own body to reach. Each was caught on video and fixed.
Our grip check counted any two-sided contact — so the policy pinched the top of the box (2.3 cm above its center) and scored a grasp without the box ever being in the jaws. Fix: a grip only counts when the box is centered between the pads, plus a reward that pulls it to center before closing. Grasp quality went from a top-pinch to a real 95% centered grip.
To reach a box sitting right at its waist, the policy buried its upper arm inside its own torso (elbow 2.2 cm from the body axis; physics didn’t catch it because the collision capsules are thinner than the visuals). It looked wrong, and it was. Fix: give the arm room — move the box further onto the table so an extended reach exists — and penalize the elbow entering the torso. A natural pose (elbow 14.3 cm clear) now reaches the box.
Replay a table grasp in 3D — drag to orbit, scrub the timeline:
This replay is an early table run (the contorted arm pose shown above) — it’ll update to the fixed run as training lands. Replayed from recorded MuJoCo state.
Real deterministic rollouts, MuJoCo · fixed base at the workstation · still open: the lift — it grips the box but won’t raise it yet · next: solve the lift, then GPU-accelerate training (MJX) for far faster iteration.
A plain-English incident is retrieved to an SOP, planned into a skill chain, and executed — each physical step checked against measured physics, and any step the robot cannot complete reported as not verified rather than passed. Full run:
INCIDENT "fetch the screwdriver from the workbench" ↓ retrieve → plan → execute (each step checked against measured physics) 1. walk_to(workbench) [VERIFIED] arrived upright — solid, repeatable 2. grasp(screwdriver) [VERIFIED] this run latched + lifted... but ~2 of 3 runs it knocks the object off 3. press_button(red) [NOT VERIFIED] can't reach a flush button standing (controller recoil) ✅ one solid skill (walk) · ⚑ a verifier that reports the hard ones honestly — even its own misses
The logged walk-and-grasp trajectory, replayed in-browser from the recorded MuJoCo state. Orbit and scrub the timeline; toggle walk+grasp vs grasp-only.
Behavior cloning — the single biggest lever. RL from scratch on a thin-margin target finds degenerate optima (hover, jab). Cloning a demonstration first drops the policy into the basin of the real behavior; RL only has to refine. Hover → deep reliable press.
The verifier — every physical step is judged by measured physics (button displacement held ≥25 steps, base upright), never the policy's own claim. It caught a reward-hacking jab and a wrong button model. A step is reported as not verified rather than passed when the physics disagrees.