SEQUENT ROBOTICS
Project reportRev 2026.07
Project report

Ticket to task-execution, physics-verified.

The full project, one page: what the system does, the measured numbers behind every claim, the videos of every skill, and the failures — reported the same way as the wins. All simulation (MuJoCo), all numbers measured by us.

01 · Overview — the loop

One ticket, one robot, one take.

An operator types a plant-floor incident in their own words. A fine-tuned 22.7M-parameter bi-encoder retrieves the governing Standard Operating Procedure from a corpus of 1,012 SOPs; a deterministic faithful planner turns the SOP into a skill chain (one skill per written step); and a simulated Unitree G1 humanoid executes that chain in one continuous MuJoCo simulation — walk, pick, carry, place, press, lever — with a verifier measuring every step against simulator state, not the policy’s own claim. A step the physics doesn’t confirm is reported as failed. The finale below is exactly that: the system retrieves the right SOP at rank 1 (score 0.897), runs the five-action plan in one take, and honestly reports the steps that failed — verdict: “partially resolved.”

Finale · ticket → brain → robot · one takeverdict: partially resolved
The finale, take 3 of 3 — the take we recorded, not the best take. Retrieval rank 1 (0.897) and the pick + 2.8 m carry pass; in this take the place drifted, the press missed (0.0 mm), and the lever sprang back (0.636→0.953 rad vs. a <0.25 success bar) — so the system says so, on camera. Take 1 (the paper’s headline run) passed pick, place, and a 33.8 mm press and failed only the lever. A demo that cannot fail on camera is not verifying anything.
02 · The brain — retrieval over 1,012 SOPs

Recipe beat size.

The benchmark was rebuilt to be hard: 1,012 SOPs across 46 equipment families with within-family hard negatives, and 2,338 incidents written in operator voice that deliberately avoid the SOP’s title words. On the held-out test (269 incidents, full index), the fine-tuned MiniLM bi-encoder nearly doubles keyword search at R@1. Two negative results shaped the system more than the headline: a 10×-larger from-scratch bert-base retriever lost to TF-IDF, and an off-the-shelf cross-encoder reranker hurt the strong retriever — it is excluded.

Method
R@1
R@5
MRR
TF-IDF (lexical baseline)
0.301
0.535
0.408
From-scratch bert-base†
0.312
0.604
0.453
Fine-tuned MiniLM bi-encoder (ours)
0.584
0.851
0.697
+ off-the-shelf reranker
0.442
0.833
0.604

† bert-base row measured on the smaller 418-SOP validation corpus (where TF-IDF scores 0.396 R@1); all other rows are the full 1,012-SOP / 269-incident held-out test, reproduced locally. Reranker row from the original training-repo evaluation (0.572 base → 0.442 reranked).

Held-out retrieval · full 1,012-SOP indexmeasured, not quoted
Grouped bar chart: R@1, R@5, MRR for TF-IDF, the fine-tuned bi-encoder, and the bi-encoder plus reranker
The reranker bar is the honest one: adding a plausible-sounding component made the system worse (R@1 0.584→0.442). Components must be justified by measurement.
03 · The skills — controllers + RL, verified above both

Controllers for kinematics, RL for contact.

The architecture thesis was forced on us by evidence (see §04): where the task is geometric (walk to a pose, servo a gripper), a controller wins; where contact dynamics dominate (press a spring-loaded button, drive a latched lever), RL earns its complexity. The verifier doesn’t care which produced the motion — it measures outcomes.

Pick — walk up, grip, lift

Controller · DLS-IK reach + force-gated latch · on the walking robot
walk → pick · continuous11.9 cm held lift
The latch is the honesty mechanism: it engages only when measured pad contact forces confirm a real two-sided pinch — it cannot grab across a gap. 11/12 held lifts in standalone validation; 11.9 cm held lift in this continuous walk→pick run.

Press — reach from rest and press once

RL · PPO + success-gated curriculum · BC warm-start in its ancestry
walk → press · end to end8/8 det · PUMPS=0
The learned reach covers ~19 cm from the true rest pose — no IK seed — and presses 29–40 mm, held, deterministic 8/8 across near and far starts. Press-once is enforced by episode termination (the pump-cycling exploit is closed) and measured live: PUMPS=0.

Lever — the hardest skill, partially solved

RL · BC from scripted arc demos → pull-only DAgger → PPO + entropy crush
lever · curriculum diagnostic~50% reliable · jerky
Honest caveat: the motion is jerky and laggy-looking, and reliability across stances is ~50%. First full deterministic throw landed 51° into the success band, but the root cause of the flakiness is structural — a per-episode arm-reach bias the policy cannot observe — identified, fix designed, not yet retrained. Reported as a partial skill.

G6 — the one-take loco-manipulation chain

walk → pick → carry 2.8 m (180° turn) → place → press → rest · one simulation, no cuts
G6 · single unbroken takeall criteria pass
Box never dropped, robot never fell, RL press 34.5 mm with PUMPS=0, every criterion measured in-run. The take also surfaced and fixed a real bug (a stale carry force corrupting the place) — which is why gates are run as takes, not stitched from clips.

Gate 7 — the ticket loop

ticket → trained retriever → faithful plan → robot · non-embodied steps captioned, not mimed
G7 · ticket → walk + pressSOP retrieved rank 1
First ticket-to-robot run: a plain-language ticket goes through the trained 1,012-SOP retriever (semantic MiniLM interleaved with lexical TF-IDF), the faithful planner emits the chain, and the robot walks up and executes the RL press. SOP steps the robot can’t embody (wait, read, notify) are captioned on screen rather than faked.

Finale — the full five-action plan

SOP-1013 (authored for the demo, disclosed) · retrieved rank 1 over the full corpus
finale · take 3 · one takepartially resolved — honest
Same video as §01, placed here for the record: this recording is take 3 — the weakest of the three takes (pick + carry pass; place, press, and lever fail; the lever springs back 0.636→0.953 rad). Take 1 passed 4/5 steps (33.8 mm press) and failed only the lever (0.883→0.443 rad vs. <0.25). We publish the take we recorded and label it, rather than re-rolling until the video flatters the system.
04 · What failed and why

The failures are the contribution.

RL grasping failed outright and became a controller. Across ~10 reward/configuration variants, deterministic verified success was 0. The training metrics said 30–60% “lift” — the videos showed the box being squeezed out of a crushing pinch and popping upward ballistically, with the height check firing mid-air. Three compounding causes: a contact-fidelity wall in the simulator’s pinch resolution, a dithering stochastic grip that never achieves force closure, and a reward that cannot tell held from launched. The fix was to stop using RL: the IK + force-gated-latch controller held 11/12 lifts on its first day. RL went where it earns its complexity — the press and the lever.

parking
Proximity income near the button → the policy hovered 5 cm short, farming the band forever. Fix: narrow the band 0.20→0.05 m, double the distance pull.
seat farming
The lever’s seat gives ~0.17 rad of free travel; linear progress income paid an annuity for sitting in it. Fix: re-baseline income past the seat, make progress convex.
pumping
Per-step depth income made re-pressing profitable — press, release, press again. Fix: success terminates the episode. Nothing left to farm.
air-swinging
With press-once in place, motion after the press was free — the arm windmilled while the hold counter ran. Fix: post-press stillness cost + dip-tolerant hold counter.
flail-to-press
The anti-jerk penalty was out-earned by fast flailing that bumped the button (jerk 0.83). Fix: a low-pass filter on the command — a constraint, not a suggestion (jerk 0.001).
ballistic pop
Any instantaneous lift signal is satisfied by launching the box. Fix: removed RL; the verifier’s held-state check (grip maintained AND supported, sustained) is the only signal that can’t be hacked.

Every one of these was caught by a human watching the rendered rollout — none by a metric. Watch the video, not the metric.

05 · The paper

The full write-up, with every number.

The complete paper — system, retrieval benchmark, the RL post-mortem, the reward-hack catalog, limitations, and everything this page summarizes — lives in the repo. All numbers in it are measured; failures are first-class results.

Read the paper ↗ Training log →