A Unified Model for Motion-Conditioned Robot Co-design

Transformer Transformer Huy Ha, Karen Liu, Shuran Song

Not all robots are created equal Not all robots are created equal — but what if you could design one for a specific task? We propose Transformer Transformer, a unified model that does exactly this: hand it a manipulation demonstration, and it generates a complete robot — every link, joint, motor, and inertial property — optimized for that motion. We fabricated one such design for cloth flinging on an ALOHA2 bimanual platform; it reduced tracking error by 73% and max joint speed by 30% versus the original. Behind that result is a diffusion transformer trained on RoboTokens, a unified tokenization of robot embodiments, states, and actions. The same architecture spans embodiment spaces (wheeled bimanual, quadrupeds, humanoids) and use cases (embodiment generation, cross-embodiment control). Rather than overfitting to one reward function, it is a dynamics model whose reward-agnostic predictions are converted into reward-specific value predictions at inference time, then used to steer embodiment diffusion through a procedure we call Dynamics Self-Guidance. Experiments across three design spaces show zero-shot optimization of unseen rewards and trajectories, improving performance and runtime over an evolutionary baseline.

Technical Summary Video A 17-minute walkthrough — or scroll for the figure-and-caption version.

What is the best robot for a given manipulation task?

A robot's embodiment decides what tasks it can do well. We pour effort into making robots smarter — bigger datasets, better algorithms, more capable policy architectures — but a bad embodiment can hamper even a great policy. You can collect all the tossing data in the world, but if your robot's shape is far from optimal, it might toss itself instead of the ball.

So let's frame the problem concretely. Given as input a target end-effector motion and a reward function, we want to generate a complete robot embodiment for the task — by complete we mean every link, joint, motor, and inertial property, plus a controller that drives the whole thing. We call this motion-conditioned robot co-design.

A randomly procedurally-generated robot attempting the target motion: it falls before it can finish 🙃

Demonstrate, Generate, Validate

Our framework reframes robot co-design as a three-step process: you demonstrate the desired end-effector motion (e.g., from a UMI3 human demonstration), our model generates an optimized embodiment, and the same model validates the design by directly controlling it. The same network plays three roles — generator, critic, and cross-embodiment controller — by training jointly on one unified tokenization of robots.

Three steps, one model. Given target end-effector motions (e.g., from a human demo) and a user-defined reward, Transformer Transformer generates a complete robot — geometry, kinematics, inertia, motors — then controls that robot to track the motion. No separate optimizer, critic, or controller pipeline.

The paper has three parts: a unified robot representation (RoboTokens), a unified architecture (Transformer Transformer), and a unified training objective (Dynamics Self-Guidance). You have until the architecture section to figure out the pun 🥸

PART 1 RoboTokens A Unified Robot Representation

Every robot becomes a typed token sequence. Blue tokens encode embodiment (links, joints, motors); orange tokens encode dynamics (states and actions).

If we want one model to co-design any robot, the network needs a shared vocabulary that spans every embodiment. RoboTokens are that vocabulary: typed tokens for every link, joint, motor, state, and action — organized so a single sequence describes both a robot's embodiment (time-invariant) and its dynamics (time-varying).

RoboTokens has the following key design goals: Complete. A RoboToken sequence captures everything about a rigid articulated robot: 5 embodiment token types (links, fixed joints, rotating/sliding joints, ball joints, motors) plus state and action tokens. Flexible.