Two 24 joint Shadow Hands, an 88 key spring loaded piano, and a MIDI file of Nettspend's we not like you. No demonstrations, no behaviour cloning, no pretrained weights. Only reinforcement learning against the score.
The policy is handed a MIDI file and a pair of robot hands. At every control step it sees which of the 88 keys should be sounding now and over the next ten steps, where its own fingers are, and how far each key is currently depressed. It outputs joint targets. If the right keys go down at the right time, it scores.
The metric is F1 over key-steps. A key that should be sounding and is counts as a hit, a key that sounds when it should not is a false press. This is the same measure RoboPianist and RP1M report, which makes the number comparable, with one caveat noted in the results.
The constraint that shaped the whole project: no pretraining of any kind. RP1M's pipeline bootstraps from a motion captured or planner generated prior. Here the hands start as a ready pose hovering above the keys, and everything after that, which finger takes which note, when to travel, when to lift, comes out of PPO.
MuJoCo builds a world out of four pieces: a body that can move, a geom stuck to it that has shape and collides, a joint that says how it may move relative to its parent, and a site, an invisible marker you can measure from. The piano is 88 copies of the same four line idea.
<body name="key_0" pos="0 0 0">
<joint name="joint_0" type="hinge" axis="0 -1 0"
range="-0.066 0" stiffness="3" damping="0.1"/>
<geom type="box" size="0.07 0.011 0.006" rgba="1 1 1 1"/>
<site name="key_site_0" pos="0.07 0 0.006"/>
</body>
Each key is a hinge at the back, a box extending forward, a torsion spring holding it up, a damper, and a travel limit so it stops at the bottom. The site on top is the point the environment measures: how far down the key is, and whether it has crossed the threshold where it sounds. In the panel one finger comes down on one key and the key turns orange the instant it crosses that threshold. Nothing else in the hand is moving.
A spring alone stores energy and gives it all back. Hooke's law says the torque is proportional to displacement and points the other way, and the energy it holds grows with the square of that displacement.
A finger presses, does work against the spring, and that work becomes U. Let go and the energy has nowhere to put itself. It sloshes between spring and kinetic energy, and the key keeps crossing back through the depth where it counts as sounding.
The panel is that experiment in the sim. The same key, the same downward flick, run twice with nothing changed but the damper. With damping = 0.1 the key is back on its rest stop within 0.15 s and never leaves it again. With damping = 0 it comes off the stop twice more and is still moving at 0.29 s. The orange is the environment's own sounding latch: struck past the sound angle, released below 80% of it.
The damper stores nothing. It turns motion into heat with a torque opposing velocity, bleeding b q̇² out of the system every cycle, so the key returns to the top once and stays there.
Both hands come from MuJoCo Menagerie's Shadow Hand, loaded into the same scene as the piano and bolted to a rail. That is the hand in the panel, rendered from the actual rollout, not a stand-in.
The gap between 24 joints and 20 actuators is the interesting part. On the index, middle, ring and little fingers, the last two joints are driven together by a single tendon. You cannot command the fingertip independently of the middle knuckle.
This is not a modelling shortcut, it copies your hand. Try bending only the tip of your index finger while keeping the middle joint straight. It is impossible, unless you hold the finger with your other hand.
In the panel only the coupled actuator is being driven. The knuckle is held where it was, and the middle and tip joints swing down together, far enough to sound the key on their own.
The practical consequence is that a fingertip cannot be placed freely. Asking for a specific tip height also decides the middle knuckle, which is why black keys, which sit higher and further back, are much harder for the short fingers than white keys are.
A fixed base Shadow Hand covers maybe two octaves. Sliding the whole hand converts "reach an unreachable key" into "be in the right place when the note arrives", which is a planning problem rather than a kinematics one.
It also creates the failure mode that cost the most score for the longest time. A hand in motion drags its fingers across every key it passes, and each one of those is a false press. The panel runs the same 26 cm traverse twice: on the left the fingers stay at the extension they press from and two keys sound on the way past, neither of them asked for. On the right the flexors curl away as soon as the rail moves and nothing sounds at all.
The actor sees the state of both hands, the state of every key, and what the score is about to ask for. Everything is normalised by a running mean and standard deviation before it hits the network, (x-μ)/(σ+0.01), so raw radians, metres and booleans arrive on a comparable scale instead of letting the 88 dimensional key block dominate.
The 20 hand outputs are not absolute joint positions. Each is a number in [-1, 1], scaled by 0.8 rad and added on top of the ready pose, so an all zero action leaves the hand hovering over the keys instead of collapsing. That one choice is why exploration from scratch starts in a sane place.
The rail is not driven directly either. A scripted servo moves the hand to wherever the fingering plan says the palm should be, and the policy adds a residual of at most 5 cm on top. The policy is responsible for the fingers, the plan is responsible for the geography.
The ten hold flags replace the sustain pedal. If a finger's hold output exceeds 0.5, the last key it struck keeps ringing after it lifts. One flag per finger rather than one global pedal means the policy can release one note while sustaining another, and if it holds a wrong note, it still pays the false press penalty.
The critic gets the same 1272 dimensions plus the contact force on each of the ten fingertips and a hand versus hand collision flag. Its entire job is to predict incoming reward and it is thrown away at deployment, so it is free to use information a real robot could never measure. Cleaner value estimates mean lower variance advantages, which means less noisy actor updates, and none of it leaks into the policy.
Everything starts from one MuJoCo state in one environment. Two things get read off it. The first is the observation the policy is allowed to see, built by _get_obs_ego: the joints, the rails, all 88 key positions, the piano roll, the 12 key windows around each palm, the upcoming notes, the pedal, and the previous action. The second is critic_extras, which is the fingertip contact forces and the hand versus hand collision flag. That second block never touches the actor.
From there the two networks split. The actor gets the policy group on its own. The critic gets the policy group and the privileged group concatenated together, so it runs on a slightly wider vector than the actor does. Each network has its own normalizer, and each one subtracts its own running mean and divides by its own running standard deviation. They are separate normalizers because they are seeing different vectors, so the statistics are not the same.
Then it is just a stack of linear layers with ELU in between. The actor goes down to a vector of means, one mean per action. The critic goes down to a single number, which is V(s), its guess at how much reward is still coming from this state.
The actor's output is not the action yet. It is the mean of a Gaussian. There is a separate learned standard deviation, one per action dimension, that is not a function of the observation at all, it is just parameters. To get an action you draw from a standard normal, scale each number by its std, and add its mean. That is the whole sampling step. You also keep log π(a|s) for that exact sample, because the update later needs to know how likely the old policy thought this action was.
The action goes into the env as a residual on the ready pose, which becomes the actuator targets, and MuJoCo runs ten physics substeps per control step. Out comes a reward, a done flag and a timeout flag. Every one of those pieces gets written into rollout storage as a single row. Then the next MuJoCo state comes back around to the top, and it repeats for 32 steps across all 1024 environments before anything is learned from.
The panel shows the dimensions of the pedal era network, 1175 observations and 43 actions into a 512-256-128 MLP. The shape of the loop never changed. By v15 the observation had grown to 1272, the action to 52 because the pedal became ten per finger hold flags, and the hidden layers to 1024-512-256.
The model is an actor critic, trained with proximal policy optimization. The two networks are built the same way, a stack of linear layers with ELU in between, but they answer different questions. The actor generates actions. Given the state of the hands, the keys and the score, it outputs the Gaussian that the action is sampled from. The critic evaluates the state. Its output is a single number, V(s), its estimate of how much reward is still coming from here.
V never plays a note. Its job is to give the actor something to compare against. A reward of +1.4 on its own says little about whether the action was any good, because some moments in the song pay well whatever the hands do and some are hard whatever they do. What the actor needs to know is whether this step went better or worse than the critic expected from this state.
If the advantage is positive, the action beat the critic's expectation, and the update makes it more likely by moving the actor's mean toward it. If it is negative, the update moves the mean away. At the same time the critic is regressed toward that same target, r + 0.99·V(s′), so its next guess for a state like this one is a little closer.
The two depend on each other. A better critic gives sharper advantages, sharper advantages give the actor a cleaner signal, and a better actor reaches states the critic then has to learn to value. That loop is why the critic gets the privileged contact forces from section 04. Anything that makes V more accurate makes every actor update less noisy.
The panel shows two samples from the same state. The first earns more than the critic predicted, A = +0.8, and the Gaussian moves toward it. The second is a false press, A = −0.7, and the Gaussian moves away. The numbers are illustrative.
Greedy, per chord fingering assignment kept producing travel the hands could not execute. A finger would be handed a note 16 cm away 0.1 s before its onset. The fix was to solve the whole song at once, a Viterbi pass over palm positions that scores every assignment against palm travel, finger switching, minimum hand separation, and a hard per finger per key reachability mask measured in the sim.
The planner's output does two things. It drives the rail servo's setpoint, and it tells the reward which finger was supposed to play each note.
The leap curriculum handles the dragging problem. Fingers are lifted mechanically whenever rail speed exceeds 0.10 m/s, then that scaffold decays to zero over 2000 steps and becomes a penalty instead, so the policy keeps the behaviour after the help is withdrawn. Brushed keys dropped from 101 per song to 5.
The composite reward follows the RoboPianist recipe. Key presses are scored through a sigmoid around the sounding threshold rather than a hard boolean, so there is a gradient to climb before the key actually sounds.
False presses are counted per intended note, not averaged over all 87 non goal keys. That denominator matters more than it looks. Averaging dilutes a wrong note to 1/87 of a unit and precision rots quietly. Dividing by the number of goal keys instead makes one misclick cost roughly what one hit earns.
The fingering term is the one the task does not work without. For every finger assigned a note this step, it rewards closing the distance from fingertip to the key's target point, through a Gaussian tolerance kernel. RoboPianist's ablation has F1 pinned at 0 without it, and the same is true here. A 50 DoF system exploring from key press reward alone never finds its first note.
Its falloff width was a real bug. At a 25x margin the term paid 0.996 for a fingertip hovering 2 cm above the key, close enough to collect almost full shaping reward, with nothing pulling it the last 2 cm down. Little fingers hovered over key 28 for an entire run without ever pressing it.
After 32 steps the rollout buffer holds, for every step, the value the critic predicted, the reward that actually arrived, and the done flag. One more critic call gets the value of the state after the last step, because you need something to bootstrap from at the end of a truncated rollout.
Then it walks backward through those 32 steps computing the surprise at each one. The surprise is what I got this step, plus what the critic now thinks the next state is worth, minus what the critic thought this state was worth. The advantage is that surprise plus a discounted share of the advantage that came after it, which is why it has to run backward. The return is the advantage plus the value that was already predicted.
The advantages get normalized over the whole batch, so the size of a gradient step does not depend on how big the rewards happen to be at this point in training. Then the same data is reused five times, in four minibatches of 512 each.
Inside each minibatch the actor runs forward again on the stored observations and produces a new mean and std, which gives a new log probability for the action that was already taken. The ratio between the new log probability and the old one is how much the policy has moved on that specific action. Multiply the advantage by the clipped ratio, flip the sign because the optimizer minimizes, and that is the policy loss. The entropy bonus, weighted at 0.001, pushes back against the std collapsing too early.
The critic runs forward on the same observations and its loss is just the squared error between the value it now predicts and the return computed earlier, in the clipped variant so a single update cannot move a value estimate too far.
Both losses get summed and handed to one Adam optimizer that owns the actor and critic parameters together. Gradients are clipped at norm 0.5, and the learning rate is adjusted up or down based on the KL divergence actually measured between the old and new policy, targeting 0.01. Too much drift and the rate drops, too little and it rises.
1024 parallel environments by 32 steps per rollout, clip 0.2, adaptive learning rate targeting KL 0.01. The entropy coefficient was the second largest single lever in the project. At 0.006 the action std drifted from 0.5 to 1.47 and F1 sat flat from iteration 800 onward. The rewards and values in the panel are illustrative. γ, λ, the clip range and the batch layout are the trained ones.
That split says the remaining loss is almost entirely notes that never get played, not wrong ones. Three causes, measured:
Every number below is a deterministic whole song rollout of the full 88 key piano on the real song. Not a folded keyboard, not an excerpt.
| Run | F1 | Recall | Prec. | What changed |
|---|---|---|---|---|
| v5 | 0.673 | 0.548 | n/a | first full keyboard baseline |
| v10 | 0.750 | 0.662 | 0.864 | chords rolled over 4 steps instead of 2 |
| v11 | 0.702 | n/a | n/a | pedal goal span narrowed, regression |
| v12 | 0.776 | n/a | 0.97 | leap curriculum, real pedal |
| v15 | 0.803 | 0.715 | 0.914 | reach mask, thumbs excluded, 1024-512-256 |
Audio is synthesised from the MIDI the policy actually produced in simulation, the key press events, not the original file. Everything it fails to play is simply absent.
Across fifteen runs, PPO's hyperparameters moved the score once, through the entropy coefficient. Everything else that mattered was a statement about the task: how a chord is spread across time, whether a finger can physically reach a key, what the reward counts as a mistake, where the hand should be before the note arrives.
The failures were informative in the same direction. Stickiness weighting, hand base placement, larger networks, contact stiffness tuning, six measured dead ends. The thing that finally moved recall was measuring, finger by finger and key by key, what the hardware could actually do, and then refusing to let the planner ask for anything else.