Fermat's Library | It Takes Two Neurons To Ride a Bicycle annotated/explained version.

It Takes Two Neurons To Ride a Bicycle Matthew Cook ∗ Abstract Past attempts to get computers to ride bicycles have required an inor- dinate amount of learning time (1700 practice rides for a reinforcement learning approach [1], while still failing to be able to ride in a straight line), or have required an algebraic analysis of the exact equations of motion for the specific bicycle to be controlled [2, 3]. Mysteriously, hu- mans do not need to do either of these when learning to ride a bicycle. Here we present a two-neuron network 1 that can ride a bicycle in a de- sired direction (for example, towards a desired goal or along a desired path), which may be chosen or changed at run time. Just as when a person rides a bicycle, the network is very accurate for long range goals, but in the short run stability issues dominate the behav- ior. This happens not by explicit design, but arises as a natural conse- quence of how the network controls the bicycle. 1 Introduction The task of riding a bicycle presents an interesting challenge, whether for human or for computer. We do not have great insight as to how we ride a bicycle, and we do not have much useful advice for someone who is learning. In fact, in the course of this project, I had the chance to ride a “virtual bicycle” on the computer, and I was surprised to find how counterintuitive it is. I had thought that, knowing perfectly well how to ride a bicycle in real life, it would be no problem in simulation. However, in real life there must be additional inertial cues that I sense or leaning actions that I make which are missing from the simulation, since I had to learn, as if from scratch, what cues to attend to and how to react to them. I even thought at first that there must be a bug in the simulator, since to turn right I found I had to push the handlebars to the left. Of course, if you stop to think about it, that is exactly correct.

To turn right, the bicycle has to lean to the right, and the only way to make that happen

2 is to shift the point of contact with the ground to the left, which requires an initial push to the left. But then, once the bicycle is leaning to the right, it will itself push the handlebars to the right due to how it is constructed for stability, 3 with a force even greater than your initial push to the left, so maintaining a ∗ California Institute of Technology, Mail Stop 136-93, Pasadena, CA 91125 cook@paradise.caltech.edu

1 Actually, the title of this paper is unproven. We have not ruled out the possibility that a single neuron could ride a bicycle. 2 This is ignoring the torque effect due to the spinning of the front wheel, but if you take that into account, it too has exactly the same effect as the effect described above (pushing to the left makes you lean to the right). 3 See footnote 7 on page 6.

constant gentle leftward push does indeed cause the bicycle to turn to the right. Similarly, to come out of the rightward turn (or even to maintain it), you need to push the handlebars gently to the right. In this paper we outline the various portions of our project, which has led us to find a surprisingly competent two-neuron network. Different portions of this work are likely to be of interest to different people. The reader should feel free to skip over sections that are not of interest—the paper has been organized so that skipping ahead should not result in a loss of understanding when reading later sections. 2 Methodology: Overview of the Simulator System 2.1 The Physics In order to allow us to experiment with different bicycle controllers, we first have to set up a virtual bicycle for them to control. The equations of motion for a bicycle are somewhat complex [2], so it seems no more complicated and much more useful to just write a general robot simulator, which can read a description of an arbitrary robot (rigid bodies linked by hinge-like connections), and simulate how that robot will move given the forces being applied to it.

This entails calculating the moments of inertia for each rigid body, simulating the motion of a single rigid body given forces acting on it [5], and solving a system of equations at each step for how the hinge-like connections can apply forces to the parts of the robot so that the alignment and co-location requirements of the hinges are met. 4 2.2 The Bicycle Robot Once we have such a general purpose physics simulator, then we can turn to setting up a robot, in this case a bicycle. A bicycle is composed of four rigid bodies: the two wheels, the frame, and the front fork (the steering column). Each adjacent pair of parts is connected with a joint that allows rotation along a defined axis, and the wheels are connected to the ground by requiring that their lowest point must have zero height and no horizontal motion (no sliding). Figure 1: The virtual bicycle.

Beyond specifying the construction and connections that form the bicycle, we need to de- cide what sensory input should be available to the controller, and how the controller’s outputs should be converted into forces on the bicycle (in robotics terms, what the sensors 4 Even the support of the wheel by the ground counts as a hinge-like connection for this purpose.

and actuators should be for this non-holonomic under-actuated system). For our bicycle, we allow all the easily perceivable quantities to be available to an interested controller: Position, heading, speed, angle of the handlebars (and its rate of change), and the amount the bicycle is leaning (and its rate of change). For actuators, we allow a torque on the back wheel and a torque on the handlebars. Humans also make good use of leaning to one side or the other when they ride, but we will not have such a control on the riderless bicycle. Also, we do not allow the controller to know the specifics of the bicycle, such as its exact proportions or the masses of its parts. 2.3 The Controller Once we have set up the robot bicycle, we can turn to the task of interest: Designing a controller for the bicycle. We want the controller to solve the same problem that a human solves when riding the bicycle.

The human knows where they are, which way they are going, how fast they are going, how the bicycle is leaning, and so on, but as we know from experience the human does not need to know the specifics of the construction of the bicycle. Here we are finally faced with a problem that we do not, a priori, know how to solve. So we stare at the ceiling for a while, and whenever we are struck with some inspiration, we quickly write a controller based on it. There are three main styles of controller (prescient, human, and two-neuron) that have led us to interesting results or observations, and we will discuss them in the next three sections. None of them made significant use of the speed—they all managed to control the bicycle using just the handlebars. We will not discuss here those controllers which did nothing but crash the bicycle at every opportunity. 3 The Prescient Controller: A Look at Reinforcement Learning One interesting idea for a controller, given that the entire system is being simulated, is to let the controller cheat by giving it access to the simulator. This could not be done with a controller for a bicycle in the real world, so it is not of interest for applications, but we can certainly try it in the simulated world to see what happens. In particular, we can try the following algorithm for the controller: At each step, first simulate and compare three actions. The actions only differ in how the handlebars are pushed at the first instant: pushed left, pushed right, or not touched. The remainder of each of the three actions is to do nothing until the bicycle crashes. These three actions can then be compared on the basis of which one causes the bicycle to remain upright for the longest time, which one results in the most progress to the right, or whatever other criterion one decides to optimize. After simulating the results of the three actions, the controller decides what to do at this instant based on those results. (Each different criterion is thus the basis for a different controller.) These simulations were tried with and without random mild forces (“wind”) being applied to the bicycle.

The original motivation for this was so that the controller would not be able to rely on an absolutely perfect prediction of the future. It might also help the controller to have a more “continuous” behavior, since over the course of several consecutive instants, it would be getting a rough estimation of the probability distribution for success of each of its actions, leading to the controller taking a similarly distributed action. However, such wind turned out in fact to have no significant effect on the results. In the language of reinforcement learning, such a controller is exactly what you would get after one step of policy iteration, if you start with the null policy of never touching the handlebars, and allow yourself three actions at each step (push left, push right, or no

Figure 2: Instability of an unsteered bicycle. This shows 800 runs of a bicycle being pushed to the

right. For each run, the path of the front wheel on the ground is shown until the bicycle has fallen over. The unstable oscillatory nature is due to the subcritical speed of the bicycle, which loses further speed with each oscillation. push). Then if the controller learns the value function for this policy (which in practice would require lots of experience with not touching the handlebars, but which we simulate by giving the controller access to the simulator), it can then act greedily with respect to that value function. This amounts to one step of policy iteration, and at least for the goal of not falling over, an optimal policy is indeed obtained after a single iteration (i.e., it successfully doesn’t fall down). However, it does not do this in a conventional way, say by riding in a straight line, but rather manages to maintain stability at near-zero speed by doing stunts with the front wheel, for example by spinning the handlebars in circles (the handlebars and front wheel do not bump into the frame for our bicycle, and there are no cables to get twisted, so why not?). A movie of this bizarre behavior can be seen at: http://www.paradise.caltech.edu/∼cook/Warehouse/RecursiveBike.avi Despite many attempts at formulating a sensible value function, we found it difficult to get sensible behavior out of the bicycle.