15 Reasons Robotics is Hard

AI progress is racing along, but virtually all of the visible progress is in the realm of knowledge work, i.e. activities that can take place inside a computer.In the San Francisco AI scene, there is a widespread belief that robots will soon enter the picture. In parallel with the race to develop broadly capable AI, there is an equally aggressive race to develop broadly capable robots – humanoid machines imbued with physical intelligence. Artificial workers that can cook and clean, fetch and carry… and do everything else, including building more of themselves, leading (in many forecasts) to economic growth best characterized as an “explosion”.In other words, the thinking goes, AI in the data center will soon subsume all intellectual labor, and AI in humanoid bodies will soon subsume all physical labor. However, there is an important difference: while we can see progress in the intellectual realm, the physical side of AI is mostly confined to test facilities and demo videos. There is no robot equivalent to ChatGPT – nothing that you or I, or even most people in the AI community, can get our hands on.So we’re stuck with demo videos. Unfortunately, they are a poor tool for assessing progress. We might be seeing the one successful task achieved in 100 attempts. The scenario might have been carefully arranged to avoid challenges the robot isn’t ready for. The video might be edited to make it look like the robot is acting with more speed and reliability than is actually the case. Here’s one very impressive demo… with a suspiciously large number of camera cuts.(I have not yet had much chance to watch videos from the recent World Humanoid Robot Games. These are valuable for providing a public platform less amenable to cherry-picking. The handful of videos I’ve watched include some impressive feats, but don’t address many of the challenges I list below… and there are also a lot of spectacular failures.)Demos draw attention to the things a robot can already do. The question then becomes: what’s missing? In today’s post, I’ll catalog the technical challenges that will have to be overcome along the road to broadly capable artificial workers. The next time you watch a robot doing something impressive, ask yourself: which of these capabilities has the robot demonstrated, and which challenges might the demo scenario be avoiding?(Note that some challenges get easier if we consider wheeled robots rather than strictly humanoid robots. A wheeled robot can carry more weight, meaning that strength, endurance, and power for electronics are less of a challenge. And wheeled robots are less likely to fall over. But they can’t climb stairs1, step over clutter, or angle themselves to reach into a cupboard.)Maybe one of the last human jobs will be close-up magicThe human hand is an engineering miracle – opposable thumbs, and all that. It has roughly two dozen “degrees of freedom” (distinct joints and/or directions in which each joint can bend), and approximately 17,000 tactile sensors. Our brains can control our hands with exquisite grace, using touch, sight, and even auditory cues to carry out all manner of delicate tasks, precisely and reliably.Current robot “manipulators” are a pale imitation. Some existing robot hands can match the human standard on one or another physical attribute. For example, some have as many as 27 degrees of freedom. However, none come close to matching the overall package of flexibility, sensitivity, strength, reliability, and other physical attributes. It is the combination of factors that is especially difficult to match, even if the demos are getting more impressive. For instance, some companies have managed to cram thousands of tactile sensors into a robotic fingertip, but none have managed to make these tiny sensors able to stand up to heavy use2.The control problem may be as challenging as the problem of physical construction. A competent robot must be able to find the right set of joint positions to grasp a complicated object; plan out the sequence of motions to fold a shirt, flip an omelette, or tighten a bolt in a constrained space; and handle squishy or floppy materials (which can require reacting instantly to a sudden shift).Computer vision has made incredible strides over the last decade or two (and is responsible for kicking off the deep learning boom that led to LLMs). But making sense of complicated visual scenes – picking out an object from a crowded environment, understanding where it should be grasped, determining where it’s safe to put your feet and how to avoid knocking something over – is not a solved problem.A general-purpose robot must be able to break down a task into individual steps, and relate those steps to its environment. How do you maneuver your arm to get a screwdriver into a piece of machinery? What’s the quickest way to clear a path to the spice bottle at the back of the shelf? In what order should you pick up the items on the living room floor?True autonomy will require planning tasks of greater scale and complexity: cooking a meal, plumbing a bathroom, repairing an engine. Not to mention the need to re-plan in the face of surprises – a stuck bolt, a rotten piece of produce, a child darting into the kitchen.When current AIs fail at a knowledge work task, it’s often because they weren’t provided with sufficient context. Robots will need context, too: where are supplies kept? How do you like your meals cooked? How much assistance does that nursing home resident need, and is that hitch in their stride normal, or a sign that they’re about to stumble?Once they have context, robots will need to reason, plan, and exercise judgement and common sense. LLM-based systems like ChatGPT and Claude are making great strides in these areas, but the physical domain brings additional challenges3. The success of LLMs has been greatly assisted by the massive pools of pre-existing data that were available for training – a substantial fraction of all books ever written, the web, and other massive pools of pre-existing data. It will be difficult to match this scale of breadth and depth of data for physical tasks. There’s no straightforward equivalent of “just Efficient learning, generalization, and adaptability / on-the-job learning seem like requirements.AI agents mostly operate in isolation, and in static environments. We rarely put them in situations where things are changing out from under them, or ask them to coordinate. When we do, things often go haywire. Isolation is easier to arrange in the virtual world, where private workspaces can be created at will, and nothing is too heavy to lift on your own. Robots will often need to cooperate with people, or with one another.From an article which notes “it took the H1 nearly a full two minutes to very slowly move to a couch, pick up a single item of clothing and put it into the washing machine”.Today’s general-purpose robots often move much more slowly than human beings. Challenges include strength, control (higher speed means less time to plan and react), and safety (a fast-moving robot will whack you harder and is harder to dodge).For some applications, slow and steady may be perfectly acceptable: I may not care if my household robot takes all night to tidy up and fold the laundry. But a slow-motion robot won’t be much use as a cook or nursing-home aide. It might get in the way at a warehouse. And it will have a harder time getting enough work done to pay for itself.That… is just not an impressive amount of weight for a full-grown robot.Some industrial robots are extremely strong. But humanoid robots – or other highly mobile, “general-purpose” robots – usually aren’t. It’s difficult to combine strength with manageable weight, a large number of joints, and a maneuverable frame. Powerful motors generate more heat and deplete batteries faster – two areas where robots already struggle (see below). And a strong, heavy robot poses greater safety challenges.ED-209 may have autocannons and a rocket launcher, but it was no match for the staircaseThe jury is still out on the appropriate form factor for general-purpose robots, especially with regard to their lower half. Should they have wheels or legs? Two legs, four, or some other number? Wheels are cheaper, more stable, and more reliable; legs are better for stepping over obstacles and climbing stairs. A bipedal frame is more maneuverable, but also more likely to topple if something goes wrong. In any case, the question is: can the robot reliably get around its work environment?Sadly, yes, that is a robot karate-kicking a child in the stomach (video). Fortunately the kid was OK.Safety considerations for general-purpose robots are almost limitless. A glitchy or malfunctioning robot could bump into someone, topple onto them, drop something on them, spill something on them, break a glass, or start a fire.Safety for LLM-based agents relies in part on review of discrete actions, such as attempts to send an email or delete a file. Robots move constantly, and it’s not so easy to single out a few specific motions as the potentially dangerous ones requiring review.If a self-driving car finds itself in a situation it can’t handle or suffers a glitch, it can pull over or, in the worst case, just hit the brakes. A general-purpose robot that suddenly freezes might leave something on the stove, topple mid-step, or trip the person it was assisting.And of course danger can be initiated by human action, such as a child darting in front of a robot. I’d much rather my kid be bumped into by a squishy person than a metal robot; and as things stand today, I’d much rather depend on human reflexes and adaptability to avoid tripping over the little rascal.(The stronger, heavier, and more capable the robot, the greater the risks.)Today’s bipedal robots can typically run for a few hours before recharging. I suspect this won’t be a limiting factor: if a workaround is needed, we’ll find one, whether that means swapping battery