Thought as a Technology

Have you ever felt awe and delight upon first experiencing a computer interface? An interface that surprised you with its strangeness, with a sense of entering an alien world? Some people experience this when they play imaginative video games, such as Monument Valley, Braid, or Portal. For some people, it occurs when they first understand how a spreadsheet program can be used to model a company, an industry, or even an entire country. And for some people, it occurs when they first use a programming language based on particularly powerful ideas, such as Haskell or Lisp. My own first experience of this awe and delight was when I used the program MacPaint. I was 11 years old. Up to that point, my experience of computers was with command line interfaces, such as the Apple II, IBM PC, and Commodore 64: To do graphical work was complicated. On my Commodore 64, I would sometimes make games, using graph paper to sketch out the pixels in my game characters, before translating them into a sequence of numeric values for input into the computer. It was detailed, painstaking work. One day, my parents took me to an Apple dealership. There, we saw a computer called a Macintosh, running a program called MacPaint. Using a new-to-me pointing device called a mouse, I could sketch much more directly on a virtual canvas. I remember feeling awe when I clicked on a region, and MacPaint filled the region in. It knew where the borders were! MacPaint also provided FatBits, a way to magnify the drawing, so it could be edited pixel by pixel. Most magical of all: if I made a mistake, I could undo the mistake with a click of the mouse button, returning the canvas to its previous state. MacPaint gave me a more direct connection to my drawing, similar to using pencils or paints. But it also provided new tools making it easy to do things that were formerly difficult or impossible. Today, such tools are familiar, and perhaps seem banal. But for me they were an epiphany, transforming what it meant to draw. Of course, most people never had the particular transformative experience I had with MacPaint. But I believe many people have been awed and delighted by some interface. In extreme cases, to use such an interface is to enter a new world, containing objects and actions unlike any you've previously seen. At first these elements seem strange. But as they become familiar, you internalize the elements of this world. Eventually, you become fluent, discovering powerful and surprising idioms, emergent patterns hidden within the interface. You begin to think with the interface, learning patterns of thought that would formerly have seemed strange, but which become second nature. The interface begins to disappear, becoming part of your consciousness. You have been, in some measure, transformed. What makes an interface transformational? Most interfaces are not so striking. But the existence of such extreme examples poses a question: what qualities make an interface transformational? To answer that question, it helps to consider another transformational technology, namely, language. Children acquire language in just a few years. It's remarkable to watch a child hear an unfamiliar word, then later speak that word aloud, and gradually come to use the word in more complex ways. Familiarity makes us take the process for granted, but it's marvellous that they can internalize an external phenomenon, and use it as a vehicle for their own thought. Language is an example of a cognitive technology: an external artifact, designed by humans, which can be internalized, and used as a substrate for cognition. That technology is made up of many individual pieces – words and phrases, in the case of language – which become basic elements of cognition. These elements of cognition are things we can think with. Language isn't the only cognitive technology we internalize. Consider visual thinking. If, like me, you sometimes think visually, it's tempting to suppose your mind's eye is a raster display, capable of conceiving any image. But while tempting, this is wrong. In fact, our visual thinking is done using visual cognitive technologies we've previously internalized. For instance, one of the world's best-known art teachers, Betty Edwards, explains that the visual thinking of most non-artist adults is limited to what she refers to as a simple “symbol system”, and that this constrains both what they see and what they can visually conceive: [A]dult students beginning in art generally do not really see what is in front of their eyes — that is, they do not perceive in the way required for drawing. They take note of what's there, and quickly translate the perception into words and symbols mainly based on the symbol system developed throughout childhood and on what they know about the perceived object. It requires extraordinary imagination to conceive new forms of visual meaning – i.e., new visual cognitive technologies. Many of our best-known artists and visual explorers are famous in part because they discovered such forms. When exposed to that work, other people can internalize those new cognitive technologies, and so expand the range of their own visual thinking. For example, cubist artists such as Picasso developed the technique of using multiple points of view in a single painting. Once you've learnt to see cubist art, it can give you a richer sense of the structure of what's being shown: Another example is the work of Doc Edgerton, a pioneer of high-speed photography, whose photographs revealed previously unsuspected structure in the world. If you study such photographs, you begin to build new mental models of everyday phenomena, enlarging your range of visual thought: Another class of examples comes from the many cartographers who've developed ways to visually depict geography. Consider, for example, the 1933 map of the London Underground, developed by Harry Beck. In the early 1930s, Beck noticed that the official map of the Underground was growing too complex for readers to understand. He simplified the map by abandoning exact geographic fidelity, as was commonly used on most maps up to that point. He concentrated instead on showing the topological structure of the network of stations, i.e., what connects to what: Images such as these are not natural or obvious. No-one would ever have these visual thoughts without the cognitive technologies developed by Picasso, Edgerton, Beck, and many other pioneers. Of course, only a small fraction of people really internalize these ways of visual thinking. But in principle, once the technologies have been invented, most of us can learn to think in these new ways. Let's come back to computer interfaces. In a similar way to language, maps etc, a computer interface can be a cognitive technology. To master an interface requires internalizing the objects and operations in the interface; they become elements of cognition. A sufficiently imaginative interface designer can invent entirely new elements of cognition: I believe this is what made MacPaint so exciting to 11 year-old me: it expanded the range of thoughts I could think. As a practical matter, this expressed itself as an expansion in the range of visual images I could create. In general, what makes an interface transformational is when it introduces new elements of cognition that enable new modes of thought. More concretely, such an interface makes it easy to have insights or make discoveries that were formerly difficult or impossible. At the highest level, it will enable discoveries (or other forms of creativity) that go beyond all previous human achievement. Alan Kay has asked*:* Alan Kay, What is a Dynabook? (2013). “what is the carrying capacity for ideas of the computer?” Similarly, we may ask: what is the carrying capacity for discovery of the computer? How can we invent new elements of cognition? Of course, most interfaces are re-combinations of standard elements, and don't introduce any new elements of cognition. Are there heuristics we can use to invent new elements of cognition? As a way of getting insight into that question, I will begin by showing a prototype interface. It's a prototype for exploring one-dimensional motion, that is, the motion of a particle on a line. To avoid disappointment, let me say that this prototype certainly isn't transformative in the same way as MacPaint! It's rough, a first sketch of an idea. But, as we'll discuss below, it illustrates two useful heuristics which can help us invent new elements of cognition. Note that this prototype is aimed at people who've taken an introductory physics class. That means familiarity with ideas such as the potential and kinetic energy of a particle. If you've less background in physics, I hope the gist is accessible. If you've more background, please put yourself back in the mindframe of a relative beginner. Note also that the prototype begins with some background explanation, before showing an actual interface. Here it is** An archived copy of this video is available here.: Heuristic 1: Reify hidden representations in the interface To understand the motivation behind this prototype, consider the following question from the mathematician William Thurston: How big a gap is there between how you think about mathematics and what you say to others? Do you say what you're thinking?… I'm under the impression that mathematicians often have unspoken thought processes guiding their work which may be difficult to explain, or they feel too inhibited to try… Once I mentioned this phenomenon to Andy Gleason; he immediately responded that when he taught algebra courses, if he was discussing cyclic subgroups of a group, he had a mental image of group elements breaking into a formation organized into circular groups. He said that 'we' never would say anything like that to the students. His words made a vivid picture in my head, because it