Originally presented as a lecture August 1999. Published on KurzweilAI.net February 22, 2001.

The human mind is a remarkable organ. It has allowed us to walk on the moon, to discover the physical basis of life and the universe, and to play chess almost as well as a computer. But the mind presents us with a paradox. On the one hand, many everyday tasks that we take for granted–walking around a room, picking up an object, recognizing a face, remembering information-are feats that scientists and engineers have been unable to duplicate in robots and computers. Nonetheless, these feats can be accomplished by any four-year-old, and we tend to be blasé about them.

On the other hand, for all its engineering excellence, the mind has many apparent quirks. For example, why is the thought of eating worms disgusting when worms are perfectly safe and nutritious? Why do men do insane things like challenge each other to duels and murder their ex-wives? Why do people believe in ghosts and spirits? Why do fools fall in love?

How the Mind Works is an attempt to answer those kinds of questions, using three key ideas: computation, evolution, and specialization.

The first idea, computation, is meant to explain how intelligence is possible in a physical system. But what is intelligence? Few people today are satisfied with the traditional psychologist’s definition, “whatever it is that IQ tests measure.” A better definition comes from William James himself, who tried to put his finger on the difference between intelligent behavior and superficially similar behavior that we would not ascribe to intelligence:

Romeo wants Juliet as the filings want the magnet; and if no obstacles intervene he moves toward her by as straight a line as they. But Romeo and Juliet, if a wall be built between them, do not remain idiotically pressing their faces against the opposite sides like the magnet and filings with the card. Romeo soon finds a circuitous way, by scaling the wall or otherwise, of touching Juliet’s lips directly. With the filings the path is fixed; whether it reaches the end depends on accidents. With the lover it is the end which is fixed; the path may be modified indefinitely.”

James identifies intelligence, then, with the pursuit of goals by means of inference, or the satisfaction of desires by beliefs about how the world works.

It is not just Romeo’s behavior that we need to explain by invoking beliefs and desires, but virtually all human behavior. If I were to ask you, “Why did Bill just get on the bus?,” to answer that question you wouldn’t run a neural network simulation, and you wouldn’t need to put Bill’s head in a brain scanner. You could just ask him, and you might discover that the explanation for his behavior is that he wants to visit his grandmother, and he knows that the bus will take him to his grandmother’s house. No science of the future is likely to provide an explanation with greater predictive power than that. If Bill hated the sight of his grand- mother, or if he knew the route had changed, his body would not be on that bus.

But this excellent theory raises a puzzle. The beliefs and desires that cause Romeo’s behavior, or Bill’s behavior, are colorless, odorless, tasteless, and weightless. Nevertheless, they are as potent a cause of action as any billiard ball clacking into another billiard ball.

How do we explain this seeming paradox? The solution, I believe, is that beliefs and desires are information. Information is another commodity that is colorless, odorless, tasteless, and weightless yet can have physical effects without resorting to any occult or mysterious process. Information consists of patterns in matter or energy, namely symbols, that correlate with states of the world. That’s what we mean when we say that something carries information. A second part of the solution is that beliefs and desires have their effects in computation–where computation is defined, roughly, as a process that takes place when a device is arranged so that information (namely, patterns in matter or energy inside the device) causes changes in the patterns of other bits of matter or energy, and the process mirrors the laws of logic, probability, or cause and effect in the world. The result is that if the old patterns are accurate or true, or correlate with some aspect of reality, the new arrangements of matter or energy will as well. The cascade gives the device an ability to deduce new truths from old truths in pursuit of a goal, which comes pretty close to William James’ characterization of intelligence.

This idea, the computational theory of mind, is the only theory that I know of that can explain how it is that patterns of physical change in a device–be it a computer or a brain, or, for that matter, some extraterrestrial intelligent life–might accomplish something we would dignify with the term “thinking.” It’s the only explanation we have for how physical changes actually do something we would be willing to call intelligent.

A few comments must be added to this claim. One is that the computational theory of mind is very different from the computer metaphor of the mind.. There are many ways in which commercially available computers are radically different from brains. Computers are serial; brains are parallel. Computers are fast; brains are slow. Computers have deterministic components; brains have noisy components. Computers are assembled by an external agent; brains have to assemble themselves. Computers display screen-savers with flying toasters; brains do not.

The claim is not that commercially available computers are a good model for the brain. Rather, the claim is that the answer to the question “What makes brains intelligent?” may overlap with the question “What makes computers intelligent?” The common feature, I suggest, is information-processing, or computation. An analogy is that when we want to understand how birds fly, we invoke principles of aerodynamics that also apply to airplanes. But that doesn’t mean that we are committed to an airplane metaphor for birds and should ask whether birds have complimentary beverage service. It’s a question of isolating the key component of the best explanation.

Another comment is that the computational theory of mind, explicitly or not, has set the agenda for brain science for decades. An old example from introductory neuroscience classes describes the naive person who asks, “Since the image on the retina is upside-down but we see the world right-side up, is there some part of the brain that turns the image right-side up?” We all realize that this question rests on a fallacy, that there is no such process in the brain, and that there doesn’t need to be any such process. Why is it a fallacy? Because the orientation of the image on the retina makes no difference to how the brain processes information. Since information-processing is the relevant aspect of what goes on in the brain, the orientation on the retina–and, for that matter, on the visual cortex–is irrelevant; that is why the above is a pseudoquestion.

Similarly, the search for the neural basis of psychological functions is guided, from beginning to end, by invoking information-processing. One of the great frontiers of science is the search for the molecular basis of learning and memory. Well, of the hundreds or thousands of metabolic processes in the brain, how will we know when we’ve identified the one that corresponds to memory? We will know we have it when the process meets the requirements of the storage and retrieval of information. So again, it is information that sets the interesting questions in neuroscience.

A third comment is that the computational theory of mind is a radical challenge to our everyday way of thinking about the mind, because the theory says that the life-blood of thought is information. That goes against our folk notion that the lifeblood of thought is energy or pressure. Why did the disgruntled postal worker shoot up the post office? Well, for many years, we say, pressure had been building up until he finally burst; if only he had had an alternative outlet to which to divert all of that energy, he could have released it in more constructive ways. The metaphor is that thought and emotion are animated by some superheated fluid or gas under pressure. Now, there is no doubt that this hydraulic metaphor captures something about our experience. But we know that it is not literally how the brain works: there is no container full of fluid and channels through which the fluid flows. And that raises an important scientific question: Why is the brain going to so much trouble to simulate energy and pressure, given that it doesn’t literally work that way? I will return to that question later.

Let me continue with the second key idea: evolution. How do we understand a complex device? Imagine that you are rummaging through an antique store and you come across a contraption bristling with gears and springs and a handle and hinges and blades. You have no idea how to explain it until someone tells you what it’s for–say, an olive–pitter. Once you realize what the device is for–what its function is–suddenly all the parts and their arrangements become clear in a satisfying rush of insight. This is an activity called “reverse engineering.” In forward engineering, you start off with an idea for what you want a device to do and you go and build the device. In reverse engineering, you stumble across a device and try to figure out what it was designed to do. Reverse engineering is what the technicians at Panasonic do when Sony comes out with a new product. They go to the store, buy one, bring it back to the lab, take a screwdriver to it, and try to figure out what all the little widgets and gizmos are for.

For the last few hundred years, the science of physiology has been a kind of reverse engineering. Living bodies are complex devices and pose questions like “Why, in the eye, do we find the most transparent tissue in the body that just happens to be shaped like a lens, behind the lens an iris that expands and contracts in response to light, and a layer of light-sensitive tissue that happens to be at the focal plane of the lens?” Questions like these can be answered only by the idea that the eye was in some sense “designed” to form an image. We analyze it just as if it were a machine. For centuries, the complexity of the eye and other organs was taken as conclusive proof of the existence of God. If the eye shows signs of design, it must have a designer-namely, God. Darwin’s great accomplishment was to explain signs of engineering in the natural world through a purely physical force, namely, the differential replication rates among replicators competing for resources in a finite environment, iterated over hundreds and thousands of generations.

Of course, the eye doesn’t just sit by itself, isolated in the skull. Rather, the eye is connected to the brain. In fact, the eye can validly be considered to be an extension of the brain. And that naturally leads us to treat the mind as a complex natural device–in this case, a complex computational device–which makes the science of psychology a kind of reverse engineering. Just as in the case of the olive-pitter, we can understand the brain only once we have correctly identified its function. If we thought that the olive-pitter was a wrist-exerciser, we would have a very different explanation for what the parts are for. The crucial place to begin explaining the mind, therefore, is to understand its function. Since the mind is a product of natural selection, not of a conscious engineer, we have an answer to that question: the ultimate function of the mind is survival and reproduction in the environment in which the mind evolved–that is, the environment of hunting and gathering tribes in which we have spent more than 99% of our evolutionary history, before the recent invention of agriculture and civilizations only 10,000 years ago.

The third key idea is specialization. The mind is designed to solve many kinds of problems, such as seeing in three dimensions, moving arms and legs, understanding the physical world, finding and keeping mates, securing allies, and many others. These are very different kinds of problems, and the tools for solving them are bound to be different as well. We know that specialization is ubiquitous in biology. The body is not made of Spam, but is divided into systems and organs and tissues, each designed to perform a special function or functions. The heart has a different structure from the kidney because a device that pumps blood has to be different from a device that filters blood. This specialization continues all the way down: to the different tissues that the heart and the kidney ate made from, all the way down to differences in the molecules that they are made from. The mind, like the body is organized into mental systems, organs, and tissues. I doubt that the the mind will ever be explained in terms of some special essence or wonder tissue or almighty mathematical principle. Rather, the mind is a system of computational organs that allowed our ancestors to understand and outsmart objects, animals, plants, and each other.

I will try to give you a glimpse of how three of these organs of computation might be dissected, by presenting examples of seeing, thinking, and feeling.

Let’s begin with seeing. The problem of vision can be made vivid by imagining what the world looks like from the brain’s point of view. It is not what we whole, functioning human beings experience, namely, a showcase of three–dimensional objects arrayed in space. Rather, the brain “sees” a million activation levels corresponding to the brightnesses of tiny patches on the retina; the retinal image as a whole is a two-dimensional-projection of the three-dimensional world. The task for the visual system of the brain is to recover information about three-dimensional shapes and their arrangements from the pattern of intensities on the retinal image. The brain has evolved a number of tricks for solving this problem, and I am going to talk about one of them–sometimes called “shape-from-shading.” Each of these tricks exploits a regularity of optics that is true by virtue of physical law, and the brain can, in a sense, run these laws “backward” to try to make intelligent guesses about what is out there in the world based on the information that is coming in from the retina.

One important bit of physics is (roughly) that the steeper the angle formed by a surface with respect to a light source, the less light the surface reflects. So as I shine a flashlight perpendicularly to a card, it projects a concentrated, bright spot of light. But when I rotate the card, the beam is smeared across a large area, and any particular part of the area must be dimmer. Now, the shape-from-shading algorithms bit of psychology–more or less runs the law backward and says that the dimmer a patch on the retina, the steeper the angle of the surface in the world. And with that algorithm, the brain can reconstruct the shape of an object by estimating the angles of the thousands of tiny facets or tangent planes that make up the surface.

This process works reasonably well, but it depends on a key assumption. Since it interprets differences in brightness as coming from differences in surface angle, it implicitly assumes a uniformly colored world, or at least a randomly colored world. That means that the process is vulnerable, because surfaces that are colored in clever ways should fool the shape-from-shading module and cause us to see things that aren’t there. In fact, it does happen. One example is television. If alien anthropologists visited this planet, they would be puzzled by the fact that the average American spends four hours a day staring at a piece of glass on the front of a box. Why do we do this? Because the television set has been arranged to violate the assumption of uniform or random coloration. It has been engineered to display a highly nonrandom pattern that fools the shape-from-shading module of the brain into hallucinating a three-dimensional world behind the pane of glass.

Another example is makeup. A person who is skilled at applying makeup might put a little blush on the sides of the nose, because the eye of the beholder is attached to a shape-from-shading module that interprets darker surfaces as steeper angles, making the sides of the nose look more parallel and the nose smaller and more attractive. Conversely, if you put light powder on the upper lip, the brain says that lighter equals a flatter angle, which makes the lip took fuller, giving that desirable pouty look that models strive so hard to attain.

More generally, these examples offer an explanation for many of the seemingly inexplicable quirks of modern human thought and behavior. Many illusions, fallacies, and maladaptive behaviors may come not from some inherent defect or design flaw but from a mismatch: a mismatch between assumptions about an ancestral world that were built into our mental modules over millions of years and the structure of the current world (which we have tamed topsy-turvy by technology in our recent history). It has long been a puzzle for biologists why people do maladaptive things like eat junk food, use contraception (which, when you think of it, is a kind of Darwinian suicide), or gamble. But if you posit that our mental modules assume a world in which sweet foods are nutritious (namely, ripe fruit), in which sex leads to babies (as it tended to do until the invention of reliable contraceptives), and in which statistical patterns have underlying causes, then these activities no longer seem quite so mysterious.

Next, let me turn to the problem of thinking. There is an old puzzle that has worried philosophers and biologists ever since it was pointed out by Alfred Russel Wallace, the co-discoverer, with Darwin, of natural selection: What do illiterate, technologically primitive hunter-gatherers do with their capacity for abstract intelligence? In fact, this question might be more justly asked by hunter-gatherers about modem American couch potatoes. After all, life for hunters and gatherers was like a camping trip that never ended, but without Swiss army knives and tents and freeze-dried pasta. Our ancestors had to live by their wits and eke out a living from an eco-system in which most of the plants and animals whose bodies we consume as food would just as soon keep their bodies for themselves.

Our species succeeded by entering what a biologist might call the “cognitive niche”: the ability to overtake the fixed defenses of other organisms by cause-and-effect reasoning. In all human societies, no matter how supposedly primitive, people use a variety of tools; traps; poisons; various ways of detoxifying plants by cooking, soaking, and leaching; methods of extracting medicines from plants to combat parasites and pathogens; and an ability to act cooperatively to accomplish what a single person acting alone could not achieve. These accomplishments show that the mind must be equipped with ways of grasping the causally significant parts of the world. The world is a heterogeneous place, and it is likely that we have several different intuitive theories or varieties of common sense that are adapted to figure out the causal structure of different aspects of the world. We can think of them as a kind of intuitive physics, intuitive biology, intuitive engineering, and intuitive psychology, each based on a core intuition.

The most basic is intuitive physics, an appreciation of how objects fall, roll, and bounce. The core intuition behind our intuitive physics is the existence of stable objects that obey some kind of physical laws. This is not a banal claim. William James said that the world of the infant is a “blooming, buzzing confusion’!–a kaleidoscope of shimmering pixels–and that knowledge of stable objects is an achievement only of late infancy. Yet one of the first things we learn in introductory courses in philosophy is that unless one has an assumption that the multitude of sensory impressions is caused by an underlying stable object, one could experience the blooming and buzzing confusion all of one’s life. Indeed, the more we know about the world of the infant, the more we see that William James, at least in this case, didn’t have it quite right. The youngest infants that can be tested (about three months old) already are expecting a world that contains stable objects, and they are surprised when an experimenter rigs up a display in which an object vanishes, passes through another object, flies apart, or moves without an external push. As the psychologist David Geary summed up the literature: A “blooming, buzzing confusion” is a better description of the world of the parents of an infant than of the world of the infant.

But there are many objects that we encounter that seem to violate our intuitive physics. As the biologist Richard Dawkins has pointed out, if you throw a dead bird in the air, it will describe a graceful parabola and come to rest on the ground, just like the physics books say it should. But if you throw a live bird in the air, it won’t describe a graceful parabola, and it might not touch land this side of the county boundary. In other words, we interpret living things such as birds not through our intuitive physics but through an intuitive biology. We do not assume that birds are some kind of weird, springy object that violates the laws of physics; we assume, rather, that birds follow a different kind of law altogether–the laws of biology. The core intuition of folk biology is that plants and animals have an internal essence that contains a renewable supply of energy or oomph, that gives the animal or plant its form, that drives its growth, and that orchestrates its bodily functions.

This deep-rooted intuition is found in all peoples and explains why hunter-gatherers are such excellent amateur biologists. Botanists and zoologists who do field work with hunter-gatherers are often astonished to learn that hunter-gatherers have remarkably de- tailed knowledge about local plants and animals and that their names for these plants and animals usually match the Linnaean genus or species of the professional biologists. These categorizations often involve lumping animals that, from surface appearance, look very different–for example, a caterpillar and a butterfly, or a male and a female bird with different plumage. Hunter-gatherers, using their intuitions about the hidden essences in animals or plants, predict their future behavior. They may, from a set of tracks, deduce the kind of animal and where it is likely to be heading so that they can surprise it at a resting place; or they might notice a flower in the spring and return to it in the fall to dig out a hidden tuber that the flower portends. They extract juices and powders of living things and try them out as medicines, poisons, and food additives.

The third kind of intuition, different from the first two, is an intuitive engineering. Our species is famous for exploiting and using tools or artifacts, and the core intuition behind tools is their function. If I ask you to define a “chair,” you might say it is a stable horizontal surface supported by four legs. But that will not work for bean-bags, cubes, severed elephant’s feet, and other objects that we can call chairs. The only thing that chairs have in common is that someone intended them to hold up a human behind. The core intuition behind our faculty to appreciate tools involves their function, or the intention of a designer. Young children, before they’ve entered school, sharply distinguish artifacts from living things. For example, in one experiment, children were told that doctors took a raccoon, spray-painted it black with a white stripe down its back, and implanted into it a sack of smelly stuff. The children were then shown a picture of a skunk and asked what it was. Most of them said that it was still a raccoon. But if they were told that doctors took a coffee pot, sawed off its handle, cut a hole through it, and filled it with birdseed, and then are shown a picture of a bird feeder and asked what it is, they say it’s a bird feeder. This experiment shows that even young children appreciate that an artifact such as a bird feeder is anything that feeds birds, but a natural object such as a raccoon has an internal constitution that cannot be changed by superficial manipulations.

And finally, people have an intuitive version of psychology. I mentioned earlier that all of us explained Bill’s behavior in getting on the bus not by assuming there is some kind of magnetic force that pulls him aboard or that he is a kind of artifact like a windup doll, but that he acts out of beliefs and desires, a kind of entity we cannot help but posit even though it is not directly observable. Again, this ability is displayed early by young children, who can, for example, deduce what an adult knows and wants just from observing what the adult is looking at.

There is evidence, apart from developmental psychology, that our reasoning ability really is divided into these intuitive theories or ways of thinking. For example, functional neuroimaging has shown that different parts of the brain are active when people think about tools or about living things. Moreover the presumably genetic syndrome of autism can be pretty well characterized by saying that it impairs a person’s intuitive psychology: Autistic children really do interpret humans as if they were windup dolls, and have no concept that other people have beliefs and desires.

Misapplications of the four forms of thinking, or a shift from one way of thinking to another, can also explain certain puzzling behaviors and beliefs. One example is slapstick humor. We laugh when someone slips on a banana peel because of the sudden shift from thinking of the person in the usual way (using our intuitive psychology and thinking of him as a locus of beliefs and desires) to thinking of him as an object ignominiously obeying the laws of physics. Belief in souls and ghosts consists of taking our intuitive psychology and divorcing it from our intuitive biology, so that we think of minds that have an existence independent of bodies. And animistic beliefs do the opposite: They marry our intuitive psychology to our intuitive biology, physics, or engineering and allow us to think of trees, mountains, or idols as having minds.

I will now proceed to my final example: emotions elicited by other people. The main puzzle about our feelings toward other people is why they are often so passionate and seemingly irrational. Why do people pursue vengeance past the point of any value to themselves? Why do people defend their honor in crazy ways such as challenging each other to duels? Why do people fall head over heels in love?

The most common theory, among both scientists and lay people, is the “romantic theory”–the idea that the emotions come from a vestigial force (part of our heritage from nature), and that they are maladaptive and dangerous unless they are channeled into art and creativity. I’m going to explain a very different alternative, the “strategic theory,” which proposes that passion is a “paradoxical tactic” wired into us. The basic idea is that a sacrifice of freedom and rationality can actually give one a strategic advantage when one is interacting with others whose interests are partly competing and partly overlapping with one’s own. The theory applies particularly well to instances of promises, threats, and bargains. Just to show how unromantic this theory is, I am going to illustrate it by reverse-engineering romantic love.

Cynical social scientists and veterans of the dating scene agree on one thing: that love is a marketplace. There is a certain rationality to love–smart shopping. All of us at some point in our lives have to search for the nicest, smartest, richest, stablest, funniest, best-looking person who will settle for us. But that person is a needle in a haystack, and we might die single if we held out indefinitely for him or her. So we trade off value against time, and after a certain period set up house with the best per- son we have found up to that point. Good evidence for this sequence of events is the phenomenon called “assortative mating” by mate value: the overall desirabilities of a husband and a wife or a boyfriend and a girlfriend are approximately equally matched, as if each was trying to get the best partner he or she could.

Needless to say, that does not explain all there is to falling in love. There is an irrational part of love, an involuntariness and caprice to it. You cannot will yourself to fall in love. Many people can recall being fixed up with a person who looked perfect on paper, but when they met, they just didn’t hit it off. Cupid’s arrow didn’t strike; the earth didn’t move. It isn’t a list of desirable traits that steals the heart; it’s often something capricious like the way a person walks, talks, or laughs.

Is this any way to design a rational organism? As a matter of fact, it might be. Entering a partnership through totally “rational” shopping poses a problem. If you have set up house with the best person you have found up to a certain point, then by the law of averages, sooner or later someone even better will come along. At that point a rational agent would be tempted to drop a wife or husband like a hot potato. But now think of it from the spouse’s point of view. Entering a partnership requires sacrifices–forgone opportunities with other potential partners and the time and energy put into child-rearing, among many other things. Rational spouses could anticipate that their partner would drop them when someone better came along, and they would be foolish to enter the relationship in the first place. Thus we would have the paradoxical situation in which what is in the interest of both parties–that they stick with each other–cannot be effected because neither one can trust the other if the other is acting as a rational, smart shopper.

Here is one solution to the problem. If we are wired so that we don’t fall in love for rational reasons, perhaps we are less likely to decide to fall out of love for rational reasons. When Cupid strikes, it makes one’s promise credible in the eyes of the object of desire. Romantic love is a guarantor of the implicit promise one makes in starting a romantic relationship, in the face of the problem that it may be rational to break that promise in the future.

Romantic love is an example of a concept from game theory called “paradoxical tactics,” in which a lack of freedom and rationality can be an advantage. An analogy from a nonpsychological domain is the rationale for laws and contracts. When we apply for a mortgage from a bank, the law states that if we default on our payments, the bank has the power to foreclose on the mortgage and seize our house. It is only this law that makes it worth the bank’s while to lend us the money, and therefore the law, paradoxically, works to the advantage of the borrower as well as the lender. Likewise, leases work to both the tenant’s and the landlord’s advantage by constraining the freedom of each. In this sense, many passions, such as romantic love, could be viewed as the neural equivalents of laws and contracts.

And by symmetrical logic, if passionate love and loyalty are guarantors that our promises are not double-crosses, so passionate vengeance and honor serve as guarantors that our threats are not bluffs. The problem with issuing a threat, such as “If you steal my goats, I will beat you up,” is that carrying out a threat can be dangerous: you could get hurt beating someone up. The only value of the threat is as a deterrent; once it has to be carried out, it serves no one’s purposes. Since the target of the threat is aware of that fact, he can threaten the threatener right back by calling his bluff and daring him to go through with the vengeance. How does one prevent one’s bluff from being called? By being forced to carry out the threat. If we are wired to interpret defiance or trespass as an intolerable insult for which we demand vengeance regardless of the cost to ourselves, that emotion serves as a credible deterrent. One gets the reputation of being someone that people don’t want to mess with.

Let me conclude. Is this a cynical view? Granted, most people don’t like to think of themselves as a system of computers “designed” by natural selection to promote survival and reproduction. On the other hand, the view is based on facts that I think can no longer be denied by any scientifically literate person. It is becoming increasingly clear, in particular, that the mind is a product of the brain; that the brain is a product of evolution, and that evolution is not guaranteed to produce niceness.