Originally published on Edge.org
July 16, 2002. Published on KurzweilAI.net July 17, 2002.

STEPHEN KOSSLYN: For the last 30 years I’ve been obsessed with a question: What shape are a German Shepherd’s ears? Of course, I’m not literally interested in that question, since if I were I could just go out and look at dogs; I’m really interested in how people answer the question from memory. Most people report that they visualize the dog’s head and mentally "look at" its ears. But what does it mean to visualize something? What does it mean to "look at it in your mind"? It’s a bit absurd, because there can’t be a little man in there that is actually looking at a picture. If there were, there would have to be a little man inside that man’s head, and so forth, and it doesn’t make any sense.

For many years we tried to collect objective evidence to show that when you have the experience of visualizing, there’s actually something pictorial in your head. It turned out that the best way to approach this was by turning to the brain. There are parts of the brain that are physically organized such that when you look at something, a corresponding pattern is physically laid out on the cortex. Even the first visual area in the processing stream is often activated during visual imagery — even if your eyes are closed when you visualize. Moreover, the way it’s activated depends on what you’re visualizing. If you visualize something that’s vertical, you find activation along the so-called vertical meridian; if it’s horizontal, the activation flips over on its side. It’s absolutely amazing. Similarly, visualizing objects at different sizes changes the pattern of activation in ways very much like what occurs if you are actually seeing objects at the corresponding sizes.

But I’ve been working on this for over 30 years now and I want to move on. Instead of trying just to establish that there actually are mental images and that these images are bona fide representations that have a functional role in processing systems, I want to ask: So what? Who cares? Why should my mother be interested in this kind of thing?

Lately I’ve been working on something that I’m tentatively calling the "Reality Simulation Principle." It is built on my lab’s findings that about two-thirds of the same brain areas are involved in visual mental imagery and visual perception. This finding occurs even when the tasks seem very different on the surface (for example, visualizing an upper case letter in a grid and deciding whether an X mark would fall on the letter if it were actually in the grid versus deciding whether a spoken name is appropriate for a picture). This is a huge amount of overlap, which leads us to suspect that an object seen in a mental image can have the same impact on the mind and body that the actual object would have. My notion is that once the brain systems are engaged, they don’t know where the impetus came from. This means that they can produce the same effects whether you activated it endogenously (from information in memory) or exogenously (from looking at something).

The "Reality Simulation Principle" describes how to use mental images as stand-ins for actual objects-to manipulate yourself, basically. It is useful to understand it in conjunction with what I call the GITI cycle, which stands for Generate, Inspect, Transform, Inspect. If mental images can simulate or stand in for actual objects and scenes, you can generate the image, inspect what you’ve got, transform it, and inspect the result. This can be done iteratively, meaning that you can use imagery to take advantage of the "Reality Simulation Principle "to do all sorts of good things for yourself.

What kinds of good things am I talking about? Memory is one obvious example. From the work of Alan Paivio and countless others, we know that you’re able to remember objects better than pictures of objects, and pictures of objects better than words. It also turns out that if you visualize the objects named by words you do better than you would otherwise. Consequently, we’re interested now in things like hypnosis. We can hypnotize you, have you visualize an object, and imagine that it’s actually a three-dimensional object, appearing in glorious vivid detail. In this case, your memory would be boosted even further.

Mental practice is another candidate. Neuroscientists such as Marc Jeannerod and Jean Decety have shown that imagining doing something recruits most of the brain mechanisms that would guide the corresponding actual movements. And people in sports psychology have shown that by imagining that you’re engaging in some activity you’ll actually get better at doing it. This process involves generating an image, inspecting the image, transforming it by imagining your movements, seeing what the result would be, and then cycling through again. The next time through you can change the image as a function of the result you saw. If you imagine you’re playing golf, for example, and your ball doesn’t get in the hole, you can imagine what would happen if you whacked it a little more softly. Mental practice clearly works. By understanding how mechanisms of imagery works we can actually optimize this mental practice.

The "Reality Simulation Principle" can also be used to acquire self-knowledge. Try this one out. Imagine it’s dusk, you’re walking alone, and you’re late. You start to walk faster and then notice a short-cut through an alley. It’s getting a little dark, but you really don’t want to be late, so you start to go towards it, and you notice that there are three guys lingering near the mouth of the alley. Now think about a first scenario: The three guys look like they’re 20 years old, are wearing long droopy shorts, dirty t-shirts, baseball caps that are on backwards, and are smoking cigarettes. As you get close, they stop talking, and all three heads swivel and fixate on you and start tracking you. How do you feel?

Now try the same thing, except instead of those three guys, make them three balding middle age, overweight accountants wearing suits. They’re standing there smoking cigarettes, and their heads swivel as they track you. How do you feel now?

You can start simulating the effects of different attributes. For example, what if the guys are black or Latino teenagers. How do you feel? If you can start actually sorting out your own emotional landscape by running these kinds of mental simulations, you may, in fact, discover certain things about yourself that may be surprising.

Some people who confront their own beliefs may think they’ve got some racial issues, and it may turn out they’re actually class issues. Make those middle-aged accountants black and see how you feel. Those kinds of simulations can help produce self-knowledge, and can help a person to improve his emotional intelligence.

You can also manipulate your body. It’s obvious that if you have a sexual fantasy you manipulate your body by imagery. Also, if you imagine something scary — an anticipated encounter with an authority figure or a walk along a narrow path in the mountains that is starting to crumble — your palms will sweat and your heart beat will change. It’s clear that mental imagery can affect the body, but it turns out that it may be more interesting than that. For example, one of the things we’re studying now is how to change your hormonal landscape by manipulating your images.

There’s something called the victory effect, where if you’re a male and you win some sort of contest your testosterone goes up afterwards. If you lose, it goes down. This is not a surprise. It also turns out that if you watch your favorite team win your testosterone will go up. If your team loses, it’ll go down. This even works if you’re watching chess, so it’s not about being aroused. In fact, it works for the chess players and for the people who are watching chess games.

Why is this interesting? With men it turns out that spatial abilities vary as a function of testosterone levels. In the fall, males’ testosterone levels are relatively high. They go down thereafter, and then they pick up again. Much research suggests that the relation between testosterone levels and spatial abilities is a U-shaped function; your spatial abilities are not as good if you have too much testosterone or too little testosterone. As you get older, both testosterone levels and spatial abilities drop. There is a lot of evidence that there is a connection between the two. The question is, can we manipulate one’s spatial abilities by having you run simulations of watching yourself win or lose? If "Reality Simulation Principle" is correct, manipulating your own testosterone levels would in turn affect your spatial abilities. This is work in progress in my lab, in collaboration with Peter Ellison and Carole Hooven; stay tuned.

My point is that you can use "Reality Simulation Principle" in lot of different ways, including some ways that are not intuitively obvious, such as manipulating your hormonal landscape. Mental imagery is also important in creativity and problem solving. Einstein reported that most of his thinking was done with images prior to any kind of verbal or mathematical statement. We know quite a bit now about how to use images in the service of solving problems and being creative. In fact, Ron Finke has written a couple of unusually creative books on this topic.

Continued at: http://www.edge.org/3rd_culture/kosslyn/kosslyn_index.html
Copyright © 2002 by Edge Foundation, Inc.