This week at The Junkyard, we’re hosting a symposium on Bence Nanay’s recent book Mental Imagery (Oxford University Press, 2023). See here for an introduction from Bence. Commentaries and replies will follow Tuesday through Thursday.

* * *

Commentary from Daniel Munro

Bence Nanay’s Mental Imagery is a pleasure to read. It’s an impressively wide-ranging book, bringing together large swathes of research from philosophy, psychology, and neuroscience. It does so in a way that’s admirably accessible to readers of all stripes.  

The book’s breadth isn’t just due to how much existing scholarship it brings together. It’s also because of the wide range of mental phenomena Bence discusses, all while aiming to convince us mental imagery is fundamental for understanding each. To name just a few:

• Amodal completion: Representing parts of a perceived object from which we get no sensory stimulation (e.g., parts of a cat occluded behind slats of a fence).

• Perceptual filling-in: Filling in parts of a perceptual field where we get no, or indeterminate, sensory stimulation (e.g., visual blind spots; peripheral vision).

• Navigating in the dark: Using mental imagery to guide action when we can’t see our environment (e.g., visualizing the layout of your dark bedroom to get around). 

• Future planning: Imagining hypothetical future actions while deciding which to undertake.

• Episodic memory: Representing past events one experienced.

• Language processing: Understanding bits of language (by, e.g., hearing “dog” automatically activating mental imagery of dogs).

And that’s far from an exhaustive list!

My commentary will focus on Bence’s definition of mental imagery. While I think the core of his definition has a lot going for it, I have some worries about how he frames it.

Mental imagery: a functional definition

Bence defines mental imagery as “perceptual representation that is not directly triggered by sensory input” (4). If a perceptual representation is directly caused by stimulation of a sense organ—e.g., when light bounces off an apple and hits your retina, causing you to see the apple—then it counts as perception proper. Whenever a perceptual representation is caused another way (as in the examples of mental imagery listed above), it counts as mental imagery.

There’s something obviously right about grouping mental imagery and perception under a broader category like “perceptual representation.” However, Bence’s definition of mental imagery is purely negative: it simply says mental imagery is any perceptual representation not directly triggered by sensory input. So, to flesh it out, we’d need to know more about the broader category under which mental imagery falls—i.e., about what it means for a representation to be perceptual, not of some other kind (e.g., linguistic).

There are a few possible types of definition of “perceptual representation”:

• Functional: Characterizes perceptual representation in terms of the function for which it was selected.

• Neuroanatomical: Characterizes perceptual representation in terms of the physical brain areas in which it’s realized.

• Phenomenal: Characterizes perceptual representation in terms of its distinctive phenomenology.

• Format-based: Characterizes perceptual representation in terms of its distinctive format (e.g., picture-like rather than language-like).

At various points, Bence clarifies that he’s interested in functional definitions, not the other types (see, e.g., chapter 14). (Sidenote: There’s another kind of “functionalist” definition with which philosophers are sometimes concerned, which types mental states purely by their causal inputs and outputs. Bence’s discussions suggest to me he has the evolutionary one in mind; regardless, my arguments in this post should also apply, with a few tweaks, to the other type.)

So, if Bence’s definition of mental imagery is to be viable, we first need a plausible account of the function of perceptual representation itself. Specifically, we need to know what functionally unifies perception proper and mental imagery, since both are meant to be sub-types of the functional kind “perceptual representation.”

A unified functional account of perceptual representation?

However, I’m pessimistic about the prospects for finding a unified, overarching function of both perception proper and mental imagery.

Some of the processes Bence classes as mental imagery do seem to be functionally like perception proper. Take amodal completion, filling-in, and navigating in the dark. It’s controversial how to define the function of perception; however, there are various plausible candidates under which these types of mental imagery would have the same function as perception. Perhaps perception has the function of informing us about what’s in our immediate external environment, and/or of directly guiding our actions as we navigate that environment; mental imagery serves these same functions during amodal completion, filling-in, and navigating in the dark. 

Other times, though, mental imagery’s function seems quite distinct from perception’s. Mental imagery sometimes helps us plan future actions, rather than directly guiding present ones. In episodic memory, it functions to inform us about past events, rather than about our present, immediate environment. Still other times, it serves functions related to language processing.

More problems for a functional definition

Now, it’s open to Bence to respond that many of these uses of mental imagery are unrelated to its proper, evolved function. Perhaps mental imagery serves its true function only during processes in which it’s functionally unified with perception, such as amodal completion or filling-in. Maybe the various other uses I’ve mentioned simply involve co-opting imagery for other purposes, much as one might use a rolling pin to hammer a nail (Bence seems to hint at something like this in the final paragraph of chapter 8 on amodal completion).

However, some of Bence’s own arguments give us reason to doubt this picture. That’s because he grants that at least some other uses for mental imagery involve functions selected by evolution.

Take language processing. Bence argues (quite plausibly) that the evolution of language put to new use our ability to form imagistic representations, which might’ve previously only featured in processes like perception, amodal completion, or filling-in. However, it’s unclear why we should say that, as a result, a single functional type (i.e., perceptual representation) is now involved both in language processing and in these other, pre-existing processes. If we’re individuating types of representation in terms of the functions for which they were selected, it seems we’d instead have an altogether new type.

This generalizes to any case in which pre-existing imagistic abilities are repurposed by evolution for some novel function (e.g., in episodic memory or future planning). We’d end up with a distinct functional type when this occurs, rather than a functionally unified category of “perceptual representation” that crosscuts both old and new processes.

A different type of definition?

For the above reasons, I’m skeptical that the category of “perceptual representation” really is a unified functional category. Still, it seems we should be able to say something about what unifies various perceptual and imagistic representations. To do so, we may need to employ a different kind of definition, perhaps neuroanatomical, phenomenal, or format-based.

These types of definition end up being more unified, since they allow us to say that a single type of representation is involved in carrying out various functions. If we define perceptual representation as a neuroanatomical type, then the same neuroanatomical type is used to realize various functions (that of perception proper and the various functions in which mental imagery is involved). If we define perceptual representation as a format type, we can say that the same format type is used to realize these various functions. Ditto for classifying perceptual representations as a phenomenal kind.

In his book, Bence offers his own reasons for rejecting neuroanatomical and phenomenal definitions of mental imagery. So, I suspect he thinks some of these alternatives are hopeless. Still, I look forward to reading his response to my worries about the functional approach—perhaps, due to my limited powers of imagination, I’ve failed to envision a clear way around them.

Response to Munro

Daniel Munro pushes me to clarify a key aspect of my definition of mental imagery: the perceptual nature of perceptual representations. I’m extremely grateful for his very helpful comments.

I define mental imagery as perceptual representation that is not directly triggered by sensory input. I spend a lot of time in the book clarifying what I took to be the most unclear part of this definition, namely, what counts as the input directly (rather than indirectly) triggering the representation and what counts as the lack of such direct triggering. But Daniel is absolutely right that I said much less about an equally pressing issue, namely, what makes a perceptual representation perceptual.

And if I am completely honest, this is no coincidence: keeping perceptual and non-perceptual representations apart is a huge question in the philosophy of perception and I wanted to stay away from the vicious debates about this. So I took a convenient shortcut I laid out in Chapter 1, namely, that throughout the book I was focusing on perceptual representations that are unquestionably perceptual: early cortical representations, like representations in the primary visual cortex. There may be debates about whether, say, semantic representations in speech perception are perceptual representations, but I only rely on less controversial examples of perceptual representations.

Nonetheless, Daniel is absolutely right: in order for this definition to be of any use, I need to say more about what I mean by perceptual representations. What is then the difference between perceptual representations and non-perceptual representations? Daniel offers me four options: a functional difference, a neuroanatomical difference, a phenomenal difference or a format-based difference.

I want to set the phenomenal one aside right at the beginning. As perception can be conscious or unconscious, phenomenal criteria will be not a very good bet for keeping perceptual and non-perceptual representations apart. Further, as I argued long ago (for example, here), even in conscious perceptual episodes, there is no good introspective way of keeping apart perceptual and non-perceptual phenomenology (whereas there must be good ways of keeping apart perceptual and non-perceptual representations). So that won’t work.

How about a neuroanatomical criterion? The problem here is the plasticity of the human brain: brain regions can be and often are reallocated to do some other job, different from what they usually do. The primary visual cortex, which is at the back of your head is where the first step of visual processing happens. But this part of the brain can be reallocated, for example, in visually impaired people, to perform some other tasks. So while neuroanatomical criteria can be indicative of perceptual representations, as a general division-line between perceptual and non-perceptual representations, this won’t work either.

Daniel assumes that I am going for a functional criterion. And I am, but not in the sense that he understands ‘functional’. What he has in mind is something like selected function: perceptual representations have been selected to perform a certain function and non-perceptual representations have been selected to perform a different function. This is an unfortunate misunderstanding as I definitely did not mean to suggest that the difference between perceptual and non-perceptual representations is that of evolutionary function.

What I had in mind is function in the sense of functionalism. Functionally speaking, what the primary visual cortex does is convert a certain kind of input (roughly, a two-dimensional array of activation) to a certain kind of output (roughly, segmentation into contours). But because of the plasticity of the brain, other brain regions could also do so: wherever this processing is implemented, as long as it is the conversion of this kind of input to this kind of output, it counts as perceptual representation. So what keeps perceptual and non-perceptual representations apart is not to be found on Marr’s implementation level (this would be the neuroanatomical criterion) but on Marr’s algorithm level.

And this is actually quite close to Daniel’s preferred candidate for keeping perceptual and non-perceptual representations apart, that of the representational format, which is something I am quite sympathetic to. In short, Daniel and I fully agree where a convincing criterion for the difference between perceptual and non-perceptual representations should come from (we only disagree about what to call ‘functional’).

Having said all this, Daniel is absolutely right that I owe the reader a story about how to draw the line between perceptual and non-perceptual representations. And that could be another book, probably much longer than this one.