A post by Michael Barkasi

We tend to think of imagination and perception as separate things. The sort of thing happening when I imagine a field of yellow daffodils in my “mind's eye” is not at all happening when I look at and see the bowl of cracked pecans on my desk. The latter involves the processing of sensory inputs (for sure), but not the unrestrained internal generation of figments found in the former.

A lot of philosophers and psychologists think this naïve view is wrong. The idea goes that sensory input is often incomplete, and imagination fills in the missing gaps (Kosslyn 1994, Addis 2020). Seeing occluded objects as complete, i.e. “amodal completion” (Nanay 2010, Kind 2018), resolving ambiguous figures like the Necker cube (Macpherson 2018), and seeing colors despite insufficient light for color-sensing cone cells, i.e. “memory-colors” (Macpherson 2012, Brown 2018) have all been proposed as specific cases in which imagination augments perception with phenomenology-affecting representations. I’ll call this the sprinkle view, since it holds that imagination sprinkles content into perception as needed to afford a complete experience of the world.

The sprinkle view is opposed to what I’ll call the radical Kantian view. (I make no claim here to getting my Kant exegesis correct.) According to the Kantian view, perception is just more imagination. All perceptual content is imagined content—figments generated by the brain which stand between us and the real reality.

The most recent incarnation of radical Kantianism is the controlled hallucination thesis (CHT), according to which perception is just controlled hallucination (Revonsuo 1995). The idea, advocated by some proponents of predictive processing (Hobson and Friston 2014, Hohwy 2016), denies any fundamental distinction between experiences had during the normal, successful use of our sensory systems (perceptual experience) and the experiences had while using our imagination or during dreams. The only difference, according to CHT, is that during perception the neural representations underlying our experiences are being used to predict and suppress the cascade of input coming from our sensory receptors. In this way, the world exerts influence on perceptual experience—experience must adequately account for receptor input—but perceptual experience is otherwise arbitrary. What's perceived is just as much a figment of the mind as what's dreamed or imagined.

My position is that the sprinkle view fails to capture the full, rich contribution of imagination to perception, while the radical Kantian view goes too far (Barkasi 2021). The reality is actually much weirder than that painted by radical Kantianism and CHT. Space precludes me from explaining why I reject radical Kantianism, but I accept standard philosophical treatments of experience which construe it as a genuine connection to the world (Anscombe 1965, Dretske 2003), not a connection to mere phenomena, percepts, figments, sense-data, or mental images. I endorse standard critiques of predictive coding (e.g., Cao 2020), and even from within the predictive coding camp itself, there are good critiques of radical Kantianism (Clark 2012).

To get a sense for the real contribution of imagination to perception, first consider memory-colors, a standard example given by the sprinkle view. Suppose I'm looking at my bowl of pecans in dim evening light. I’m likely to still see the pecans as a rich caramel color, despite my retinal activity lacking that information. Why? Because my brain has a high-level association (somewhere in the hippocampus or inferotemporal cortex) between cracked pecans and caramel, an association formed through previous encounters with pecans and stored in a memory trace. The visual neural representation of pecans cascading up from my photoreceptors triggers that association—it generates (“imagines”) a representation of caramel which gets injected into my overall visual representation, into my conscious experience. 

Now consider the color processing which happens when I look at my bowl of pecans under normal light. My eyes receive a rich stream of color information, but my visual system still must correctly process (or interpret) the encoded color. Caramel, like all colors, doesn't just produce one pattern of retinal activation—there are countless possible activation patterns produced by caramel, depending on factors like ambient lighting and shadows. My previous encounters with other instances of caramel created a sophisticated web of neural connections (memory traces) sensitive to the subtle similarities common to the many different neural patterns caused by caramel. These connections aren't as crude as “pecans = caramel”, and they are likely localized to neural circuits close to my retinas (e.g., in V1), but otherwise they function just like the high-level associations behind memory-colors. They lead me to a certain interpretation (“caramel”) of the color information gathered under good light conditions, just as my association between pecans and caramel causes me to see pecans as caramel even when that color information is missing.

The point is that even with a rich stream of information about a sensible property (color, shape, texture, flavor, etc.), suitable representations of that property must be generated. The brain must extract the relevant information, identify it as signaling the relevant property (e.g., “caramel”), and integrate it into the perceptual experience. Now, it may sound like I'm just talking about regular old sensory processing, not imagination, but the point of comparing memory-colors to normal color processing was to point out the similarities. A representation of color, or any sensible property, never comes for free—it is always generated on the basis of a raw neural signal that fits a pattern for that property encoded in stored memory traces (see also Brown 2018).

So, as I look at my bowl of cracked pecans, the colors, shapes, and other sensible properties entering my experience are all imagined. But, pace the radical Kantian view, this does not mean they are mere figments. To see why, consider again the process involved in their generation. My brain is not creating properties out of nothing. The neural representation of each sensible property is a memory trace, a pattern of firing shaped by previous encounters with that property. While it abstracts away from the specifics of any one past encounter, it still encodes the overall appearance common to those past encounters. So, what’s imagined, i.e. generated, is the overall past appearance common to previous encounters with that property.

It’s helpful to make a comparison with sensory modalities. The same property (e.g., roundness) can be experienced through both vision and touch. These experiences will be quite different: What it’s like to see a sphere is not at all what it’s like to touch one. This does not mean that the experienced roundness is a mere figment—it merely means that different modes of sensing roundness lead to different experiences. Think of how viewing yellow daffodils through blue-tinted lenses gives them a green appearance; you don’t suddenly stop seeing the daffodils themselves because you put on the lenses. The lenses merely reshape (or recolor) your experience of the daffodils.

Imagination—the interpretation of sensory input through memory traces—functions much like its own sensory modality. It doesn’t cut us off from reality or present to us mere figments. Instead, it reshapes or recolors how we experience the world. We experience properties like shape and color through the lens of our memories of our past encounters with those properties. The caramel color I see as I look at my cracked pecans reflects how I have encountered caramel in the past. If you looked at my pecans under identical viewing conditions, your experience of their color would differ slightly, reflecting your past encounters with it. As William James (1962/1892, p. 334) wrote, “the brain reacts by [nerve-fiber] paths which the previous experiences have worn, and which make us perceive the probable thing, i.e., the thing by which on previous occasions the reaction was most frequently aroused.”

References

Addis, D. R. (2020). Mental time travel? A neurocognitive model of event simulation. Review of Philosophy and Psychology 11, 233–259.

Anscombe, G. (1965). The intentionality of sensation: A grammatical feature. In R. J. Butler (Ed.), Analytic Philosophy, pp. 158–80. Oxford: Blackwell.

Barkasi, M. (2021). Memory as sensory modality, perception as experience of the past. Review of Philosophy and Psychology, 1–19.

Brown, D. H. (2018). Infusing perception with imagination. In F. Macpherson and F. Dorsch (Eds.), Perceptual Imagination and Perceptual Memory, pp. 133–160. Oxford: Oxford University Press.

Cao, R. (2020). New labels for old ideas: Predictive processing and the interpretation of neural signals. Review of Philosophy and Psychology 11, 517–546.

Clark, A. (2012). Dreaming the whole cat: Generative models, predictive processing, and the enactivist conception of perceptual experience. Mind 121(483), 753–771.

Dretske, F. (2003). Experience as representation. Philosophical Issues 13(1), 67–82.

Hobson, J. A. and K. J. Friston (2014). Consciousness, dreams, and inference: The Cartesian theatre revisited. Journal of Consciousness Studies 21(1-2), 6–32.

Hohwy, J. (2016). The self-evidencing brain. Noûs 50(2), 259–85.

James, W. (1962/1892). Psychology: Briefer Course. New York: Collier Books.

Kind, A. (2018). Imaginative presence. In F. Dorsch and F. Macpherson (Eds.), Phenomenal Presence, pp. 165–180. Oxford: Oxford University Press.

Kosslyn, S. M. (1994). Image and Brain: The Resolution of the Imagery Debate. Cambridge: The MIT Press.

Macpherson, F. (2012). Cognitive penetration of colour experience: Rethinking the issue in light of an indirect mechanism. Philosophy and Phenomenological Research 84(1), 24–62.

Macpherson, F. (2018). Perceptual imagination and perceptual memory: An overview. In F. Macpherson and F. Dorsch (Eds.), Perceptual Imagination and Perceptual Memory, pp. 1–6. Oxford: Oxford University Press.

Nanay, B. (2010). Perception and imagination: Amodal perception as mental imagery. Philosophical Studies 150, 239–254.

Revonsuo, A. (1995). Consciousness, dreams and virtual reality. Philosophical Psychology 8(1), 35–58.