A post by Sofiia Rappe

Recent philosophy and cognitive sciences literature increasingly treats various phenomenal experiences as arising from constructive, “simulation” processes. This is the case for episodic remembering (Sutton, 2009; Sutton & O’Brien, 2022; Michaelian, 2016; Werning, 2020), perception (e.g., when viewed through the predictive processing lens, see Clark, 2016; Friston, 2010; Hohwy, 2013), as well as counterfactual, future-oriented, and fantastical imagination, which are all characterized through construction of hypothetical or imagined scenarios (De Brigard & Parikh, 2019; Kind & Kung, 2016). This “trend,” in turn, has reignited a series of “continuity debates,” e.g., to what extent do perception and imagination, perception and episodic remembering, or imagination and dreaming (as some examples) share the exact constructive mechanisms and processes?[1] Are the differences between them differences in degree (e.g., of reliance on sensory input) or kind?

Adding complexity to this debate is the observation of non-trivial neural overlaps in brain activity across experimental tasks we associate with perception and imagination, episodic remembering and counterfactual reasoning, and others. These overlaps potentially suggest reliance on some shared elements of the brain architecture. However, dissimilarities require a more nuanced treatment (Cavedon-Taylor, 2021). What strikes me is that the discussions around continuity typically focus on the pairwise comparison (e.g., between perception vs. imagination or imagination vs. episodic remembering) rather than examining these experiences as part of a more extensive experiential repertoire and cognitive system. On the one hand, this is a sensible strategy—it makes things more manageable and focused (and produces publishable results). On the other hand, this approach already presupposes a certain kind of partitioning of experiences that may obscure the relevant differences related to the underlying cognitive machinery. It also potentially neglects the opportunity to acquire some insights regarding broader cognitive functions in the regions where the overlapping neural activity occurs across multiple types of experiences in a meaningful way (e.g., hippocampus in counterfactual simulation and episodic remembering). 

The alternative (or complementary, if you wish) approach that could be worth exploring starts with collecting distinct types of experiences and attempting to flesh them out according to certain “dimensions” tied to the specific features of the brain architecture they exploit, highlighting the similarities and differences. By doing so, we might uncover new groupings or relationships between experiences traditionally considered distinct or, conversely, categorize multiple experiences as relating to the same type of constructive processes (e.g., hallucinations and illusions as two types of (mis-) perceiving). This could lead to a new, more robust taxonomy of subjective experiences and underlying processes grounded in the specific features of the brain architecture/cognitive machinery rather than traditional psychological definitions. 

One problem, of course, is identifying the relevant dimensions, although this does not have to be set in stone from the beginning. It may be helpful to be overly inclusive first (I encourage various proposals in the commentaries) and then narrow down the list to the minimal set that seems the most instrumental. Below, as a starting point, I propose four dimensions that, in my opinion, may bring useful insights. I formulate them in predictive processing terms, primarily because of convenient “language” at the algorithmic level and specific links to neurobiology. In addition, predictive processing has already been applied to explain or model typical and atypical cases of perception, imagination, counterfactual cognition, and, with some modifications, episodic memory (see, e.g., Werning’s 2020 trace minimalism). That said, the dimensions below can be formulated in more general terms suitable for different theoretical frameworks. The assumption I make, however, is that constructive processes underpinning the experiences mentioned above involve a (somewhat) hierarchical network of predictions/representations, which roughly follows the cortical hierarchy. 

1. Online vs. offline processing — This dimension corresponds to the reliance on sensory information in simulation/construction (in predictive processing terms, using sensory input as a source of prediction error).

2. Generative model — This dimension relates to whether the construction process may be based on the current generative model or requires an alternative model, which, for example, could be constructed through significant inhibition/alteration of the relevant predictions, especially at the higher levels.

3. Involvement of  “time-travel” to the past — This dimension relates to the need for direct access to individual past experiences/states of the model. One way to think about this is in terms of reliance of the constructive process on memory traces as a source of prediction error (at least for some part of construction).

4.  Primary levels of representation/prediction engaged in the constructive process. 

To clarify points 2 and 3, there are different types of detachment from reality in the constructive processes. The first type is when the process disregards sensory stimuli but still operates with the existing generative model of the current world. An example would be short-term, future-oriented imagination. In these cases, future hypotheticals “may be handled by the temporal updating system if one allows for imagined updates to the current world model” (Beck & Rafetseder 2019, p 2). In other words, short-term future-oriented imagining and planning may be handled by the brain by unfolding the “simulation” of the current state of the agent and their environment in time. The second kind of offline process completely departs from the current-world generative model. Such departure cannot be realized by merely updating the model—either because what is required is a very different model altogether (fantastical imagination) or access to the model’s past states (see the discussion on temporal reasoning in Hoerl & McCormack, 2019).  

Counterfactual reasoning is an example of a process involving temporal departure from reality that requires access to (or a way to reconstruct) the past states of the generative model or its parts (Beck & Rafetseder, 2019). One way to do so (which is a view I currently subscribe to) is through re-constructing a generative model of past events, for example, using some residual neural activity (memory traces) as the source of prediction errors  (Werning, 2020).

My intuition is that some experiences/mental states may be uniquely described using the four dimensions above. In contrast, others may arise through a more complex combination of processes (e.g., long-term future-oriented imagination as a combination of paradigmatic, (fantastical) imagination with elements of episodic remembering).

This framing may reveal the possible sources of common phenomenological elements of experience (e.g., mental imagery, sense of reality) and experiential properties (e.g., epistemic reliability). For example, unsurprisingly, the processes involving mental imagery seem to engage the lower (perceptual) levels of prediction, which are more proximate to the sensory surfaces. 

The experiences typically involving the sense of reality (perception and episodic remembering) directly rely on the identifiable sources of prediction error in scenario construction (sensory input and memory traces, respectively). This is notably different from the future-based instructive imaginings (“Would the sofa I saw in Ikea fit into my living room?”) or counterfactual causal reasoning (“Would I still have been late if I took another road?”) (Rappe & Werning, 2024). In both cases, the traces may serve as a basis of epistemic reliability but are indirectly involved in the process (in the first case, traces are combined, and in the second case, the relevant predictions are later inhibited). 

Ultimately, by adopting a systematic approach to mapping out various types of experiences across these (and other) dimensions, we may refine our understanding of the underlying constructive processes and establish a more coherent exploratory framework. The idea is not just to provide a new taxonomy but to look at what various experiences may share in common in terms of the kind of construction required. This process could also potentially generate new testable hypotheses. 

Notes:

[1] For some frameworks that postulate a single principle for most if not all of cognitive activity (like predictive processing), the question is better reformulated not in terms of continuity, but relevant dissimilarities that seem to fall thorugh the cracks in the strife for unification (see, e.g., Cavedon-Taylor, 2022).

References

Beck, S. R., & Rafetseder, E. (2019). Are counterfactuals in and about time? Behavioral and Brain Sciences, 42.

De Brigard, F., & Parikh, N. (2019). Episodic counterfactual thinking. Current Directions in Psychological Science, 28(1), 59-66.

Cavedon-Taylor, D. 2021. Untying the Knot: Imagination, Perception and their Neural Substrates. Synthese 199, no. 3: 7203-7230.

Cavedon-Taylor, D. 2022. Predictive processing and perception: What does imagining have to do with it? Consciousness and Cognition 106: 103419.

Clark, A. (2015). Surfing uncertainty: Prediction, action, and the embodied mind. Oxford University Press.

Hoerl, C., & McCormack, T. (2019). Thinking in and about time: A dual systems perspective on temporal cognition. Behavioral and Brain Sciences, 42, e244.

Hohwy, J. (2013). The predictive mind. OUP Oxford.

Friston, K. (2010). The free-energy principle: a unified brain theory? Nature Reviews Neuroscience, 11(2), 127-138.

Kind, A., & Kung, P. (Eds.). (2016). Knowledge through imagination. Oxford University Press.

Michaelian, K. (2016). Mental time travel: Episodic memory and our knowledge of the personal past. Cambridge, MA: MIT Press.

Rappe, S., & Werning, M. (2024). Episodic memory in causal reasoning about singular events. In Proceedings of the Annual Meeting of the Cognitive Science Society (Vol. 46).

Sutton, J. (2006). Remembering (Handbook of Situated Cognition). In forthcoming in Robbins & Aydede (eds), Cambridge Handbook of Situated Cognition.

Sutton, J., & O’Brien, G. (2022). Distributed traces and the causal theory of constructive memory. In Current controversies in philosophy of memory (pp. 82-104). Routledge.

Werning, M. (2020). Predicting the past from minimal traces: Episodic memory and its distinction from imagination and preservation. Review of Philosophy and Psychology, 11, 301-333.