Joshua Myers is a postdoctoral fellow at York University, specializing in epistemology and the philosophy of mind and cognitive science. His work investigates the nature and epistemic role of the imagination and other representations that occur outside of discursive thought.
A post by Joshua Myers and Johannes Mahr.
Imagination and episodic memory are intimately linked. Imagination draws on previous experience—when picturing yourself on a beach in Portugal sipping mango juice, you will draw on memories of other beaches you’ve visited and other mango juices you’ve had. Conversely, memory involves imaginatively filling in missing detail—when you remember your last trip to the beach you will likely draw on elements from other beach trips to populate the scene.
A popular way to explain the intimate relationship between memory and imagination has been to say that they both rely on a process of episodic construction that (re)combines elements of previous experiences (Addis 2020, Schacter & Addis 2007, Hassabis & Maguire 2007). But it is much less common to cash out the notion of episodic construction or (re)combination in any detail. According to what principles does this process unfold?
Johannes Mahr is an Assistant Professor of Philosophy at York University in Toronto, specializing in the philosophy of cognitive science and psychology. His work combines theoretical analysis with empirical methods from cognitive psychology to understand episodic memory, imagination, and social cognition.
We propose that episodic construction should be thought of as a compositional process. More specifically, we defend Episodic Compositionality: the view that episodic construction generates complex episodic representations by combining basic representational elements according to syntactic rules.
Episodic Compositionality contrasts with widely-endorsed approaches that deny that episodic construction—and the episodic representations that it outputs—have any interesting compositional structure. Most notable and influential among these competitors is Episodic Associationism, which is the view that episodic construction combines basic elements according to associative rather than syntactic relations. For example, Addis emphasizes that ”the modus operandi of the constructive simulation system is associative, with different associative processes occurring at different levels of the simulation process” (2020, 11).
Associations are causal links between mental representations formed purely as a function of repeated co-occurrence. Because association is a causal rather than semantic relationship between representations, it does not ‘say’ anything about the world. For example, while ‘apples are red,’ has truth-conditional content, an association between ‘apples’ and ‘red’ does not. It is merely a mechanism by which one representation tends to activate the other, without being combined into a meaningful semantic structure.
While episodic associationism is motivated by a variety of empirical results such as the holistic nature of memory retrieval (Horner & Burgess 2013), various false memory effects (Roediger & McDermott 1995), and hippocampal architecture (Marr 1971), it also runs into several challenges:
The Challenge from Accuracy: Memories and imaginings are evaluable for accuracy. However, associations are not evaluable for accuracy. So, episodic associationism cannot account for episodic content. The associationist might attempt to reverse-engineer accuracy conditions into their account. For example, they might hold that the associated elements are themselves accuracy-evaluable. But associations themselves are not accuracy-evaluable no matter how much accuracy-evaluable content one builds into their relata. Moreover, associationism is meant to explain how episodic construction produces complex contents. If the basic elements already express complex contents, then associationism lacks explanatory power.
The Challenge from Novelty: We can imagine contents that have never co-occurred in our previous experiences, such as pigs with wings. Novel content outstrips our stored associations. Associationists might attempt to account for novel content by appealing to associative inhibition. But to account for novel content such as flying pigs we need to not only explain why we fail to imagine contents that are associated with wings, such as birds, but also why we succeed at imagining content that isn’t associated with wings, such as pigs. Instead, the associationist might appeal to schemas, which are associative structures that abstract away from particular elements and instead encode general regularities (Ghosh & Gilboa 2014, Addis 2020). But abstraction is not the only kind of novelty present in episodic construction. It also involves the novel recombination of features, regardless of whether those features are generalized from previous experience.
The Challenge from Inference: Because associations are causal rather than rational relationships between mental states, Episodic Associationism predicts that episodic construction should be modulated by associative learning, but not by rational inference. However, episodic construction is rife with inferential effects. Consider the misinformation effect, which occurs when misleading information presented after an event causes witnesses to misremember the event (Loftus 1979). The more credibility a subject assigns to the misinformation source, the more susceptible they become to the misinformation effect (Gabbert, Memon, & Wright 2007, Skagerberg & Wright 2009). Associationism struggles to explain this result because the misleading information has the same associative strength regardless of source credibility. Instead, this result suggests that subjects are inferring the truth of the misinformation based on the reliability of the source. Other studies suggest that episodic construction is driven by causal inferences (Strickland & Keil 2011) and narrative coherence (Cohn-Sheehy et al. 2021, 2022) as opposed to mere association.
Our objections to episodic associationism all hinge on the idea that episodic construction does not merely associate basic elements, but instead composes those elements in ways that are meaningful. This provides the initial motivation for Episodic Compositionality, which is the view that episodic construction operates according to syntactic rules rather than mere associations.
What does it mean to say that episodic construction has a syntax? First, while any two elements can be associated with each other, a syntax will impose constraints on which combinations of elements yield well-formed representations. Second, it entails a syntactic hierarchy. Complex expressions are composed of simpler ones which are in turn composed of yet simpler ones, and so on. In language, for example, sentences are composed of phrases which are composed of words. Third, it entails role-filler independence: lexical elements are independent from the syntactic roles that they play and vice versa. For example, ‘Josh high-fives Johannes’ and ‘Johannes high-fives Josh’ both feature the same elements (‘Johannes’, ‘high-fives’, ‘Josh’), and the same syntactic roles (agent and patient), even though different elements play different roles in each.
There are a variety of motivations for Episodic Compositionality beyond just the failure of associationism:
The Argument from Compression: Experiences are informationally rich. This poses a problem for episodic cognition: how should this information be encoded to be retrievable at a later date? Encoding all information in a one-to-one fashion would be computationally intractable and neurally inefficient. Episodic Compositionality provides a way to address this problem: episodic construction relies on re-using and recombining representational primitives - objects, spatial relations, actions, etc. Rather than storing each episode in its entirety, the system encodes how these elements were bound together in a particular configuration. This achieves compression because the same primitives can be reused across many episodes: the system need not redundantly encode the full perceptual detail each time.
The Argument from Hierarchical Organization: There is empirical evidence that episodic representations are hierarchically organized (Andermane et al. 2021). The levels of episodic cognition range from representations of individual objects, to representations of small-scale, self-contained events involving those objects, to representations of large-scale narratives comprising several events. Moving up the hierarchy involves combining items at the lower level according to distinct, contentful relations. Importantly, these different levels involve different representational resources, different underlying brain regions, and different mechanisms for retrieval and forgetting. This vindicates a core prediction of Episodic Compositionality: that episodic representations form a hierarchical structure akin to a syntactic tree.
The Arguments from Productivity and Systematicity: Productivity and systematicity form the basis for influential arguments in favor of the compositionality of thought (Fodor 1975, Fodor & Pylyshyn 1988). We contend that these classical arguments can be extended to support Episodic Compositionality. First, episodic construction is productive: it exhibits an unbounded competence for constructing novel representations. As an example of episodic productivity: if you can imagine a circle inside of a square, then you can imagine a circle inside of a square inside of a circle, and a circle inside of a square inside of a circle inside of a square, and so on. Second, episodic construction is systematic: it produces representations that reuse the same parts and bear systematic relationships to each other. As an example of episodic systematicity: if you can imagine an apple to the left of an orange, then you can also imagine an orange to the left of an apple. Episodic Compositionality provides a compelling explanation of these phenomena. Episodic construction can yield an indefinite number of novel representations by iteratively applying syntactic combinatorial rules to a finite lexicon. Episodic construction can result in representations that are systematically related to each other by rearranging primitives within the same syntactic structure. There is empirical evidence to support this: Thakral et al. (2020) showed that in imagination, the hippocampus reinstates and recombines elements previously tokened in episodic memory.
Does this mean that episodic construction is necessarily ‘language-like’? No. Episodic Compositionality is the claim that episodic construction involves a lexicon and syntax. It is compatible with various more specific accounts of what such a lexicon and syntax might look like. Indeed, many hold that pictures and other non-linguistic representations are compositional (e.g. Kulvicki 2020, Lande 2024). Similarly, Episodic Compositionality is compatible with associative effects on retrieval in episodic memory and imagination. Just as the compositional nature of sentence-construction is compatible with associative effects governing word-choice, so too is Episodic Compositionality compatible with the idea that association regularly governs which episodic elements are being retrieved as inputs to construction. Indeed, much of the empirical evidence commonly cited in favor of Episodic Associationism is better construed as evidence for associative episodic retrieval.
Episodic Compositionality provides a useful framework for formulating hypotheses about the nature, syntax, and semantics of the episodic construction system—a framework that views episodic construction as not merely recapitulating co-occurrence statistics, but as making rational inferences over contentful representations.
References
Addis, D. R. (2020). Mental time travel? A neurocognitive model of event simulation. Review of Philosophy and Psychology, 11(2), 233-259.
Andermane, N., Joensen, B. H., & Horner, A. J. (2021). Forgetting across a hierarchy of episodic representations. Current Opinion in Neurobiology, 67, 50-57.
Cohn-Sheehy, B. I., Delarazan, A. I., Reagh, Z. M., Crivelli-Decker, J. E., Kim, K., Barnett, A. J., ... & Ranganath, C. (2021). The hippocampus constructs narrative memories across distant events. Current Biology, 31(22), 4935-4945.
Cohn-Sheehy, B. I., Delarazan, A. I., Crivelli-Decker, J. E., Reagh, Z. M., Mundada, N. S., Yonelinas, A. P., ... & Ranganath, C. (2022). Narratives bridge the divide between distant events in episodic memory. Memory & Cognition, 50(3), 478-494.
Fodor, J. (1975). The language of thought. Harvard University Press.
Fodor, J. A., & Pylyshyn, Z. W. (1988). Connectionism and cognitive architecture: A critical analysis. Cognition, 28(1-2), 3-71.
Gabbert, F., Memon, A., & Wright, D. B. (2007). I saw it for longer than you: The relationship between perceived encoding duration and memory conformity. Acta psychologica, 124(3), 319-331.
Ghosh, V. E., & Gilboa, A. (2014). What is a memory schema? A historical perspective on current neuroscience literature. Neuropsychologia, 53, 104-114.
Hassabis, D., & Maguire, E. A. (2007). Deconstructing episodic memory with construction. Trends in cognitive sciences, 11(7), 299-306.
Horner, A. J., & Burgess, N. (2013). The associative structure of memory for multi-element events. Journal of Experimental Psychology: General, 142(4), 1370.
Kulvicki, J. V. (2020). Modeling the Meanings of Pictures: Depiction and the philosophy of language. Oxford University Press.
Lande, K. J. (2024). Pictorial syntax. Mind & Language, 39(4), 518-539.
Loftus, E. F. (1979). The malleability of human memory: Information introduced after we view an incident can transform memory. American scientist, 67(3), 312-320.
Marr, D. (1971). Simple memory: a theory for archicortex. Philosophical Transactions of the Royal Society of London. B, Biological Sciences, 262(841), 23-81.
Roediger, H. L., & McDermott, K. B. (1995). Creating false memories: Remembering words not presented in lists. Journal of experimental psychology: Learning, Memory, and Cognition, 21(4), 803.
Schacter, D. L., & Addis, D. R. (2007). The cognitive neuroscience of constructive memory: Remembering the past and imagining the future. Philosophical Transactions of the Royal Society B: Biological Sciences, 362(1481), 773-786.
Skagerberg, E. M., & Wright, D. B. (2009). Susceptibility to postidentification feedback is affected by source credibility. Applied Cognitive Psychology: The Official Journal of the Society for Applied Research in Memory and Cognition, 23(4), 506-523.
Strickland, B., & Keil, F. (2011). Event completion: Event based inferences distort memory in a matter of seconds. Cognition, 121(3), 409-415.
Thakral, P. P., Madore, K. P., Addis, D. R., & Schacter, D. L. (2020). Reinstatement of event details during episodic simulation in the hippocampus. Cerebral Cortex, 30(4), 2321-2337.