Abstract
Free full text
Hierarchical process memory: memory as an integral component of information processing
Abstract
Models of working memory commonly focus on how information is encoded into and retrieved from storage at specific moments. However, in the majority of real-life processes, past information is used continuously to process incoming information across multiple timescales. Considering single unit, electrocorticography, and functional imaging data, we argue that (i) virtually all cortical circuits can accumulate information over time, and (ii) the timescales of accumulation vary hierarchically, from early sensory areas with short processing timescales (tens to hundreds of milliseconds) to higher-order areas with long processing timescales (many seconds to minutes). In this hierarchical systems perspective, memory is not restricted to a few localized stores, but is intrinsic to information processing that unfolds throughout the brain on multiple timescales.
“The present contains nothing more than the past, and what is found in the effect was already in the cause.”
Henri L Bergson
Memory as a component of all neural processes
In real life, multiple timescales of prior information continuously influence the processing of information in the present. Consider, for example, how prior information shapes language comprehension: each phoneme achieves its meaning in the context of a word, each word in the context of a sentence, and each sentence in the context of a discourse. Thus, past information gathered over milliseconds-, seconds-, and minutes-long timescales all contribute to comprehension. More generally, memories of recent events continuously support the processing of incoming information.
Working memory as a specialized memory store
When information from the recent past is needed for task performance, it is conventionally described as being stored in working memory (WM) [1, 2]. Theories of working memory traditionally focus on memory stores: how information enters and leaves them, their capacity, and the robustness of stored information to interference and decay. The separation between information storage and information processing is rooted in analogies with digital computer architectures, where the systems that perform information processing (e.g., CPUs) are separated from the memory systems that store information (e.g., RAM, caches, and hard-disks). Thus, in computer-inspired models of memory, new information is temporarily stored in limited-capacity working memory buffers, or old information is made available for present processing when it is loaded into the buffers from long-term memory (LTM) storage (Figure 1A). In such models, the systems of memory storage (WM and LTM) are functionally distinct (and in some cases physically separated, Figure 1B) from the systems that support online information processing tasks, such as visual and auditory object recognition, biological motion perception, decision making, and the organization of movement [3].
The innovation provided by the multi-store model (Figure 1) was in specifying how a general purpose working memory resource was instantiated via a control system (“central executive”) operating on a set of functionally specialized buffers (the phonological loop, visuo-spatial sketchpad, and episodic buffers); this clarified how working memory might relate to task performance in wide-ranging task domains. However, as researchers came to consider the number of working memory subsystems that would be required to support memory for different kinds of information over multiple timescales, these subdivisions of working memory began to appear inadequate [4]. Moreover, the neural circuits identified as working memory buffers (e.g., the phonological loop) appear in many cases to be the same as the neural circuits that perform the relevant processing (e.g., of phonological and linguistic information) [5].
Newer perspectives on working memory no longer require a physical separation between memory storage and ongoing information processing, but they still maintain a functional separation between stores and processing [4, 6]. For example, in contemporary theories of visual working memory, the visual memory representations are located in the visual processing stream itself. Nonetheless, the representations in visual WM are functionally separated from new visual input, as top-down fronto-parietal signals are required to shield the contents of WM from interference. Thus, information is still considered to be stored in and retrieved from working memory, which has a distinct functional status from the representations of incoming information.
Although working memory models effectively capture behavioral and neural data related to goal-directed control of prior information (e.g., maintaining a visual array over a delay period), we propose that they can also conceptually obscure our understanding of ubiquitous real-life processes in which memory has to be integrated with ongoing processing. We argue that memory and online processing are entangled in many everyday situations, such as reading a book or conversing with a friend. Thus, we suggest replacing the question “how is information stored and then retrieved for later processing?” with the question “how does prior information continuously shape processing in the present moment?”
Questioning the separation between memory and ongoing processing
The separation of the contents of memory from ongoing information processing has long been questioned. A prominent example is the network memory model [7], which embraced the idea that there are no dedicated memory systems and that memory is an integral part of all neural networks, arguing for a shift in research focus from “systems of memory” to the “memory of systems”. The network memory model also incorporated a hierarchical organization of the systemic memory, as higher order areas combined and abstracted memories accumulated from lower areas in the hierarchy. In a similar spirit, the theory of active memory questioned the separation between short-term memory and LTM [8]. In this framework, memory is considered a single entity, either active or inactive, with active memories envisioned as a subset of especially labile memories that are currently being used by the brain to process incoming information [9]. More recently, the separation between memory units and processing units has been questioned by neuroscientists who propose local interactions between memory and perception within visual areas [10–12] and beyond [13], as well as by some long-term memory researchers who argue for the involvement of the medial temporal lobe memory system in perceptual processes [14–17].
Neurobiologically, the idea of functional separation of memory into processes and stores is not well substantiated. In neural circuits, there does not appear to be a separation between neurons that process information and neurons that store information. For example, in the sea slug Aplysia Californica, there are no systems specialized for memory that are separated from perceptual and motor systems [18]; in contrast, short-term and long-term changes in the synaptic efficacy of sensory and motor neurons support learning and memory. More generally, in mammalian circuits, it is known that patterns of prior information reshape synapses over minutes and years [19–21] and can alter levels of activation, potentiation, and excitability over milliseconds and minutes [22–24]. Thus, at the biological level, prior information continually influences information processing in the present, and memory is intrinsic to virtually all neural processes.
Psychologists have also critiqued the segregation of memory and ongoing processing. An early step in this direction came in linking memory performance to the hierarchical depth of processing of a stimulus [25, 26]. Following the rise of connectionist models [27], even more radical theories arose, proposing that the memory needed for in task performance (e.g., in language comprehension) is an intrinsic property of distributed circuits that continuously process the language input [28]. In such models, individual differences in language comprehension are ascribed not to differences in capacity, but to differences in linguistic expertise (i.e. the organization of information processing within the language processing circuits). Another model that emphasized the importance of expertise for explaining mnemonic performance was the long-term working memory (LTWM) model [29], which aimed to account for variations in memory as a function of domain expertise and stimulus semantics.
Finally, computational modeling work has shown in a variety of ways how memory and information processing can be combined in the same circuits [30]. Abstract connectionist models provided some of the earliest and most influential examples [31], while more recent dynamical systems models of information processing have also posited varying integration timescales, leading each level of a hierarchical system to exhibit differing capacities to integrate information over time [32–34].
Process memory framework
Synthesizing these prior ideas with recent empirical work from neurophysiology and neuroimaging, we now outline a framework for how memory serves online information processing. In this framework, we emphasize that traces of past information should not be segregated from ongoing neural processes. To dissociate our notion of memory from the traditional notion of encapsulated memory stores, we will use the term “process memory” (PM) throughout the paper. We use the term process memory, in a broad sense, to mean active traces of past information that are used by a neural circuit to process incoming information in the present moment. Furthermore, we argue for a hierarchical organization of process memory, in which the timescale of memory-dependent processing gradually increases from early sensory areas to high order areas. The new framework is broadly consistent with the family of distributed memory models described above [7, 8, 10].
In the process memory framework, virtually all cortical circuits have the ability to accumulate information over time (Figure 2, red circles). We operationalize the processing timescale of each brain region by measuring its temporal receptive window (TRW): the window of time in which prior information from an ongoing stimulus can affect the processing of newly arriving information. The TRW is defined as a temporal analogue of the spatial receptive field. Some areas have a short TRW (e.g., tens to hundreds of milliseconds), enabling them to integrate a few phonemes to detect a word. Other areas have a medium TRW (e.g., several seconds), enabling the integration of sequences of words while parsing a sentence. Still other areas have a long TRW (e.g., tens to hundreds of seconds) needed for the integration of sentences over time while comprehending a narrative.
The TRW increases in an orderly hierarchical fashion from early sensory areas to higher-order perceptual and cognitive areas (Figure 2, size of red circles). Memories of the recent past are not stored in a few dedicated memory stores, but are organized hierarchically across cortical regions that process incoming information. We will now outline evidence for the process memory model, before returning to discuss the relationship between process memory and existing models of working memory (WM) and long term memory (LTM).
A method for mapping processing timescales
To characterize the TRW for each area of the cerebral cortex, we measured the extent to which traces of prior events (recent memory) influenced moment-to-moment neural activity (online processes) during minutes-long real-life stimuli (such as stories and movies – see Box 1). First, minutes-long stimuli were broken into smaller and smaller temporal units (e.g., paragraphs, sentences, words) and the order of the units was scrambled, thereby varying the temporal structure of the stimulus at multiple timescales while preserving the atomic elements (e.g., using the identical movie frames or elementary sound clips in all conditions). Next, neural activity during the intact and scrambled stimuli was examined for evidence of whether online responses changed as a function of the structure of prior events. Areas with short processing timescales (i.e., short TRWs) were expected to respond in the same way at any given moment regardless of the prior context. Areas with long processing timescales (i.e. long TRWs) were expected to modulate their responses to a given event as a function of prior context over many seconds; for example, responses to a particular word would be affected by information from a previous paragraph. Areas with intermediate processing timescales (i.e., intermediate TRWs) were expected to be sensitive to the structure of past events over a few seconds, e.g., at a sentence-level timescale.
Neural response dynamics during these temporally-extended real-life stimuli were assessed using inter-subject correlation [35]. The response reliability of any given brain region was measured by calculating the correlation of that region’s timecourse across multiple subjects during exposure to the same stimulus. High correlations between subjects at any given location in the brain indicate the presence of stimulus-driven reliable responses at that location. For more details about the inter-subject correlation method see [35, 36].
A hierarchical topography of process memory
Mapping temporal receptive windows using fMRI, electrocorticography (ECoG), and single unit recording has revealed a large-scale topographic organization of processing timescales along the auditory and visual processing streams [37–39]. Figure 3A presents the gradual transition along the superior temporal gyrus, from short TRWs in early auditory cortex to long TRWs in the temporoparietal junction and angular gyrus, as measured using fMRI while subjects listened to a story scrambled over multiple timescales. Early auditory areas (A1+) responded reliably (i.e. high inter-subject correlation) at all scrambling levels, from the intact full story (FS), to scrambled paragraphs (P), scrambled sentences (S), scrambled words (W), and backward speech (B). These sensory regions were denoted as having short process memory (short TRWs). Further up the processing hierarchy, more and more of the stimulus history was found to affect processing in the present moment. In areas with especially long process memory (long TRWs), such as the temporoparietal junction (TPJ), angular gyrus (AG), and medial prefrontal cortex (mPFC), the cortical activity at each moment depended on information that had previously arrived over tens of seconds. In these higher order areas the responses were reliable only at the full story (FS) and paragraphs (P) levels [38]. The process memory hierarchy was not confined to the processing of temporally extended linguistic input, as a similar topographic organization was found in the visual system when subjects viewed silent movies [39]. Finally, a similar topographic gradient of TRWs was observed in the visual and auditory processing streams for an audio-visual movie using ECoG (Figure 3B), which replicated the fMRI findings with a direct neurophysiological measurement [37].
Slow neural dynamics are more pronounced in areas with long process memory
In the primate brain, the TRW of an area (i.e., its process memory timescale) covaries with the timescale of its intrinsic neural dynamics. In other words, intrinsically faster neural dynamics are observed in areas with shorter TRWs, whereas intrinsically slower neural dynamics are observed in areas with longer TRWs. Recently, the spike-count autocorrelation during short resting periods was measured in seven cortical areas in the macaque monkey, revealing a hierarchical ordering in which sensory and prefrontal areas exhibited shorter and longer timescales, respectively [40] (Figure 4). Similarly, using ECoG [37] it was observed that neuronal population activity in higher-order regions exhibited a greater proportion of low frequency fluctuations (and increased temporal autocorrelation), while in early sensory areas there was a greater proportion of high frequency fluctuations (and decreased temporal autocorrelation). In both studies, the gradient of timescales of neural dynamics was observed in the absence of any stimulus, suggesting that it may be an intrinsic property of neural circuits. Similar dynamical organizations have been reported using fMRI [41].
Together, these results suggest that areas with faster neural dynamics accumulate information over shorter timescales, whereas areas with slower neural dynamics accumulate information over longer timescales. Thus, the abundant slow fluctuations of neural dynamics [42], which are commonly ignored or treated as entirely artifactual, can actually be connected to the processing of real-life information, which is structured on timescales of milliseconds, seconds and minutes [37, 41, 43]. Recent modeling has connected the hierarchical organization of dynamical timescales to changes in excitatory-inhibitory balance (via changes in spine density) [44]; other models have proposed a role for large-scale anatomical organizations [45] and the topography of neuromodulators also appears likely to also play a role.
Scaling of temporal receptive window size as a function of information rate
The results presented so far suggest that TRWs increase gradually from sensory areas with short TRWs up to higher-order areas with long TRWs. The next study asked whether the size of the TRW should be defined in fixed temporal units (e.g., milliseconds, seconds, and minutes) or informational units (e.g., phonemes, words, and sentences). Fortunately, temporal units and informational units are easily dissociated in real-life speech. The fastest speakers of American English will articulate a sentence about twice as fast as the slowest speakers [46]. If temporal integration windows are defined based on informational units, then neural response timecourses should be rescaled in time when the incoming information is rescaled in time. In accordance with such a prediction, a temporal scaling of neural responses was observed throughout auditory, linguistic and extra-linguistic brain areas in response to a linear scaling (speeding or slowing) of the incoming speech rate [47]. These data suggest that the process memory integration window should be measured on relative rather than absolute timescales, with the brain scaling its neural dynamics in response to compression or dilation of the input.
The temporal rescaling of neural responses in accordance with rescaled stimuli began to break down when stimuli were presented at double speed (50% duration stimulus); this is also when intelligibility began to be impaired [47]. Furthermore, it is important to emphasize that, even when the timescales of processing were re-scaled, the TRW hierarchy was preserved. Thus, just as spatial receptive fields can spatially rescale as a function of task, context, and attention demands [48–51], process memory integration windows can temporally rescale according to the rate at which information is arriving [52, 53].
How process memory can be linked with other types of memory
We have argued here for the parsimonious idea that process memory is an integral feature of many cortical processing systems. Moreover, we proposed that process memory increases in a hierarchical manner across the cerebral cortex. How does the process memory framework relate to the classic distinctions [54] between sub-types of memories, both declarative (semantic and episodic) and non-declarative (procedural, priming and conditioning)?
The frameworks are compatible, but answer different questions. Traditional memory categories are defined based on the kinds of stimuli remembered (e.g., visual or auditory), the type of learning (e.g., one-shot or repeated exposures), and the kinds of behavior that the memory supports (e.g., recollection or recognition). The process memory framework is concerned with active memory that is intrinsic to the information processing taking place within a circuit. Thus, there may be similar amounts of process memory, supported by similar circuit mechanisms, even across circuits with very different functions. For example, premotor circuits may contain some of the procedurally learned memories for steering a bicycle, whereas visual cortex may contain the memories that later support recognition of an emotional facial expression. Nevertheless, both organizing a steering movement and recognizing a dynamic facial expression may require integrating information over a second of time. Thus, the type of processes performed in the circuit will define the functional properties of the stored information, but even very different processes may have a common functional signature and some common mechanisms for integrating information. Our model is also compatible with alternative processing-based parcellation of memory, such as [55].
In the process-memory framework, the same cortical neurons that process information (e.g. in sensory perception) also store the information. Nonetheless, additional processes are needed to manipulate, control, and consolidate these process-memories in specific contexts. In particular, there are two major modulatory processes that act on the primary process-memories: attentional control processes supported by fronto-parietal circuits (related to traditional working memory) and binding and consolidation processes supported by medial temporal lobe circuits (related to episodic memory) (Figure 2C).
Process Memory and the Attentional Perspective of Working Memory
In many contemporary perspectives, the term “working memory” has become almost synonymous with attention and cognitive control [56]. The deep connection between working memory and attention arises because, in classic working memory paradigms such as digit manipulation and delayed-match-to-sample, subjects are asked to actively preserve some aspects of the incoming information and task goals in mind (see Box I). When the to-be-remembered content is fragile and labile, attention must be used to shield prior information from interference with new information [3, 57]. Shielding against processing of new input while actively maintaining information in memory is a special case of the more common situations discussed above, in which memory and processing of incoming information are intertwined (see Box I).
Although attentional control is a fundamental aspect of cognition, and we incorporate it as a key modulator in our model (Figure 2C), its role in ongoing perception and comprehension may be obscured by the label “working memory”. The work of memory is performed in virtually every neural circuit, and attentional systems modulate this ongoing processing in accordance with rule- or goal-related constraints. Thus, both the designated memory buffers framework and the attentional-control framework of working memory fail to account for the tight reliance of online neural processes on memory across multiple levels of the processing hierarchy. One perspective on top-down modulation that is more consistent with our distributed model is provided by hierarchical predictive coding models (see Box II Q4).
Process Memory and the Medial Temporal Lobe
Our framework suggests that processing timescales increase gradually along the cortical hierarchy, from early sensory areas with short (milliseconds-long) integration windows up to higher order areas with minutes-long integration windows (see Figure 2A–B). The responses in areas with the longest processing timescales, at the top of the hierarchy, seem to be influenced by information accumulated over many minutes. These areas overlap broadly with the default mode network (DMN), and include the angular gyrus, retrosplenial cortex, precuneus, posterior cingulate cortex, and mPFC. Many studies implicate these areas in the encoding and retrieval of episodic memories (memory for situated experiences tied to a particular time and place [58, 59]), as well as in a variety of high-level cognitive processes such as decision making [60–62], self-representation [63–65], prospective planning [63], and social reasoning [65].
The DMN is functionally and anatomically connected to the medial temporal lobe (MTL) and hippocampus [66, 67], but it remains unclear to what extent the hippocampus is needed in order for the DMN to retain information over many minutes. Hippocampal damage strikingly impacts the ability to retain episodic memories [68], but exactly how long new information can be maintained in cortical areas without hippocampal involvement is not clear. Behaviorally, under natural conditions, it appears that information can be sustained for a few minutes without reliance on the hippocampus. For example, hippocampal amnesics can retain stimulus information for long enough to engage in a conversation [68], summarize a short passage of prose [69, 70], and play a complex communicative game [71]. Such observations suggest that a meaningful continuous context (e.g., a conversation or listening to a story) may enable information to persist in cortical areas for few minutes without relying on the hippocampus (though hippocampal circuits do appear to contribute to ongoing processing when they are intact [72–74]). Furthermore, the important role of the hippocampus in encoding and retrieval of episodic memories over long timescales [75] does not exclude its involvement in binding of relational information over short time scales [72, 76–78]. Measurements of neural responses to scrambled narratives in MTL lesion patients will provide important constraints on how these regions modulate process memory.
Links between process memory and active long-term memory
Stored knowledge is just as crucial for online processing of incoming information as is recently acquired information. After all, understanding the meaning of a word, sentence, or idea relies heavily on information gathered throughout the lifetime of the individual. Because online processing, at each level of the hierarchy, relies both on recent memories (e.g., information accumulated during an ongoing conversation) and distant memories (e.g., long term knowledge), both types of memory have to be integrated within each neural circuit. Thus, the argument against dedicated working memory stores can naturally be extended to question the notion of dedicated LTM stores that are separate from the circuits that process information [79].
Inspired by prior researchers [8], we suggest that memory should be conceived as a single entity that can be either in an active (process memory) state or an inactive (long-term memory) state. Process memory refers to prior information that is currently used by an area to process incoming information; to influence ongoing processing, the prior information must be in an active state. The active information is composed of the stimulus information accumulated in a given circuit throughout the event, as well as a subset of long term memories activated during the processing of the incoming information. Note that the word “active” is not necessarily synonymous with sustained elevation in firing rates [80], given that information can be sustained in a neuronal circuit by short-term calcium-mediated synaptic facilitation in the absence of recurrent activity [22]. Inactive long-term memories, by contrast, are simply the long-lasting structural features of a circuit (e.g., synaptic patterning) that are not currently affecting the processing of incoming information in that circuit. Inactive long-term memories may be brought into an active state via the influence of other active process-memories in the circuit, or via modulation from medial temporal lobe input or fronto-parietal control areas (see Box II Q2 and Q5).
Concluding Remarks
In this article, we have argued that the traditional dissociation between memory and ongoing information processing is artificial. Diverse cortical functions, ranging from the smooth pursuit of a swiftly moving object, to resolving an anaphoric reference, up to the integration of information across multiple paragraphs of text, all require active integration of past information with new information. We have reviewed data from multiple sources indicating that timescales of processing vary in a hierarchical fashion across the cerebral cortex, with shorter (milliseconds to seconds) timescales in sensory regions and a gradient of lengthening timescales (seconds to minutes) in higher order cortices. Instead of compartmentalizing memory into increasingly specialized storage systems, we highlight the fact that memory is an integral component of the processing conducted in each neural circuit. Process memory is an especially important factor in real-world cognition and perception, which requires continuous information integration, not only maintenance over delays. We propose the process memory framework as a biologically motivated model of the memory that is intrinsic to ongoing, integrative information processing in naturalistic settings.
Acknowledgments
U.H. and J.C were supported by the National Institute of Mental Health award (R01-MH094480). C.J H. was supported by the Natural Sciences and Engineering Research Council of Canada (RGPIN-2014-04465). For insightful comments and contributions to the ideas in this manuscript, we thank Mariam Aly, Bradley R. Buchsbaum, Alin I. Coman, Jarrod A. Lewis-Peacock, Kenneth A. Norman, Rosanna K. Olsen, and Jordan Poppenk.
Footnotes
Publisher's Disclaimer: This is a PDF file of an unedited manuscript that has been accepted for publication. As a service to our customers we are providing this early version of the manuscript. The manuscript will undergo copyediting, typesetting, and review of the resulting proof before it is published in its final citable form. Please note that during the production process errors may be discovered which could affect the content, and all legal disclaimers that apply to the journal pertain.
References
Full text links
Read article at publisher's site: https://doi.org/10.1016/j.tics.2015.04.006
Read article for free, from open access legal sources, via Unpaywall: https://europepmc.org/articles/pmc4457571?pdf=render
Citations & impact
Impact metrics
Citations of article over time
Alternative metrics
Smart citations by scite.ai
Explore citation contexts and check if this article has been
supported or disputed.
https://scite.ai/reports/10.1016/j.tics.2015.04.006
Article citations
Small vessel disease and cognitive reserve oppositely modulate global network redundancy and cognitive function: A study in middle-to-old aged community participants.
Hum Brain Mapp, 45(5):e26634, 01 Apr 2024
Cited by: 1 article | PMID: 38553856 | PMCID: PMC10980841
Large-scale evidence for logarithmic effects of word predictability on reading time.
Proc Natl Acad Sci U S A, 121(10):e2307876121, 29 Feb 2024
Cited by: 1 article | PMID: 38422017 | PMCID: PMC10927576
Neural timescales reflect behavioral demands in freely moving rhesus macaques.
Nat Commun, 15(1):2151, 09 Mar 2024
Cited by: 0 articles | PMID: 38461167 | PMCID: PMC10925022
Cortical timescales and the modular organization of structural and functional brain networks.
Hum Brain Mapp, 45(2):e26587, 01 Feb 2024
Cited by: 1 article | PMID: 38339903 | PMCID: PMC10823764
Mixed Selectivity Coding of Content-Temporal Detail by Dorsomedial Posterior Parietal Neurons.
J Neurosci, 44(3):e1677232023, 17 Jan 2024
Cited by: 0 articles | PMID: 37985178
Go to all (224) article citations
Similar Articles
To arrive at the top five similar articles we use a word-weighted algorithm to compare words from the Title and Abstract of each citation.
Neuronal timescales are functionally dynamic and shaped by cortical microarchitecture.
Elife, 9:e61277, 23 Nov 2020
Cited by: 81 articles | PMID: 33226336 | PMCID: PMC7755395
Short-Term Memory Impairment
StatPearls Publishing, Treasure Island (FL), 20 Aug 2019
Cited by: 0 articles | PMID: 31424720
Books & documents Free full text in Europe PMC
A resource model of the neural basis of executive working memory.
Proc Natl Acad Sci U S A, 97(7):3573-3578, 01 Mar 2000
Cited by: 121 articles | PMID: 10725372 | PMCID: PMC16281
Information processing with population codes.
Nat Rev Neurosci, 1(2):125-132, 01 Nov 2000
Cited by: 324 articles | PMID: 11252775
Review
A single microcircuit with multiple functions: state dependent information processing in the hippocampus.
Curr Opin Neurobiol, 22(4):704-708, 03 Apr 2012
Cited by: 23 articles | PMID: 22480878 | PMCID: PMC3438355
Review Free full text in Europe PMC
Funding
Funders who supported this work.
NIMH NIH HHS (2)
Grant ID: R01 MH094480
Grant ID: R01-MH094480