Neural mechanisms of credit assignment for delayed outcomes during contingent learning.

Adaptive behavior in complex environments critically relies on the ability to appropriately link specific choices or actions to their outcomes. However, the neural mechanisms that support the ability to credit only those past choices believed to have caused the observed outcomes remain unclear. Here, we leverage multivariate pattern analyses of functional magnetic resonance imaging (fMRI) data and an adaptive learning task to shed light on the underlying neural mechanisms of such specific credit assignment.

Temporal Dynamics of Uncertainty Cause Anxiety and Avoidance.

Alfred Hitchcock, film director and "Master of Suspense," observed that terror is not driven by a negative event, but "only in the anticipation of it." This observation is not restricted to the movies: Anxiety builds when we anticipate uncertain negative events, and heightened reactivity during uncertain threat anticipation is a transdiagnostic marker of anxiety (Grupe & Nitschke, 2013; Holley & Fox, 2022; Hur et al., 2020; Krain et al.

PET-measured human dopamine synthesis capacity and receptor availability predict trading rewards and time-costs during foraging.

Foraging behavior requires weighing costs of time to decide when to leave one reward patch to search for another. Computational and animal studies suggest that striatal dopamine is key to this process; however, the specific role of dopamine in foraging behavior in humans is not well characterized. We use positron emission tomography (PET) imaging to directly measure dopamine synthesis capacity and D and D receptor availability in 57 healthy adults who complete a computerized foraging task.

Cognitive maps: Constructing a route with your snout.

Humans construct cognitive maps of the physical, imagined, and abstract world around us based on visually sampled information. A new study shows how the human brain can also use olfactory cues to form and use cognitive maps.

Goal-oriented representations in the human hippocampus during planning and navigation.

Recent work in cognitive and systems neuroscience has suggested that the hippocampus might support planning, imagination, and navigation by forming cognitive maps that capture the abstract structure of physical spaces, tasks, and situations. Navigation involves disambiguating similar contexts, and the planning and execution of a sequence of decisions to reach a goal. Here, we examine hippocampal activity patterns in humans during a goal-directed navigation task to investigate how contextual and goal information are incorporated in the construction and execution of navigational plans.

Neural mechanisms of credit assignment for inferred relationships in a structured world.

Animals abstract compact representations of a task's structure, which supports accelerated learning and flexible behavior. Whether and how such abstracted representations may be used to assign credit for inferred, but unobserved, relationships in structured environments are unknown. We develop a hierarchical reversal-learning task and Bayesian learning model to assess the computational and neural mechanisms underlying how humans infer specific choice-outcome associations via structured knowledge.

The hippocampus and orbitofrontal cortex jointly represent task structure during memory-guided decision making.

The hippocampus, well known for its role in episodic memory, might also be an important brain region for extracting structure from our experiences in order to guide future decisions. Recent evidence in rodents suggests that the hippocampus supports decision making by representing task structure in cooperation with the orbitofrontal cortex (OFC). Here, we examine how the human hippocampus and OFC represent task structure during an associative learning task that required learning of both context-determined and context-invariant probabilistic associations.

Inferences on a multidimensional social hierarchy use a grid-like code.

Generalizing experiences to guide decision-making in novel situations is a hallmark of flexible behavior. Cognitive maps of an environment or task can theoretically afford such flexibility, but direct evidence has proven elusive. In this study, we found that discretely sampled abstract relationships between entities in an unseen two-dimensional social hierarchy are reconstructed into a unitary two-dimensional cognitive map in the hippocampus and entorhinal cortex.

The orbital frontal cortex, task structure, and inference.

The orbital frontal cortex (OFC) has long been linked to goal-directed, flexible behaviors. Recent evidence suggests the OFC plays key roles in representing the abstracted structure of task spaces, and using this representation for flexible inferences during both learning and choice. Here, we review convergent evidence from studies in animal models and humans in support of this view.

Protocol for building a cognitive map of structural knowledge in humans by integrating abstract relationships from separate experiences.

Humans are adept at learning the latent structure of the relationship between abstract concepts and can build a cognitive map from limited experiences. However, examining internal representations of the cognitive map is challenging because they are unobservable and differ across individuals. Here, we introduce a behavioral training protocol designed for human participants to implicitly build a map of two-dimensional social hierarchies while making a series of binary choices and analytic tools for measuring the internal representation of this structural knowledge.

Map Making: Constructing, Combining, and Inferring on Abstract Cognitive Maps.

Cognitive maps enable efficient inferences from limited experience that can guide novel decisions. We tested whether the hippocampus (HC), entorhinal cortex (EC), and ventromedial prefrontal cortex (vmPFC)/medial orbitofrontal cortex (mOFC) organize abstract and discrete relational information into a cognitive map to guide novel inferences. Subjects learned the status of people in two unseen 2D social hierarchies, with each dimension learned on a separate day.

Neural computations underlying strategic social decision-making in groups.

When making decisions in groups, the outcome of one's decision often depends on the decisions of others, and there is a tradeoff between short-term incentives for an individual and long-term incentives for the groups. Yet, little is known about the neurocomputational mechanisms at play when weighing different utilities during repeated social interactions. Here, using model-based fMRI and Public-good-games, we find that the ventromedial prefrontal cortex encodes immediate expected rewards as individual utility while the lateral frontopolar cortex encodes group utility (i.

Value, search, persistence and model updating in anterior cingulate cortex.

Dorsal anterior cingulate cortex (dACC) carries a wealth of value-related information necessary for regulating behavioral flexibility and persistence. It signals error and reward events informing decisions about switching or staying with current behavior. During decision-making, it encodes the average value of exploring alternative choices (search value), even after controlling for response selection difficulty, and during learning, it encodes the degree to which internal models of the environment and current task must be updated.

Observational learning computations in neurons of the human anterior cingulate cortex.

When learning from direct experience, neurons in the primate brain have been shown to encode a teaching signal used by algorithms in artificial intelligence: the reward prediction error (PE)-the difference between how rewarding an event is, and how rewarding it was expected to be. However, in humans and other species learning often takes place by observing other individuals. Here, we show that, when humans observe other players in a card game, neurons in their rostral anterior cingulate cortex (rACC) encode both the expected value of an observed choice, and the PE after the outcome was revealed.

Two Anatomically and Computationally Distinct Learning Signals Predict Changes to Stimulus-Outcome Associations in Hippocampus.

Complex cognitive processes require sophisticated local processing but also interactions between distant brain regions. It is therefore critical to be able to study distant interactions between local computations and the neural representations they act on. Here we report two anatomically and computationally distinct learning signals in lateral orbitofrontal cortex (lOFC) and the dopaminergic ventral midbrain (VM) that predict trial-by-trial changes to a basic internal model in hippocampus.

Unilateral medial frontal cortex lesions cause a cognitive decision-making deficit in rats.

The medial frontal cortex (MFC) is critical for cost-benefit decision-making. Generally, cognitive and reward-based behaviour in rodents is not thought to be lateralised within the brain. In this study, however, we demonstrate that rats with unilateral MFC lesions show a profound change in decision-making on an effort-based decision-making task.

The behavioral and neural mechanisms underlying the tracking of expertise.

Evaluating the abilities of others is fundamental for successful economic and social behavior. We investigated the computational and neurobiological basis of ability tracking by designing an fMRI task that required participants to use and update estimates of both people and algorithms' expertise through observation of their predictions. Behaviorally, we find a model-based algorithm characterized subject predictions better than several alternative models.

Ventromedial prefrontal and anterior cingulate cortex adopt choice and default reference frames during sequential multi-alternative choice.

Although damage to the medial frontal cortex causes profound decision-making impairments, it has been difficult to pinpoint the relative contributions of key anatomical subdivisions. Here we use function magnetic resonance imaging to examine the contributions of human ventromedial prefrontal cortex (vmPFC) and dorsal anterior cingulate cortex (dACC) during sequential choices between multiple alternatives--two key features of choices made in ecological settings. By carefully constructing options whose current value at any given decision was dissociable from their longer term value, we were able to examine choices in current and long-term frames of reference.

Counterfactual choice and learning in a neural network centered on human lateral frontopolar cortex.

Decision making and learning in a real-world context require organisms to track not only the choices they make and the outcomes that follow but also other untaken, or counterfactual, choices and their outcomes. Although the neural system responsible for tracking the value of choices actually taken is increasingly well understood, whether a neural system tracks counterfactual information is currently unclear. Using a three-alternative decision-making task, a Bayesian reinforcement-learning algorithm, and fMRI, we investigated the coding of counterfactual choices and prediction errors in the human brain.

Frontal cortex and reward-guided learning and decision-making.

Reward-guided decision-making and learning depends on distributed neural circuits with many components. Here we focus on recent evidence that suggests four frontal lobe regions make distinct contributions to reward-guided learning and decision-making: the lateral orbitofrontal cortex, the ventromedial prefrontal cortex and adjacent medial orbitofrontal cortex, anterior cingulate cortex, and the anterior lateral prefrontal cortex. We attempt to identify common themes in experiments with human participants and with animal models, which suggest roles that the areas play in learning about reward associations, selecting reward goals, choosing actions to obtain reward, and monitoring the potential value of switching to alternative courses of action.

Distributed and causal influence of frontal operculum in task control.

It has been suggested that the frontal operculum (fO) is a key node in a network for exerting control over cognitive processes. How it exerts this influence, however, has been unclear. Here, using the complementary approaches of functional MRI and transcranial magnetic stimulation, we have shown that the fO regulates increases and decreases of activity in multiple occipitotemporal cortical areas when task performance depended on directing attention to different classes of stimuli held in memory.

A network centered on ventral premotor cortex exerts both facilitatory and inhibitory control over primary motor cortex during action reprogramming.

Ventral premotor cortex (PMv) is widely accepted to exert an important influence over primary motor cortex (M1) when hand movements are made. Although study of these interactions has typically focused on their excitatory nature, given its strong connections with both ventral and opercular frontal regions, one feature of the influence of PMv over M1 may be inhibitory. Paired-pulse transcranial magnetic stimulation (ppTMS) was used to examine functional interactions between human PMv and M1 during the selection and reprogramming of a naturalistic goal-directed action.

Conceptual representation and the making of new decisions.

A key feature of an adaptive decision making mechanism is its ability to guide behavior even in new situations. In this issue of Neuron, Kumaran et al. report that conceptual representations, which allow generalization from one situation to another through their shared features, can guide decisions even when new problems are encountered via the hippocampus.

How green is the grass on the other side? Frontopolar cortex and the evidence in favor of alternative courses of action.

Behavioral flexibility is the hallmark of goal-directed behavior. Whereas a great deal is known about the neural substrates of behavioral adjustment when it is explicitly cued by features of the external environment, little is known about how we adapt our behavior when such changes are made on the basis of uncertain evidence. Using a Bayesian reinforcement-learning model and fMRI, we show that frontopolar cortex (FPC) tracks the relative advantage in favor of switching to a foregone alternative when choices are made voluntarily.