Value representations: Fast and slow.

Corticostriatal circuits represent value and choice during value-guided decision making. In this issue of Neuron, Balewski et al. (2022) show that caudate nucleus and orbitofrontal cortex use distinct value signals during choice, which are consistent with two parallel valuation mechanisms, one fast, one slow.

Brain signals of a Surprise-Actor-Critic model: Evidence for multiple learning modules in human decision making.

Learning how to reach a reward over long series of actions is a remarkable capability of humans, and potentially guided by multiple parallel learning modules. Current brain imaging of learning modules is limited by (i) simple experimental paradigms, (ii) entanglement of brain signals of different learning modules, and (iii) a limited number of computational models considered as candidates for explaining behavior. Here, we address these three limitations and (i) introduce a complex sequential decision making task with surprising events that allows us to (ii) dissociate correlates of reward prediction errors from those of surprise in functional magnetic resonance imaging (fMRI); and (iii) we test behavior against a large repertoire of model-free, model-based, and hybrid reinforcement learning algorithms, including a novel surprise-modulated actor-critic algorithm.

Testing models at the neural level reveals how the brain computes subjective value.

Decisions are based on the subjective values of choice options. However, subjective value is a theoretical construct and not directly observable. Strikingly, distinct theoretical models competing to explain how subjective values are assigned to choice options often make very similar behavioral predictions, which poses a major difficulty for establishing a mechanistic, biologically plausible explanation of decision-making based on behavior alone.

Information Theoretic Characterization of Uncertainty Distinguishes Surprise From Accuracy Signals in the Brain.

Uncertainty presents a problem for both human and machine decision-making. While utility maximization has traditionally been viewed as the motive force behind choice behavior, it has been theorized that uncertainty minimization may supersede reward motivation. Beyond reward, decisions are guided by belief, i.

A Bird's eye view from below: Activity in the temporo-parietal junction predicts from-above Necker Cube percepts.

The temporo-parietal junction (TPJ) consistently emerges in other-regarding behavior, including tasks probing affective phenomena such as morality and empathy. Yet the TPJ is also recruited in processes with no affective or social component, such as visuo-spatial processing and mathematical cognition. We present serendipitous findings from a perceptual decision-making task on a bistable stimulus, the Necker Cube, performed in an MRI scanner.

Risk prediction error signaling: A two-component response?

Organisms use rewards to navigate and adapt to (uncertain) environments. Error-based learning about rewards is supported by the dopaminergic system, which is thought to signal reward prediction errors to make adjustments to past predictions. More recently, the phasic dopamine response was suggested to have two components: the first rapid component is thought to signal the detection of a potentially rewarding stimulus; the second, slightly later component characterizes the stimulus by its reward prediction error.

Anterior insula reflects surprise in value-based decision-making and perception.

The brain has been theorized to employ inferential processes to overcome the problem of uncertainty. Inference is thought to underlie neural processes, including in disparate domains such as value-based decision-making and perception. Value-based decision-making commonly involves deliberation, a time-consuming process that requires conscious consideration of decision variables.

One-shot learning and behavioral eligibility traces in sequential decision making.

In many daily tasks, we make multiple decisions before reaching a goal. In order to learn such sequences of decisions, a mechanism to link earlier actions to later reward is necessary. Reinforcement learning (RL) theory suggests two classes of algorithms solving this credit assignment problem: In classic temporal-difference learning, earlier actions receive reward information only after multiple repetitions of the task, whereas models with eligibility traces reinforce entire sequences of actions from a single experience (one-shot).

Balancing New against Old Information: The Role of Puzzlement Surprise in Learning.

Surprise describes a range of phenomena from unexpected events to behavioral responses. We propose a novel measure of surprise and use it for surprise-driven learning. Our surprise measure takes into account data likelihood as well as the degree of commitment to a belief via the entropy of the belief distribution.

Apathy and noradrenaline: silent partners to mild cognitive impairment in Parkinson's disease?

Purpose Of Review: Mild cognitive impairment (MCI) is a comorbid factor in Parkinson's disease. The aim of this review is to examine the recent neuroimaging findings in the search for Parkinson's disease MCI (PD-MCI) biomarkers to gain insight on whether MCI and specific cognitive deficits in Parkinson's disease implicate striatal dopamine or another system.

Recent Findings: The evidence implicates a diffuse pathophysiology in PD-MCI rather than acute dopaminergic involvement.

Neural correlates of anticipation risk reflect risk preferences.

Individual risk preferences have a large influence on decisions, such as financial investments, career and health choices, or gambling. Decision making under risk has been studied both behaviorally and on a neural level. It remains unclear, however, how risk attitudes are encoded and integrated with choice.

Optimizing experimental design for comparing models of brain function.

This article presents the first attempt to formalize the optimization of experimental design with the aim of comparing models of brain function based on neuroimaging data. We demonstrate our approach in the context of Dynamic Causal Modelling (DCM), which relates experimental manipulations to observed network dynamics (via hidden neuronal states) and provides an inference framework for selecting among candidate models. Here, we show how to optimize the sensitivity of model selection by choosing among experimental designs according to their respective model selection accuracy.

Pupil Dilation Signals Surprise: Evidence for Noradrenaline's Role in Decision Making.

Our decisions are guided by the rewards we expect. These expectations are often based on incomplete knowledge and are thus subject to uncertainty. While the neurophysiology of expected rewards is well understood, less is known about the physiology of uncertainty.

Neural responses to ingroup and outgroup members' suffering predict individual differences in costly helping.

Little is known about the neurobiological mechanisms underlying prosocial decisions and how they are modulated by social factors such as perceived group membership. The present study investigates the neural processes preceding the willingness to engage in costly helping toward ingroup and outgroup members. Soccer fans witnessed a fan of their favorite team (ingroup member) or of a rival team (outgroup member) experience pain.

A common role of insula in feelings, empathy and uncertainty.

Although accumulating evidence highlights a crucial role of the insular cortex in feelings, empathy and processing uncertainty in the context of decision making, neuroscientific models of affective learning and decision making have mostly focused on structures such as the amygdala and the striatum. Here, we propose a unifying model in which insula cortex supports different levels of representation of current and predictive states allowing for error-based learning of both feeling states and uncertainty. This information is then integrated in a general subjective feeling state which is modulated by individual preferences such as risk aversion and contextual appraisal.

Explicit neural signals reflecting reward uncertainty.

The acknowledged importance of uncertainty in economic decision making has stimulated the search for neural signals that could influence learning and inform decision mechanisms. Current views distinguish two forms of uncertainty, namely risk and ambiguity, depending on whether the probability distributions of outcomes are known or unknown. Behavioural neurophysiological studies on dopamine neurons revealed a risk signal, which covaried with the standard deviation or variance of the magnitude of juice rewards and occurred separately from reward value coding.

Investigating signal integration with canonical correlation analysis of fMRI brain activation data.

How the brain integrates signals from specific areas has been a longstanding critical question for neurobiologists. Two recent observations suggest a new approach to fMRI data analysis of this question. First, in many instances, the brain analyzes inputs by decomposing the information along several salient dimensions.

Human insula activation reflects risk prediction errors as well as risk.

Understanding how organisms deal with probabilistic stimulus-reward associations has been advanced by a convergence between reinforcement learning models and primate physiology, which demonstrated that the brain encodes a reward prediction error signal. However, organisms must also predict the level of risk associated with reward forecasts, monitor the errors in those risk predictions, and update these in light of new information. Risk prediction serves a dual purpose: (1) to guide choice in risk-sensitive organisms and (2) to modulate learning of uncertain rewards.

Adding prediction risk to the theory of reward learning.

This article analyzes the simple Rescorla-Wagner learning rule from the vantage point of least squares learning theory. In particular, it suggests how measures of risk, such as prediction risk, can be used to adjust the learning constant in reinforcement learning. It argues that prediction risk is most effectively incorporated by scaling the prediction errors.

Neural differentiation of expected reward and risk in human subcortical structures.

In decision-making under uncertainty, economic studies emphasize the importance of risk in addition to expected reward. Studies in neuroscience focus on expected reward and learning rather than risk. We combined functional imaging with a simple gambling task to vary expected reward and risk simultaneously and in an uncorrelated manner.