Towards Human Sensory Augmentation: A Cognitive Neuroscience Framework for Evaluating Integration of New Signals within Perception, Brain Representations, and Subjective Experience.

New wearable devices and technologies provide unprecedented scope to augment or substitute human perceptual abilities. However, the flexibility to reorganize brain processing to use novel sensory signals during early sensitive periods in infancy is much less evident at later ages, making integration of new signals into adults' perception a significant challenge. We believe that an approach informed by cognitive neuroscience is crucial for maximizing the true potential of new sensory technologies.

Correctly establishing evidence for cue combination via gains in sensory precision: Why the choice of comparator matters.

Studying how sensory signals from different sources (sensory cues) are integrated within or across multiple senses allows us to better understand the perceptual computations that lie at the foundation of adaptive behaviour. As such, determining the presence of precision gains - the classic hallmark of cue combination - is important for characterising perceptual systems, their development and functioning in clinical conditions. However, empirically measuring precision gains to distinguish cue combination from alternative perceptual strategies requires careful methodological considerations.

Different types of uncertainty in multisensory perceptual decision making.

Efficient decision-making requires accounting for sources of uncertainty (noise, or variability). Many studies have shown how the nervous system is able to account for perceptual uncertainty (noise, variability) that arises from limitations in its own abilities to encode perceptual stimuli. However, many other sources of uncertainty exist, reflecting for example variability in the behaviour of other agents or physical processes.

Sensory augmentation for a rapid motor task in a multisensory environment.

Background: Sensory substitution and augmentation systems (SSASy) seek to either replace or enhance existing sensory skills by providing a new route to access information about the world. Tests of such systems have largely been limited to untimed, unisensory tasks.

Objective: To test the use of a SSASy for rapid, ballistic motor actions in a multisensory environment.

Multisensory perception and decision-making with a new sensory skill.

It is clear that people can learn a new sensory skill-a new way of mapping sensory inputs onto world states. It remains unclear how flexibly a new sensory skill can become embedded in multisensory perception and decision-making. To address this, we trained typically sighted participants (N = 12) to use a new echo-like auditory cue to distance in a virtual world, together with a noisy visual cue.

Newly learned shape-color associations show signatures of reliability-weighted averaging without forced fusion or a memory color effect.

Reliability-weighted averaging of multiple perceptual estimates (or cues) can improve precision. Research suggests that newly learned statistical associations can be rapidly integrated in this way for efficient decision-making. Yet, it remains unclear if the integration of newly learned statistics into decision-making can directly influence perception, rather than taking place only at the decision stage.

Internal biases are linked to disrupted cue combination in children and adults.

Cue combination describes the use of two sensory cues together to increase perceptual precision. Internal relative bias describes a situation in which two cues to the same state of the world are perceived as signaling different states of the world on average. Current theory and evidence have difficulty accounting for many instances where cue combination is absent, such as in children under 10 years old, and in a variety of tasks.

Developmental changes in colour constancy in a naturalistic object selection task.

When the illumination falling on a surface change, so does the reflected light. Despite this, adult observers are good at perceiving surfaces as relatively unchanging-an ability termed colour constancy. Very few studies have investigated colour constancy in infants, and even fewer in children.

Newly learned novel cues to location are combined with familiar cues but not always with each other.

Mature perceptual systems can learn new arbitrary sensory signals (novel cues) to properties of the environment, but little is known about the extent to which novel cues are integrated into normal perception. In normal perception, multiple uncertain familiar cues are combined, often near-optimally (reliability-weighted averaging), to increase perceptual precision. We trained observers to use abstract novel cues to estimate horizontal locations of hidden objects on a monitor.

An adaptive cue selection model of allocentric spatial reorientation.

After becoming disoriented, an organism must use the local environment to reorient and recover vectors to important locations. A new theory, adaptive combination, suggests that the information from different spatial cues is combined with Bayesian efficiency during reorientation. To test this further, we modified the standard reorientation paradigm to be more amenable to Bayesian cue combination analyses while still requiring reorientation in an allocentric (i.

Merging familiar and new senses to perceive and act in space.

Our experience of the world seems to unfold seamlessly in a unitary 3D space. For this to be possible, the brain has to merge many disparate cognitive representations and sensory inputs. How does it do so? I discuss work on two key combination problems: coordinating multiple frames of reference (e.

Central tendency biases must be accounted for to consistently capture Bayesian cue combination in continuous response data.

Observers in perceptual tasks are often reported to combine multiple sensory cues in a weighted average that improves precision-in some studies, approaching statistically optimal (Bayesian) weighting, but in others departing from optimality, or not benefitting from combined cues at all. To correctly conclude which combination rules observers use, it is crucial to have accurate measures of their sensory precision and cue weighting. Here, we present a new approach for accurately recovering these parameters in perceptual tasks with continuous responses.

Specular highlights improve color constancy when other cues are weakened.

Previous studies suggest that to achieve color constancy, the human visual system makes use of multiple cues, including a priori assumptions about the illumination ("daylight priors"). Specular highlights have been proposed to aid constancy, but the evidence for their usefulness is mixed. Here, we used a novel cue-combination approach to test whether the presence of specular highlights or the validity of a daylight prior improves illumination chromaticity estimates, inferred from achromatic settings, to determine whether and under which conditions either cue contributes to color constancy.

Combining the senses: The role of experience- and task-dependent mechanisms in the development of audiovisual simultaneity perception.

The brain's ability to integrate information from the different senses is essential for decreasing sensory uncertainty and ultimately limiting errors. Temporal correspondence is one of the key processes that determines whether information from different senses will be integrated and is influenced by both experience- and task-dependent mechanisms in adults. Here we investigated the development of both task- and experience-dependent temporal mechanisms by testing 7-8-year-old children, 10-11-year-old children, and adults in two tasks (simultaneity judgment, temporal order judgment) using audiovisual stimuli with differing degrees of association based on prior experience (low for beep-flash vs.

Bayesian transfer in a complex spatial localization task.

Prior knowledge can help observers in various situations. Adults can simultaneously learn two location priors and integrate these with sensory information to locate hidden objects. Importantly, observers weight prior and sensory (likelihood) information differently depending on their respective reliabilities, in line with principles of Bayesian inference.

The Difficulty of Effectively Using Allocentric Prior Information in a Spatial Recall Task.

Prior information represents the long-term statistical structure of an environment. For example, colds develop more often than throat cancer, making the former a more likely diagnosis for a sore throat. There is ample evidence for effective use of prior information during a variety of perceptual tasks, including the ability to recall locations using an egocentric (self-based) frame.

Coding Locations Relative to One or Many Landmarks in Childhood.

Cognitive development studies how information processing in the brain changes over the course of development. A key part of this question is how information is represented and stored in memory. This study examined allocentric (world-based) spatial memory, an important cognitive tool for planning routes and interacting with the space around us.

Boundaries in spatial cognition: Looking like a boundary is more important than being a boundary.

Large walls and other typical boundaries strongly influence neural activity related to navigation and the representations of spatial layouts. They are also major aids to reliable navigation behavior in young children and nonhuman animals. Is this because they are physical boundaries (barriers to movement), or because they present certain visual features, such as visually extended 3D surfaces? Here, these 2 factors were dissociated by using immersive virtual reality and real boundaries.

Backtracking during navigation is correlated with enhanced anterior cingulate activity and suppression of alpha oscillations and the 'default-mode' network.

Successful navigation can require realizing the current path choice was a mistake and the best strategy is to retreat along the recent path: 'back-track'. Despite the wealth of studies on the neural correlates of navigation little is known about backtracking. To explore the neural underpinnings of backtracking we tested humans during functional magnetic resonance imaging on their ability to navigate to a set of goal locations in a virtual desert island riven by lava which constrained the paths that could be taken.

Sensory cue combination in children under 10 years of age.

Cue combination occurs when two independent noisy perceptual estimates are merged together as a weighted average, creating a unified estimate that is more precise than either single estimate alone. Surprisingly, this effect has not been demonstrated compellingly in children under the age of 10 years, in contrast with the array of other multisensory skills that children show even in infancy. Instead, across a wide variety of studies, precision with both cues is no better than the best single cue - and sometimes worse.

Efficient visual information sampling develops late in childhood.

It is often unclear which course of action gives the best outcome. We can reduce this uncertainty by gathering more information, but gathering information always comes at a cost. For example, a sports player waiting too long to judge a ball's trajectory will run out of time to intercept it.

Prefrontal Dynamics Associated with Efficient Detours and Shortcuts: A Combined Functional Magnetic Resonance Imaging and Magnetoencenphalography Study.

Central to the concept of the "cognitive map" is that it confers behavioral flexibility, allowing animals to take efficient detours, exploit shortcuts, and avoid alluring, but unhelpful, paths. The neural underpinnings of such naturalistic and flexible behavior remain unclear. In two neuroimaging experiments, we tested human participants on their ability to navigate to a set of goal locations in a virtual desert island riven by lava, which occasionally spread to block selected paths (necessitating detours) or receded to open new paths (affording real shortcuts or false shortcuts to be avoided).

Observer models of perceptual development.

We agree with Rahnev & Denison (R&D) that to understand perception at a process level, we must investigate why performance sometimes deviates from idealised decision models. Recent research reveals that such deviations from optimality are pervasive during perceptual development. We argue that a full understanding of perception requires a model of how perceptual systems become increasingly optimised during development.

Bayes-Like Integration of a New Sensory Skill with Vision.

Humans are effective at dealing with noisy, probabilistic information in familiar settings. One hallmark of this is Bayesian Cue Combination: combining multiple noisy estimates to increase precision beyond the best single estimate, taking into account their reliabilities. Here we show that adults also combine a novel audio cue to distance, akin to human echolocation, with a visual cue.

Effects of two-dimensional versus three-dimensional landmark geometry and layout on young children's recall of locations from new viewpoints.

Spatial memory is an important aspect of adaptive behavior and experience, providing both content and context to the perceptions and memories that we form in everyday life. Young children's abilities in this realm shift from mainly egocentric (self-based) to include allocentric (world-based) codings at around 4 years of age. However, information about the cognitive mechanisms underlying acquisition of these new abilities is still lacking.