Suppression of the auditory N1 by visual anticipatory motion is modulated by temporal and identity predictability.

The amplitude of the auditory N1 component of the event-related potential (ERP) is typically suppressed when a sound is accompanied by visual anticipatory information that reliably predicts the timing and identity of the sound. While this visually induced suppression of the auditory N1 is considered an early electrophysiological marker of fulfilled prediction, it is not yet fully understood whether this internal predictive coding mechanism is primarily driven by the temporal characteristics, or by the identity features of the anticipated sound. The current study examined the impact of temporal and identity predictability on suppression of the auditory N1 by visual anticipatory motion with an ecologically valid audiovisual event (a video of a handclap).

Comment on "Differential Effects of the Temporal and Spatial Distribution of Audiovisual Stimuli on Cross-Modal Spatial Recalibration".

Bruns et al. (2020) provide new research that suggests that the ventriloquism after-effect (VAE: an enduring shift of the perceived location of a sound toward a previously seen visual stimulus) and multisensory enhancement (ME: an improvement in the precision of sound localization) may dissociate depending on the rate at which exposure stimuli are presented. They reported that the VAE, but not the ME, was diminished when exposure stimuli were presented at 10 Hz rather than at 2 Hz.

Atypical visual-auditory predictive coding in autism spectrum disorder: Electrophysiological evidence from stimulus omissions.

Many autistic individuals experience difficulties in processing sensory information (e.g. increased sensitivity to sound).

Are alpha oscillations instrumental in multisensory synchrony perception?

Different inputs from a multisensory object or event are often integrated into a coherent and unitary percept, despite differences in sensory formats, neural pathways, and processing times of the involved modalities. Presumably, multisensory integration occurs if the cross-modal inputs are presented within a certain window of temporal integration where inputs are perceived as being simultaneous. Here, we examine the role of ongoing neuronal alpha (i.

Speech-specific audiovisual integration modulates induced theta-band oscillations.

Speech perception is influenced by vision through a process of audiovisual integration. This is demonstrated by the McGurk illusion where visual speech (for example /ga/) dubbed with incongruent auditory speech (such as /ba/) leads to a modified auditory percept (/da/). Recent studies have indicated that perception of the incongruent speech stimuli used in McGurk paradigms involves mechanisms of both general and audiovisual speech specific mismatch processing and that general mismatch processing modulates induced theta-band (4-8 Hz) oscillations.

Increased sub-clinical levels of autistic traits are associated with reduced multisensory integration of audiovisual speech.

Recent studies suggest that sub-clinical levels of autistic symptoms may be related to reduced processing of artificial audiovisual stimuli. It is unclear whether these findings extent to more natural stimuli such as audiovisual speech. The current study examined the relationship between autistic traits measured by the Autism spectrum Quotient and audiovisual speech processing in a large non-clinical population using a battery of experimental tasks assessing audiovisual perceptual binding, visual enhancement of speech embedded in noise and audiovisual temporal processing.

Electrophysiological alterations in motor-auditory predictive coding in autism spectrum disorder.

The amplitude of the auditory N1 component of the event-related potential (ERP) is typically attenuated for self-initiated sounds, compared to sounds with identical acoustic and temporal features that are triggered externally. This effect has been ascribed to internal forward models predicting the sensory consequences of one's own motor actions. The predictive coding account of autistic symptomatology states that individuals with autism spectrum disorder (ASD) have difficulties anticipating upcoming sensory stimulation due to a decreased ability to infer the probabilistic structure of their environment.

Multisensory integration of speech sounds with letters vs. visual speech: only visual speech induces the mismatch negativity.

Numerous studies have demonstrated that the vision of lip movements can alter the perception of auditory speech syllables (McGurk effect). While there is ample evidence for integration of text and auditory speech, there are only a few studies on the orthographic equivalent of the McGurk effect. Here, we examined whether written text, like visual speech, can induce an illusory change in the perception of speech sounds on both the behavioural and neural levels.

Temporal and identity prediction in visual-auditory events: Electrophysiological evidence from stimulus omissions.

A rare omission of a sound that is predictable by anticipatory visual information induces an early negative omission response (oN1) in the EEG during the period of silence where the sound was expected. It was previously suggested that the oN1 was primarily driven by the identity of the anticipated sound. Here, we examined the role of temporal prediction in conjunction with identity prediction of the anticipated sound in the evocation of the auditory oN1.

From face to hand: Attentional bias towards expressive hands in social anxiety.

The eye-region conveys important emotional information that we spontaneously attend to. Socially submissive individuals avoid other's gaze which is regarded as avoidance of others' emotional face expressions. But this interpretation ignores the fact that there are other sources of emotional information besides the face.

The effect of synesthetic associations between the visual and auditory modalities on the Colavita effect.

The Colavita effect refers to the phenomenon that when confronted with an audiovisual stimulus, observers report more often to have perceived the visual than the auditory component. The Colavita effect depends on low-level stimulus factors such as spatial and temporal proximity between the unimodal signals. Here, we examined whether the Colavita effect is modulated by synesthetic congruency between visual size and auditory pitch.

Predictive coding of visual-auditory and motor-auditory events: An electrophysiological study.

The amplitude of auditory components of the event-related potential (ERP) is attenuated when sounds are self-generated compared to externally generated sounds. This effect has been ascribed to internal forward modals predicting the sensory consequences of one's own motor actions. Auditory potentials are also attenuated when a sound is accompanied by a video of anticipatory visual motion that reliably predicts the sound.

Motor-induced visual motion: hand movements driving visual motion perception.

Visual perception can be changed by co-occurring input from other sensory modalities. Here, we explored how self-generated finger movements (left-right or up-down key presses) affect visual motion perception. In Experiment 1, motion perception of a blinking bar was shifted in the direction of co-occurring hand motor movements, indicative of motor-induced visual motion (MIVM).

A bias-free two-alternative forced choice procedure to examine intersensory illusions applied to the ventriloquist effect by flashes and averted eye-gazes.

We compared with a new psychophysical method whether flashes and averted eye-gazes of a cartoon face induce a ventriloquist illusion (an illusory shift of the apparent location of a sound by a visual distracter). With standard psychophysical procedures that measure a direct ventriloquist effect and a ventriloquist aftereffect, we found in human subjects that both types of stimuli induced an illusory shift of sound location. These traditional methods, though, are probably contaminated by response strategies.

Emotional signals from faces, bodies and scenes influence observers' face expressions, fixations and pupil-size.

We receive emotional signals from different sources, including the face, the whole body, and the natural scene. Previous research has shown the importance of context provided by the whole body and the scene on the recognition of facial expressions. This study measured physiological responses to face-body-scene combinations.

Electrophysiological evidence for speech-specific audiovisual integration.

Lip-read speech is integrated with heard speech at various neural levels. Here, we investigated the extent to which lip-read induced modulations of the auditory N1 and P2 (measured with EEG) are indicative of speech-specific audiovisual integration, and we explored to what extent the ERPs were modulated by phonetic audiovisual congruency. In order to disentangle speech-specific (phonetic) integration from non-speech integration, we used Sine-Wave Speech (SWS) that was perceived as speech by half of the participants (they were in speech-mode), while the other half was in non-speech mode.

Sound improves diminished visual temporal sensitivity in schizophrenia.

Visual temporal processing and multisensory integration (MSI) of sound and vision were examined in individuals with schizophrenia using a visual temporal order judgment (TOJ) task. Compared to a non-psychiatric control group, persons with schizophrenia were less sensitive judging the temporal order of two successively presented visual stimuli. However, their sensitivity to visual temporal order improved as in the control group when two accessory sounds were added (temporal ventriloquism).

Deficient multisensory integration in schizophrenia: an event-related potential study.

Background: In many natural audiovisual events (e.g., the sight of a face articulating the syllable /ba/), the visual signal precedes the sound and thus allows observers to predict the onset and the content of the sound.

Perception of face and body expressions using electromyography, pupillometry and gaze measures.

Traditional emotion theories stress the importance of the face in the expression of emotions but bodily expressions are becoming increasingly important as well. In these experiments we tested the hypothesis that similar physiological responses can be evoked by observing emotional face and body signals and that the reaction to angry signals is amplified in anxious individuals. We designed three experiments in which participants categorized emotional expressions from isolated facial and bodily expressions and emotionally congruent and incongruent face-body compounds.

Electrophysiological correlates of predictive coding of auditory location in the perception of natural audiovisual events.

In many natural audiovisual events (e.g., a clap of the two hands), the visual signal precedes the sound and thus allows observers to predict when, where, and which sound will occur.

Electrophysiological evidence for a multisensory speech-specific mode of perception.

We investigated whether the interpretation of auditory stimuli as speech or non-speech affects audiovisual (AV) speech integration at the neural level. Perceptually ambiguous sine-wave replicas (SWS) of natural speech were presented to listeners who were either in 'speech mode' or 'non-speech mode'. At the behavioral level, incongruent lipread information led to an illusory change of the sound only for listeners in speech mode.

Neural correlates of motor-sensory temporal recalibration.

The relative timing of a motor-sensory event can be recalibrated after exposure to delayed visual feedback. Here we examined the neural consequences of lag adaptation using event-related potentials (ERPs). Participants tapped their finger on a pad, which triggered a flash either after a short delay (0 ms/50 ms) or a long delay (100 ms/150 ms).

Perception of intersensory synchrony in audiovisual speech: not that special.

Perception of intersensory temporal order is particularly difficult for (continuous) audiovisual speech, as perceivers may find it difficult to notice substantial timing differences between speech sounds and lip movements. Here we tested whether this occurs because audiovisual speech is strongly paired ("unity assumption"). Participants made temporal order judgments (TOJ) and simultaneity judgments (SJ) about sine-wave speech (SWS) replicas of pseudowords and the corresponding video of the face.

Visual anticipatory information modulates multisensory interactions of artificial audiovisual stimuli.

The neural activity of speech sound processing (the N1 component of the auditory ERP) can be suppressed if a speech sound is accompanied by concordant lip movements. Here we demonstrate that this audiovisual interaction is neither speech specific nor linked to humanlike actions but can be observed with artificial stimuli if their timing is made predictable. In Experiment 1, a pure tone synchronized with a deformation of a rectangle induced a smaller auditory N1 than auditory-only presentations if the temporal occurrence of this audiovisual event was made predictable by two moving disks that touched the rectangle.

Neural correlates of audiovisual motion capture.

Visual motion can affect the perceived direction of auditory motion (i.e., audiovisual motion capture).