Repeatedly experiencing the McGurk effect induces long-lasting changes in auditory speech perception.

In the McGurk effect, presentation of incongruent auditory and visual speech evokes a fusion percept different than either component modality. We show that repeatedly experiencing the McGurk effect for 14 days induces a change in auditory-only speech perception: the auditory component of the McGurk stimulus begins to evoke the fusion percept, even when presented on its own without accompanying visual speech. This perceptual change, termed fusion-induced recalibration (FIR), was talker-specific and syllable-specific and persisted for a year or more in some participants without any additional McGurk exposure.

The noisy encoding of disparity model predicts perception of the McGurk effect in native Japanese speakers.

In the McGurk effect, visual speech from the face of the talker alters the perception of auditory speech. The diversity of human languages has prompted many intercultural studies of the effect in both Western and non-Western cultures, including native Japanese speakers. Studies of large samples of native English speakers have shown that the McGurk effect is characterized by high variability in the susceptibility of different individuals to the illusion and in the strength of different experimental stimuli to induce the illusion.

Synthetic faces generated with the facial action coding system or deep neural networks improve speech-in-noise perception, but not as much as real faces.

The prevalence of synthetic talking faces in both commercial and academic environments is increasing as the technology to generate them grows more powerful and available. While it has long been known that seeing the face of the talker improves human perception of speech-in-noise, recent studies have shown that synthetic talking faces generated by deep neural networks (DNNs) are also able to improve human perception of speech-in-noise. However, in previous studies the benefit provided by DNN synthetic faces was only about half that of real human talkers.

The Effect on Speech-in-Noise Perception of Real Faces and Synthetic Faces Generated with either Deep Neural Networks or the Facial Action Coding System.

The prevalence of synthetic talking faces in both commercial and academic environments is increasing as the technology to generate them grows more powerful and available. While it has long been known that seeing the face of the talker improves human perception of speech-in-noise, recent studies have shown that synthetic talking faces generated by deep neural networks (DNNs) are also able to improve human perception of speech-in-noise. However, in previous studies the benefit provided by DNN synthetic faces was only about half that of real human talkers.

Benefits of sharing neurophysiology data from the BRAIN Initiative Research Opportunities in Humans Consortium.

Sharing human brain data can yield scientific benefits, but because of various disincentives, only a fraction of these data is currently shared. We profile three successful data-sharing experiences from the NIH BRAIN Initiative Research Opportunities in Humans (ROH) Consortium and demonstrate benefits to data producers and to users.

YAEL: Your Advanced Electrode Localizer.

Intracranial electroencephalography (iEEG) provides a unique opportunity to record and stimulate neuronal populations in the human brain. A key step in neuroscience inference from iEEG is localizing the electrodes relative to individual subject anatomy and identified regions in brain atlases. We describe a new software tool, Your Advanced Electrode Localizer (YAEL), that provides an integrated solution for every step of the electrode localization process.

Multivariate fMRI responses in superior temporal cortex predict visual contributions to, and individual differences in, the intelligibility of noisy speech.

Humans have the unique ability to decode the rapid stream of language elements that constitute speech, even when it is contaminated by noise. Two reliable observations about noisy speech perception are that seeing the face of the talker improves intelligibility and the existence of individual differences in the ability to perceive noisy speech. We introduce a multivariate BOLD fMRI measure that explains both observations.

A Comparison of Neuroelectrophysiology Databases.

As data sharing has become more prevalent, three pillars - archives, standards, and analysis tools - have emerged as critical components in facilitating effective data sharing and collaboration. This paper compares four freely available intracranial neuroelectrophysiology data repositories: Data Archive for the BRAIN Initiative (DABI), Distributed Archives for Neurophysiology Data Integration (DANDI), OpenNeuro, and Brain-CODE. The aim of this review is to describe archives that provide researchers with tools to store, share, and reanalyze both human and non-human neurophysiology data based on criteria that are of interest to the neuroscientific community.

Percepts evoked by multi-electrode stimulation of human visual cortex.

Background: Direct electrical stimulation of early visual cortex evokes the perception of small spots of light known as phosphenes. Previous studies have examined the location, size, and brightness of phosphenes evoked by stimulation of single electrodes. While it has been envisioned that concurrent stimulation of many electrodes could be used as the basis for a visual cortical prosthesis, the percepts resulting from multi-electrode stimulation have not been fully characterized.

Functional group bridge for simultaneous regression and support estimation.

This paper is motivated by studying differential brain activities to multiple experimental condition presentations in intracranial electroencephalography (iEEG) experiments. Contrasting effects of experimental conditions are often zero in most regions and nonzero in some local regions, yielding locally sparse functions. Such studies are essentially a function-on-scalar regression problem, with interest being focused not only on estimating nonparametric functions but also on recovering the function supports.

Ethical commitments, principles, and practices guiding intracranial neuroscientific research in humans.

Leveraging firsthand experience, BRAIN-funded investigators conducting intracranial human neuroscience research propose two fundamental ethical commitments: (1) maintaining the integrity of clinical care and (2) ensuring voluntariness. Principles, practices, and uncertainties related to these commitments are offered for future investigation.

Biased Orientation and Color Tuning of the Human Visual Gamma Rhythm.

Narrowband γ oscillations (NBG: ∼20-60 Hz) in visual cortex reflect rhythmic fluctuations in population activity generated by underlying circuits tuned for stimulus location, orientation, and color. A variety of theories posit a specific role for NBG in encoding and communicating this information within visual cortex. However, recent findings suggest a more nuanced role for NBG, given its dependence on certain stimulus feature configurations, such as coherent-oriented edges and specific hues.

Intelligibility of audiovisual sentences drives multivoxel response patterns in human superior temporal cortex.

Regions of the human posterior superior temporal gyrus and sulcus (pSTG/S) respond to the visual mouth movements that constitute visual speech and the auditory vocalizations that constitute auditory speech, and neural responses in pSTG/S may underlie the perceptual benefit of visual speech for the comprehension of noisy auditory speech. We examined this possibility through the lens of multivoxel pattern responses in pSTG/S. BOLD fMRI data was collected from 22 participants presented with speech consisting of English sentences presented in five different formats: visual-only; auditory with and without added auditory noise; and audiovisual with and without auditory noise.

Raising the stakes for cortical visual prostheses.

In this issue of the JCI, the dream of restoring useful vision to blind individuals with neurotechnology moves one step closer to realization. Fernández et al. implanted an electrode array with 96 penetrating electrodes in the visual cortex of a blind patient who had been without light perception for 16 years due to optic neuropathy.

Multi-electrode stimulation evokes consistent spatial patterns of phosphenes and improves phosphene mapping in blind subjects.

Background: Visual cortical prostheses (VCPs) have the potential to restore visual function to patients with acquired blindness. Successful implementation of VCPs requires the ability to reliably map the location of the phosphene produced by stimulation of each implanted electrode.

Objective: To evaluate the efficacy of different approaches to phosphene mapping and propose simple improvements to mapping strategy.

3D-printed microchip electrophoresis device containing spiral electrodes for integrated capacitively coupled contactless conductivity detection.

In this work, we demonstrate for the first time the design and fabrication of microchip electrophoresis devices containing cross-shaped channels and spiral electrodes around the separation channel for microchip electrophoresis and capacitively coupled contactless conductivity detection. The whole device was prepared in a digital light processing-based 3D printer in poly(ethylene glycol) diacrylate resin. Outstanding X-Y resolution of the customized 3D printer ensured the fabrication of 40-μm cross section channels.

Face and voice perception: Monkey see, monkey hear.

Primate brains contain specialized areas for perceiving social cues. New research shows that only some of these areas integrate visual faces with auditory voices.

Weak observer-level correlation and strong stimulus-level correlation between the McGurk effect and audiovisual speech-in-noise: A causal inference explanation.

The McGurk effect is a widely used measure of multisensory integration during speech perception. Two observations have raised questions about the validity of the effect as a tool for understanding speech perception. First, there is high variability in perception of the McGurk effect across different stimuli and observers.

Crossmodal Phase Reset and Evoked Responses Provide Complementary Mechanisms for the Influence of Visual Speech in Auditory Cortex.

Natural conversation is multisensory: when we can see the speaker's face, visual speech cues improve our comprehension. The neuronal mechanisms underlying this phenomenon remain unclear. The two main alternatives are visually mediated phase modulation of neuronal oscillations (excitability fluctuations) in auditory neurons and visual input-evoked responses in auditory neurons.

RAVE: Comprehensive open-source software for reproducible analysis and visualization of intracranial EEG data.

Direct recording of neural activity from the human brain using implanted electrodes (iEEG, intracranial electroencephalography) is a fast-growing technique in human neuroscience. While the ability to record from the human brain with high spatial and temporal resolution has advanced our understanding, it generates staggering amounts of data: a single patient can be implanted with hundreds of electrodes, each sampled thousands of times a second for hours or days. The difficulty of exploring these vast datasets is the rate-limiting step in discovery.

Responses to Visual Speech in Human Posterior Superior Temporal Gyrus Examined with iEEG Deconvolution.

Experimentalists studying multisensory integration compare neural responses to multisensory stimuli with responses to the component modalities presented in isolation. This procedure is problematic for multisensory speech perception since audiovisual speech and auditory-only speech are easily intelligible but visual-only speech is not. To overcome this confound, we developed intracranial encephalography (iEEG) deconvolution.

Dynamic Stimulation of Visual Cortex Produces Form Vision in Sighted and Blind Humans.

A visual cortical prosthesis (VCP) has long been proposed as a strategy for restoring useful vision to the blind, under the assumption that visual percepts of small spots of light produced with electrical stimulation of visual cortex (phosphenes) will combine into coherent percepts of visual forms, like pixels on a video screen. We tested an alternative strategy in which shapes were traced on the surface of visual cortex by stimulating electrodes in dynamic sequence. In both sighted and blind participants, dynamic stimulation enabled accurate recognition of letter shapes predicted by the brain's spatial map of the visual world.