The Benefit of Bimodal Training in Voice Learning.

It is known that talkers can be recognized by listening to their specific vocal qualities-breathiness and fundamental frequencies. However, talker identification can also occur by focusing on the talkers' unique articulatory style, which is known to be available auditorily and visually and can be shared across modalities. Evidence shows that voices heard while seeing talkers' faces are later recognized better on their own compared to the voices heard alone.

Semantic priming from McGurk words: Priming depends on perception.

The McGurk effect is an illusion in which visible articulations alter the perception of auditory speech (e.g., video 'da' dubbed with audio 'ba' may be heard as 'da').

Selective adaptation in speech: Measuring the effects of visual and lexical contexts.

Speech selective adaptation is a phenomenon in which repeated presentation of a speech stimulus alters subsequent phonetic categorization. Prior work has reported that lexical, but not multisensory, context influences selective adaptation. This dissociation suggests that lexical and multisensory contexts influence speech perception through separate and independent processes (see Samuel & Lieblich, 2014).

Cross-modal transfer of talker-identity learning.

A speech signal carries information about meaning and about the talker conveying that meaning. It is now known that these two dimensions are related. There is evidence that gaining experience with a particular talker in one modality not only facilitates better phonetic perception in that modality, but also transfers across modalities to allow better phonetic perception in the other.

The role of speech fidelity in the irrelevant sound effect: Insights from noise-vocoded speech backgrounds.

The Irrelevant Sound Effect (ISE) is the finding that background sound impairs accuracy for visually presented serial recall tasks. Among various auditory backgrounds, speech typically acts as the strongest distractor. Based on the changing-state hypothesis, speech is a disruptive background because it is more complex than other nonspeech backgrounds.

Neural Mechanisms Underlying Cross-Modal Phonetic Encoding.

Audiovisual (AV) integration is essential for speech comprehension, especially in adverse listening situations. Divergent, but not mutually exclusive, theories have been proposed to explain the neural mechanisms underlying AV integration. One theory advocates that this process occurs via interactions between the auditory and visual cortices, as opposed to fusion of AV percepts in a multisensory integrator.

Influences of selective adaptation on perception of audiovisual speech.

Research suggests that selective adaptation in speech is a low-level process dependent on sensory-specific information shared between the adaptor and test-stimuli. However, previous research has only examined how adaptors shift perception of unimodal test stimuli, either auditory or visual. In the current series of experiments, we investigated whether adaptation to cross-sensory phonetic information can influence perception of integrated audio-visual phonetic information.

Visibility of speech articulation enhances auditory phonetic convergence.

Talkers automatically imitate aspects of perceived speech, a phenomenon known as phonetic convergence. Talkers have previously been found to converge to auditory and visual speech information. Furthermore, talkers converge more to the speech of a conversational partner who is seen and heard, relative to one who is just heard (Dias & Rosenblum Perception, 40, 1457-1466, 2011).

Experience with a talker can transfer across modalities to facilitate lipreading.

Rosenblum, Miller, and Sanchez (Psychological Science, 18, 392-396, 2007) found that subjects first trained to lip-read a particular talker were then better able to perceive the auditory speech of that same talker, as compared with that of a novel talker. This suggests that the talker experience a perceiver gains in one sensory modality can be transferred to another modality to make that speech easier to perceive. An experiment was conducted to examine whether this cross-sensory transfer of talker experience could occur (1) from auditory to lip-read speech, (2) with subjects not screened for adequate lipreading skill, (3) when both a familiar and an unfamiliar talker are presented during lipreading, and (4) for both old (presentation set) and new words.

Is speech alignment to talkers or tasks?

Speech alignment, or the tendency of individuals to subtly imitate each other's speaking styles, is often assessed by comparing a subject's baseline and shadowed utterances to a model's utterances, often through perceptual ratings. These types of comparisons provide information about the occurrence of a change in subject's speech, but they do not indicate that this change is toward the specific shadowed model. In three experiments, we investigated whether alignment is specific to a shadowed model.

Visual influences on interactive speech alignment.

Speech alignment describes the unconscious tendency to produce speech that shares characteristics with perceived speech (eg Goldinger, 1998 Psychological Review 105 251-279). In the present study we evaluated whether seeing a talker enhances alignment over just hearing a talker. Pairs of participants performed an interactive search task which required them to repeatedly utter a series of keywords.

Alignment to visual speech information.

Speech alignment is the tendency for interlocutors to unconsciously imitate one another's speaking style. Alignment also occurs when a talker is asked to shadow recorded words (e.g.

Visual influences on alignment to voice onset time.

Purpose: Speech shadowing experiments were conducted to test whether alignment (inadvertent imitation) to voice onset time (VOT) can be influenced by visual speech information.

Method: Experiment 1 examined whether alignment would occur to auditory /pa/ syllables manipulated to have 3 different VOTs. Nineteen female participants were asked to listen to 180 syllables over headphones and to say each syllable out loud quickly and clearly.

Speech Perception as a Multimodal Phenomenon.

Speech perception is inherently multimodal. Visual speech (lip-reading) information is used by all perceivers and readily integrates with auditory speech. Imaging research suggests that the brain treats auditory and visual speech similarly.

Lip-read me now, hear me better later: cross-modal transfer of talker-familiarity effects.

There is evidence that for both auditory and visual speech perception, familiarity with the talker facilitates speech recognition. Explanations of these effects have concentrated on the retention of talker information specific to each of these modalities. It could be, however, that some amodal, talker-specific articulatory-style information facilitates speech perception in both modalities.

Look who's talking: recognizing friends from visible articulation.

We tested whether isolated visible articulatory information can be used for identifying familiar speakers. A facial point-light methodology was used to isolate the visible articulation of seven speakers. These point-light video clips were then shown to nine participants who had long-term personal interactions with the speakers.

Hearing a face: cross-modal speaker matching using isolated visible speech.

An experiment was performed to test whether cross-modal speaker matches could be made using isolated visible speech movement information. Visible speech movements were isolated using a point-light technique. In five conditions, subjects were asked to match a voice to one of two (unimodal) speaking point-light faces on the basis of speaker identity.

Effects of intrastimulus modality change on audiovisual time-to-arrival judgments.

In this research, anticipatory perception of an approaching vehicle was examined. By varying the availability of visual and acoustic media, conditions occurred in which the input to the modalities was repetitively interrupted while the presentation of the approach event continued. In these conditions, the audio and the visual signals were presented either in short and concurrent segments at regular intervals or in alternation (interleaved) at regular intervals.

Visual speech information for face recognition.

Two experiments test whether isolated visible speech movements can be used for face matching. Visible speech information was isolated with a point-light methodology. Participants were asked to match articulating point-light faces to a fully illuminated articulating face in an XAB task.