Using network science to examine audio-visual speech perception with a multi-layer graph.

To examine visual speech perception (i.e., lip-reading), we created a multi-layer network (the AV-net) that contained: (1) an auditory layer with nodes representing phonological word-forms and edges connecting words that were phonologically related, and (2) a visual layer with nodes representing the viseme representations of words and edges connecting viseme representations that differed by a single viseme (and additional edges to connect related nodes in the two layers).

Interpreting tone of voice: musical pitch relationships convey agreement in dyadic conversation.

Previous research has found that the musical intervals found in speech are associated with various emotions. Intervals can be classified by their level of consonance or dissonance-how pleasant or unpleasant the combined tones sound to the ear. Exploratory investigations have indicated that in an agreeable conversation, the pitches of the last word in an utterance and the first word of a conversation partner's utterance are consonantly related; in a disagreeable conversation, the two pitches are dissonantly related.

Specification of cross-modal source information in isolated kinematic displays of speech.

Information about the acoustic properties of a talker's voice is available in optical displays of speech, and vice versa, as evidenced by perceivers' ability to match faces and voices based on vocal identity. The present investigation used point-light displays (PLDs) of visual speech and sinewave replicas of auditory speech in a cross-modal matching task to assess perceivers' ability to match faces and voices under conditions when only isolated kinematic information about vocal tract articulation was available. These stimuli were also used in a word recognition experiment under auditory-alone and audiovisual conditions.

Cross-modal source information and spoken word recognition.

In a cross-modal matching task, participants were asked to match visual and auditory displays of speech based on the identity of the speaker. The present investigation used this task with acoustically transformed speech to examine the properties of sound that can convey cross-modal information. Word recognition performance was also measured under the same transformations.

Crossmodal Source Identification in Speech Perception.

Four experiments examined the nature of multisensory speech information. In Experiment 1, participants were asked to match heard voices with dynamic visual-alone video clips of speakers' articulating faces. This cross-modal matching task was used to examine whether vocal source matching can be accomplished across sensory modalities.

Talker and lexical effects on audiovisual word recognition by adults with cochlear implants.

The present study examined how postlingually deafened adults with cochlear implants combine visual information from lipreading with auditory cues in an open-set word recognition task. Adults with normal hearing served as a comparison group. Word recognition performance was assessed using lexically controlled word lists presented under auditory-only, visual-only, and combined audiovisual presentation formats.

AUDIOVISUAL INTEGRATION OF SPEECH BY CHILDREN AND ADULTS WITH COCHEAR IMPLANTS.

The present study examined how prelingually deafened children and postlingually deafened adults with cochlear implants (CIs) combine visual speech information with auditory cues. Performance was assessed under auditory-alone (A), visual- alone (V), and combined audiovisual (AV) presentation formats. A measure of visual enhancement, R, was used to assess the gain in performance provided in the AV condition relative to the maximum possible performance in the auditory-alone format.

Use of Partial Stimulus Information by Cochlear Implant Users and Listeners with Normal Hearing in Identifying Spoken Words: Some Preliminary Analyses.

An error analysis of the word recognition responses of cochlear implant users and listeners with normal hearing was conducted to determine the types of partial information used by these two populations when they identified spoken words under auditory-alone and audiovisual conditions. The results revealed that the two groups used different types of partial information in identifying spoken words under auditory-alone or audiovisual presentation. Different types of partial information were also used in identifying words with different lexical properties.