Are We Moving Too Fast?: Representation of Speed in Static Images.

Despite pictures being static representations, they use various cues to suggest dynamic motion. To investigate the effectiveness of different motion cues in conveying speed in static images, we conducted 3 experiments. In Experiment 1, we compared subjective speed ratings given for motion lines trailing behind movers, suppletion lines replacing parts of the movers and backfixing lines set in the background against the baseline of having no extra cue.

Processing Incongruity for Mental Events in Comics: Contours of Character Inferences.

Visual narratives, like comics, at times show depictions of characters' imagination, dreams, or flashbacks, which seem incongruent with the ongoing primary narrative. Such "domain constructions" thus integrate an auxiliary domain (e.g.

The electrophysiology of lexical prediction of emoji and text.

As emoji often appear naturally alongside text in utterances, they provide a way to study how prediction unfolds in multimodal sentences in direct comparison to unimodal sentences. In this experiment, participants (N = 40) read sentences in which the sentence-final noun appeared in either word form or emoji form, a between-subjects manipulation. The experiment featured both high constraint sentences and low constraint sentences to examine how the lexical processing of emoji interacts with prediction processes in sentence comprehension.

Anaphoric distance dependencies in visual narrative structure and processing.

Linguistic syntax has often been claimed as uniquely complex due to features like anaphoric relations and distance dependencies. However, visual narratives of sequential images, like those in comics, have been argued to use sequencing mechanisms analogous to those in language. These narrative structures include "refiner" panels that "zoom in" on the contents of another panel.

Semantic processing of verbal narratives compared to semantic processing of visual narratives: an ERP study of school-aged children.

There is a misconception that pictures are easy to comprehend, which is problematic in pedagogical practices that include pictures. For example, if a child has difficulties with verbal narration to picture sequences, it may be interpreted as specific to spoken language even though the child may have additional difficulties with comprehension of visual narratives in the form of picture sequences. The purpose of the present study was therefore to increase our understanding of semantic processing in the pictorial domain in relation to semantic processing in the verbal domain, focusing on 9-13 years-old children with typical language development.

Background check: cross-cultural differences in the spatial context of comic scenes.

Cognitive research points towards cultural differences in the way people perceive and express scenes. Whereas people from Western cultures focus more on focal objects, those from East Asia have been shown to focus on the surrounding context. This paper examines whether these cultural differences are expressed in complex multimodal media such as comics.

The Meaning of Motion Lines?: A Review of Theoretical and Empirical Research on Static Depiction of Motion.

Static depiction of motion, particularly lines trailing behind a mover, has long been of interest in the psychology literature. Empirical research has demonstrated that these "motion lines" benefited motion comprehension in static images by disambiguating the direction of movement. Yet, there is no consensus on how those lines derive their meaning.

Visual languages and the problems with ideographies: A commentary on Morin.

Morin argues that ideographies are limited because graphic codes lack a capacity for proliferating standardization. However, natural graphic systems display rich standardization and can be placed in sequences using complex combinatorial structures. In contrast, ideographies are not natural, and their limitations lie in their attempts to artificially force a graphic system to behave like a writing system.

Individual differences in the neural dynamics of visual narrative comprehension: The effects of proficiency and age of acquisition.

Understanding visual narrative sequences, as found in comics, is known to recruit similar cognitive mechanisms to verbal language. As measured by event-related potentials (ERPs), these manifest as initial negativities (N400, LAN) and subsequent positivities (P600). While these components are thought to index discrete processing stages, they differentially arise across participants for any given stimulus.

Sequences in harmony: Cognitive interactions between musical and visual narrative structure.

From film and television to graphic storytelling, tonal music can accompany visual narratives in a variety of contexts. The apprehension of both musical and narrative sequences involves temporal categories in ordered patterning, which raises an interesting question: Do musical progressions and visual narratives rely on shared sequence processing mechanisms? If this is the case, then cues from music and sequential static images, when presented simultaneously, should interact during audiovisual online processing. We tested this question by measuring reaction times to target picture panels appearing in visual narrative (comic strip) sequences, which were presented panel by panel and synchronized with musical chord progressions.

The framing of subjectivity: Point-of-view in a cross-cultural analysis of comics.

In visual narratives like comics, the most overt form of perspective-taking comes in panels that directly depict the viewpoints of characters in the scene. We therefore examined these subjective viewpoint panels (also known as point-of-view panels) in a corpus of over 300 annotated comics from Asia, Europe, and the United States. In line with predictions that Japanese manga use a more 'subjective' storytelling style than other comics, we found that more manga use subjective panels than other comics, with high proportions of subjective panels also found in Chinese, French, and American comics.

The Lexicon of Emoji? Conventionality Modulates Processing of Emoji.

Emoji have been ubiquitous in communication for over a decade, yet how they derive meaning remains underexplored. Here, we examine an aspect fundamental to linguistic meaning-making: the degree to which emoji have conventional lexicalized meanings and whether that conventionalization affects processing in real-time. Experiment 1 establishes a range of meaning agreement levels across emoji within a population; Experiment 2 measures accuracy and response times to word-emoji pairings in a match/mismatch task.

Linguistic typology of motion events in visual narratives.

Languages use different strategies to encode motion. Some use particles or "satellites" to describe a path of motion (Satellite-framed or S-languages like English), while others typically use the main verb to convey the path information (Verb-framed or V-languages like French). We here ask: might this linguistic variation lead to differences in the way paths are depicted in visual narratives like comics? We analyzed a corpus of 85 comics originally created by speakers of S-languages (comics from the United States, China, Germany) and V-languages (France, Japan, Korea) for both their depictions of path segments (source, route, and goal) and the visual cues signaling these paths and manner information (e.

Reimagining Language.

Since its inception, the study of language has been a central pillar to Cognitive Science. Despite an "amodal view," where language is thought to "flow into" modalities indiscriminately, speech has always been considered the prototypical form of the linguistic system. However, this view does not hold up to the evidence about language and expressive modalities.

An electrophysiological investigation of co-referential processes in visual narrative comprehension.

Visual narratives make use of various means to convey referential and co-referential meaning, so comprehenders must recognize that different depictions across sequential images represent the same character(s). In this study, we investigated how the order in which different types of panels in visual sequences are presented affects how the unfolding narrative is comprehended. Participants viewed short comic strips while their electroencephalogram (EEG) was recorded.

Meaning above (and in) the head: Combinatorial visual morphology from comics and emoji.

Compositionality is a primary feature of language, but graphics can also create combinatorial meaning, like with items above faces (e.g., lightbulbs to mean inspiration).

Remarks on Multimodality: Grammatical Interactions in the Parallel Architecture.

Language is typically embedded in multimodal communication, yet models of linguistic competence do not often incorporate this complexity. Meanwhile, speech, gesture, and/or pictures are each considered as indivisible components of multimodal messages. Here, we argue that multimodality should not be characterized by whole interacting behaviors, but by interactions of similar substructures which permeate across expressive behaviors.

The electrophysiology of audiovisual processing in visual narratives in adolescents with autism spectrum disorder.

We investigated the semantic processing of the multimodal audiovisual combination of visual narratives with auditory descriptive words and auditory sounds in individuals with ASD. To this aim, we recorded ERPs to critical auditory words and sounds associated with events in visual narrative that were either semantically congruent or incongruent with the climactic visual event. A similar N400 effect was found both in adolescents with ASD and neurotypical adolescents (ages 9-16) when accessing different types of auditory information (i.

A starring role for inference in the neurocognition of visual narratives.

Research in verbal and visual narratives has often emphasized backward-looking inferences, where absent information is subsequently inferred. However, comics use conventions like star-shaped "action stars" where a reader knows events are undepicted at that moment, rather than omitted entirely. We contrasted the event-related brain potentials (ERPs) to visual narratives depicting an explicit event, an action star, or a "noise" panel of scrambled lines.

Zooming in on the cognitive neuroscience of visual narrative.

Visual narratives like comics and films often shift between showing full scenes and close, zoomed-in viewpoints. These zooms are similar to the "spotlight of attention" cast across a visual scene in perception. We here measured ERPs to visual narratives (comic strips) that used zoomed-in and full-scene panels either throughout the whole sequence context or at specific critical panels.

Zooming in on visual narrative comprehension.

The comprehension of visual narratives requires paying attention to certain elements and integrating them across a sequence of images. To study this process, we developed a new approach that modified comic strips according to where observers looked while viewing each sequence. Across three self-paced experiments, we presented sequences of six panels that were sometimes automatically "zoomed-in" or re-framed in order to highlight parts of the image that had been fixated by another group of observers.

Predictability modulates neurocognitive semantic processing of non-verbal narratives.

Predictability is known to modulate semantic processing in language, but it is unclear to what extent this applies for other modalities. Here we ask whether similar cognitive processes are at play in predicting upcoming events in a non-verbal visual narrative. Typically developing adults viewed comics sequences in which a target panel was highly predictable ("high cloze"), less predictable ("low cloze"), or incongruent with the preceding narrative context ("anomalous") during EEG recording.

Visual and Verbal Narrative Comprehension in Children and Adolescents with Autism Spectrum Disorders: An ERP Study.

We examined semantic processing in ASD children by presenting sentences with congruent or incongruent final words and visual narratives with congruent or incongruent final panels. An N400 effect to incongruent words appeared as compared to congruent ones, which was attenuated for the ASD children. We observed a negativity sustained to incongruous than congruous words, but only for the TD children.

Editors' Introduction and Review: Visual Narrative Research: An Emerging Field in Cognitive Science.

Drawn sequences of images are among our oldest records of human intelligence, appearing on cave paintings, wall carvings, and ancient pottery, and they pervade across cultures from instruction manuals to comics. They also appear prevalently as stimuli across Cognitive Science, for studies of temporal cognition, event structure, social cognition, discourse, and basic intelligence. Yet, despite this fundamental place in human expression and research on cognition, the study of visual narratives themselves has only recently gained traction in Cognitive Science.

Visual narrative comprehension: Universal or not?

Visual narratives of sequential images - as found in comics, picture stories, and storyboards - are often thought to provide a fairly universal and transparent message that requires minimal learning to decode. This perceived transparency has led to frequent use of sequential images as experimental stimuli in the cognitive and psychological sciences to explore a wide range of topics. In addition, it underlines efforts to use visual narratives in science and health communication and as educational materials in both classroom settings and across developmental, clinical, and non-literate populations.