Interpreting vision and language generative models with semantic visual priors.

When applied to Image-to-text models, explainability methods have two challenges. First, they often provide token-by-token explanations namely, they compute a visual explanation for each token of the generated sequence. This makes explanations expensive to compute and unable to comprehensively explain the model's output.

Rethinking symbolic and visual context in Referring Expression Generation.

Situational context is crucial for linguistic reference to visible objects, since the same description can refer unambiguously to an object in one context but be ambiguous or misleading in others. This also applies to Referring Expression Generation (), where the production of identifying descriptions is always dependent on a given context. Research in REG has long represented visual domains through information about objects and their properties, to determine identifying sets of target features during content determination.

Conceptualization in reference production: Probabilistic modeling and experimental testing.

In psycholinguistics, there has been relatively little work investigating conceptualization-how speakers decide which concepts to express. This contrasts with work in natural language generation (NLG), a subfield of artificial intelligence, where much research has explored content determination during the generation of referring expressions. Existing NLG algorithms for conceptualization during reference production do not fully explain previous psycholinguistic results, so we developed new models that we tested in three language production experiments.

Reference Production as Search: The Impact of Domain Size on the Production of Distinguishing Descriptions.

When producing a description of a target referent in a visual context, speakers need to choose a set of properties that distinguish it from its distractors. Computational models of language production/generation usually model this as a search process and predict that the time taken will increase both with the number of distractors in a scene and with the number of properties required to distinguish the target. These predictions are reminiscent of classic findings in visual search; however, unlike models of reference production, visual search models also predict that search can become very efficient under certain conditions, something that reference production models do not consider.

Automatic generation of natural language nursing shift summaries in neonatal intensive care: BT-Nurse.

Introduction: Our objective was to determine whether and how a computer system could automatically generate helpful natural language nursing shift summaries solely from an electronic patient record system, in a neonatal intensive care unit (NICU).

Methods: A system was developed which automatically generates partial NICU shift summaries (for the respiratory and cardiovascular systems), using data-to-text technology. It was evaluated for 2 months in the NICU at the Royal Infirmary of Edinburgh, under supervision.

Are all Semitic languages immune to letter transpositions? The case of Maltese.

Recent research using the rapid serial visual presentation (RSVP) paradigm with English sentences that included words with letter transpositions (e.g., jugde) has shown that participants can readily reproduce the correctly spelled sentences with little cost; in contrast, there is a dramatic reading cost with root-derived Hebrew words (Velan & Frost, Psychonomic Bulletin & Review 14:913-918, 2007, Cognition 118:141-156, 2011).

Toward a computational psycholinguistics of reference production.

This article introduces the topic ''Production of Referring Expressions: Bridging the Gap between Computational and Empirical Approaches to Reference'' of the journal Topics in Cognitive Science. We argue that computational and psycholinguistic approaches to reference production can benefit from closer interaction, and that this is likely to result in the construction of algorithms that differ markedly from the ones currently known in the computational literature. We focus particularly on determinism, the feature of existing algorithms that is perhaps most clearly at odds with psycholinguistic results, discussing how future algorithms might include non-determinism, and how new psycholinguistic experiments could inform the development of such algorithms.

Generation of referring expressions: assessing the Incremental Algorithm.

A substantial amount of recent work in natural language generation has focused on the generation of ''one-shot'' referring expressions whose only aim is to identify a target referent. Dale and Reiter's Incremental Algorithm (IA) is often thought to be the best algorithm for maximizing the similarity to referring expressions produced by people. We test this hypothesis by eliciting referring expressions from human subjects and computing the similarity between the expressions elicited and the ones generated by algorithms.

BT-Nurse: computer generation of natural language shift summaries from complex heterogeneous medical data.

The BT-Nurse system uses data-to-text technology to automatically generate a natural language nursing shift summary in a neonatal intensive care unit (NICU). The summary is solely based on data held in an electronic patient record system, no additional data-entry is required. BT-Nurse was tested for two months in the Royal Infirmary of Edinburgh NICU.

Summarising complex ICU data in natural language: demonstration of the BT-45 system.

As ICUs generate increasing amounts of information, writing medical reports involves complex time-consuming reasoning to build a coherent text which will be meaningful to those who will use it for decision making (e.g.: for nurse handover).

Summarising complex ICU data in natural language.

It has been shown that summarizing complex multi-channel physiological and discrete data in natural language (text) can lead to better decision-making in the intensive care unit (ICU). As part of the BabyTalk project, we describe a prototype system (BT-45) which can generate such textual summaries automatically. Although these summaries are not yet as good as those generated by human experts, we have demonstrated experimentally that they lead to as good decision-making as can be achieved through presenting the same data graphically.