METACOGNITIVE SCAFFOLDS IMPROVE SELF-JUDGMENTS OF ACCURACY IN A MEDICAL INTELLIGENT TUTORING SYSTEM.

In this study, we examined the effect of two metacognitive scaffolds on the accuracy of confidence judgments made while diagnosing dermatopathology slides in SlideTutor. Thirty-one ( = 31) first- to fourth-year pathology and dermatology residents were randomly assigned to one of the two scaffolding conditions. The cases used in this study were selected from the domain of Nodular and Diffuse Dermatitides.

Design of electronic medical record user interfaces: a matrix-based method for improving usability.

This study examines a new approach of using the Design Structure Matrix (DSM) modeling technique to improve the design of Electronic Medical Record (EMR) user interfaces. The usability of an EMR medication dosage calculator used for placing orders in an academic hospital setting was investigated. The proposed method captures and analyzes the interactions between user interface elements of the EMR system and groups elements based on information exchange, spatial adjacency, and similarity to improve screen density and time-on-task.

Comparative effectiveness research designs: an analysis of terms and coverage in Medical Subject Headings (MeSH) and Emtree.

Objectives: We analyzed the extent to which comparative effectiveness research (CER) organizations share terms for designs, analyzed coverage of CER designs in Medical Subject Headings (MeSH) and Emtree, and explored whether scientists use CER design terms.

Methods: We developed local terminologies (LTs) and a CER design terminology by extracting terms in documents from five organizations. We defined coverage as the distribution over match type in MeSH and Emtree.

Automated detection of heuristics and biases among pathologists in a computer-based system.

The purpose of this study is threefold: (1) to develop an automated, computer-based method to detect heuristics and biases as pathologists examine virtual slide cases, (2) to measure the frequency and distribution of heuristics and errors across three levels of training, and (3) to examine relationships of heuristics to biases, and biases to diagnostic errors. The authors conducted the study using a computer-based system to view and diagnose virtual slide cases. The software recorded participant responses throughout the diagnostic process, and automatically classified participant actions based on definitions of eight common heuristics and/or biases.

A system for coreference resolution for the clinical narrative.

Objective: To research computational methods for coreference resolution in the clinical narrative and build a system implementing the best methods.

Methods: The Ontology Development and Information Extraction corpus annotated for coreference relations consists of 7214 coreferential markables, forming 5992 pairs and 1304 chains. We trained classifiers with semantic, syntactic, and surface features pruned by feature selection.

Coreference resolution: a review of general methodologies and applications in the clinical domain.

Coreference resolution is the task of determining linguistic expressions that refer to the same real-world entity in natural language. Research on coreference resolution in the general English domain dates back to 1960s and 1970s. However, research on coreference resolution in the clinical free text has not seen major development.

The feasibility of automating audit and feedback for ART guideline adherence in Malawi.

Objective: To determine the feasibility of using electronic medical record (EMR) data to provide audit and feedback of antiretroviral therapy (ART) clinical guideline adherence to healthcare workers (HCWs) in Malawi.

Materials And Methods: We evaluated recommendations from Malawi's ART guidelines using GuideLine Implementability Appraisal criteria. Recommendations that passed selected criteria were converted into ratio-based performance measures.

Anaphoric relations in the clinical narrative: corpus creation.

Objective: The long-term goal of this work is the automated discovery of anaphoric relations from the clinical narrative. The creation of a gold standard set from a cross-institutional corpus of clinical notes and high-level characteristics of that gold standard are described.

Methods: A standard methodology for annotation guideline development, gold standard annotations, and inter-annotator agreement (IAA) was used.

Touchscreen task efficiency and learnability in an electronic medical record at the point-of-care.

The objective of this study was to determine the relative efficiency of novices compared to a prediction of skilled use when performing tasks using the touchscreen interface of an EMR developed in Malawi. We observed novice users performing touchscreen tasks and recorded timestamp data from their performances. Using a predictive human performance modeling tool, the authors predicted the skilled task performance time for each task.

Natural Language Processing methods and systems for biomedical ontology learning.

While the biomedical informatics community widely acknowledges the utility of domain ontologies, there remain many barriers to their effective use. One important requirement of domain ontologies is that they must achieve a high degree of coverage of the domain concepts and concept relationships. However, the development of these ontologies is typically a manual, time-consuming, and often error-prone process.

caTIES: a grid based system for coding and retrieval of surgical pathology reports and tissue specimens in support of translational research.

The authors report on the development of the Cancer Tissue Information Extraction System (caTIES)--an application that supports collaborative tissue banking and text mining by leveraging existing natural language processing methods and algorithms, grid communication and security frameworks, and query visualization methods. The system fills an important need for text-derived clinical data in translational research such as tissue-banking and clinical trials. The design of caTIES addresses three critical issues for informatics support of translational research: (1) federation of research data sources derived from clinical systems; (2) expressive graphical interfaces for concept-based text mining; and (3) regulatory and security model for supporting multi-center collaborative research.

Ranked Levels of Influence model: selecting influence techniques to minimize IT resistance.

Implementation of electronic health records (EHR), particularly computerized physician/provider order entry systems (CPOE), is often met with resistance. Influence presented at the right time, in the right manner, may minimize resistance or at least limit the risk of complete system failure. Combining established theories on power, influence tactics, and resistance, we developed the Ranked Levels of Influence model.

Effect of a limited-enforcement intelligent tutoring system in dermatopathology on student errors, goals and solution paths.

Objectives: Determine effects of a limited-enforcement intelligent tutoring system in dermatopathology on student errors, goals and solution paths. Determine if limited enforcement in a medical tutoring system inhibits students from learning the optimal and most efficient solution path. Describe the type of deviations from the optimal solution path that occur during tutoring, and how these deviations change over time.

Security and privacy requirements for a multi-institutional cancer research data grid: an interview-based study.

Background: Data protection is important for all information systems that deal with human-subjects data. Grid-based systems--such as the cancer Biomedical Informatics Grid (caBIG)--seek to develop new mechanisms to facilitate real-time federation of cancer-relevant data sources, including sources protected under a variety of regulatory laws, such as HIPAA and 21CFR11. These systems embody new models for data sharing, and hence pose new challenges to the regulatory community, and to those who would develop or adopt them.

Factors affecting feeling-of-knowing in a medical intelligent tutoring system: the role of immediate feedback as a metacognitive scaffold.

Previous studies in our laboratory have shown the benefits of immediate feedback on cognitive performance for pathology residents using an intelligent tutoring system (ITS) in pathology. In this study, we examined the effect of immediate feedback on metacognitive performance, and investigated whether other metacognitive scaffolds will support metacognitive gains when immediate feedback is faded. Twenty-three participants were randomized into intervention and control groups.

Assessing the use of cognitive heuristic representativeness in clinical reasoning.

We performed a pilot study to investigate use of the cognitive heuristic Representativeness in clinical reasoning. We tested a set of tasks and assessments to determine whether subjects used the heuristics in reasoning, to obtain initial frequencies of heuristic use and related cognitive errors, and to collect cognitive process data using think-aloud techniques. The study investigates two aspects of the Representativeness heuristic - judging by perceived frequency and representativeness as causal beliefs.

Negative CPOE attitudes correlate with diminished power in the workplace.

Power changes have been identified as a frequent and unintended consequence of the implementation of computerized physician order entry (CPOE). However, no previous study has described the degree or direction of power change, or even confirmed that such a relationship exists. Using a validated, standardized instrument for measuring personal power, we collected data from 276 healthcare workers in two different hospitals before and after implementation of CPOE.

Towards a data sharing culture: recommendations for leadership from academic health centers.

Rebecca Crowley and colleagues propose that academic health centers can and should lead the transition towards a culture of biomedical data sharing.

Development of an instrument for measuring clinicians' power perceptions in the workplace.

We report on the development of an instrument to measure clinicians' perceptions of their personal power in the workplace in relation to resistance to computerized physician order entry (CPOE). The instrument is based on French and Raven's six bases of social power and uses a semantic differential methodology. A measurement study was conducted to determine the reliability and validity of the survey.

A natural language intelligent tutoring system for training pathologists: implementation and evaluation.

Introduction: We developed and evaluated a Natural Language Interface (NLI) for an Intelligent Tutoring System (ITS) in Diagnostic Pathology. The system teaches residents to examine pathologic slides and write accurate pathology reports while providing immediate feedback on errors they make in their slide review and diagnostic reports. Residents can ask for help at any point in the case, and will receive context-specific feedback.

The importance of pathology informatics in translational research.

Pathology informatics involves management and analysis of large complex data sets derived from various tests performed in clinical and anatomic pathology laboratories, annotated biorepositories, image analysis, telepathology, and large scale experiments, including gene expression analysis, proteomics, and tissue array studies. It facilitates intelligent use of computing technologies to improve patient care and understand the natural history of disease. Herein, we describe the various bioinformatics tools used to support translational research at the University of Pittsburgh Medical Center.

Heuristic sample selection to minimize reference standard training set for a part-of-speech tagger.

Part-of-speech tagging represents an important first step for most medical natural language processing (NLP) systems. The majority of current statistically-based POS taggers are trained using a general English corpus. Consequently, these systems perform poorly on medical text.

A system for sharing routine surgical pathology specimens across institutions: the Shared Pathology Informatics Network.

This report presents an overview for pathologists of the development and potential applications of a novel Web enabled system allowing indexing and retrieval of pathology specimens across multiple institutions. The system was developed through the National Cancer Institute's Shared Pathology Informatics Network program with the goal of creating a prototype system to find existing pathology specimens derived from routine surgical and autopsy procedures ("paraffin blocks") that may be relevant to cancer research. To reach this goal, a number of challenges needed to be met.

How primary care physicians' attitudes toward risk and uncertainty affect their use of electronic information resources.

Objective: The research sought to determine if primary care physicians' attitudes toward risk taking or uncertainty affected how they sought information and used electronic information resources when answering simulated clinical questions.

Methods: Using physician-supplied data collected from existing risk and uncertainty scales, twenty-five physicians were classified as risk seekers (e.g.

Evaluation of an intelligent tutoring system in pathology: effects of external representation on performance gains, metacognition, and acceptance.

Objective: Determine effects of computer-based tutoring on diagnostic performance gains, meta-cognition, and acceptance using two different problem representations. Describe impact of tutoring on spectrum of diagnostic skills required for task performance. Identify key features of student-tutor interaction contributing to learning gains.