Factors affecting inter-rater agreement in human classification of eye movements: a comparison of three datasets.

Behav Res Methods

Derrick M5, Department of Computer Science, Texas State University, 601 University Drive, San Marcos, Texas, 78640, USA.

Published: January 2023

Manual classification of eye-movements is used in research and as a basis for comparison with automatic algorithms in the development phase. However, human classification will not be useful if it is unreliable and unrepeatable. Therefore, it is important to know what factors might influence and enhance the accuracy and reliability of human classification of eye-movements. In this report we compare three datasets of human manual classification, two from earlier datasets and one, our own dataset, which we present here for the first time. For inter-rater reliability, we assess both the event-level F1-score and sample-level Cohen's κ, across groups of raters. The report points to several possible influences on human classification reliability: eye-tracker quality, use of head restraint, characteristics of the recorded subjects, the availability of detailed scoring rules, and the characteristics and training of the raters.

Download full-text PDF

Source
http://dx.doi.org/10.3758/s13428-021-01782-4DOI Listing

Publication Analysis

Top Keywords

human classification
16
three datasets
8
manual classification
8
classification eye-movements
8
classification
6
human
5
factors inter-rater
4
inter-rater agreement
4
agreement human
4
classification eye
4

Similar Publications

Background: The PalliPed project is a nationwide, observational, cross-sectional study designed with the aim of providing a constantly updated national database for the census and monitoring of specialized pediatric palliative care (PPC) activities in Italy. This paper presents the results of the first monitoring phase of the PalliPed project, which was developed through the PalliPed 2022-2023 study, to update current knowledge on the provision of specialized PPC services in Italy.

Methods: Italian specialized PPC centers/facilities were invited to participate and asked to complete a self-reporting, ad-hoc, online survey regarding their clinical activity in 2022-2023, in the revision of the data initially collected in the first PalliPed study of 2021.

View Article and Find Full Text PDF

Optical techniques, such as functional near-infrared spectroscopy (fNIRS), contain high potential for the development of non-invasive wearable systems for evaluating cerebral vascular condition in aging, due to their portability and ability to monitor real-time changes in cerebral hemodynamics. In this study, thirty-six healthy adults were measured by single channel fNIRS to explore differences between two age groups using machine learning (ML). The subjects, measured during functional magnetic resonance imaging (fMRI) at Oulu University Hospital, were divided into young (age ≤ 32) and elderly (age ≥ 57) groups.

View Article and Find Full Text PDF

Streptococcus dysgalactiae (S. dysgalactiae ) is a common pathogen of humans and various animals. However, the phylogenetic position of animal S.

View Article and Find Full Text PDF

We aimed to build a robust classifier for the MGMT methylation status of glioblastoma in multiparametric MRI. We focused on multi-habitat deep image descriptors as our basic focus. A subset of the BRATS 2021 MGMT methylation dataset containing both MGMT class labels and segmentation masks was used.

View Article and Find Full Text PDF

An automatic cervical cell classification model based on improved DenseNet121.

Sci Rep

January 2025

Department of Biomedical Engineering, School of Life Science and Technology, Changchun University of Science and Technology, Changchun, 130022, China.

The cervical cell classification technique can determine the degree of cellular abnormality and pathological condition, which can help doctors to detect the risk of cervical cancer at an early stage and improve the cure and survival rates of cervical cancer patients. Addressing the issue of low accuracy in cervical cell classification, a deep convolutional neural network A2SDNet121 is proposed. A2SDNet121 takes DenseNet121 as the backbone network.

View Article and Find Full Text PDF

Want AI Summaries of new PubMed Abstracts delivered to your In-box?

Enter search terms and have AI summaries delivered each week - change queries or unsubscribe any time!