In this paper, we present and discuss two new measures of inter- and intra-rater agreement to assess the reliability of the raters, and hence of their labeling, in multi-rater setings, which are common in the production of ground truth for machine learning models. Our proposal is more conservative of other existing agreement measures, as it considers a more articulated notion of agreement by chance, based on an empirical estimation of the precision (or reliability) of the single raters involved. We discuss the measures in light of a realistic annotation tasks that involved 13 expert radiologists in labeling the MRNet dataset.
Download full-text PDF |
Source |
|---|---|
| http://dx.doi.org/10.3233/SHTI200167 | DOI Listing |
Publication Analysis
Top Keywords
Similar Publications
Validity and Reliability of a Linear Position Transducer to Measure Barbell Velocity, Duration, and Displacement During the Bench Press.
Int J Exerc Sci
December 2024
Exercise Science and Exercise Physiology Program, Kent State University, Kent, OH, USA.
This investigation evaluated validity and reliability of the HUMAC360 linear position transducer (LPT) compared to the Tendo Sport Weightlifting Analyzer (TENDO) for measuring mean velocity (MV), peak velocity (PV), and displacement (D) during the bench press. Seventeen recreationally active individuals completed three visits. During visit one, participants were assessed for their one repetition maximum (1RM) bench press.
View Article and Find Full Text PDFDIAMACC: protocol of a prospective diagnostic accuracy study of the maximal systolic acceleration to detect peripheral arterial disease in patients with diabetes-related foot ulceration in the Netherlands.
BMJ Open
December 2024
Department of Vascular Surgery, Leids Universitair Medisch Centrum, Leiden, The Netherlands.
Introduction: Foot ulcers are one of the most serious complications of diabetes, leading to significant risks on amputation and mortality. Peripheral arterial disease (PAD) is an important factor for the development and the outcome of diabetic foot ulcers (DFU). Although prompt and accurate detection of PAD is critical to reduce complications, its diagnosis can be challenging with currently used bedside tests (such as ankle-brachial index and toe pressure) due to medial arterial calcification.
View Article and Find Full Text PDFBackground: In 2018, a nationwide survey carried out in 387 acute care hospitals from 16 out of 21 Italian regions, allowed defining an extended checklist for the participatory evaluation of person-centredness in hospital care. We aimed to validate a reduced set of core items for continuous use across the country.
Methods: Factor analysis was used to validate the construct of the checklist.
Optimizing hip MRI: enhancing image quality and elevating inter-observer consistency using deep learning-powered reconstruction.
BMC Med Imaging
January 2025
Department of Magnetic Resonance Imaging, The First Affiliated Hospital, Zhengzhou University, Zhengzhou, 450052, China.
Background: Conventional hip joint MRI scans necessitate lengthy scan durations, posing challenges for patient comfort and clinical efficiency. Previously, accelerated imaging techniques were constrained by a trade-off between noise and resolution. Leveraging deep learning-based reconstruction (DLR) holds the potential to mitigate scan time without compromising image quality.
View Article and Find Full Text PDFOpportunistic Hounsfield units measurements on computed tomography scans can predict the number of prevalent vertebral compression fractures in older adults.
Eur Spine J
January 2025
Department of Orthopaedic Surgery, AO Spine Fellowship, Clinica Alemana Santiago - Universidad del Desarrollo, Santiago, Chile.
Purpose: There is a growing interest in using computed tomography (CT) scans to opportunistically assess bone mineral density via Hounsfield units (HU). Previous studies have shown lower HU in patients with vertebral compression fractures (VCFs) and that HU can predict pre-existing VCFs. This study evaluated whether HU from CT scans can predict the number of prevalent VCFs.
View Article and Find Full Text PDFWant AI Summaries of new PubMed Abstracts delivered to your In-box?
Enter search terms and have AI summaries delivered each week - change queries or unsubscribe any time!