Analysis of intra-observer and inter-observer variability of pathologists for non-benign thyroid fine needle aspiration cytology according to Bethesda system categories.

Aim: To examine the intra- and inter-observer variability for non-benign thyroid cytological subcategories according to the Bethesda classification system after the second review.

Methods: Between November 2018 and May 2019, thyroid fine needle aspiration biopsies of 381 nodules were retrospectively evaluated. Among them, 74 non-benign (category III-VI) thyroid biopsies, analyzed according to the Bethesda system (pathologist 1:40 vs pathologist 2:34) by two independent pathologists, were reassessed by the same pathologists and by a cytopathologist. In this observer-blinded study, weighted Cohen's kappa was used to assess the intra-observer agreement, and Krippendorff's alpha was used to assess the inter-observer agreement.

Results: At the first and second evaluations of pathologists 1 and 2, the percentage agreement was 62.5% for pathologist 1 and 58.8% for pathologist 2. The intra-observer agreement was substantial (κ = 0.705) for pathologist 1, and moderate (κ = 0.447) for pathologist 2. In the second evaluation of pathologist 1 and 2, which was compared with the cytopathologist, the agreement percentage of pathologist 1 with the cytopathologist was 50.0%, and that of pathologist 2 was 56.8%. The inter-observer agreement was below the lowest acceptable limit for an overall agreement (α = 0.634) among the three raters. The inter-observer agreement was only acceptable between the cytopathologist and the second pathologist, while it was low between the other raters. In the evaluation of the non-benign nodules, the mean category score of the cytopathologist was 3.22 and lower than both pathologists (3.73 and 3.58, respectively).

Conclusions: The intra-observer agreement of pathologists was moderate-to-substantial in the evaluation of non-benign thyroid biopsies according to the Bethesda reporting system. However, the inter-observer agreement was below the lowest acceptable limit when the cytopathologist was taken as a reference.