The New York City Rheumatology Objective Structured Clinical Examination: five-year data demonstrates its validity, usefulness as a unique rating tool, objectivity, and sensitivity to change.

Objective: Traditional means of testing rheumatology fellows do not adequately assess some skills that are required to practice medicine well, such as humanistic qualities, communication skills, or professionalism. Institution of the New York City Rheumatology Objective Structured Clinical Examination (ROSCE) and our sequential 5 years of experience have provided us with a unique opportunity to assess its usefulness and objectivity as a rheumatology assessment tool.

Methods: Prior to taking the examination, all of the fellows were rated by their program directors. Fellows from the participating institutions then underwent a multistation patient-interactive examination observed and rated by patient actors and faculty raters. Assessments were recorded by all of the participants using separate but overlapping sets of instruments testing the Accreditation Council of Graduate Medical Education (ACGME) core competencies of patient care, interpersonal and communication skills, professionalism, and overall medical knowledge.

Results: Although the program directors tended to rate their fellows more highly than the ROSCE raters, typically there was agreement between the program directors and the ROSCE faculty in distinguishing between the highest- and lowest- performing fellows. The ROSCE faculty and patient actor assessments of individual trainees were notable for a high degree of concordance, both quantitatively and qualitatively.

Conclusion: The ROSCE provides a unique opportunity to obtain a patient-centered assessment of fellows' ACGME-mandated competencies that traditional knowledge-based examinations, such as the rheumatology in-service examination, cannot measure. The ability of the ROSCE to provide a well-rounded and objective assessment suggests that it should be considered an important component of the rheumatology training director's toolbox.