A problem-oriented approach to resident performance ratings.

Background: Global, end-of-rotation evaluations are often difficult to interpret due to their high level of abstraction (eg, excellent, good, poor) and the bias toward high ratings. This study documents the utility of and measurement characteristics of serious problem items, an alternative item format.

Methods: This report is based on 4,234 faculty performance ratings for 105 general surgery residents. Faculty members reported whether each resident had a serious problem for each of 8 areas of clinical performance and 6 areas of professional behavior.

Results: A total of 263 serious problems were reported. The performance category with the most total serious problems noted was knowledge and that with the least problems noted was relations with patients and family members. Seven residents accounted for 86.9% of all serious problem reports. Each resident had serious problems in multiple performance areas. Problems were reported most frequently in knowledge, management, technical/procedural skills, ability to assume responsibility within level of competence, and problem identification. Citations of these serious problems were most common in year 3. Traditional ratings of global performance were not an adequate means for identifying residents with serious performance problems.

Conclusion: Serious problem ratings can communicate faculty concerns about residents more directly and can be used as a complement to conventional global rating scales without substantially increasing faculty workload.