Do certain phrases in letters of recommendation predict success in fellowship performance?

Background: Letters of recommendation (LORs) are a valued, yet imperfect tool. Program directors (PDs) score phrases such as and % of students as positive. Although positive phrases are valued by PDs, there is no evidence that these phrases predict performance. We attempt to identify whether 12 specific phrases found in letters of recommendation predict future performance of fellows.

Methods: LORs were evaluated for 12 select phrases and statements. Alpha Omega Alpha (AOA) status, Step 2 Clinical Knowledge (CK) score, and whether the letter writer was personally known to our admission's committee were also categorized. Logistic regressions were performed to evaluate the relationship of the independent variables with fellow performance.

Results: Using multivariate logistic regression, (OR = 4.02, 95% CI (1.0, 15.9),  < 0.05), (OR = 4.74, 95% CI (1.4, 16.3),  = 0.01), and (OR = 3.87, 95% CI (1.3, 11.7),  = 0.02) predicted positive performance. H (OR = 0.31, 95% CI (0.1, 1.0),  < 0.05) and (OR = 0.05, 95% CI (0.0, 0.6),  = 0.02) predicted negative performance. The remaining phrases did not correlate to fellowship performance.

Conclusion: The current LOR evaluation process may place undo importance on phrases that have limited bearing on a candidate's success in training. Training both letter readers and writers to avoid using coded language or avoid assigning improper importance to select phrases may help improve the candidate selection process.