How well do commonly used data presentation formats support comparative effectiveness evaluations?

Background: Good decisions depend on an accurate understanding of the comparative effectiveness of decision alternatives. The best way to convey data needed to support these comparisons is unknown.

Objective: To determine how well 5 commonly used data presentation formats convey comparative effectiveness information.

Methods: The study was an Internet survey using a factorial design. Participants consisted of 279 members of an online survey panel. Study participants compared outcomes associated with 3 hypothetical screening test options relative to 5 possible outcomes with probabilities ranging from 2 per 5000 (0.04%) to 500 per 1000 (50%). Data presentation formats included a table, a "magnified" bar chart, a risk scale, a frequency diagram, and an icon array. Outcomes included the number of correct ordinal judgments regarding the more likely of 2 outcomes, the ratio of perceived versus actual relative likelihoods of the paired outcomes, the intersubject consistency of responses, and perceived clarity.

Results: The mean number of correct ordinal judgments was 12 of 15 (80%), with no differences among data formats. On average, there was a 3.3-fold difference between perceived and actual likelihood ratios (95% confidence interval = 3.0-3.6). Comparative judgments based on flowcharts, icon arrays, and tables were all significantly more accurate and consistent than those based on risk scales and bar charts (P < 0.001). The most clearly perceived formats were the table and the flowchart. Low subjective numeracy was associated with less accurate and more variable data interpretations and lower perceived clarity for icon displays, bar charts, and flow diagrams.

Conclusions: None of the data presentation formats studied can reliably provide patients, especially those with low subjective numeracy, with an accurate understanding of comparative effectiveness information.