Consequences of event rate heterogeneity across non-randomized study sub-groups.

Analyses to compare non-randomized groups are more and more common in both post hoc analyses of randomized clinical trials data and in analyses of long-term observational data. In such cases, it is quite likely that there are unknown or uncollected sources of heterogeneity in event rates. Research has shown that an underlying source of heterogeneity in event rates which is not included in proportional hazards regression models leads to biased estimates for included covariate effect estimates and lower power to test them whether the source of heterogeneity is assumed to be fixed or random. We demonstrate here using several post hoc analyses of clinical trials data that a potentially common problem may be that the non-randomized groups which are to be compared have differential variability in their event rates. We then show through simulation that such underlying heterogeneity which varies across the groups, when ignored in the modelling, can lead to an attenuated regression effect estimate for comparing the two groups to each other, lower rejection rates for the effect, and Wald-based confidence intervals with potentially much lower coverage than nominal. When the groups are not significantly different, but heterogeneity differs between them, an analysis ignoring the heterogeneity can even result in a significant negative comparison.