Context matters: emergent variability in an effectiveness trial of online teaching modules.

Context: Conducting research in real life settings (effectiveness studies) can introduce many confounding factors. Efficacy studies seek to control for researcher bias and data quality rather than considering how the efficacy of an intervention is changed by the contexts in which it is used. Relatively little is known about the impact of context on educational interventions, in particular on multimedia learning.

Methods: An effectiveness study to understand implementation variance of online educational modules in surgery clerkships was conducted in six US medical schools participating in an efficacy trial of different multimedia designs. Student and teacher experiences were captured through focus groups and one-to-one interviews with trial participants and their teachers. Audio-recordings of these sessions were transcribed and analysed using grounded theory techniques.

Results: Differences were identified in student and teacher perceptions of how the educational intervention had been implemented and how its uptake had been influenced by context-dependent factors: (i) the intervention was implemented in different ways to suit different educational contexts and this influenced how students and teachers responded to it; (ii) the ways students and teachers interacted with, and behaved around, the intervention influenced its uptake; (iii) the way the intervention was perceived by students and teachers influenced its uptake; and (iv) the medium and design of the intervention had a directing influence on its uptake.

Conclusions: It was observed that each institutional context formed a complex educational ecology. The intervention became interwoven with different educational ecologies so that it could no longer be considered a stable variable across the study. We suggest that researchers should conduct implementation-profiling studies in advance of any intervention-based research to account for the constructing nature of educational ecologies on their interventions and in doing so to more clearly differentiate between efficacy and effectiveness studies.