Checking for interviewer bias in outcome assessment: a method for strengthening the design of prospective, randomised trials in surgery.

Background: Blind, randomised trials are conceived as the gold standard in clinical research, but this ideal, in its strict sense, can rarely be achieved in surgical settings. One way to strengthen the study design is to check for observer bias in the assessment and evaluation of surgical outcome.

Method: In a randomised, prospective trial comparing nasogastric versus gastrostomy tubes the primary endpoint was the subjective inconvenience induced by the tube system and was assessed in the context of a standardised face-to-face interview. These interviews were tape-recorded on a pocket memo. Two independent raters listened to these interviews and judged--on the basis of how the interviewer formulated the questions--which treatment arm they thought the patients were assigned to and how confident they were in their judgement.

Results: The overall proportion of correct judgements was 50.5% for rater 1 and 53.2% for rater 2. In other words, both judgement performances were not greater than chance. Nevertheless, the raters' confidence in their judgements increased significantly ( P<0.05) in the course of the rating procedure, whereas the actual proportion of correct judgements did not. There was no overlap between the two raters [kappa = 0.022, not significant (NS)] and between actual group assignment and both raters' judgements (kappa = 0.012, NS and kappa = 0.110, NS).

Conclusion: The two independent raters were not able to detect systematic variations in the interviewing style that were contingent on treatment arm assignment. This gives further credence to the results of the randomised trial showing greater patient-reported discomfort and inconvenience with the nasogastric tube than with the gastrostomy tube. The present report describes a feasible method to monitor subtle biases that may occur in trial settings. This helps to strengthen the design of randomised clinical trials in surgery.