OUTCOME BIAS IN JUDGEMENTS OF THE QUALITY OF EXPERIMENTAL DESIGNS

Abstract

This thesis is concerned with the evaluation of experimental design. It reviews design research in general and notes some methodological limitations. The process of experimental design is considered in the light of this review, particularly in terms of the ability of designers to evaluate potential designs. Experiment 1, a prospective study involving the creative design of psychology experiments, reports inadequate assessment of experimental power and little or no evidence of explicit evaluation processes. Experiment 2 assesses the evaluation of existing experimental designs and demonstrates that judgements of the quality of experimental design are influenced by the presence of outcome information more than any other factor. Following this result a review of the hindsight and outcome bias literature is presented. Experiments 3, 4 and 5 demonstrate that outcome bias is a pervasive effect not mediated by task presentation, explicit definitions of quality or statistical expertise. Also, by manipulating subjects' perspective, Experiment 5 differentiates between the effects of outcomes themselves and their financial implications. The term outcome salience is defined as "the relative importance of the implications of an outcome from the point of view of the subject". It is shown that both the size and direction of an outcome bias are determined by (and could be predicted from) the associated outcome salience. Experiment 6 explores subjects' beliefs about the relevance of experimental factors, outcome information and the financial implications of outcomes to quality judgements. Ratings of relevance are shown to be in direct opposition to the actual use of these factors in judgements. Relevance ratings are also shown to be influenced by a subject's perspective. Experiment 7 tests the links between outcome bias effects and traditional, memory based, hindsight bias effects using a memory based paradigm. Results show that, in addition to biases of judgement in foresight, the same outcome information will also bias the memory of earlier judgements and the memory of relevant task details in hindsight. The practical implications of outcome bias are discussed. Using a motivational account based on the concept of outcome salience, hindsight bias is redefined as one particular form of outcome bias. This account unifies two previously separate research areas and is shown to explain a number of previously unexplained effects in the hindsight literature. Accounts of reasoning are reviewed, as are information processing and motivational accounts of hindsight bias. The theoretical implications of the present results are discussed in the light of these accounts.