Relatively IdiosyncraticExploring variations in assessors' performance judgements within medical education

Access to files

•   FULL-TEXT.PDF (pdf)

Abstract

Title: Relatively idiosyncratic: Exploring variations in assessors' performance judgements within medical education.Background: Whilst direct-observation, workplace-based (or performance) assessments, sit at the conceptual epitome of assessment within medical education, their overall utility is limited by high-inter-assessor score variability. We conceptualised this issue as one of problematic judgements by assessors. Existing literature and evidence about judgements within performance appraisal and impression formation, as well as the small evolving literature on raters’ cognition within medical education, provided the theoretical context to study assessor’s judgement processes.Methods and Results: In this thesis we present three studies. The first study adopted an exploratory approach to studying assessors’ judgements in direct observation performance assessments, by asking assessors to describe their thoughts whilst assessing standard videoed performances by junior doctors. Comments and follow up interviews were analysed qualitatively using grounded theory principles. Results showed that assessors attributed different levels of salience to different aspects of performances, understood criteria differently (often comparing performance against other trainees) and expressed their judgements in unique narrative language. Consequently assessors’ judgements were comparatively idiosyncratic, or unique.The two subsequent follow up studies used experimental, internet based, experimental designs to further investigate the comparative judgements demonstrated in study 1. In study 2, participants were primed with either good or poor performances prior to watching intermediate (borderline) performances. In study 3 a similar design was employed but participants watched identical performances in either increasing or decreasing levels of proficiency. Collectively, the results of these two studies showed that recent experiences influenced assessors’ judgements, repeatedly showing a contrast effect (performances were scored unduly differently from earlier performances). These effects were greater than participants’ consistent tendency to be either lenient or stringent and occurred at multiple levels of performance. The effect appeared to be robust despite our attempting to reduce participants’ reliance on the immediate context. Moreover, assessors appeared to lack insight into the effect on their judgements.Discussion: Collectively, these results indicate that assessors score variations can be substantially explained by idiosyncrasy in cognitive representations of the judgement task, and susceptibility to contrast effects through comparative judgements. Moreover, assessors appear to be incapable of judging in absolute terms, instead judging normatively. These findings have important implications for theory and practice and suggest numerous further lines of research.

Layman's Abstract

Within medical education, medical students and junior doctors are required to complete various asessments - both to determine whether they are competent for their role, but also to guide their development. Many of these rely on their performance being observed and judged by a more senior person, who gives them scores and feedback. Unfortunately current approaches suffer from being very subjective: the same student or trainee can receive quite different reports depending on who has assessed them. This is considered problematic. More to point, attempts to improve this through training assessors, or providing them with better guidance have been at best partially successful. This lead us to study the thought processess that assessors use as they make judgements.Across three studies we found a couple of explanations for these variations. They arise in part because assessors have their own individual, fairly unique approaches to understanding and making judgements. This causes them to focus on different parts of trainees performances, and to reach differing judgements. On top of this, they are also influenced by their recent experienced of other students or trainees, in a way that alters the scores that they give. This means that their individual, fairly unique approaches aren't very "fixed", but quite susceptible to being influenced. This has various implications for the way that we understand the judgement process, but needs further research before it is clear exactly what this means for practice.

Additional content not available electronically

As the thesis is presented in alternative format, it contains material in a format suitable for publication. Two of the papers that make up the thesis cannot be published here for reasons of copyright, but should be accessed and inserted at the points that are described in the thesis. These papers are:Seeing the same thing differently: Mechanisms that contribute to assessor differences in directly-observed performance assessmentsYeates P, O’Neill P, Mann K, Eva K.Advances in Health Science Education (2012)DOI 10.1007/s10459-012-9372-1Published on-line ahead of print, Available at http://link.springer.com/article/10.1007/s10459-012-9372-1Effect of Exposure to Good vs Poor Medical Trainee Performance on Attending Physician Ratings of Subsequent Performances Yeates P, O’Neill P, Mann K, Eva K.Journal of the American Medical Association (JAMA) 2012;308(21):2226-2232. doi:10.1001/jama.2012.36515

Keyword(s)

Assessment; Judgement; Medical Education; Rater Cognition; Workplace Based Assessment

Bibliographic metadata

Institutional metadata

Record metadata