Composite Scores and Decision-Making in Undergraduate Medical Education

Abstract

Composite scores are frequently used in medical education to reflect aggregate information about a student’s performance. Combining assessment scores into composites has been shown to be a successful practice in traditional medical education models and is normally driven by the educational system in place. The recent paradigm shift to competency-based medical education has been associated with many implications for assessment. A major challenge emerged about the procedure of combining assessment information in competency-based models, and the validity of decision-making based on composites. In this study, we examined validity evidence associated with traditional composite scores and consequential decision-making, and that associated with reformulated composites based on the competency framework. Furthermore, a third decision model about students’ academic progress was built from deliberations among education experts. All assessment data about third-year medical students were collected, in addition to scores on International exams and information about residency placement. Our results showed that the reliability of composite scores is adequate for the scope of their use, irrespective of the medical education system that drove their formulation. However, associations were more meaningful and interpretable in the decision model based on the competency framework, in comparison to the traditional model. The three models yielded an absolute agreement in 67.4% of cases, and a re-classification of students’ academic status in the rest. Correlations with external criteria (performance on International exams and residency placement) demonstrated that decisions ensuing from the three models are supported by consequential validity evidence, and that the second model, using competency-guided composite scores, provided a better classification accuracy, especially in the borderline spectrum of performance. Finally, our findings suggest that the use of composite scores is associated with defensible decisions about student advancement irrespective of the medical education model. However, decision models differ with their ability to address the challenge of identifying struggling students. Although the advancement of competency-based medical education had implications over assessment, formulating composite scores as measures of competencies is feasible and seems to yield better classification decisions.