EXPLORING THE VALIDITY OF MULTIMEDIA WRITTEN ASSESSMENT IN SAUDI ARABIA

Abstract

Introduction: Multiple choice questions are widely used in high-stakes written examinations and are continuously being challenged for testing recall facts rather than higher cognition. Innovative MCQ formats (Multimedia-enhanced) can test such skills. To evaluate this format a recent validity framework “The Cambridge Framework” was used to assess and explore this intervention.

Aim: Explore the validity of multimedia MCQs for testing higher cognition in Emergency Medicine (EM) in Saudi Arabia using the Cambridge framework, and evaluate the use of this framework.

Methods: A total of 164 EM residents (seniors and juniors) from three regions of Saudi Arabia took a total of 80 multimedia-text matched items in an end of year exam. A mixed-methods approach triangulating quantitative (pilot test, parallel forms of multimedia and text items, item psychometrics and characteristics, questionnaires) and qualitative methods (semi-systematic literature review, focus-group discussions, Cambridge validity framework implementation and research legitimation), were applied using systematic guidelines for each method to explore multimedia items.

Results: Discrimination was significantly higher for multimedia than text items (DI= 0.19 +18, 0.14 +17, p= .03), and took significantly longer to answer (p=.01). Both formats had a moderate difficulty level (Diff = 0.75,0.74). Multimedia-items had a higher reliability and G-coefficient than text items. Focus group results revealed seven main themes of multimedia that effect item characteristics. Review of the Cambridge Framework demonstrated areas of gaps in sources of validity evidence and external-related factors not covered in other frameworks that have implications on validity.

Conclusion: Multimedia questions were more discriminating and took longer time to answer than the text questions. They test higher cognition and have certain characteristics that effect difficulty level and how they are perceived by the examinees altering their thinking process towards the answer. Gap areas identified in the Cambridge framework and external factors that were contextual, may give room to explore the issue of international validity in assessment.