W33 – Eliminating Trick Questions: Creating Multiple Choice Questions That Accurately Reflect Learners’ Competencies in Psychiatry, Not Competencies in Test Taking
Friday, Oct. 28
15:45 – 16:45 (1 hr)
Meeting Room: Linden (Mezzanine)
Kien Dang*, MD, FRCPC; Carla Garcia, MD, FRCPC; Chloe Leon, MD, FRCPC; Finola D’Souza, MD; Katina Tzanetos, MD, FRCPC
CanMEDS Role:
- Scholar
At the end of this session, participants will be able to: 1) Describe the anatomy of a multiple choice question (MCQ); 2) Identify common flaws in MCQs that affect reliability in measuring competence; and 3) Apply an MCQ creation template in generating one high-quality multiple choice question.
Assessments are an integral part of measuring the competencies of our undergraduate and postgraduate trainees, as well as in self-assessment in continuing professional development. The use of single-best-answer multiple choice questions (MCQs) can allow for the assessment of high volumes of learners with good reliability if questions are constructed appropriately. In contrast, flawed MCQs allow for skilled “test takers” to identify the “trick” in the question. As a result, flawed MCQs are more likely to perform poorly on psychometric analysis and will not accurately measure trainees’ competencies. Undergraduate and postgraduate learners need experience with high-quality MCQs, because MCQs are an integral part of high-stakes examinations, including the United States Medical Licensing Examination and Medical Council of Canada licensing exams at the undergraduate level and the Royal College of Physicians and Surgeons of Canada specialty exam at the postgraduate level. MCQs are also an effective teaching tool, especially in engaging large numbers of learners in virtual platforms; however, few faculty are trained in the construction of high-quality MCQs.
In this interactive workshop, we intend to discuss the goals of assessment questions and common pitfalls in MCQ creation and describe a structure of MCQs that are likely to be reliable assessment tools. Participants will participate in a process that can ensure the creation of high-quality MCQs.
References:
- Boland RJ, Lester NA, Williams E. Writing multiple-choice questions. Acad Psychiatry 2010;34:310–6.
- Medical Council of Canada. Guidelines for the development of multiple-choice questions [Internet]. Ottawa (ON): Author; 2010. Available from: https://mcc.ca/media/Multiple-choice-question-guidelines.pdf
