The future of online testing and
assessment: question quality in MOOCs
Costello, EamonORCID: 0000-0002-2775-6006, Holland, Jane and Kirwan, Colette
(2018)
The future of online testing and
assessment: question quality in MOOCs.
International Journal of Educational Technology in Higher Education, 15
(42).
If MOOCs are to play a role in the future of higher education it is imperative that we
critically examine how they are currently functioning. In particular, questions persist
about the role MOOCs will play in the future of formal accredited learning. As the
focus turns from informal and free to formal, accredited and paid, greater scrutiny
will be brought to bear on the quality of the courses themselves. Although there
have been some empirical studies into the quality of MOOCs, a notable gap exists in
that such research has not examined Multiple Choice Questions (MCQs) which are a
key component of much MOOC assessment and testing. Previous research suggests
that flawed MCQ items may compromise the reliability and validity of these
assessments, potentially leading to inconsistent outcomes for students. This study
was hence designed to examine MCQ quality in MOOCs. 204 MCQs were analysed,
from a selection of 18 MOOCs, sampling the domains of computing, social science
and health sciences. Over 50% of MCQs (112) contained at least one item flaw; 57
MCQs contained multiple flaws. A large proportion of MOOC MCQs violated itemwriting
guidelines, which is comparable with previous studies examining the
prevalence of flaws in assessments in more traditional educational contexts. The
problem of low quality MCQs can be ameliorated by appropriate faculty training and
pre- and post-test quality checks. These activities are essential if MOOCs are to
become a force that can enable enhanced and improved pedagogies in the future
of higher education, instead of simply proceeding to replicate existing poor practices
at scale.