Multiple Choice Questions (MCQs) are a core building block of many MOOCs. In this exploratory study we analyze a sample of MCQs from a number of MOOCs and evaluate their quality. We conducted this analysis using a framework informed by a body of empirical research, which describes several common flaws that may occur in the way MCQs are written or phrased. Studies have shown that the presence of these flaws are likely to compromise the reliability and validity of tests containing these MCQs, potentially leading to poorer pedagogical outcomes. Through our study we contribute to the broad debate of whether MOOCs are a force that can enable enhanced and improved pedagogies or whether they will be susceptible to replicating existing poor pedagogies or practises at scale.