Cochrane is a non-profit organization which mainly relies on volunteer health professionals for the production, simplification, evaluation, and multilingual dissemination of high-quality health content. The approach that Cochrane volunteers adopt for the simplification (or intralingual translation) of English health content is non-automated and involves the manual checking and implementation of plain language guidelines. This study investigated whether and to what extent the introduction of a controlled language (CL) checker—which would make the simplification approach semi-automated—increased authors’ satisfaction and machine translation (MT) quality. Twelve Cochrane authors completed a standardized questionnaire and answered follow-up questions on their level of satisfaction and preferences. Forty-one Cochrane evaluators assessed the quality of the Spanish MT outputs of simplified texts. Authors showed a preference for the introduction of a CL checker. Differences in MT quality scores were slight.