Educational technology as a broad and applied interdisciplinary research field faces challenges in achieving consensus on what constitutes good quality research. As the field is embedded in many other disciplines, considering what evidence matters and the optimal methodologies to conduct inquiry is continually evolving and maturing. Inhabiting a boundary between education, and computer science, and viewed through numerous theoretical lenses ranging from disciplines of sociology, politics, psychology, the learning systems, curriculum development, digital humanities, and beyond, the number of approaches contributing to the field is vast. The validity, trustworthiness and integrity of over two decades of research in this domain is continually questioned. Furthermore, as technology itself also changes, there are differing opinions on how best to explore and understand the role it plays in education. How we define, research and evaluate our evidence is central to our understanding of how we learn and how this is enhanced with and through technology in various ways. Whilst scholars continue to critique and debate the veracity of findings, educational technology journals play an important role in allowing us to collectively peer review, and publish the best quality research studies. Changes in the open access publishing world and in the open science movement have the potential to address some of shortfalls in how our understandings are evaluated, critiqued and judged in this domain.