In a lifelog, data from various sources are combined to form a record from which one can retrieve information about oneself and the environment in which one is situated. It could be considered similar to an automated biography. Lifelog technology is still at an early stage of development. However, the history of lifelogs so far shows a clear academic, corporate and governmental interest. Therefore, a thorough inquiry into the ethical aspects of lifelogs could prove beneficial to the responsible development of this field. This article maps the main ethically relevant challenges and opportunities associated with the further development of lifelog technologies as discussed in the scholarly literature. By identifying challenges and opportunities in the current debate, we were able to identify other challenges and opportunities left unmentioned. Some of these challenges are partly explained by a blind spot in the current debate. Whilst the current debate focuses mainly on lifelogs held by individuals, lifelogs held by governmental institutions and corporations pose idiosyncratic ethical concerns as well. We have provided a brief taxonomy of lifelog technology to show the variety in uses for lifelogs. In addition, we provided a general approach to alleviate the ethical challenges identified in the critical analysis.