The ability to learn a hierarchical and compact representation from data stands as a fundamental principle behind the rapid growth of Deep Learning, particularly evident in Computer Vision. Despite the significant progress on the perception tasks such as recognition and detection, these models still fall short in terms of reasoning and planning capabilities, and cannot generalise systematically despite being trained with extensive amount of data and compute resources. How to effectively scale up a representation learning system in terms of computation and data, and extend the capabilities of the visual representations toward high-level tasks is the central research topic of this thesis. First we focus on contrastive representation learning, a general approach for learning representation by comparison. We survey and analyse more than 100 recent works and provide a framework to categorise and understand research in this direction, not only in the context of self-supervised visual learning but also for other domains and applications. We then turn towards the problem of object-centric representation learning, a promising approach to learn structured representations in a complex visual scene for planning and reasoning tasks. We first explore using discrete representation for object-centric learning, motivated by the common goal of decomposing the continuous visual signal into individual discrete components. Understanding the importance and challenges of scaling in learning representations from data, we propose an efficient architecture for decoding object-centric representations, a ubiquitous but memory-intense component present in most object- centric learning methods. Finally, to address the challenge of learning these object-centric representations in complex and realistic data, we capitalise on the advancements in pre-trained mod- els for visual representations, enabling the learning of higher-level representations. Inspired by human cognitive development, we further study the effects of depth information and geometry contained in these representations, exploring their influence on the process of unsupervised object discovery.