Much of the recent hype around artificial intelligence stems from recent advances in Neural Networks, currently the most widely used algorithm that succeeded where other approaches failed for decades. Neural Networks today can leverage large amounts of data to be trained to perform hard tasks such as recognising objects in an image or translating languages. The process they use to perform these tasks is equivalent to a complex pattern recognition procedure which uses some clever mathematics to expose the underlying structure in a body of data. Humans think in a more conceptual way. We build a mental model of our world. We have the ability to extract relationships such as causality between elements involved in learning to perform a task, and the ability to use background knowledge when learning. One of the key challenges in making more human-like artificial intelligence is incorporating these properties of natural learning into the neural network paradigm. Designing such a system which could utilise background knowledge in learning a new task would enable the networks to be trained on much less data, opening up a new world of opportunities for Neural Networks to be applied to tasks which were previously not feasible due to the scarce availability of data. In identifying these challenges, we have been inspired by recent seminal papers within the Deep Learning community, which call for new approaches to enhance deep representations with (common-sense) background knowledge. This is considered as a key enabler to significantly improve the ability of machines to learn new tasks faster and in a domain invariant way. The main practical challenges involved in this research are finding how best to extract and format relevant knowledge from a trained network, and finding how best to inject this knowledge into an untrained network.