The field of Explainable Artificial Intelligence (XAI ) has recently gained prominence, driven by the demands for transparency and accountability in the application of AI models in critical decision-making scenarios. Although Deep Neural Networks (DNNs) have achieved remarkable success, specifically in computer vision tasks, understanding the steps pertaining to their decision-making processes remains a significant hurdle due to the opaque nature of the model. Most existing methods for explanation in computer vision focus on low-level features (such as pixels) and their influence on the final classification. This relevance is then used to generate explanations based on visual cues (such as saliency maps), rather than providing higher-level human concepts and the relations among them. This thesis aims to address this issue by proposing approaches to extract high-level human concepts, relations and rules from deep representations, thus enhancing the transparency of the model’s decision-making process. Specifically, we focus on Convolutional Neural Networks (CNNs) for image classification tasks and have leveraged our approach to make assumptions about what semantic concepts and relations are effectively learned and hidden in deep representations for computer vision models. Our results demonstrate the approach’s ability to provide a semantic understanding of what a trained image classification model has learned in a way that humans can comprehend without a deeper knowledge of machine learning techniques and concepts, thereby providing transparency and acceptance of the model’s results.