Deploying deep learning models on severely resource-constrained edge platforms, such as microcontroller-based systems, remains a significant challenge due to strict limitations on memory, computation, latency, and power consumption. These constraints necessitate the use of ultra-compact neural networks, which typically suffer from reduced representational capacity and degraded performance compared to standard large-scale models. Knowledge distillation offers a promising solution by transferring information from a high-capacity teacher model to a compact student, improving not only accuracy but also robustness, which is critical for real-world tiny machine learning (TinyML) applications operating under distributional shifts. However, when student models are extremely compact, the large capacity gap between teacher and student can hinder effective knowledge transfer. In standard distillation settings, this gap can be narrowed by simply using a larger student. Resource-constrained deployments, however, impose strict upper bounds on model size, making large capacity gaps inherent and unavoidable. In this work, we systematically investigate the effectiveness of three main categories of offline knowledge distillation including response-based, feature- based, and relation-based across heterogeneous teacher–student convolutional neural network (CNN) pairs. We specifically target ultra-compact student architectures representative of TinyML deployments, examining how varying capacity gaps affect distillation in this setting. We first evaluate classification accuracy on the CIFAR-100 benchmark and then assess robustness on the CIFAR-100-C dataset without additional fine-tuning. The study is then extended to a practical Driver Monitoring System (DMS) application using the State Farm Distracted Driver Detection dataset, followed by robustness evaluation on the unseen 100-Driver dataset. Our results demonstrate that knowledge distillation remains effective even for ultra-compact CNNs designed for resource-constrained platforms. We further show that even when capacity gaps are consistently large, the relative difference between teacher and student still meaningfully influences distillation effectiveness, with smaller gaps yielding larger accuracy improvements. Among the evaluated methods, relation-based distillation, particularly contrastive representation distillation (CRD), consistently provides the strongest gains in both accuracy and robustness, making it well suited for resource-constrained edge deployments.