A novel approach to lossless convolutional neural network compression via progressive knowledge distillation-incorporated low-rank compression

Abstract

Model compression is widely used to deploy large neural networks on resource-constrained edge devices. Among existing techniques, low-rank composition is theoretically grounded in approximation theory and provides a strong basis for preserving model performance after compression. However, in practice, even state-of-the-art low-rank methods such as Tucker, Tensor Train, and Tensor Ring decomposition may still introduce substantial reconstruction errors. These errors are difficult to recover using conventional fine-tuning strategies, especially under high compression ratios. To address this issue, this paper proposes Progressive Knowledge Distillation-Incorporated Low-Rank Compression (PKD-LRC). The proposed framework introduces three key components. First, a residual-compensation mechanism is incorporated into convolutional compression to reduce information loss. Second, a dynamic temperature adjustment strategy is used to improve the fine-tuning of low-rank models and enhance prediction accuracy. Third, a progressive learning paradigm is developed to balance compression ratio and classification performance. Theoretical analysis is conducted to investigate the effectiveness of the residual compensation mechanism for compressing convolutional neural networks (CNNs). The experimental results on the CIFAR-100 dataset show that PKD-LRC achieves a 0.35% higher Top-1 accuracy with a 7.75 × compression ratio compared to the uncompressed VGG-16 model. On the ImageNet dataset, the proposed method yields a 0.20% Top-1 accuracy gain with a 2.78 × compression ratio over the baseline ResNet-18 model. Above results indicate that PKD-LRC effectively mitigates reconstruction errors and enhances the performance of compressed models through adaptive fine-tuning.