Optimised Deep Learning for Gastrointestinal Polyp Classification: A Controlled Benchmark of Five CNN and Transformer Architectures with Grad-CAM Interpretability

Abstract

Background/Objectives: Colorectal cancer (CRC) is the second leading cause of cancer-related mortality worldwide, with polyp miss rates of up to 26% reported during colonoscopy and classification accuracy remaining highly operator-dependent. Accurate multi-class polyp subtype classification is clinically critical, as it directly determines treatment decisions: adenomatous polyps require resection, whereas hyperplastic lesions may warrant only surveillance. This study aims to systematically compare five deep learning architectures for five-class gastrointestinal polyp classification and to provide clinically interpretable diagnostic insights through Grad-CAM visualisation. Methods: ResNet50, VGG16, EfficientNet-B3, DenseNet121, and Vision Transformer (ViT-B/16) were evaluated on the Kvasir Dataset V2 (5000 images, five classes) under a unified training and evaluation protocol on common GPU hardware. All models employed ImageNet transfer learning with a redesigned multi-layer classification head. Two optimisation strategies were applied: SGD with cosine annealing for CNN architectures, and AdamW with linear warmup for ViT-B/16. Gradient-weighted Class Activation Mapping (Grad-CAM) was applied to generate spatial attention heatmaps for qualitative clinical interpretation. Results: Under a single 80/10/10 split, ViT-B/16 attained the highest accuracy (97.2%); however, because a single split is sensitive to sampling, the evaluation was strengthened with stratified five-fold cross-validation (mean ± SD). Under cross-validation, EfficientNet-B3 achieved the highest accuracy at 95.90 ± 0.35%, followed closely by ViT-B/16 (95.12 ± 0.72%), then ResNet50 (91.74 ± 0.74%), DenseNet121 (90.32 ± 0.70%), and VGG16 (88.50 ± 1.72%); the small standard deviations indicate that all models, including ViT-B/16, were stable across folds. Pairwise McNemar tests with Holm correction found that every difference was statistically significant (p < 0.05), including the EfficientNet-B3 advantage over ViT-B/16 (p = 0.010). ViT-B/16 thus remained a strong, stable performer that significantly outperformed the three remaining CNNs, while the cross-validated ranking placed the most compact model, EfficientNet-B3, first: a Vision Transformer was highly competitive with, but not superior to, the strongest CNN. A consistent, architecture-agnostic misclassification pattern was identified between dyed-lifted polyps and dyed-resection margins across all five models, consistent with a task-level visual ambiguity that may also reflect overlapping class definitions and annotation factors, with direct clinical implications. Grad-CAM analysis, quantified by attention-entropy and concentration metrics, showed that model attention remained focused on relevant stained tissue regardless of whether predictions were correct, indicating that the dyed-class confusions reflect genuine visual ambiguity rather than a localisation failure. Conclusions: Under cross-validation, EfficientNet-B3 achieved the highest accuracy on the Kvasir V2 five-class task, significantly outperforming all other architectures, with ViT-B/16 being a close and competitive second. The identified confusion between post-procedural chromoendoscopic classes is unlikely to be fully resolved by architectural changes alone and may require higher-resolution imaging or domain expert re-annotation. These findings contribute to the evidence base for explainable deep learning in gastrointestinal endoscopy; external, multi-centre validation remains necessary before clinical adoption.