Interpreting networks via semantic probes with correction and counterfactuals

Abstract

Convolutional neural networks (CNNs) have achieved remarkable success in image-related tasks, yet their deployment in high-risk domains remains constrained by the lack of interpretable and empirically testable decision evidence. Existing gradient- and saliency-based methods typically highlight coarse influential regions without explicitly disclosing the semantic attributes driving predictions, whereas concept-based approaches often suffer from imprecise visual grounding and weak coupling to localized visual evidence. Moreover, even when expert feedback is incorporated to refine explanations, it is often unclear whether the corrected evidence remains consistently associated with model decisions under targeted interventions. To address these limitations, an Explain–Correct–Verify framework, termed ECV, is proposed for network interpretation. The framework integrates a Semantic Visual Probe (SVP), an Interpretation Corrector, and Counterfactual Verification into a unified closed-loop pipeline. By incorporating prior knowledge into gradient computation, SVP identifies decision-relevant textual attributes together with their corresponding visual evidence. The Interpretation Corrector further adjusts the probe's internal parameters according to expert knowledge, thereby improving semantic consistency and spatial alignment. To move beyond correlational agreement, localized counterfactual interventions are introduced to assess whether the identified evidence remains relevant to predictions under controlled edits. Extensive experiments on pre-trained models and public datasets demonstrate that the proposed approach effectively reveals decision-relevant evidence and improves explanation quality in terms of semantic consistency and spatial grounding. Across VGG-16, ResNet-50, and ResNet-101, the BBox scores reach 64.85%, 77.02%, and 77.26%, respectively. Compared with competing methods, the proposed approach improves mean top-5 textual attribute accuracy by an average of 1.7% and top-ranked textual attribute accuracy by 2.2%.