Abstract
Visual recognition models have achieved unprecedented success in various tasks. While researchers aim to understand the underlying mechanisms of these models, the growing demand for deployment in safety-critical areas like autonomous driving and medical diagnostics has accelerated the development of eXplainable AI (XAI). Distinct from generic XAI, visual recognition XAI is positioned at the intersection of vision and language, which represent the two most fundamental human modalities and form the cornerstones of multimodal intelligence. This paper provides a systematic survey of XAI in visual recognition by establishing a multi-dimensional taxonomy from a human-centered perspective based on intent, object, presentation, and methodology. Beyond categorization, we summarize critical evaluation desiderata and metrics, conducting an extensive qualitative assessment across different categories and demonstrating quantitative benchmarks within specific dimensions. Furthermore, we explore the interpretability of Multimodal Large Language Models and practical applications, identifying emerging trends and opportunities. By synthesizing these diverse perspectives, this survey provides an insightful roadmap to inspire future research on the interpretability of visual recognition models.
Taxonomy
This section shows the proposed taxonomy and corresponding method groups of XAI in visual recognition, with a three-level hierarchy: dimensions, groups, and subgroups.
Four dimensions are defined to categorize existing methods: intent, object, presentation, and methodology.
- Intent: What is the purpose of bringing in interpretability?
- Object: What does the generated explanation focus on?
- Presentation: What does the generated explanation look like?
- Methodology: How is the explanation generated?
Tags for Paper List
The table below introduces various badges used to tag papers in terms of the four key dimensions. The badges help to efficiently identify research papers based on their interpretability approaches.
| Badge | Description |
|---|---|
 |
Intent is passive. Methods that explain already trained models by revealing their recognition process. |
 |
Intent is active. Methods that integrate interpretability during model construction, making the process inherently interpretable. |
 |
Object is local. Explanation focused on individual samples, such as diagnostic suggestions for each patient. |
 |
Object is semilocal. Explanation that highlights common characteristics within a class of samples. |
 |
Object is global. Explanation of the entire model's decision rules, often category-independent. |
 |
Presentation is scalar. Explanation presented in quantitative forms, such as numerical scores. |
 |
Presentation is attention. Used to highlight important features or regions contributing to a decision. |
 |
Presentation is structured. Explanation involving structured representations such as graphs. |
 |
Presentation is semantic unit. Explanation decomposed into human-understandable semantic concepts. |
 |
Presentation is exemplar. Explanation through examples that illustrate specific model behaviors. |
 |
Methodology is association. Methods that model correlations to show the relationships and patterns between inputs and outputs. |
 |
Methodology is intervention. Methods predicting outcomes after making active changes to the model or its inputs. |
 |
Methodology is conterfactual. Simulates alternative scenarios by perturbing inputs to explore the potential outcomes that could arise under different conditions. |
The full paper list with tags can be found in our GitHub repository .
Metrics
In this section we provide the full tables of metrics (i.e. Table 4 in the paper).
| Type | Metric | Undst. | Fidel. | Conti. | Effic. | Description |
|---|---|---|---|---|---|---|
| Localization Metrics | AOPC [132] | ✔ | ✔ | Measure explanation quality by the confidence drop when perturbing salient regions | ||
| Pointing Game (PG) [133] | ✔ | ✔ | Measure localization accuracy by calculating the hit rate of the attention map's peak point falling within ground-truth regions | |||
| Deletion, Insertion [134] | ✔ | ✔ | Track class probability changes as the most important pixels are removed and added | |||
| MCS, IDR [135] | ✔ | Introduce BAM metrics to evaluate attribution methods across models and inputs | ||||
| IIR [135] | ✔ | |||||
| Bias of Attribution Map, Unexplainable Feature [133] | ✔ | Develop four metrics for attribution maps to enable ground-truth-free evaluation | ||||
| Robustness, Mutual Verification [133] | ✔ | |||||
| Faithfulness (F) [136] | ✔ | Measure Pearson correlation between pixel relevance and changes after perturbation | ||||
| HI score [137] | ✔ | ✔ | ✔ | Assess heatmaps by rewarding meaningful activations and penalizing irrelevant ones | ||
| POMPOM [138] | ✔ | ✔ | Calculate the percentage of meaningful pixels leakage outside target regions | |||
| FP Error, FN Error [139] | ✔ | Quantify pixel-wise errors between saliency maps and ground truth masks | ||||
| iAUC, IntIoSR [140] | ✔ | Modify PG to use intersection ratio between salient area and ground truth mask | ||||
| CS, SENSmax [140] | ✔ | |||||
| GTC, SC, IoU [141] | ✔ | Quantify overlap between model’s saliency map and human-defined ground truth | ||||
| MeGe, ReCo [125] | ✔ | ✔ | Assess generalizability and consistency of explanations for quality and trustworthiness | |||
| RMA, RRA [142] | ✔ | Propose mass accuracy and rank accuracy for heatmap evaluation by CLEVR-XAI dataset | ||||
| AR, AP [68] | ✔ | Measure how much of the relevant parts of test images are considered relevant by a model | ||||
| Semantic Metrics | Completeness Score [76] | ✔ | Measure how well concept scores can reconstruct the model's original predictions | |||
| Nfgconcept, Nbgconcept, λ ratio [77] | ✔ | Quantify “dark-matter” visual concepts encoded during the knowledge distillation | ||||
| ρ value [77] | ✔ | |||||
| Dmean, Dstd [77] | ✔ | |||||
| Fidc, Fidr [78] | ✔ | Measure fidelity of concept-based explanations for classification and regression models | ||||
| Faithfulness, Fidelity, Intervention on Concepts (IoC) [80] | ✔ | Metrics to assess faithfulness, fidelity, explanation error, and concept intervention | ||||
| Explanation Error [80] | ✔ | ✔ | ||||
| AIPD, AIFD [66] | ✔ | Compute average inter-class distance for prototypes and nearest local representations | ||||
| Factuality, Groundability [143] | ✔ | ✔ | Measure concept accuracy and vision-language alignment with human interpretations | |||
| CDR [55] | ✔ | Summarize participants' responses in the user study of discovered concepts | ||||
| CC, MIC [55] | ✔ | |||||
| TCPC, TOPC [144] | ✔ | Measure concept weight stability and output prediction stability under perturbation | ||||
| CUE [145] | ✔ | ✔ | Use both average length and quantity of concepts to evaluate concepts’ efficiency | |||
| RC, IC [82] | ✔ | ✔ | Evaluate concept importance and correctness during the concept extraction process |
[55] B. Wang, L. Li, Y. Nakashima, and H. Nagahara, “Learning Bottleneck Concepts in Image Classification,” in CVPR, 2023, pp. 10962-10971.
[66] C. Wang et al., “Learning Support and Trivial Prototypes for Interpretable Image Classification,” in ICCV, 2023, pp. 2062-2072.
[68] M. T. Hagos, N. Belton, K. M. Curran, and B. Mac Namee, “Distance-Aware Explanation Based Learning,” in 2023 IEEE 35th International Conference on Tools with Artificial Intelligence (ICTAI), 2023, pp. 279-286.
[76] C.-K. Yeh, B. Kim, S. Arik, C.-L. Li, T. Pfister, and P. Ravikumar, “On Completeness-Aware Concept-Based Explanations in Deep Neural Networks,” NeurIPS, vol. 33, pp. 20554-20565, 2020.
[77] X. Cheng, Z. Rao, Y. Chen, and Q. Zhang, “Explaining Knowledge Distillation by Quantifying the Knowledge,” in CVPR, 2020, pp. 12925-12935.
[78] R. Zhang, P. Madumal, T. Miller, K. A. Ehinger, and B. I. Rubinstein, “Invertible Concept-Based Explanations for CNN Models with Non-negative Concept Activation Vectors,” in AAAI, vol. 35, 2021, pp. 11682-11690.
[80] A. Sarkar, D. Vijaykeerthy, A. Sarkar, and V. N. Balasubramanian, “A Framework for Learning Ante-hoc Explainable Models Via Concepts,” in CVPR, 2022, pp. 10286-10295.
[82] A. F. Posada-Moreno, N. Surya, and S. Trimpe, “ECLAD: Extracting Concepts with Local Aggregated Descriptors,” Pattern Recognition, vol. 147, p. 110146, 2024.
[125] T. Fel, D. Vigouroux, R. Cad`ene, and T. Serre, “How Good Is Your Explanation? Algorithmic Stability Measures to Assess the Quality of Explanations for Deep Neural Networks,” in WACV, 2022, pp. 720-730.
[132] W. Samek, A. Binder, G. Montavon, S. Lapuschkin, and K.-R. M¨ uller, “Evaluating the Visualization of What a Deep Neural Network Has Learned,” IEEE Transactions on Neural Networks and Learning Systems, vol. 28, no. 11, pp. 2660-2673, 2016.
[133] J. Zhang, S. A. Bargal, Z. Lin, J. Brandt, X. Shen, and S. Sclaroff, “Top-Down Neural Attention by Excitation Backprop,” International Journal of Computer Vision, vol. 126, no. 10, pp. 1084-1102, 2018.
[134] V. Petsiuk, A. Das, and K. Saenko, “RISE: Randomized Input Sampling for Explanation of Black-Box Models,” arXiv:1806.07421, 2018.
[135] M. Yang and B. Kim, “Benchmarking Attribution Methods with Relative Feature Importance,” arXiv:1907.09701, 2019.
[136] R. Tomsett, D. Harborne, S. Chakraborty, P. Gurram, and A. Preece, “Sanity Checks for Saliency Metrics,” in AAAI, vol. 34, 2020, pp. 6021-6029.
[137] A. Theodorus, M. Nauta, and C. Seifert, “Evaluating CNN Interpretability on Sketch Classification,” in Twelfth International Conference on Machine Vision (ICMV 2019), vol. 11433, 2020, pp. 475-482.
[138] I. Rio-Torto, K. Fernandes, and L. F. Teixeira, “Understanding the Decisions of CNNs: An In-Model Approach,” Pattern Recognition Letters, vol. 133, pp. 373-380, 2020.
[139] S. Mohseni, J. E. Block, and E. Ragan, “Quantitative Evaluation of Machine Learning Explanations: A Human-Grounded Benchmark,” in Proceedings of the 26th International Conference on Intelligent User Interfaces, 2021, pp. 22-31.
[140] X.-H. Li, Y. Shi, H. Li,W. Bai, C. C. Cao, and L. Chen, “An Experimental Study of Quantitative Evaluations on Saliency Methods,” in Proceedings of the 27th ACM SIGKDD Conference on Knowledge Discovery & Data Mining, 2021, pp. 3200-3208.
[141] A. Boggust, B. Hoover, A. Satyanarayan, and H. Strobelt, “Shared Interest: Measuring Human-AI Alignment to Identify Recurring Patterns in Model Behavior,” in Proceedings of the 2022 CHI Conference on Human Factors in Computing Systems, 2022, pp. 1-17.
[142] L. Arras, A. Osman, and W. Samek, “CLEVR-XAI: A Benchmark Dataset for the Ground Truth Evaluation of Neural Network Explanations,” Information Fusion, vol. 81, pp. 14-40, 2022.
[143] Y. Yang, A. Panagopoulou, S. Zhou, D. Jin, C. Callison-Burch, and M. Yatskar, “Language in a Bottle: Language Model Guided Concept Bottlenecks for Interpretable Image Classification,” in CVPR, 2023, pp. 19187-19197.
[144] S. Lai, L. Hu, J. Wang, L. Berti-Equille, and D. Wang, “Faithful Vision-Language Interpretation Via Concept Bottleneck Models,” in ICLR, 2023.
[145] C. Shang, S. Zhou, H. Zhang, X. Ni, Y. Yang, and Y. Wang, “Incremental Residual Concept Bottleneck Models,” in CVPR, 2024, pp. 11030-11040.
Related Survey
This section summarizes the related surveys of our work, organized into three groups.
Generic AI Models
XAI techniques, classification methods, evaluation metrics, and future challenges for various AI models.
| Type | Reference | Year | Literature Coverage | Description |
|---|---|---|---|---|
| XAI Surveys on Generic AI Models | abusitta2024survey | 2024 | 2018-2024 | Propose a classification of XAI techniques, emphasizing applications in cybersecurity and future challenges |
| schwalbe2024comprehensive | 2024 | 2017-2024 | Propose a unified taxonomy of XAI methods and provide use-case-oriented insights | |
| hassija2024interpreting | 2024 | 2016-2024 | Review XAI models, evaluation metrics, challenges, and trends to enhance transparency | |
| mersha2024explainable | 2024 | 2015-2024 | Deliver a review of XAI, examining methods, applications, and principles to enhance transparency | |
| mazhar2024survey | 2024 | 2018-2023 | Survey XAI methods for deep learning, investigating techniques, tools, and future research directions | |
| chander2024toward | 2024 | 2017-2023 | Discuss trustworthy AI, focusing on robustness and explainability to enhance safety and reliability | |
| chamola2023review | 2023 | 2019-2023 | Evaluate trustworthy and explainable AI, focusing on transparency and reliability across industries | |
| yang2023survey | 2023 | 2017-2023 | Review XAI research, focusing on key areas, taxonomies, applications, and future directions | |
| saranya2023systematic | 2023 | 2020-2022 | Conduct a systematic review of XAI applications across various fields and suggest directions for future research | |
| saeed2023explainable | 2023 | 2017-2022 | Conduct a systematic meta-survey of XAI challenges and future research across development phases | |
| samek2023explainable | 2023 | 2017-2022 | Introduce XAI techniques for deep neural networks, emphasizing challenges and future directions | |
| dwivedi2023explainable | 2023 | 2017-2021 | Survey XAI techniques, classify approaches, and guide framework selection for interpretable AI systems | |
| holzinger2022explainable | 2022 | 2019-2022 | Present a concise overview of 17 key XAI methods for beginners | |
| saleem2022explaining | 2022 | 2019-2022 | Review global interpretation methods in XAI, highlighting strengths, weaknesses, and future opportunities | |
| ding2022explainability | 2022 | 2016-2022 | Discuss XAI principles, taxonomy, evaluation, and challenges, suggesting future research directions | |
| ras2022explainable | 2022 | 2014-2022 | Guide readers through explainable deep learning, discussing key methods, evaluations, and future directions | |
| rudin2022interpretable | 2022 | 2013-2022 | Identify 10 technical challenge areas and provide historical and background context | |
| hanif2021survey | 2021 | 2017-2021 | Survey XAI techniques, emphasizing the need for transparent and responsible AI to build trust | |
| zhang2021survey | 2021 | 2015-2021 | Propose a taxonomy of neural network interpretability based on engagement, explanation type, and focus | |
| samek2021explaining | 2021 | 2014-2021 | Offer an overview of post-hoc explanations, evaluate interpretability methods, and demonstrate XAI applications | |
| islam2021explainable | 2021 | 2018-2020 | Analyze XAI methods for credit default prediction, compare advantages, and suggest future research | |
| fan2021interpretability | 2021 | 2014-2020 | Propose a taxonomy and discuss applications and future directions | |
| das2020opportunities | 2020 | 2017-2020 | Deliver an overview of XAI techniques, including taxonomy, methods, principles, and evaluation | |
| huang2020survey | 2020 | 2015-2020 | Review research on making DNNs safe, concentrating on verification, testing, attacks, and interpretability | |
| arrieta2020explainable | 2020 | 2007-2020 | Explore the importance of explainability in AI and present a taxonomy of XAI techniques | |
| ferreira2020people | 2020 | 2018-2019 | Explore AI explainability from CS and HCI perspectives, concentrating on goals, recipients, and context |
Vision-Related Fields
XAI research in visual tasks, visualization techniques, architectures, multimodal models, and reinforcement learning.
| Type | Reference | Year | Literature Coverage | Description |
|---|---|---|---|---|
| Visual Task | gipivskis2024explainable | 2024 | 2017-2024 | Survey XAI in semantic segmentation, categorizing evaluation metrics and future challenges |
| poeta2023concept | 2023 | 2018-2023 | Review concept-based XAI methods, offering taxonomy, guidelines, and evaluations for the future | |
| bai2021explainable | 2021 | 2015-2021 | Introduce papers on explainable deep learning, efficiency, and robustness in pattern recognition | |
| Visualization | baniecki2024adversarial | 2024 | 2017-2024 | Survey adversarial attacks on XAI, outlining security challenges and suggesting future directions |
| alicioglu2022survey | 2022 | 2018-2021 | Review trends and challenges in visual analytics for XAI, focusing on deep learning model interpretation | |
| seifert2017visualizations | 2017 | 2014-2016 | Survey visualization methods for DNNs, focusing on insights gained in computer vision | |
| Architecture | fantozzi2024explainability | 2024 | 2017-2024 | Survey transformer explainability, categorizing by components, applications, and visualization |
| kashefi2023explainability | 2023 | 2021-2023 | Review explainability methods for vision transformers, categorizing approaches and evaluation criteria | |
| stassin2023explainability | 2023 | 2019-2023 | Evaluate XAI for vision transformers, highlighting challenges in metrics, convergence, and adaptation | |
| Multimodal | kazmierczak2025explainability | 2025 | 2017-2025 | Survey integration of foundation models with explainable AI in the vision domain |
| dang2024explainable | 2024 | 2017-2024 | Analyze recent advances in Multimodal XAI, focusing on methods, datasets, and evaluation metrics | |
| rodis2024multimodal | 2024 | 2016-2024 | Survey interpretability of MLLMs, categorizing evaluations and future directions | |
| sun2024review | 2024 | 2015-2024 | Review research on interpretability of MLLMs, focusing on challenges, metrics, and future directions | |
| Reinforcement Learning | milani2024explainable | 2024 | 2017-2024 | Survey XRL techniques, introduce a taxonomy, and outline challenges and future directions |
| glanois2024survey | 2024 | 2015-2024 | Survey interpretability approaches in reinforcement learning, focusing on inputs, models, and decisions | |
| dazeley2023explainable | 2023 | 2015-2023 | Introduce the Causal XRL Framework, unify XRL research, and explore future Broad-XAI development | |
| puiutta2020explainable | 2020 | 2017-2019 | Survey XRL methods and highlight the need for interdisciplinary human-centered explanations | |
| Others | hu2024interpretable | 2024 | 2021-2024 | Review explainable clustering methods, emphasizing transparency and ethics in high-stakes applications |
| schneider2024explainable | 2024 | 2019-2024 | Review XAI in generative AI, addressing challenges, categories, criteria, and future research directions | |
| burkart2021survey | 2021 | 2015-2021 | Survey explainable supervised learning methods, highlighting principles, methodologies, and directions | |
| li2020survey | 2020 | 2015-2020 | Survey explanation methods, focusing on data-driven and knowledge-aware approaches, and applications |
Visual Applications
Applications of XAI in medical imaging, industrial manufacturing, smart cities, and cybersecurity.
| Type | Reference | Year | Literature Coverage | Description |
|---|---|---|---|---|
| Medical Imaging | bhati2024survey | 2024 | 2017-2024 | Survey interpretability and visualization techniques for deep learning in medical imaging |
| hou2024self | 2024 | 2017-2024 | Survey self-explainable AI for medical image analysis: methods, challenges, and future research | |
| klauschen2024toward | 2024 | 2015-2023 | Explore diagnostic pathology: classification, biomarker quantification, transparency, XAI solutions | |
| borys2023explainable | 2023 | 2020-2022 | Review non-saliency XAI methods in medical imaging for clinicians | |
| chaddad2023survey | 2023 | 2019-2022 | Survey XAI techniques, categorize challenges, and suggest future directions in medical imaging | |
| nazir2023survey | 2023 | 2017-2022 | Survey XAI for medical imaging diagnostics, analyze challenges, and propose future directions | |
| sheu2022survey | 2022 | 2018-2022 | Survey medical XAI: evaluations, case studies, human-machine collaboration, and the future | |
| teng2022survey | 2022 | 2019-2021 | Review interpretability in medical diagnosis: methods, applications, challenges, and future directions | |
| jin2022explainable | 2022 | 2016-2021 | Review interpretability in healthcare: methods, advantages, applications, and future directions | |
| tjoa2020survey | 2020 | 2015-2020 | Categorize AI interpretability approaches to guide cautious application in medical practices | |
| pocevivciute2020survey | 2020 | 2015-2020 | Survey XAI in digital pathology: techniques, uncertainty estimation, and cross-disciplinary insights | |
| Industry/Manufacturing | naqvi2024survey | 2024 | 2001-2023 | Survey ontology-based and semantic-based XAI for transparent AI decisions in manufacturing |
| alexander2024interrogative | 2024 | 2020-2022 | Survey explainable AI applications in manufacturing, highlight research, and propose future directions | |
| li2023survey | 2023 | 2016-2023 | Survey explainable anomaly detection: techniques, taxonomy, ethics, and guidance for practitioners | |
| ahmed2022artificial | 2022 | 2018-2021 | Survey AI and XAI methods in Industry 4.0 for autonomous decision-making and transparency | |
| Smart City | javed2023survey | 2023 | 2018-2023 | Survey XAI developments in smart cities, focusing on use cases, challenges, and research directions |
| kok2023explainable | 2023 | 2018-2023 | Examine XAI in IoT: transparent models, challenges, foresee future directions, and classify studies | |
| jagatheesaperumal2022explainable | 2022 | 2018-2022 | Study XAI in IoT: assess frameworks, security, IoMT, IIoT, IoCT, edge XAI, and future directions | |
| Cybersecurity | rjoub2023survey | 2023 | 2020-2023 | Survey XAI in cybersecurity: approaches, challenges, limitations, and future research directions |
| charmet2022explainable | 2022 | 2018-2022 | Survey XAI in cybersecurity: applications, security concerns, challenges, and future directions | |
| capuano2022explainable | 2022 | 2018-2022 | Conduct in-depth study on XAI in cybersecurity: applications, challenges, methods, and the future | |
| Others | hohl2024opening | 2024 | 2017-2024 | Perform systematic review of XAI methods, trends, and challenges in Remote Sensing |
| alizadehsani2024explainable | 2024 | 2020-2023 | Review XAI methods in drug discovery, identify challenges, applications, and future directions |
BibTeX
@article{wan2025survey,
title={A Survey on Interpretability in Visual Recognition},
author={Wan, Qiyang and Gao, Chengzhi and Wang, Ruiping and Chen, Xilin},
journal={arXiv preprint arXiv:2507.11099},
year={2025},
url={https://vipl-vsu.github.io/xai-recognition/}
}