A user-centric exploration of transparency, explanations, and trust in multi-model and single-model decision support systems

Abstract

Advancements in artificial intelligence (AI) have increased the demand for interpretable decision-making processes. This study explores Explainable AI (XAI) by examining the relationship between machine learning model types and explanatory mechanisms. An ensemble of models—XGBoost, Neural Network, Naive Bayes, Decision Tree, and K-Nearest Neighbour—was developed, offering a balance between transparency and accuracy. The models were evaluated using a car assessment task, where participants made decisions and were then shown predictions from single and ensemble models, sometimes accompanied by explanations like LIME and SHAP. Results show that the ensemble model outperformed most constituent models in accuracy and positively influenced user trust, particularly in scenarios of appropriate compliance and incorrect predictions. While explanations had limited effect on trust, the level of agreement within the ensemble significantly influenced user behaviour. Preconceptions such as familiarity and risk appetite also affected compliance. SHAP's waterfall plot emerged as the preferred explanation type. This research contributes methodologically with a novel ensemble model balancing accuracy and interpretability, and empirically by deepening understanding of human-AI interaction. Practical recommendations are provided for presenting explanations to improve user trust in machine learning applications.