Identifying and understanding significant change due to drift when assessing AI models in healthcare: a narrative review

Abstract

Artificial intelligence (AI) as a Medical Device (AIaMD) or medical devices that use AI algorithms—like any other medical device—must meet the requirements of medical device regulation. For regulatory purposes, the most relevant requirement is that the developer must provide evidence that the device performs as intended under normal conditions of use for its entire lifecycle. However, healthcare data are not static and underlying characteristics can change for many reasons (eg, the introduction of new technologies which improve measurement accuracy, changes in population demographics, etc). This ‘drift’ may lead to a change in performance overall or in certain subgroups in AI models. Models can be updated with new data if significant drift is identified, but in the context of AIaMD, this needs to be done transparently and within a robust regulatory framework. This paper reports on the consensus view of an expert working group hosted by the UK Medicines and Healthcare products Regulatory Agency (MHRA). It aims to highlight the challenges with identifying and assessing significant changes in the performance of a model and understanding the nature of a detected drift to preserve patient safety. We discuss distinct drift subtypes from a statistical perspective and highlight potential causes in the real world that could lead to significant changes to the performance of AI algorithms. We also outline the regulatory challenges associated with risk assessment and the characteristics of drift that are crucial to examine (such as speed and severity) to correctly address interventions and ensure the deployment of safe healthcare products on the market. Finally, we discuss a range of considerations to best identify, risk-assess and intervene for drift when assessing healthcare AI products.