Validation of Continuously Learning AI/ML Systems in Medical Devices – A Scenario-based Analysis

Abstract

This paper discusses the use of continuously learning AI/ML based medical devices, i.e. devices which optimize their performance during the product’s lifetime. For such devices, a regulatory strategy was recently proposed by the US Food & Drug Administration (FDA). The paper analyzes the options this approach provides as well as potential shortcomings it may pose. In particular, it studies the proposed concept of automated validation for these devices. In this analysis, the assessment of the relationship between technical parameters and clinical effects is a main focus. This includes the association to potential risks as well the dependencies between the algorithmic outcomes and the clinical environment. Additionally, potential issues w.r.t. bias, explainability, and fairness of the algorithms are addressed. The paper uses application scenarios, where ML based devices are utilized in the intensive care unit (ICU). In summary, ML based medical devices and especially continuously learning devices still possess considerable challenges which should be addressed thoroughly. Regarding appropriate regulatory strategies, a deliberate approach is recommended which prioritizes the collection of sufficient experience with ML based devices over amplifying their use in a rather uncontrolled fashion.