Healthcare AI, explainability, and the human-machine relationship: a (not so) novel practical challenge

This paper focuses on the lack of explainability that afflicts machine-learning-based AI systems applied in the field of healthcare. After a brief introduction to the topic, from both a technical and legal point of view, this work aims to assess the main consequences that the lack of explainability has on the human-machine relationship in clinical care, through a practical perspective. It then questions whether explainability is truly an objective worth seeking and, if so, to what extent, taking into account the current possible solutions.