FRANCE Law and Practice Contributed by: Liliana Eskenazi, Julie Ernewein and Pauline Lecrenais, Fréget Glaser et Associés
tems (eg, diagnosis, monitoring) allow human control at all key stages. Roles of Healthcare Professionals When Using AI Tools Clinicians must remain the main decision-makers. When using AI, they are required to: • understand the capabilities and limitations of the high-risk AI system; • be vigilant with respect to automation bias; • correctly interpret the outputs of the high-risk AI system; and • be able to override, disregard, or opt not to use the AI system when necessary. Level of Human Involvement The level of human control depends on risk level as follows. • High-risk AI requires strict human control and inter - vention mechanisms. • Low-risk/administrative AI requires transparency and accountability; CNIL advises enabling users to raise concerns, typically via a DPO. 6. Data Governance in Healthcare AI 6.1 Training Data Requirements For healthcare AI systems, governance obligations are imposed by both the AI Act for high-risk AI and by the MDR/IVDR, as follows. • Datasets : Datasets used for training, validation and testing must be relevant, sufficiently representa - tive, as error-free as possible, and complete with respect to the intended purpose of the AI system/ medical device. The French government empha - sises this need for access to national databases that are “high-quality and representative ” to obtain “ robust” data. • Bias : Any potential bias in the datasets must be addressed if it is likely to affect people’s health or safety, negatively impact fundamental rights, or result in discrimination prohibited under EU law. This includes managing unintended bias, dataset
drift, and identifying subgroups where the model may underperform. • Personal data : The transparency obligation con - cerning the original purpose of data collection must be respected, and stringent data governance practices must be in place to maintain data integ - rity, and address privacy and security concerns. The CNIL provides guidelines explaining how to process training data legally. In support of these requirements, the European Com - mission will issue horizontal guidelines, and a com - mittee is developing harmonised standards on data and bias. 6.2 Secondary Use of Health Data Rules Governing the Secondary Use of Health Data in France The secondary use of health data for innovation activi - ties and the training of AI algorithms, etc, is primarily governed by Chapter 4 of the EHDS and the GDPR/ French Data Protection Act. The cross-border infra - structure HealthData@EU is developed to meet the requirements of the EHDS. Based on these regulations, the French government is expected to publish national strategies for the sec - ondary use of health data aimed at developing the secondary use of health data and building trustworthy AI. The CNIL also plays a key role in this area and has issued several recommendations and decisions France has established the “Health Data Hub” as a centralised platform for accessing data from the National Health Data System (SNDS) (health data originating from national health sources). This plat - form enables the secondary use of health data for AI algorithm training/testing, research, etc. However, the use of health data may require prior authorisation from the CNIL. For instance, in July 2024, the CNIL authorised a hos - pital to process SNDS data to develop a decision- support algorithm for admission to intensive care regarding secondary data use. Health Data Hub and SNDS
76
CHAMBERS.COM
Powered by FlippingBook