USA Law and Practice Contributed by: Nadia de la Houssaye, Andy Lee, Jason Loring and Graham Ryan, Jones Walker LLP
intended patient population are represented; and ensure that results can be reasonably generalised to the intended use population. • Technical documentation: AI system developers must submit detailed technical documentation describing algorithm design, training methodolo - gies, validation processes and performance char - acteristics. • Risk assessment frameworks: Comprehensive risk assessments must include reviews of AI-specific hazards, including algorithmic bias, cybersecurity vulnerabilities and performance degradation over time. • Bias testing and mitigation: Regular testing and validation of AI tools and algorithms is required to ensure compliance with non-discrimination stand - ards. • Algorithmic transparency requirements: AI systems must provide sufficient transparency to enable healthcare providers and regulatory reviewers to understand system functionality, limitations and appropriate use cases. • Cybersecurity and data protection: The Consoli - dated Appropriations Act of 2023 added Section 524B to the FFDCA, requiring medical device manufacturers to include cybersecurity information in pre-market submissions. 3.3 Post-Market Surveillance Following market introduction, healthcare AI systems must continue to monitor performance and compli - ance. Key areas of concern include: • ongoing monitoring requirements – organisa - tions should implement multilayered approaches that include regular scanning for outdated code or anomalies in AI systems, potentially different behaviour in clinical practice compared to con - trolled development environments, key perfor - mance indicators, performance drift and potential bias or safety issues • adverse event reporting – adverse events and safety issues associated with AI system use must be reported through established FDA reporting; • algorithm update processes – post-market algo - rithm updates require ongoing evaluation to ensure safety and effectiveness. PCCPs ensure regulatory
oversight of significant changes while enabling sys - tematic updates within predefined parameters; and • real-world evidence collection – post-market surveillance programmes should be implemented to collect and analyse real-world evidence of AI system performance. 3.4 Enforcement Actions Non-compliance with relevant laws and regulations can be addressed in several ways. • FDA enforcement mechanisms: FDA enforcement actions include warning letters, product recalls, injunctions, marketing prohibitions, civil monetary penalties and criminal referrals. • HIPAA privacy and security enforcement: The OCR enforces HIPAA violations through civil monetary penalties and corrective action plans. The OCR also introduced its risk analysis initiative at the end of 2024, focusing OCR enforcement on entities that fail to properly conduct the required periodic security risk analysis (SRA). • State-level enforcement: State attorneys general increasingly enforce state-specific data privacy laws and consumer protection statutes against healthcare AI companies. • Professional licensing board actions: State medi - cal and professional licensing boards may take disciplinary action against healthcare providers for inappropriate AI use or failure to meet professional standards. 4. Liability and Risk in Healthcare AI 4.1 Liability Framework Healthcare AI liability generally operates within estab - lished medical malpractice frameworks that require the establishment of four key elements: duty of care, breach of that duty, causation and damages. When AI systems are involved in patient care, determining these elements becomes more complex. While a phy - sician must exercise the skill and knowledge normally possessed by other physicians, AI integration creates uncertainty about what constitutes reasonable care. Healthcare AI liability often involves multiple stake - holders, including healthcare providers, AI develop -
131 CHAMBERS.COM
Powered by FlippingBook