USA Trends and Developments Contributed by: Nadia de la Houssaye, Andy Lee, Jason Loring and Graham Ryan, Jones Walker LLP
AI tools for clinical documentation, decision support and administrative functions. The use of AI-powered transcription and documenta - tion tools has emerged as a particular area of con - cern. Healthcare providers utilising AI systems for automated note-taking during patient encounters face potential HIPAA violations if proper safeguards are not implemented. These systems often require access to comprehensive patient information to func - tion effectively, yet traditional HIPAA standards may conflict with AI systems’ need for extensive datasets to optimise performance. AI tools must be designed to access and use only the protected health informa - tion (PHI) strictly necessary for their purpose, even though AI models often seek comprehensive datasets to achieve their full potential. The proposed Department of HHS regulations issued in January 2025 attempt to address some of these concerns by requiring covered entities to include AI tools in their risk analysis and risk management com - pliance activities. These requirements mandate that organisations conduct vulnerability scanning at least every six months and penetration testing annually, recognising that AI systems introduce new vectors for potential data breaches and unauthorised access. Business associate agreements (BAAs) have become increasingly complex as organisations attempt to address AI-specific risks. These agreements must now encompass algorithm updates, data retention policies and security measures for ML processes while ensuring that AI vendors processing protected health information operate under robust contractual frameworks that specify permissible data uses and required safeguards. Healthcare organisations must ensure that AI vendors processing PHI operate under robust BAAs that specify permissible data uses and necessary security measures, and account for AI-spe - cific risks related to algorithm updates, data retention policies and other ML processes. Algorithmic Bias and Health Equity Concerns The potential for algorithmic bias in healthcare AI sys - tems has emerged as one of the most significant ethi - cal and legal challenges facing the industry. A 2024 review of 692 AI- and ML-enabled FDA-approved
medical devices revealed troubling gaps in demo - graphic representation, with only 3.6% of approvals reporting race and ethnicity data, 99.1% providing no socioeconomic information and 81.6% failing to report study subject ages. These data gaps have profound implications for health equity, as AI systems trained on non-representative datasets may perpetuate or exacerbate existing healthcare disparities. Training data quality and rep - resentativeness significantly – and inevitably – impact AI system performance across diverse patient popula - tions. The challenge is particularly acute given the rap - id changes in federal enforcement priorities regarding diversity, equity and inclusion (DEI) initiatives. While the April 2024 HHS final rule under Section 1557 of the Affordable Care Act established requirements for healthcare entities to ensure AI systems do not discriminate against protected classes, the current administration’s opposition to DEI initiatives has cre - ated uncertainty about enforcement mechanisms and compliance expectations. Given the rapid turnabout in executive-branch policy towards DEI and anti- discrimination initiatives, it remains to be seen how federal healthcare AI regulations with respect to bias and fairness will be affected. Healthcare organisations are increasingly implement - ing systematic bias testing and mitigation strategies throughout the AI life cycle, focusing on technical validation, promoting health equity, ensuring algo - rithmic transparency, engaging patient communities, identifying fairness issues and trade-offs, and main - taining accountability for equitable outcomes. AI sys - tem developers have, until recently, faced increasing regulatory pressure to ensure training datasets ade - quately represent diverse patient populations. Most healthcare AI developers and practitioners continue to maintain that relevant characteristics, including age, gender, sex, race and ethnicity, should be appropri - ately represented and tracked in clinical studies to ensure that results can be reasonably generalised to the intended use populations. However, these efforts often occur without clear regulatory guidance or standardised methodologies for bias detection and remediation. Special attention
144 CHAMBERS.COM
Powered by FlippingBook