USA – CALIFORNIA Trends and Developments Contributed by: Lowell Brown and Douglas Grimm, ArentFox Schiff LLP
• Using patient data to train AI models without proper authorisation or de-identification could con - stitute a CMIA violation. • The California Attorney General’s legal advisories reinforce these privacy obligations for healthcare AI. The advisories emphasise that AI applica - tions must adhere to the CCPA, CPRA, CMIA, and HIPAA requirements, where applicable. AI devel - opers and healthcare organisations deploying AI systems are required to: (a) limit the collection and use of personal data to what is reasonably necessary and proportion - ate; (b) obtain consumer consent where required and provide mechanisms for individuals to exercise their privacy rights; (c) ensure rigorous testing and validation of AI systems to prevent errors and reduce harm, including ensuring training data is free from biases that could compromise accuracy or fair - ness; (d) implement robust security measures to safe - guard patient data, consistent with state requirements; and (e) be transparent with patients about whether their information is used to train AI and how AI is utilised in decision-making. Algorithmic Bias and Fairness Requirements This previously arcane concept is now a particularly significant concern in the deployment of artificial intel - ligence in healthcare. California has taken a proac - tive stance to address algorithmic bias, recognising that biased AI systems can perpetuate or exacerbate existing health inequities. Assembly Bill 2885 (AB 2885), the Algorithmic Accountability Act, mandates that the Department of Technology conduct a comprehensive inventory of “high-risk automated decision systems” used or pro - posed by state agencies. After all, algorithms are simi - lar to recipes: They provide step-by-step instructions for accomplishing a complex task. Healthcare “reci - pes” are complex and carry profound consequences. These decision systems are deemed high-risk if they materially impact access to, or approval for, critical areas such as housing, education, employment, or healthcare.
The inventory requires a description of measures in place to mitigate risks, including the risk of inaccu - rate, unfairly discriminatory, or biased decisions. This includes performance metrics to gauge accuracy and risk assessments or audits for potential biases. The California Attorney General’s legal advisories explicitly address algorithmic bias within healthcare AI. The Attorney General emphasises that AI systems must align with state anti-discrimination laws. These laws prohibit discrimination based on protected char - acteristics such as sex, race, religion, or disability, and their applicability extends to AI systems, even if the discriminatory impact is unintentional. The Attorney General warns that AI systems making “less accurate” predictions about protected classes could be considered discriminatory, regardless of data availability. Healthcare entities are required to proactively design, acquire, and implement AI solutions that prevent past discrimination from being embedded or amplified by new technologies, and must maintain documenta - tion of their bias testing and mitigation efforts. This includes avoiding uses of AI that could lead to dis - criminatory outcomes, such as using past claims data to deny patient access or conducting cost-benefit analyses based on stereotypes that undervalue cer - tain patient populations. Informed Consent and Transparency Obligations The principles of Informed consent and transparency are woven into the fabric of healthcare. California’s regulatory framework is actively extending these prin - ciples to artificial intelligence. The state aims to ensure that patients are fully aware when AI is involved in their care and have the means to understand and question its role. The AI in Healthcare Act mandates specific disclosure requirements for health facilities, clinics, physicians’ offices, and group practices that use generative AI to communicate patient clinical information. Communications must include a prominent disclaimer indicating that generative AI produced the content. The disclaimer’s placement and format vary depend -
153 CHAMBERS.COM
Powered by FlippingBook