Publications

AI Governance in Health Care: What In-House Counsel and Compliance Teams Need to Know Now

June 8, 2026

As artificial intelligence (AI) becomes embedded across clinical and administrative workflows, health care organizations face growing pressure to establish governance frameworks that address compliance, legal, accreditation, and patient-safety considerations. AI adoption in health care reached 85 percent by the end of 2024, with the sector adopting AI 2.2 times faster than the broader economy.¹ Ambient documentation alone commanded $600 million in spending in 2025, and 66 percent of physicians reported using health AI in 2024 – a 78 percent increase from 2023.²

Yet this rapid adoption is outpacing governance framework, a troubling concern. According to recent data, 63 percent of organizations have no AI-governance policies in place, shadow AI is present in 40 percent of hospitals, and only 29 percent of providers are even aware of their organization's main AI policies. The risks are not theoretical; the myriad of issues are generating litigation, regulatory enforcement actions, and patient-safety events. Hallucinations, miscommunication to patients, clinician deskilling, privacy issues, bias, shadow AI, and data-security vulnerabilities are among the numerous issues surfacing in clinical environments. For organizations that have not yet established governance infrastructure, the risk profile grows exponentially as AI use expands without corresponding oversight and clear communication to workforce members.

This alert provides a framework for understanding the current governance landscape, the significance of the new Coalition for Health AI/Joint Commission standards, and practical steps for translating these standards into organizational action.

When AI Goes Wrong: The Case for Governance

Recent real-world incidents illustrate the risks of deploying AI without adequate governance infrastructure and underscore why this topic warrants attention from legal and compliance leadership.

Putative class actions now allege that health care providers illegally recorded patient visits using AI-powered ambient scribes without proper consent, asserting claims under state wiretap, invasion-of-privacy, and medical-confidentiality statutes.³ Statutory damages assessed per encounter or per violation mean exposure can scale quickly. Separately, Pennsylvania filed the first state enforcement action against Character.AI for the unauthorized practice of medicine after a chatbot falsely claimed to be a licensed psychiatrist and provided a fabricated license number, signaling that states are actively using existing professional-licensing and consumer-protection laws to police AI conduct.⁴ In clinical documentation, the literature consistently describes a dual risk: AI models produce errors at meaningful rates, while clinicians subject to "automation complacency" or "automation bias" often fail to detect or correct them.⁵ Together, these developments underscore why human-in-the-loop review for patient-impacting AI and robust consent infrastructure are governance baselines that should be communicated to all workforce members.

The CHAI-Joint Commission Partnership: A New Standard of Care

The collaboration between the Coalition for Health AI (CHAI) and The Joint Commission (TJC) represents one of the more significant developments in health care AI governance.⁶ In September 2025, the two organizations published joint guidance on the Responsible Use of AI in Healthcare (RUAIH). The RUAIH draws on the National Institute of Standards and Technology (NIST) AI Risk Management Framework (AI RMF 1.0), translating NIST's sector-neutral approach into health care-specific best practices. This is the first formal framework from a U.S. health care accreditation body designed to assist organizations with integrating AI safely and ethically.⁷ TJC has since launched a voluntary RUAIH Certification program focused on organizational governance rather than individual AI products used by health care organizations. Health care organizations obtaining certification would ostensibly be able to provide documented, third-party-validated evidence of responsible AI practices that may be relevant in litigation, regulatory, or accreditation contexts.⁸ As industry norms consolidate around these standards, organizations without comparable frameworks may face heightened scrutiny.

In May 2026, CHAI enhanced the RUAIH by releasing governance playbooks translating RUAIH principles into actionable implementation steps with baseline controls tailored by organizational size.⁹ Organizations seeking to demonstrate reasonable AI-governance practices often align with the NIST AI RMF, a voluntary standard increasingly referenced by federal agencies and courts as evidence of responsible AI deployment. The RUAIH builds directly on NIST's core risk-management principles – governance, risk mapping, measurement, and ongoing management – while tailoring implementation to the clinical, operational, and regulatory realities of health care delivery.¹⁰ Health care organizations can apply the NIST AI RMF as their enterprise-wide AI-governance foundation while using RUAIH and CHAI's playbooks to address clinical-workflow integration, patient-safety considerations, and health care-specific regulatory requirements.

The Regulatory Environment

Although no comprehensive federal AI legislation exists, health care organizations remain subject to a growing body of federal and state laws that apply to AI use, including privacy, wiretapping, anti-discrimination, consumer-protection, professional-licensing, and medical-device statutes and regulations.

Federal Developments

Executive Direction: The Trump administration has prioritized AI dominance largely through deregulation, including executive orders removing bureaucratic constraints (January 2025),¹¹ targeting state-law barriers (December 2025),¹² and ordering the strengthening of critical infrastructure – specifically including rural hospitals – against AI-related cyber threats (June 2026).¹³ Despite the administration's efforts to preempt certain state laws through executive power, state consumer-protection and licensing laws remain in place and may not be subject to preemption under draft legislation. Further, existing federal laws and regulations remain in place, including HIPAA and Section 1557 of the ACA, resulting in a complex state and federal landscape.
Emerging AI Products-Liability Frameworks: Both federal and state legislative proposals are moving toward applying traditional products-liability principles to AI systems – a development with significant implications for health care organizations that develop AI solutions in-house. At the federal level, the bipartisan AI LEAD Act (S. 2937), introduced in September 2025 by Senators Durbin and Hawley, would establish federal causes of action for design defects, failure to warn, and breach of express warranty against AI developers. The March 2026 Trump America AI Act discussion draft goes further, imposing a duty of care on AI developers to prevent foreseeable harm and establishing that deployers who "substantially modify" an AI system or "intentionally misuse" it may be treated as developers for liability purposes – with joint and several liability where both contribute to harm. For academic medical centers and health systems that develop, fine-tune, or substantially customize AI tools for clinical use, these frameworks raise threshold questions about whether the organization may be considered a "developer" subject to heightened liability exposure. Organizations should assess existing quality-management, product-safety, and risk-disclosure practices against these emerging frameworks and ensure vendor contracts appropriately allocate liability for AI-related claims.
FDA Classification and Oversight: The Food and Drug Administration (FDA) has authorized more than 1,400 AI-enabled medical devices as of March 2026.¹⁴ The January 2026 Clinical Decision Support Software Guidance clarifies when software qualifies for exemption from device regulation under the four-criteria test in Section 520(o)(1)(E) of the Federal Food, Drug, and Cosmetic Act (FD&C Act).¹⁵ The 2026 revision exercises enforcement discretion for tools providing a single, clinically appropriate recommendation but continues treating time-critical decision-making as higher risk. For academic medical centers developing AI clinical solutions in-house, classification as regulated Software as a Medical Device (SaMD) versus exempt Clinical Decision Support (CDS) is a threshold question that should be addressed early and often as capabilities evolve to avoid unintended regulatory exposure.¹⁶ The research, clinical, or operational classification also has significant implications beyond FDA oversight: IRB oversight may be required if patient data is used for algorithm development in ways that meet the Common Rule definition of research, even if the ultimate product is intended for clinical use. Organizations should consider developing clear classification criteria and engaging IRB and research-compliance personnel early in the development lifecycle. For regulated SaMD, Predetermined Change Control Plans allow pre-specified AI updates without new premarket submissions.¹⁷ FDA clearance does not replace local validation and monitoring.
Algorithm Transparency (HTI-1): In 2024, the Office of the National Coordinator (ONC) established the first federal requirement for AI transparency in certified health IT, though future applicability may be limited under the proposed HTI-5.¹⁸
CMS/FDA Demonstration Models: The CMS Advancing Chronic Care with Effective, Scalable Solutions (ACCESS) Model and Technology-Enabled Meaningful Patient Outcomes (TEMPO) Pilot test outcome-aligned payments for digital-health tools.¹⁹ The Wasteful and Inappropriate Service Reduction (WISeR) Model uses AI and machine learning for prior authorizations.²⁰
Proposed Updates to the HIPAA Security Rule: In early 2025, the U.S. Department of Health and Human Services proposed updates to the HIPAA Security Rule in a Notice of Proposed Rulemaking (NPRM).²¹ While the regulatory changes have not been finalized, the NPRM included notable developments that signal important considerations for regulated entities. For example, the preamble notes the importance of considering AI models as part of an organization's routine risk-management process and documenting how AI processes, stores, and outputs protected health information.

State Legislative Trends

With 45 of 50 states having introduced AI legislation as of early 2026,²² state activity is accelerating. Key themes include:

Clinical AI Oversight and Patient Disclosure: Multiple states have enacted laws addressing AI use in clinical contexts, generally requiring clinician oversight of AI outputs and patient disclosure or consent.²³ Texas requires practitioners to disclose AI use for diagnostic purposes and review all AI-created records.²⁴ Utah requires "prominent disclosure" of generative AI use in medical decisions.²⁵ California's AB 489 (effective January 2027) prohibits AI systems from using terms that imply possession of a health care license.²⁶
Payer and Utilization-Management AI: A growing number of states are prohibiting AI-only prior-authorization denials without human physician review. In 2025, Arizona, Illinois, Maryland, Nebraska, and Texas enacted such laws.²⁷ Indiana broke new ground by prohibiting insurers from using AI as the sole basis to downcode claims, a trend now reflected in bills across seven states.
Mental Health Chatbots and AI Companions: Following several high-profile incidents, several states, including California, New York, Utah, and Maine, enacted laws requiring disclosure that users are interacting with AI, protocols for detecting suicidal ideation, and crisis-referral mechanisms.²⁸ Illinois prohibits licensed therapy providers from using AI beyond administrative and supplementary support without written patient consent.²⁹
Ambient Listening and Wiretap Compliance: The ambient-scribe lawsuits illustrate how existing state privacy, wiretapping, and medical-confidentiality laws may apply to AI tools. Maine now requires mental health professionals to obtain patient consent before using ambient-listening or AI-powered recording tools.³⁰ Arizona requires documented informed consent before providing services involving AI (effective January 2027).³¹
Broad AI Transparency and High-Risk AI: States continue to pursue comprehensive AI legislation addressing algorithmic discrimination and high-risk AI systems. Colorado's original AI Act (SB 24-205) was repealed and replaced by SB 26-189, signed May 14, 2026, which takes effect January 1, 2027. The replacement law narrows the scope from "high-risk AI systems" to "covered automated decision-making technology," shifts compliance obligations from broad governance requirements toward targeted consumer disclosures and human-review rights, and exempts HIPAA-covered entities and FDA-regulated medical devices from most obligations. Other states, including California, New York, and Texas, have introduced or advanced AI bills addressing transparency, impact assessments, or algorithmic accountability, though none has yet enacted a comparably comprehensive framework.³²
Federal Preemption Tensions: The December 2025 executive order established a Department of Justice AI Litigation Task Force to challenge state AI laws. Despite these pressures, states have continued to legislate, and compliance with existing state requirements remains essential regardless of how preemption questions are ultimately resolved.³³

Practical Action Steps for In-House Counsel and Compliance Teams

Key Priorities

Consent and Patient-Facing Transparency: Health care organizations deploying any patient-facing AI – ambient documentation tools, chatbots, portal auto-replies, automated communications – should review patient intake forms, informed consent documents, and patient-facing communications, including existing website privacy policies and terms of use, to ensure they address AI. For ambient tools specifically, organizations should evaluate whether patients are clearly informed that recording will occur; whether consent is obtained, documented, and honored for all individuals present; and whether applicable state wiretap or medical confidentiality laws require authorization beyond HIPAA. Most vendor agreements place consent, notification, and compliance obligations on the health care organization, not the vendor. Health care organizations in all-party consent states or those subject to specific state AI disclosure requirements should pay particular attention to the adequacy of existing disclosures.
Address Shadow AI: Shadow AI, i.e., the unknown or unsanctioned use of AI tools within an organization without vetting, knowledge, or approval, is a growing concern. Many organizations are unaware of their workforce's adoption of AI tools or of their vendors' use of AI. Ignorance is not bliss. Without clear AI acceptable-use policies and workforce education, including a list of sanctioned and prohibited AI tools, organizations lack the ability to control what data enters AI systems, monitor for unauthorized disclosures, or hold workforce members accountable for policy violations. Similarly, without contractual limitations on vendor AI use, organizations may face vicarious liability, breach-notification obligations, and regulatory enforcement exposure for AI activities occurring within their enterprise that they neither authorized nor knew existed. The absence of these guardrails creates significant legal and operational risk: organizations could be deemed to have constructively consented to data processing, waived objections to AI-generated outputs, or failed to meet reasonable-safeguards standards under HIPAA, state privacy laws, and emerging AI-specific regulations. It is estimated that Shadow AI adds an average of $670,000 to breach costs.³⁴ With shadow AI already present in 40 percent of hospitals, health care organizations need to deploy acceptable-use policies, maintain an approved-tools inventory with single sign-on access controls, implement data-loss-prevention banners, and establish audit trails.³⁵ Governance frameworks must account for the reality that clinicians and staff will use AI tools outside of sanctioned channels.
Consider Lifecycle Governance – Intake Through Sunsetting: Effective governance extends beyond committee formation to encompass end-to-end lifecycle management. This includes formal intake and risk tiering prior to negotiation and implementation, deployment with proportionate monitoring throughout the life of the tool, change-control processes requiring approval for model updates, and defined criteria for sunsetting tools when they underperform, drift, or become noncompliant. Sunsetting decisions should address data-retention and deletion schedules and designated record-set boundaries. Health care organizations, particularly academic medical centers and health systems engaged in AI clinical research and development, should also address research-governance considerations, including: (i) whether training AI models on patient data constitutes "research" requiring IRB review, or falls under quality-improvement or clinical-operations exemptions; (ii) whether existing HIPAA authorizations or consents address secondary use of data for AI development; (iii) the interplay among the Common Rule, HIPAA, and state privacy laws when patient data is used to develop AI tools; (iv) data-access and governance frameworks that appropriately segment research, clinical, and vendor access; and (v) whether and how research-compliance frameworks apply when AI is developed internally versus acquired from vendors. Governance artifacts should be generated at every stage, including validation summaries, monitoring records, change-control approvals, training logs, and incident reports, and should serve as documented evidence of reasonable care.

Key Takeaways

The convergence of rapid AI adoption, emerging litigation, and the CHAI/TJC governance framework presents a clear opportunity for health care organizations to establish governance practices that protect patients, reduce legal exposure, and position the organization for the evolving accreditation landscape.

Start with Governance and Scale Guardrails by Risk: Establish a governance structure aligned with RUAIH, implement lifecycle management from intake through sunsetting, and maintain human oversight of patient-facing and care-impacting AI outputs.

Guardrails should be scaled based on proximity to patient care:
- Patient-Facing or Care-Impacting Outputs (High Risk): Guardrails should include mandatory human review, disclosure, near-real-time monitoring, incident reporting, and rollback criteria
- Documentation/Ambient Scribe Outputs (Moderate Risk): Guardrails should include verification steps, random audits, deskilling mitigation, change-control checkpoints, and standardized consent workflows.
- Claims/Appeals Drafting Outputs (Lower Risk): Guardrails should include performance metrics, audit logging, and use-case boundaries to prevent purpose drift.
Prioritize Consent and Transparency Across All Patient-Facing AI: The ambient scribe lawsuits and the Character.AI enforcement action demonstrate that existing state privacy, wiretapping, medical confidentiality, and professional licensing laws may apply to AI tools, and that statutory damages can scale quickly. Organizations should review patient intake forms, consent documents, and patient-facing communications to ensure they address AI use.
Strengthen Vendor Contracts and Validate Locally: Vendor contracts should define permissible uses of inputs, outputs, logs, and performance data, with prohibitions on reidentification and unauthorized secondary use. Require service-level agreements, validation commitments, update notifications, model documentation, and the ability to pause or roll back deployments. Ensure Business Associate Agreements (BAAs) and Data Use Agreements (DUAs) are in place alongside audit rights, incident-cooperation duties, and indemnities covering privacy, security, intellectual property, and regulatory violations. A critically important step is to maintain local validation regardless of FDA clearance status. Health care organizations should also reassess existing vendor relationships to confirm the vendor's current AI use aligns with evolving governance standards.
Document the Governance Process: Governance artifacts, including validation summaries, bias checks, monitoring records, training logs, and incident reports, can serve as valuable evidence that the health care organization exercised reasonable care in litigation, regulatory, and accreditation contexts.
Monitor the Regulatory Landscape: With 45 states legislating, federal preemption tensions unresolved, and FDA guidance evolving, organizations operating across multiple jurisdictions should track developments and maintain compliance with applicable requirements.
Address Research-Related AI Considerations Early: Academic medical centers and health systems engaged in AI research and development face distinct governance challenges. These include navigating complex data-use questions (e.g., whether patient data can be used for algorithm training under existing consents or authorizations, or whether secondary use requires new consent or a waiver), determining when IRB oversight is triggered (research versus quality improvement versus clinical operations), and ensuring appropriate data-access controls across research, clinical, and vendor environments. These organizations should establish clear pathways for classifying AI projects and engage research-compliance, privacy, and IRB functions early, particularly when AI development involves patient data, even if the intended use is ultimately clinical or operational rather than traditional research.

As health care organizations continue to evaluate, deploy, and govern AI tools, legal and Compliance, leadership will play a critical role in shaping responsible, defensible adoption strategies. Baker Donelson's AI and Health Care Systems team works with providers, health systems, and other stakeholders to navigate the complex and rapidly evolving landscape of AI governance, regulatory compliance, risk management, and implementation. If you have questions about how these developments may affect your organization or would like assistance assessing your current AI governance framework, we encourage you to reach out to Baker Donelson's AI and Health Care Systems team.

------

¹ Healthcare Analytics Statistics 2026: Key Data and Trends.

² Adoption of artificial intelligence in healthcare: survey of health system priorities, successes, and challenges - PMC; AI in EHR: How Ambient Documentation and Clinical AI Are Transforming Healthcare in 2026 | EHR Source; https://www.demandsage.com/ai-in-healthcare-stats/.

³ Saucedo v. Sharp Healthcare, No. 25CU063632C (Cal. Super. Ct., San Diego Cnty.); Lisota v. Heartland Dental, LLC, No. 1:25-cv-07518 (N.D. Ill.). See also Patient sues Sharp HealthCare over ambient AI use, Becker's Hosp. Rev. (Dec. 15, 2025), https://www.beckershospitalreview.com/legal-regulatory-issues/patient-sues-sharp-healthcare-over-ambient-ai-use/; Heartland Dental hit with class-action lawsuit over AI use, Becker's Dental Rev. (July 11, 2025), https://www.beckersdental.com/dso-dpms/heartland-dental-hit-with-class-action-lawsuit-over-ai-use/.

⁴ Shapiro Administration Sues Character.AI Over Fake Medical Claims | Commonwealth of Pennsylvania.

⁵ https://www.fda.gov/media/182871/download; https://www.jmir.org/2024/1/e53164; https://www.researchgate.net/publication/391722773_A_framework_to_assess_clinical_safety_and_hallucination_rates_of_LLMs_for_medical_text_summarisation.

⁶ https://www.jointcommission.org/en-us/knowledge-library/news/2025-09-jc-and-chai-release-initial-guidance-to-support-responsible-ai-adoption.

⁷ Responsible Use of AI in Healthcare - Joint Commission.

⁸ https://www.jointcommission.org/en-us/certification/responsible-use-of-ai-in-healthcare.

⁹ Coalition for Health AI (CHAI) Releases Comprehensive Governance Playbooks to Streamline AI Implementation for Health Systems | CHAI; https://www.chai.org/workgroup/cross-cutting/ai-governance.

¹⁰ https://nvlpubs.nist.gov/nistpubs/ai/NIST.AI.100-1.pdf.

¹¹ https://www.whitehouse.gov/presidential-actions/2025/01/removing-barriers-to-american-leadership-in-artificial-intelligence/.

¹² https://www.whitehouse.gov/presidential-actions/2025/12/eliminating-state-law-obstruction-of-national-artificial-intelligence-policy/.

¹³ https://www.whitehouse.gov/presidential-actions/2026/06/promoting-advanced-artificial-intelligence-innovation-and-security/.

¹⁴ https://www.fda.gov/medical-devices/software-medical-device-samd/artificial-intelligence-enabled-medical-devices.

¹⁵ https://www.fda.gov/regulatory-information/search-fda-guidance-documents/clinical-decision-support-software.