Federal regulators misunderstand the value of transparency rules for clinical AI.
By the end of 2025, the U.S. Food and Drug Administration (FDA) had authorized more than 1,450 artificial intelligence-enabled medical devices, and the agency cleared 295 devices in 2025 alone. Two-thirds of clinicians now use artificial intelligence (AI) in their work, according to the American Medical Association. Yet fewer than 2 percent of those cleared devices were supported by randomized clinical trials, and studies of large language models used for clinical decision support estimate hallucination rates of 8 to 20 percent.
Against that backdrop, the federal government is about to make oversight weaker, not stronger.
In January 2026, the Department of Health and Human Services released a proposed rule known as HTI-5. The rule would scale back the criteria for certifying AI for use in “decision support interventions”—which, just one year earlier, constituted the first federal transparency requirement for predictive AI in certified electronic health records.
The criterion, finalized in late 2023, required developers of certified health information technology to disclose 31 source attributes—the standardized disclosure categories, defined in the rule, describing how each tool was built, validated, and maintained—for every AI-powered tool – such as, training data composition, fairness testing, external validation, performance metrics, and a maintenance schedule. The criterion also obligated developers to perform intervention risk management, a structured process for identifying and mitigating potential harms across the tool’s lifecycle, covering validity, reliability, fairness, intelligibility, safety, security, and privacy. Together, these requirements were the closest that the United States had come to a model-card mandate—a requirement that all AI developers publish a standardized “nutrition label” describing how a model was trained, tested, and intended to be used, for clinical AI.
HTI-5 would remove the model-card transparency requirements, the source-attribute disclosures, and the intervention risk management provisions. The agency justifies the rollback by citing a lack of demonstrated clinical utility and alignment with the White House’s broader deregulatory posture on AI. Some industry observers welcome the change as a way to remove barriers to AI development. Others warn that, if finalized, HTI-5 would eliminate the only federal transparency mandate covering clinical AI inside certified electronic health records—the systems that support more than 96 percent of U.S. hospitals.
The proposed rollback rests on a misreading of the problem. Regulators have framed transparency as a paperwork burden whose costs exceed its benefits. But the source-attribute regime was never meant primarily to inform individual clinicians at the point of care. Its function was institutional. Hospital procurement officers, compliance leads, and quality committees needed a standardized disclosure floor to compare vendors, audit deployed AI models, and meet their own internal governance obligations. Removing that floor does not relieve a burden; it shifts the burden onto the buyer, who now must construct bespoke diligence processes for every algorithm in every workflow.
That shift collides with two other trends.
The first trend is that FDA’s own oversight, although broad in scope—more than 1,450 AI-enabled devices have been authorized to date—was never designed to govern the changing performance of AI products after they have entered the market. A 2025 analysis of cleared AI devices found that most 510(k) summaries, the public evidence manufacturers file to obtain FDA clearance through the agency’s most common premarket pathway, lack details on study design, sample sizes, and demographic representation. FDA acknowledged the gap and requested public comment on real-world performance measurement and drift detection. That work is still preliminary. In the interim, the source-attribute disclosure regime filled part of the post-market accountability void by forcing developers to publish a validation and update schedule. Removing it leaves the void exposed.
The second trend is that much of the clinical AI now entering hospital workflows is not regulated by FDA at all. Non-device clinical decision support tools, internally developed models, and generative AI assistants frequently fall outside the agency’s jurisdiction. The Decision Support Interventions criterion was deliberately drafted to include both FDA-regulated software and non-device predictive tools supplied through certified health IT. Eliminating that umbrella means that the fastest-growing category of clinical AI—generative tools built on foundation models—operates in a regulatory blind spot at exactly the moment hallucination rates remain measurable in the double digits.
A more defensible course would preserve the source-attribute disclosures and reframe them as one layer of a continuous-monitoring regime. The same nine disclosure categories that HTI-1 codified—purpose, development inputs, fairness process, external validation, performance measures, ongoing maintenance, update schedule, cautioned out-of-scope use, and intervention details—map cleanly onto the AI risk management framework published by the National Institute of Standards and Technology and onto the post-deployment surveillance questions that FDA itself is now asking. Rather than retiring the model card, regulators could require that those attributes be machine-readable, versioned, and exposed through a standard application programming interface so that hospital governance committees, payers, and accreditors can verify them continuously.
That approach treats AI governance as infrastructure rather than as a one-time disclosure. It puts the marginal cost of compliance on the developer who controls the model, where it belongs, and gives the rest of the health care system a common substrate for oversight. It also acknowledges what the past three years of deployment have demonstrated: Clinical AI may not fail at the moment of regulatory clearance, but it may fail quietly, over months, as populations shift and models drift.
The comment period on HTI-5 has closed, but the rule is not yet final. Before retiring the country’s only mandatory transparency floor for clinical AI, regulators should consider whether the burden they are removing is in fact a burden – or whether it is the scaffolding on which the rest of the governance system was just beginning to be built.
