Sooner than you think, a procurement lead will slide four pages across the table before a pilot can begin. After lengthy negotiations, when you think you are finally close, one more step appears. A standard AI attachment, she calls it. At first, you're not concerned. By page two, you realize this is not the standard security review you were expecting.

The attachment contains AI-specific provisions unlike anything in a standard security review. It prohibits you from using customer data, including anonymized and de-identified versions, to train or improve your AI. It requires you to warrant, contractually guarantee, that your AI has been assessed for bias and that documented controls are in place.

Your compliance documentation sits in your laptop bag. Some of it applies. Most of it was built for the conversation you expected to have, not the one you are actually in.

The Claim

AI has added a documentation layer to healthcare procurement that existing compliance frameworks were never designed to produce. Two years ago that was a prediction. Today it is documented in five independent sources: binding contracts at Kaiser Permanente, federally funded guidance for the country's smallest community health clinics, joint guidance from the Joint Commission whose standards govern Medicare and Medicaid eligibility for more than 23,000 organizations, survey data from 650 hospitals, and in the procurement conversations happening right now in rooms most founders are not prepared for.

The Four-Question Documentation Gap

1. How did you train your model, and on whose data?

Training data provenance is now a formal procurement requirement, and at the largest health systems, a contractual obligation from two directions. Kaiser's AI attachment prohibits using customer data, including anonymized and de-identified versions, to train or improve the AI (Section 2.2). It separately requires documentation of how the training data was collected, its sources, and the vendor's rights to use it, available upon request (Section 2.7). The Community Health Care Association of New York State (CHCANYS) requests an AI Bill of Materials documenting all model components, training data sources, and third-party dependencies, and the Joint Commission and Coalition for Health AI's (CHAI) Applied Model Card has dedicated fields for development data characterization and input data source. For founders building on foundation models, this question has a genuinely complicated answer. You may not know exactly what your model was trained on. Healthcare buyers are asking anyway.

2. How do you know your model isn't biased against our patients?

Bias assessment is no longer a recommendation. It is a warranty. Kaiser's AI attachment requires vendors to warrant that their AI has been assessed for bias, inaccuracies, and unfairness, and that documented controls are in place to mitigate each (Section 2.4(d)). The California Telehealth Resource Center (CTRC), a federally funded resource for community health clinics and critical access hospitals, asks specifically which fairness metrics, such as demographic parity, have been applied. The Joint Commission and CHAI guidance name the CHAI Applied Model Card as the recommended tool for bias and risk assessment during procurement, a structured disclosure of how bias was evaluated and for which populations. Most founders have thought about bias. Few have documented it in a form that satisfies the requirements of a contractual warranty. Those are different things.

3. What happens when you change your model?

Buyers are now putting model change requirements into contracts. Kaiser's AI attachment requires written notice of any material change to the AI model, explicitly including model drift (Section 2.7). Black Book Research, a healthcare IT market research firm that surveys hospital technology leaders, recommends a 30-day notice for material model updates, 24-hour emergency notification, and version pinning rights. The part most founders miss: this obligation extends to changes your model provider makes. If your foundation model is updated upstream by OpenAI, Anthropic, or whoever powers your core product, your outputs may change without your initiation. You may not have been notified yourself.

4. Who is liable when the AI is wrong?

Nobody has a good answer yet, and everybody knows it. CHCANYS, which represents some of the smallest and most resource-constrained healthcare organizations in the country, asks directly: who bears responsibility when the AI contributes to adverse outcomes? The CTRC asks it two ways: what is your internal accountability policy, and does your errors and omissions insurance cover AI-specific failures? Black Book Research recommends that hospitals include caps on hallucination rates and clinical quality SLAs in vendor contracts, with credits or termination rights if those caps are breached. The contracts are beginning to address it. The founders who get ahead of this conversation will be in a different negotiating position than those who wait.

The model card is the clearest example of what healthcare governance now requires from vendors. Most founders know what one is, but the question is whether theirs covers clinical validation data, demographic subgroup performance metrics, bias mitigation methodology, and ongoing monitoring documentation. The gap is not awareness. It is specificity.

Expert Signal — Matt Loar, Attorney

Former engineer and practicing attorney with experience advising on technology contracts in regulated industries.

“It's important not to overlook the fact that Kaiser requires vendors not just to make representations about testing their AI solutions and implementing controls, but to "warrant" these as well. Black's Law Dictionary, 8th Edition, defines a "warranty" in contract as "[a]n express or implied promise that something in furtherance of the contract is guaranteed by one of the contracting parties." This is important because you are not simply stating the facts you believe to be true, but accepting liability if such belief is mistaken. Are you making representations about how an open-source model was trained and tested based on the model card? If so, you are on the hook if the model card was inaccurate and your customer suffers loss as a result”

The So What

Three things to do in the next 30 days:

  1. Map your documentation against the four questions in this issue, documented, partial, or no answer. One hour. Your governance roadmap.

  2. Start with the CHAI Applied Model Card, specifically the training data characterization section. Note every field you cannot complete. Those gaps are what to build next. It costs nothing to produce.

  3. Before your next healthcare conversation, ask: what governance documentation do you currently ask AI vendors to provide? Their answer tells you where their program is. Your ability to ask tells them where you are.

Governance by design is not a compliance exercise. It is a commercial positioning decision. Build toward it before you need it. Founders who wait tend to find out in a room where the deal is already at risk.

The numbers from Black Book's 2026 survey of 650 hospitals make the case simply: 80% of hospital leaders say vendor AI claims are hard to verify without formal governance, and health systems with a governance program are 2.1 times more likely to hit ROI within 12 months of a pilot.

What’s Next

The Signal AI covers one governance or procurement question founders are encountering in real healthcare deals each issue, grounded in primary sources and our honest read on where things are heading.

Coming up: what AI agents mean for governance when your product starts making autonomous decisions, the new CHAI and Joint Commission certification quietly becoming the procurement baseline, and the AI liability insurance gap most founders discover too late.

Reply and tell us what governance question you are running into right now. The most common answers shape what we cover next.

If you found this issue useful, send it to one founder who needs to read it.If this was forwarded to you, subscribe at thesignalainews.beehiiv.com

Primary Sources

The research shows what the research shows. What you do with it is your call.

— The Signal AI

If this was forwarded to you, subscribe at thesignal.ai