HIPAA-compliant AI and LLM architecture: what PHI in prompts actually requires

The flow nobody drew on the diagram

A team building for healthcare gets to the AI feature the same way everyone else does. There is a clinical note, or a patient message thread, or a claims record, and someone says the obvious thing: we could summarize that. Or answer questions about it. Or let an agent triage it. The engineering is straightforward — assemble a prompt, call an API, render the response — and it works on the first afternoon.

What did not happen on that afternoon is anyone drawing the data flow. Because if you draw it, the picture is unambiguous: patient data left your infrastructure, crossed the public internet, and was processed by a third-party company's servers. That is a disclosure of protected health information to a vendor. It is the same category of event as sending records to a billing processor or a transcription service. The fact that it happened inside a JSON body labeled messages, in a function that looks like every other API call in the codebase, does not change its regulatory character.

This is the core misread, and it is nearly universal in teams new to healthcare. PHI in a prompt does not feel like a data transfer. It feels like a parameter. The record was already in memory; you are just passing part of it to a function. But the function is a network call to another company, and HIPAA does not care about the ergonomics of the SDK.

Two clarifications make the scope tractable. First, this is about identifiable health information — data that relates to health, care, or payment and that identifies an individual or could reasonably be used to. A prompt containing a patient name and a diagnosis is obviously in scope. A prompt containing a free-text clinical narrative with no name is very often still in scope, for reasons covered below. Second, the obligation attaches to the flow, not to the feature. One AI feature in your product may be entirely out of scope while another is squarely in it, and the useful discipline is to classify each call rather than reasoning about "our AI" as a single undifferentiated thing.

The BAA is the gate, and it is not always available

Under HIPAA, when a covered entity or business associate discloses PHI to a vendor that will create, receive, maintain, or transmit it on their behalf, that vendor is a business associate and there must be a Business Associate Agreement in place first. That is the mechanism by which obligations follow the data. Without it, the disclosure itself is the problem, independent of whether anything bad happens to the data afterward.

So the sequencing is strict: the BAA is signed before any PHI is sent, not after the feature ships. There is no retroactive cure. If PHI went to a vendor with no agreement in place, that happened, and it is an incident to be assessed rather than a paperwork gap to be tidied up later.

The part teams do not expect is that the agreement is frequently not available on the plan they are on. This is the same shape as the platform-level gotcha covered in the HIPAA-compliant Next.js on Vercel guide: the vendor supports HIPAA workloads, the marketing page says so, and the support only exists on a commercial tier that a self-serve signup does not put you on. A developer who signed up with a credit card in twenty seconds is, by default, on terms that do not include a BAA. Everything works identically. Nothing warns you.

Two further points about scope that get missed. A BAA covers the services named in it, not everything the vendor sells — if you sign for one product surface and then start sending PHI through a different one from the same company, verify that it is in scope. And the model provider is one of many vendors in the path. The prompt was assembled by code running on a hosting platform, the conversation was stored in a database, the request may have passed through a gateway, an error may have been captured by an exception tracker, and metrics may have gone to an observability vendor. Every one of those that touches PHI needs its own agreement. The audit that catches teams out is rarely the model provider — it is the error tracker nobody thought of as a data processor.

Provider reality, and why this page will not tell you the answer

Major model providers — Anthropic, OpenAI, Google, and the hyperscalers reselling models through their own cloud platforms — do have paths for customers handling PHI. What varies, and varies constantly, is which plan, which product surface, and which contracting route gets you there. The general pattern is that BAA coverage sits with enterprise or specific commercial agreements rather than with default self-serve API access, and that it requires an actual signed document rather than a checkbox.

I am deliberately not writing down which provider offers what at which tier, and you should be suspicious of any page that does. These terms are revised, tiers are renamed, and the specific answer that was correct when an article was written is a liability when read a year later. Getting this wrong is not a stale-blog-post problem; it is a compliance problem with real consequences for a real patient's data.

The verification is not hard, it just has to actually be done, per provider, before launch:

1. Confirm the provider offers a BAA for the specific product and API surface you will use — not for the company in general.
2. Confirm which plan or contracting route it requires, and confirm that the account holding your production API key is on it. This is where prototypes die: the company has an enterprise agreement, and the key in the environment variable belongs to someone's personal account.
3. Get it signed, executed, and filed where your compliance process can find it during a review or a questionnaire.
4. Read what it says about permitted uses, retention, subcontractors, and breach notification. The agreement is where the answers live, and it can differ from what the public documentation implies.
5. Re-verify on plan changes, provider changes, and material terms updates. Put a recurring review on the calendar rather than trusting that today's answer is permanent.

If a provider you want does not offer a BAA on any reachable terms, that is a design constraint, not an obstacle to route around. The options are to use a different provider, to route through a cloud platform where you already have coverage, to run a model in infrastructure you control, or to change the architecture so PHI never reaches that call — which is the next section.

Architecture that reduces what is exposed

A BAA makes a flow permissible. It does not make it wise. The independent goal is to minimize how much PHI leaves your control, because every record that never crosses the boundary is one that cannot be involved in a vendor's incident.

Send only what the task requires. The default implementation passes the whole record because the whole record is what the ORM returned. Most tasks do not need it. A model summarizing the last three visits does not need the full patient history, the insurance identifiers, the address, or the guarantor details. Building the prompt from an explicit projection — fields you deliberately selected — rather than from a serialized object is a small change with a large effect on exposure, and it also makes the flow describable in a risk assessment. "We send these seven fields" is an answer. "We send the patient object" is not.

De-identify or redact where the task allows it. Some tasks genuinely do not need identifiers: classifying urgency, extracting a structured medication list, drafting a template. For these, running the input through a de-identification or redaction step before the model call can move the flow out of scope entirely — but only if the de-identification is real. HIPAA has a defined standard here, and "we regex out names and dates" is not it. Free-text clinical narrative is the hard case: a paragraph mentioning a rare diagnosis, a named facility, a specific date, and an occupation can be re-identifying while containing no field you would think to strip. If you are relying on de-identification to take a flow out of scope, verify it against the actual standard, and understand that a partial redaction still leaves you with PHI and still needs a BAA.

Classify every LLM call as in-scope or out-of-scope, explicitly. Most healthcare products have both. The clinical summarizer touches PHI; the marketing-copy helper in the admin console does not. Write the classification down, per call site, and keep it near the code. This is the artifact that lets you answer a security questionnaire quickly, and more importantly, it is what prevents drift — the out-of-scope helper that someone later points at a patient record because it was the endpoint that already existed.

Treat prompt and response logs as PHI stores. LLM features generate enormous pressure to log everything, because prompt/response pairs are how you debug quality, evaluate changes, and build eval sets. That instinct is correct and the logs are genuinely valuable. But a log of prompts containing PHI is a PHI database, and it needs the same treatment as any other: access control, encryption at rest, a written retention period that your infrastructure actually enforces, and an audit trail of who read it. Apply the same discipline described in SOC 2 audit logging for developers — an append-only record of actor, action, resource, and outcome — to access of the prompt store itself, because "which engineer read which patient's conversation" is exactly the question you will be asked. The specific thing to avoid is the reflex to pipe prompts into whichever observability or error-tracking tool the team already uses, because that tool was chosen for convenience and almost certainly has no BAA.

Keep the boundary where you can see it. Route model calls through a single internal service or module rather than letting every feature call the provider SDK directly. One choke point is where you enforce field projection, redaction, logging policy, and provider selection — and it is the only way to answer "what data can reach the model" without reading the whole codebase. Scattered SDK calls make that question unanswerable, which is the same structural problem as scattered audit logging.

Retention and training: verify, do not assume

Two questions decide what happens to your data after the response comes back, and both must be answered from the terms currently binding your account rather than from memory or from an article.

Does the provider use API inputs to train models? Practice differs across providers and across product surfaces from the same provider — consumer chat products and API access have frequently been governed by different terms — and policies have been revised more than once. Never carry an assumption across a provider boundary or across a plan change.

How long is request data retained, and where? Providers commonly retain request and response data for some period for abuse monitoring, safety, or operational reasons, and that period, along with whether zero-retention or reduced-retention options exist, varies. This matters because it determines how long PHI you sent exists on infrastructure you do not control, which is a question your own risk analysis and your customers' security reviews will both ask.

The BAA and the enterprise terms are where the binding answers live, and they can be more restrictive than the public documentation. Read them, record the answers in your risk documentation with a date, and re-check on any plan or provider change. What makes this section different from most compliance advice is the volatility: this is not a thing you learn once. Anything you write down about a provider's training or retention behavior should carry the date you verified it.

The failure mode: the prototype key that shipped

Here is how this actually goes wrong in practice, and it is nearly always the same story.

An engineer builds the AI feature during a spike. Speed is the point, so they use whatever key they have — a personal account, a free tier, a shared team key created before anyone thought about healthcare. The demo works, everyone is happy, and the feature moves onto the roadmap. Then the work becomes about quality, latency, and cost. The compliance posture of the API key is not a thing anyone revisits, because it was never a decision anyone consciously made; it was the default that let the prototype start.

Six months later the feature is live, real patient data flows through it daily, and the account it flows to has no BAA — and possibly is not even a company account. Nobody was negligent. There was simply no point in the process where "is this vendor relationship covered" was a question with an owner.

The fix is procedural, and it is cheap if you install it early:

• Treat the model provider as a vendor from the first line of code. It goes on the vendor inventory alongside the database host and the payment processor, not into a category called "an API we call."
• Separate prototype and production credentials structurally. Production keys come from the covered account and live only in production configuration. A key that cannot be traced to a specific account under a specific agreement does not get promoted.
• Add a launch gate that names the flows. Before any feature touching patient data ships, someone signs off on: which PHI leaves the system, to which vendor, under which executed agreement, retained for how long, logged where. Four sentences. The value is entirely in the fact that someone has to write them.
• Re-run the check on change. New model, new provider, new plan, new observability tool, new place prompts get stored — each is a moment to re-ask the question rather than inherit the previous answer.

What "done" looks like

For every AI feature in the product, you can say in a paragraph: whether it touches PHI, what fields it sends, which vendor receives them, under which signed agreement, what that agreement says about retention and training, where prompts and responses are stored, who can read that store, and how long they are kept. You can point to the single code path that enforces it, and to the date each provider claim was last verified.

That is a smaller document than teams expect and a harder one to write than they expect, because writing it surfaces the flows nobody drew. The reason to write it early is the same reason to design audit logging and access control early rather than retrofitting them: the AI feature is easy to build and the compliance envelope around it is not, and the order you do them in determines whether the second one is a week of work or a rebuild. That argument generalizes beyond health data — it is the case for building compliance-ready architecture from the start — and where a product carries both HIPAA and SOC 2 obligations, the overlaps and the sharp divergences are mapped in SOC 2 vs. HIPAA.