Governance Is the Accelerator, Not the Brake

There is a reflex in most organisations to treat governance as the thing that slows AI down. Legal gets involved, the project stalls, everyone blames compliance. The reflex is wrong, and the deployments that actually reached production prove it. Governance done early is not friction. It is the reason the project survives contact with the real world.

Answer the hard question in the architecture

Wells Fargo's assistant handled 245 million interactions in a single year, and later passed a billion cumulative, without a single piece of personally identifiable information reaching the language model. The design is the lesson. Customer speech is transcribed and scrubbed locally; an internal model detects and tokenises the sensitive parts; only then is an external model called, and only to work out intent. All the sensitive computation stays inside the bank. Their CIO described the architecture plainly: they are the filters in front of and behind the model.

That is governance as an engineering decision, not a policy memo. And it did not slow the system down. It is what let the system exist at all in a regulated context.

JPMorgan took a different route to the same place. Faced with the data-leakage risk of public chatbots, it banned them and built LLM Suite, an internal portal giving roughly a quarter of a million employees governed access to frontier models behind the bank's own controls. The ban was not anti-AI. It was the precondition for AI at scale.

Merck's internal front door, GPTeal, does the same for pharma: every query and output passes through a layer that encrypts and vets it so internal data never leaves the company. Multiple models sit behind one governed door.

The pattern is consistent. Where governance is part of the build, AI scales. Where it is an afterthought, it does not.

What happens when governance is optional

The instructive failures are the organisations that skipped this step or refused to.

One large bank's analytics leadership looked at letting a model speak directly to customers and declined. The risk of a confidently wrong answer reaching a customer was judged too high. That is not a failure of nerve. It is a governance assessment that correctly concluded the system was not safe to deploy as designed. The lesson is not "be braver." It is "design the system so the assessment comes back yes."

The legal profession ran the uncontrolled experiment for everyone. More than 1,300 cases worldwide have now been flagged by courts for AI-generated hallucinations: fabricated citations and invented case law, submitted as fact. Sanctions are climbing into six figures. The detail that matters most for any regulated function is that the tools were often not free chatbots but purpose-built, enterprise legal AI. The tool did not determine the risk. The absence of a verification step did. That is a governance gap, not a technology one.

The pharma stakes

This is where the patient-safety lens stops being abstract. When the output of an AI system is submitted to an external authority as fact, whether a court or a regulator reviewing a submission, being wrong is not a quality metric. In pharma, an unverified claim that reaches a regulatory dossier, or a generated narrative that misstates an adverse event, has consequences measured in patient harm and lost trust, not rework hours.

Governance is what makes the output traceable: who generated it, from what source, reviewed by whom, against which version of the regulation. 21 CFR Part 11 asks for exactly this: audit trails, controls, signatures. GDPR asks where the data went. The EU AI Act asks you to classify the risk before you deploy. None of these are obstacles invented to slow you down. They are the structure that lets you move at all.

The takeaway

Build the governance into the system: detect and redact sensitive data before the model sees it, keep the audit trail, require human verification on anything that reaches an external authority, and classify risk before deployment rather than after the incident. Do that and governance becomes what it should be, the thing that lets you take the corner at speed. Skip it and you will meet it again at month nine, in the post-mortem.