Software & AI · Life sciences & manufacturing · Written and maintained by Haink’s AI adoption team · Updated August 2026 · 12 min read
Document extraction when the system itself has to be validated
What this page argues
- The hard part is not the extraction. It is that the system doing the extracting must itself be validated — and that is a constraint on architecture, not a document you write at the end.
- Cloud usually fails on change control, not security. A model updated on the provider's schedule is an uncontrolled change to a qualified system.
- Rules win here more often than anywhere else, because reproducibility is a requirement rather than a preference — and a rule is far cheaper to validate than a model.
- Version pinning and the evidence chain go in at the start. Retrofitted, they leave a hole covering every decision made before they existed.
- Air-gapped is a process as much as a rack. The hardware is the easy half; the transfer, approval and version-ownership procedures are what an auditor asks about.
This is not “AI in healthcare”
Search that phrase and you get diagnostics, medical imaging and predictive analytics — models that look at a patient and infer something. Those are real fields with real literature, and none of them is this.
The subject here is the regulated paperwork of manufacturing and clinical research: batch records, deviation reports, equipment logs, calibration certificates, protocols and their amendments, training records, case report forms. Documents that exist because a regulation says they must, that are reviewed because a regulation says they must be, and that are retained for years because a regulation says how long.
The distinction matters because it changes who the reader is and what the difficulty is. In diagnostics the difficulty is the model — can it see what a radiologist sees. Here the model is close to a solved problem: reading a scanned form and pulling out fields is ordinary document work, covered in document data capture methods. The difficulty is that in this environment the software is itself a regulated object.
What validation actually demands of the architecture
“Validated” gets used loosely. Concretely it means: documented evidence that the system does what its specification says, and keeps doing it. Five properties follow from that, and each one constrains how the system is built rather than how it is documented.
| Requirement | What it means for design | What breaks if it was not built in |
|---|---|---|
| Reproducibility | The same input produces the same output, always. No sampling, no temperature, no hosted endpoint whose behaviour can shift. Determinism is a design choice made at the start. | You cannot re-run a past decision to demonstrate the system was working. Every historical result becomes unverifiable. |
| Version pinning | Model weights, prompts, rules, schemas and the runtime are versioned artefacts with dates, held in a controlled repository — not “whatever the API served that day”. | You cannot state what produced a given result. The qualification package refers to a configuration that no longer exists. |
| Documented testing | A written test protocol executed before go-live against a defined data set, with expected results agreed in advance and deviations recorded. | There is nothing to show. Retrospective testing on data you have already seen is not qualification and will be read as such. |
| Change control | Every update — model, prompt, rule, dependency — enters through a procedure with impact assessment and, where warranted, requalification. Nothing deploys because someone merged it. | The system in production is not the system that was validated. This is the single most common finding in this area. |
| Traceability | From any output back to the inputs, the versions in effect, the intermediate evidence and the human who approved it — and forward again. | A result cannot be explained. The record proves an answer existed, not that it was right. |
Read the right-hand column and a pattern appears: every failure is retrospective. Nothing goes wrong at build time. It goes wrong months later, when someone asks a question the system was never designed to answer — and by then the cheap moment to have fixed it has passed by a year.
That is why the sequencing matters more here than in almost any other document project. Determinism, version pinning and the evidence chain cost very little on day one and cannot be genuinely added on day four hundred, because the period before they existed stays unrecoverable however much engineering you throw at it afterwards.
Why cloud usually fails — and it is not security
The assumed objection to a hosted AI service in pharma is confidentiality, and quality teams are used to answering it: encryption, regional hosting, a data processing agreement. That conversation is winnable and it is not the one that kills the project.
The one that kills it is change control.
There is no good answer to that question. The system in production is not the system that was qualified, the change was neither assessed nor approved, and you cannot even date it. It does not matter whether version B is better — an uncontrolled change to a qualified system is a finding on its own terms.
Three further properties of hosted services compound it:
- You cannot freeze the environment. Even where a provider offers pinned model versions, the surrounding service — tokenisation, safety filters, routing, defaults — moves underneath the pin.
- Deprecation is on their calendar. A version you qualified against will eventually be withdrawn, and the retirement notice will not align with your requalification window.
- You cannot reproduce it after the fact. Once a version is retired, a past decision cannot be re-derived at all — the evidence chain has a hole in it that no amount of your own logging fixes.
None of this makes hosted services wrong in general. It makes them wrong for the qualified part of the system, which is a narrower claim and a more useful one: many sites run a hosted assistant for non-critical work and a controlled, self-hosted stack for anything that touches a GxP record. The deployment trade-off in full is in private LLM for pharma.
What air-gapped deployment actually requires
“Air-gapped” is often used to mean “on our own servers”. Properly it means the system has no network path to anything outside the controlled perimeter — and that has consequences for everything the network would otherwise have done.
On the hardware side the requirement is straightforward: enough local compute to run the model at the throughput the process needs. For document extraction with open-weight models this is a smaller machine than people expect — the workload is bursty, the documents are not large, and a single node with modern GPUs handles a manufacturing site's daily volume comfortably. Sizing and indicative configurations are covered in private LLM for pharma and reference architectures.
On the process side is where the real work sits, and it is what an auditor will ask about:
| What the network normally does | What replaces it |
|---|---|
| Pulls model weights and updates | Physical media transfer through a defined procedure: checksum verification, approval before import, an entry in the version inventory. Nothing arrives because a package manager fetched it. |
| Resolves dependencies at build time | A frozen, mirrored dependency set imported once and versioned. A build that cannot be reproduced offline is not a validated build. |
| Ships logs and telemetry out | Retention inside the perimeter, with defined export for the rare case where it must leave — itself a controlled procedure. |
| Provides time synchronisation | A local time source. Audit-trail timestamps that cannot be trusted undermine every record they appear in. |
| Delivers security patches | A scheduled, approved patch cycle with the same import controls — and an accepted position on the lag between disclosure and application. |
Two roles have to exist by name and usually do not at the start: someone who owns the version inventory — what is installed, at which version, approved when, by whom — and someone who owns the import procedure. Without those, an air-gapped system drifts into an undocumented state within a year, which is worse than a connected one because nobody can tell what it drifted into.
The hardware half is the part we supply directly; we did exactly that for a computational chemistry team whose IP counsel ruled out any cloud path, in air-gapped AI for pharmaceutical R&D — four weeks from quote to a powered-on lab, zero cloud. That project was compute supply rather than a validated document system, and it is worth being precise about the difference: it proves the deployment model, not the qualification work that would sit on top of it.
Where the deterministic layer has to sit
In most document processes the rules-versus-models question is a trade-off. Here it is closer to a constraint, for a reason specific to validated environments: you have to validate whatever acts as the control, and a rule is enormously cheaper to validate than a model.
A rule has a specification, a finite set of test cases, and behaviour that can be demonstrated exhaustively. A model has a distribution. Qualifying a rule is an afternoon of test-protocol execution; qualifying a model means agreeing acceptance metrics in advance, holding out an independent test set, documenting the training data, and repeating some of it on every retrain.
Which produces the architecture that survives review:
The model is not the control; the rules are. That distinction is what makes the validation package tractable — you qualify a deterministic check, and the model sits upstream of it as an input-preparation step whose errors the check is designed to catch. Invert it, letting the model's output be consumed directly, and the model becomes the thing you must qualify.
What the model earns in return is real: it removes the transcription work, reads scans and handwriting, and finds the pages a human should look at. That is the same division reached from a different direction in AI-assisted batch record review, which covers what such an assistant can catch in a batch record specifically and where it has to stop.
How this lines up with Annex 22 and the wider regime
Everything above follows from ordinary computerised-system validation practice under Annex 11, and would be true if no AI-specific guidance existed. What the draft EU GMP Annex 22 adds is that it makes several of these expectations explicit for AI specifically: static and deterministic models for GMP-critical applications, documented intended use before testing begins, separation of test data from training data, explainability per inference, and change control that treats retraining as a significant change.
Status matters here. Annex 22 remains a draft. Plan against it; do not cite it as a binding requirement. The architectural consequences on this page do not depend on the final text, which is the point — a system built to be reproducible, version-pinned and traceable satisfies the draft, satisfies Annex 11, and will not need rebuilding when the final wording lands. The full analysis of what the draft says is in EU GMP Annex 22: what the draft requires.
One provision is worth repeating because it changes procurement rather than engineering: responsibility for validation evidence rests with the regulated company, whether the system was built in-house or bought. There is no vendor black box. A supplier who cannot tell you how a model was trained, on what, and how it is monitored has left you a gap that is yours to close — which is a question to ask during selection rather than during an inspection.
Frequently asked questions
What does it mean to validate an AI system in a GxP environment?
It means demonstrating, with documented evidence, that the system does what its specification says it does — and continues to. In practice that imposes five things on the architecture: the same input must produce the same output, every component version must be pinned and dated, the system must be tested against a written protocol before go-live, any change must pass through change control rather than being deployed, and any result must be traceable back to the inputs that produced it. None of those are features you add afterwards; each is a constraint on how the thing is built.
Why does a cloud AI service usually fail validation?
Usually not on security — on change control. A hosted model is updated on the provider's schedule, not yours. A system qualified against version A can be running on version B the following week with no notification, no revalidation and no record of when it changed. In a validated environment that is an uncontrolled change to a qualified system, which is a finding regardless of how good the new version is. Data residency and the inability to freeze the environment are separate problems on top of it.
What does air-gapped AI deployment actually require?
Hardware capable of running the model locally, and a process for everything that would otherwise use the network. Model weights and updates arrive on physical media through a defined transfer procedure with checksums and a documented approval; there is no package manager reaching out. Someone owns the version inventory. Monitoring and logs stay inside. The hardware is generally the easy half — the transfer, approval and version-ownership procedures are what take the time and what an auditor will ask about.
Can a language model be used at all in a validated process?
Yes, on unstructured input, provided its output passes a deterministic check before anything acts on it. The pattern that survives review is a model that reads and proposes, feeding rules that verify and a person who approves. What does not survive is a model whose output is consumed directly, because then the model is the control and the model is what has to be validated — which is a far harder problem than validating a rule that checks it.
How does this relate to EU GMP Annex 22?
Annex 22 is the draft that makes these expectations explicit for AI in GMP manufacturing: static deterministic models for critical applications, documented intended use, test-data independence, explainability and change control that extends to retraining. It is still a draft, so it should be planned against rather than cited as binding. The architectural consequences described here follow from Annex 11 and ordinary computerised-system validation practice regardless of what the final Annex 22 text says.
This page describes engineering and validation practice as it bears on system design. It is not regulatory advice and does not replace a gap assessment by your own quality unit or a qualified consultant.
Related Resources
- Clinical Document Automation in Validated Environments — what we build and deploy, and what we decline
- EU GMP Annex 22: What the Draft Requires
- AI-Assisted Batch Record Review — the process, and where the assistant has to stop
- Private LLM for Pharma: Decision, Stack and Cost
- Document Data Capture Methods: Which to Choose
Reproducibility is designed in, not documented in
Version pinning, a deterministic control layer and an evidence chain that survives a question asked a year later — decided in the first week of design, or not at all.
Sources and scope. Validation expectations here follow EudraLex Volume 4 Annex 11 on computerised systems and general CSV practice; the AI-specific expectations are from the draft Annex 22, which had not been finalised as of this update — see EudraLex Volume 4. The referenced Haink project was an air-gapped compute deployment for pharmaceutical R&D, described in the linked case study; it is cited as evidence of the deployment model rather than of a validated document system.
Reviewed: August 2026. Next review: November 2026.