Why do most enterprise AI pilots fail to reach production?

In practice the model is rarely the blocker. Pilots stall because the organisation has no owner for the data the system depends on, no evaluation set that defines what correct output looks like, and no answer for who is accountable when the system is wrong. A pilot only has to be impressive once; production has to be correct continuously, and the gap between those two standards is organisational rather than technical.

feras.me / ai-agents

// Practice Area

Private LLMs & AI Agents in Regulated Industry

Q: What is a private LLM?

A private LLM is a large language model that runs on infrastructure the organisation controls, so prompts and responses never leave its trust boundary. It can be self-hosted in an on-premise data centre, in a sovereign cloud region, or fully air-gapped. Organisations choose it for three reasons: regulation that forbids sending data to third-party processors, contractual confidentiality with customers or governments, and predictable cost at high token volume.

Q: What is ISO/IEC 42001?

ISO/IEC 42001 is the international management system standard for artificial intelligence, published in 2023. It does the same structural job for AI that ISO/IEC 27001 does for information security: it requires an organisation to define scope, assess AI-specific risks, assign accountability, and demonstrate continual improvement. For regulated operators it is becoming the practical mechanism for proving AI governance to auditors and regulators.

Q: Should an AI agent have write access to production systems?

Not initially, and in operational technology environments not without a human approval step. The useful design pattern is asymmetric permission: broad read access so the agent can reason over real system state, and narrow, explicitly enumerated write access gated behind approval. Every action should carry the agent's own identity rather than a shared service account, so the audit trail attributes actions to the agent and not to a human who never took them.

Most writing about enterprise AI assumes you can send your data to someone else's API. In regulated water, energy, and government operations you frequently cannot. This is what building AI inside that constraint actually involves.

Private LLMs RAG Agentic systems GPU infrastructure ISO/IEC 42001

Role: Manager, Industry 4.0 & Emerging Tech, TAQA Water Solutions (2023 – 2026)
Now: VP of Digital Infrastructure — owns the AI/ML roadmap alongside OT security and private 5G
Built: Private LLM architecture for secure data processing; agentic business roadmaps; optimised GPU clusters for enterprise GenAI
Credential: ISO/IEC 42001 AI Quality Infrastructure (UKAS, 2025)

Why private, and when it is actually worth it

A private LLM is a model running on infrastructure you control, so prompts and responses never cross your trust boundary. That can mean on-premise, a sovereign cloud region, or fully air-gapped. It is more expensive and more operationally demanding than calling a hosted API, and it is the right decision less often than vendors selling GPUs would like you to believe.

It is genuinely justified in three situations:

Regulation forbids third-party processing. Data classified under national information assurance rules often cannot be sent to an external processor at any price.
Contractual confidentiality. Government and utility contracts routinely prohibit disclosure to subprocessors, and a hosted model API is a subprocessor.
Token economics at scale. Above a certain sustained volume, amortised GPU capacity is cheaper than per-token pricing — but that crossover point is much higher than most business cases assume, and it moves every time hosted pricing drops.

If none of those apply, use a hosted model. Running inference infrastructure to avoid a theoretical risk you cannot articulate is an expensive way to feel secure.

What the architecture actually looks like

The GPU layer is a capacity planning problem, not a purchasing one

Optimising GPU cluster performance for enterprise GenAI is mostly about memory bandwidth and batching behaviour, not raw FLOPS. Organisations routinely over-buy on compute and then discover their real ceiling is VRAM per node, or that concurrency collapses because every request is served with batch size one. The question to answer before procurement is not "how many GPUs" but "what concurrent request profile do we need to serve, at what latency, at what context length" — three numbers that determine everything downstream.

Retrieval is where the value and the risk both live

In practice, the model matters less than what you feed it. A capable open-weights model with well-structured retrieval reliably beats a frontier model with poor retrieval on domain-specific work. But retrieval is also the point where access control quietly breaks: if your vector index does not carry the source document's permissions, you have built a system that will cheerfully summarise a document the asker was never cleared to read.

Enforce permissions at retrieval time, per user, per query. Filtering after the model has already seen the content is not access control — it is redaction, and it leaks through inference.

Agents need identity before they need autonomy

An agent that can act on systems is a principal in your security model, whether or not you have modelled it as one. Before granting any write capability I want three things in place:

Its own identity. Not a shared service account. If the audit log attributes the agent's actions to a human who never took them, the log is actively misleading.
Asymmetric permissions. Broad read so it can reason over real state; narrow, explicitly enumerated write, gated behind human approval in any OT-adjacent context.
A full action trail. Every tool call recorded with inputs and outputs. Not for compliance theatre — because this is the only way to debug an agent that did something surprising three weeks ago.

Why pilots stall

I have watched a considerable number of enterprise AI pilots produce an impressive demo and then never ship. The failure is almost never the model. It is one of these:

Symptom	Actual cause
"The answers aren't accurate enough"	No evaluation set. Nobody defined what correct looks like, so quality is a matter of opinion and the argument never resolves.
"We're waiting on data access"	The source data has no owner. This is a governance gap that predates the AI project by years and the project has simply surfaced it.
"Legal is reviewing it"	No accountability model for when the system is wrong. Legal cannot approve what nobody will own.
"It works but nobody uses it"	Built for a workflow that does not exist. The demo scenario was invented by the project team, not observed in the business.

A pilot has to be impressive once. Production has to be correct continuously. Almost everything that separates the two is organisational, not technical.

Where AI meets OT

This is the part of my work that sits at the intersection of both practice areas, and it is where I am most cautious. An AI system that recommends is straightforward. An AI system that acts on industrial control infrastructure inherits every constraint described in OT security architecture — safety first, availability second, and a change control regime that assumes any unplanned action is a potential incident.

My working position: AI belongs in the OT environment for anomaly detection, predictive maintenance, and operator decision support — all read-heavy, all advisory. Closed-loop control by a probabilistic system is a different risk category entirely, and the industry has not yet earned the right to it. I built an AI-driven SOC for threat detection under exactly this principle: the system surfaces and prioritises, humans decide.

Frequently asked

What is a private LLM?

A large language model running on infrastructure the organisation controls, so prompts and responses never leave its trust boundary — on-premise, sovereign cloud, or air-gapped. Organisations adopt it when regulation forbids third-party processing, when contracts prohibit subprocessors, or when sustained token volume makes amortised GPU capacity cheaper than API pricing.

What is ISO/IEC 42001?

The international management system standard for artificial intelligence, published in 2023. It does structurally for AI what ISO/IEC 27001 does for information security: define scope, assess AI-specific risks, assign accountability, demonstrate continual improvement. For regulated operators it is becoming the practical mechanism for evidencing AI governance to auditors.

Should an AI agent have write access to production systems?

Not initially, and never in an OT context without a human approval gate. Use asymmetric permissions — broad read, narrow and explicitly enumerated write. Give the agent its own identity so actions are attributable to it rather than to a shared account.

Open-weights or commercial models for private deployment?

For genuinely air-gapped work, open-weights models are usually the only option, and modern ones are more than capable for retrieval-grounded domain tasks. Where a sovereign cloud region is acceptable, commercial models deployed within that boundary can be the better quality-per-effort trade. The decision follows the data classification, not the benchmark leaderboard.

OT & enterprise security architecture — the constraints AI has to live inside
0xFK — offensive security research
Full career timeline and certifications