AI Architecture Isn’t a Diagram. It’s an Operator’s Checklist.

I’ve sat on both sides of this problem: the side that ships technology, and the side that carries the operational consequences.

When I ran operations in an electrification and energy storage business, I didn’t get to hide behind roadmaps. Engineering, R&D, production, supply chain, and service all met reality every day. A product that “mostly works” is not a product; it’s a liability. Later, when I ran a smart-building business unit across multiple countries, connected devices and cloud services taught the same lesson in a different costume: uptime is not a feature. It’s an operating discipline.

AI is now repeating that pattern—faster. Leaders ask for “the AI architecture.” They get a diagram: models, vector databases, agents, orchestration. It looks neat. And then it breaks in the only place that matters: in production, with customers, under time pressure.

My opinion is simple: AI architecture isn’t a diagram. It’s an operator’s checklist. If you can’t run it, you don’t have it.

The best framing I’ve seen lately is in MIT Technology Review’s overview of the foundational elements leaders need to scale AI. It reinforces what operators already know: the more capable and agentic AI becomes, the more your risk surface expands. The response is not more excitement. It’s more discipline.

The tension: AI wants to be product, but it behaves like operations

In industrial environments, we don’t confuse a prototype with a production line. We gate. We qualify. We write procedures. We measure yield. We run corrective actions. AI needs the same treatment, because AI failure modes look less like “bugs” and more like “process drift.”

And drift is deadly. Not because the model is “bad,” but because the system around it is undefined. Inputs change. Tools change. User behavior changes. Regulations change. Suddenly your AI assistant is confidently giving the wrong answer, or your agent is taking the wrong action, and everyone discovers—too late—that nobody owns the operating system around it.

If you’re a board member or operator, here’s the reframing I want you to adopt: treat AI as a production operating system. Not a demo. Not a capability. Not a slide. An operating system with contracts, quality gates, performance commitments, and an incident playbook.

The operator’s checklist: what “AI architecture” actually means

I’m deliberately avoiding a component list. You can buy components. You can’t buy operational integrity. This is the checklist I use when I review an AI program—whether it’s for a manufacturing workflow, a field-service support assistant, a fintech back office, or an AI layer in one of my own ventures like Shopeno or IBHQ.

1) Data contracts, not “data access”

In power electronics earlier in my career, quality issues were rarely mysterious. They were traceable: a tolerance stack-up, a supplier change, a test gap. AI needs the same traceability discipline, starting with the input boundary.

• Define input schemas for every AI-relevant source (tickets, sensor feeds, invoices, SOPs, chat logs). Version them.
• Assign a data owner who can approve changes and accept consequences.
• Set freshness and completeness expectations like you would for a production KPI.
• Document allowed transformations (redaction, summarization, deduplication) so you can reproduce outputs.

“We connected the model to our data lake” is not a contract. It’s a future incident.

2) Evals are your incoming inspection

In hardware and embedded systems, you qualify components before you scale. AI is no different, except the “component” changes behavior with prompts, context windows, retrieval, and tool availability.

• Maintain a living evaluation set that mirrors real work (not benchmark theater).
• Measure task success, not just accuracy: resolution time, escalation rate, rework created, customer impact.
• Test for failure modes: hallucination under uncertainty, policy violations, tool misuse, data leakage.
• Run evals on every change: model, prompt, retrieval, tool schema, routing logic.

If you can’t describe your evaluation harness as clearly as you describe a factory test station, you’re not ready to scale.

3) SLOs, not “it seems fine”

When I led R&D organizations across embedded software and cloud, we learned to translate customer pain into measurable commitments. AI needs the same translation.

• Latency SLO: time-to-first-token and time-to-action.
• Reliability SLO: tool-call success rate, retrieval success rate, fallback behavior.
• Quality SLO: pass rate on critical eval cases; “no-regret” thresholds for regulated outputs.
• Cost SLO: cost per resolved case or per workflow completion, not cost per token.

Boards understand SLOs. Operators can manage SLOs. Teams can’t manage vibes.

4) Human controls are not a weakness; they’re the control plane

AI programs often swing between two extremes: “full automation” fantasies and “human-in-the-loop” as a vague safety blanket.

Operators think differently. We define control modes.

• Advisory mode: AI recommends; human decides. Good for high-risk domains and early rollout.
• Guarded execution: AI acts within strict bounds (limits, templates, approval steps).
• Autopilot: AI executes end-to-end, but with measurable stop conditions and auditability.

The question isn’t “human in the loop or not.” The question is: where is authority, where is accountability, and what is the override?

5) Incident response: assume you will be wrong at scale

In connected building controls, every incident has two costs: the immediate customer impact and the long-term trust decay. AI adds a third: epistemic risk. People stop knowing what to trust.

• Define incident types: wrong answer, wrong action, policy breach, data leak, runaway cost, degraded performance.
• Create a kill switch: disable tool execution, disable external calls, revert to a safe model, or fall back to human-only flows.
• Log for forensics: prompts, context, retrieval results, tool calls, outputs, user actions (with privacy controls).
• Run postmortems that change the system: eval updates, contract updates, routing updates, permissions changes.

Without incident muscle, scaling AI is just scaling blast radius.

A lived pattern: why AI and industrial execution rhyme

Here’s the pattern I’ve seen repeatedly—first in industrial technology, then in SaaS and fintech, and now in applied AI:

The constraint is rarely capability. The constraint is coordination under uncertainty.

When I ran a multi-country smart-building unit, we had embedded devices in the field, cloud services evolving, and customers with different compliance expectations. What worked wasn’t “better architecture diagrams.” What worked was operational clarity: what changes were allowed, how we validated them, what we monitored, and how we recovered when something slipped through.

I’m applying the same discipline in my own venture building today—especially in platforms like Shopeno and IBHQ where trust, correctness, and cost control matter. AI features are tempting to ship fast. But the only AI that survives is the AI you can operate: versioned inputs, repeatable evals, explicit SLOs, and a clear authority model.

This is also why I like borrowing mental models from energy and infrastructure. Bankability isn’t about optimism. It’s about controls, guarantees, and response plans. The same operator logic shows up in how I think about execution risk in energy projects (I wrote more on that in The Operator’s Playbook for Making Grid-Scale Storage Bankable). AI needs its own version of bankability: decision-grade quality, bounded risk, and predictable operations.

The board-level questions that expose “demo architecture”

If you sit on a board, or you own a P&L, you don’t need to debate model families. You need to pressure-test operational readiness. These questions do it fast:

1. What is the AI allowed to do? Not “what can it do”—what is authorized in production?
2. What are the SLOs? For latency, cost, reliability, and quality. Who owns them?
3. Show me the eval suite. What tasks? What failure modes? How often does it run? What blocks release?
4. What changes trigger re-qualification? Model, prompt, retrieval corpus, tool schema, policy changes.
5. Where is the kill switch? Who can pull it? Under what conditions? What is the fallback workflow?
6. How do we investigate an incident? What logs exist? How long are they retained? What’s the privacy posture?
7. What is the rollout strategy? Advisory → guarded → autopilot, with explicit gates.

If leadership can’t answer these without hand-waving, the “architecture” is still a diagram.

My clear opinion: build less AI. Operate more AI.

The AI conversation is crowded with frameworks, stacks, and vendor promises. Operators should stay stubbornly practical.

AI scales when you treat it like production. That means contracts over access, evals over vibes, SLOs over anecdotes, control modes over blanket “human-in-the-loop,” and incident response over hope.

If you want a single action for this quarter, do this: pick one AI use case that touches real operations (service triage, procurement support, maintenance planning, customer onboarding). Then write the checklist on one page. Assign owners. Put SLOs on a dashboard. Run an incident drill. You’ll learn more in 30 days of operating than in six months of diagramming.

That’s the point. Architecture isn’t what you draw. It’s what you can run.