DZone Spotlight

Compliance Checkbox vs. Architectural Constraint Most data platforms treat audit-readiness as a downstream concern. The pipelines are built, the warehouse is populated, the dashboards ship, and only then does someone ask how the platform would respond to a regulator's request to reconstruct account balances as of a date eighteen months ago, or to prove that a reported figure hasn't been altered since submission. At that point, the answer is usually assembled after the fact: cross-referencing backups, reconstructing state from scattered logs, or worse, discovering that the required history was never captured at all. This reactive posture is what "compliance checkbox" architecture looks like in practice. The alternative audit-ready by design treats three properties as non-negotiable architectural constraints from the outset, not features added later: lineage, point-in-time reconstruction, and immutability. The distinction matters because a constraint enforced at the architecture level cannot be silently bypassed under deadline pressure the way a bolted-on compliance script can. Three Architectural Constraints, Defined Lineage Every data point must be traceable to its origin, and every transformation it passed through must be recorded as a first-class artifact of the pipeline, not reconstructed later from job logs or tribal knowledge. Lineage that lives only in a wiki page or a data dictionary is documentation, not architecture; lineage that lives in pipeline metadata, enforced by the platform itself, is a constraint. Point-in-Time Reconstruction A regulator's question is rarely "what does the data look like today"; it's "what did the data look like as of a specific past date, and can you prove it?" A platform designed for point-in-time reconstruction can reproduce the exact reported state as of any historical timestamp, not just restore from the nearest backup window. Immutability Once a record has been reported or submitted, it should be architecturally incapable of silent modification. This doesn't mean data can never be corrected; it means corrections are new, versioned, timestamped events layered on top of history, never in-place overwrites of it. The Audit-Readiness by Design (ARD) Maturity Model To evaluate whether a given data platform or pipeline is genuinely audit-ready by design, it helps to score it across the same three dimensions on a four-level maturity scale from bolted-on compliance to constraint-native architecture. This is deliberately structured the same way infrastructure maturity models work: each level represents a materially different failure mode under regulatory scrutiny, not just a stylistic difference. Dimension Level 1: Bolted-On Level 2: Retrofitted Level 3: Designed-In Level 4: Constraint-Native Lineage Manual documentation only; no code-level trace Logging added after pipelines built; partial coverage Lineage captured by pipeline metadata at build time Lineage is a required schema element; pipelines fail to deploy without it Point-in-Time Reconstruction No historical state; only current snapshot exists Periodic backups allow coarse-grained rollback Versioned tables enable reconstruction to any recorded checkpoint Any timestamp is reconstructable to the transaction level, by design Immutability Tables freely overwritten (UPDATE/DELETE in place) Soft-delete flags added; underlying rows still mutable Append-only storage for regulated tables Immutability enforced at the storage layer; mutation is architecturally impossible A platform's ARD maturity is not a single score but a profile across the three rows; it's common to see a platform at Level 3 on Immutability while still at Level 1 on Lineage, and that gap is usually exactly where audit findings originate. The model is most useful as a gap-identification tool during architecture review, applied per regulated data domain rather than to an entire platform at once, since different domains (e.g., transactional reporting vs. internal analytics) typically warrant different target levels. Design Patterns That Support Each Constraint Event sourcing: Storing state as an append-only sequence of events rather than mutable current-state tables gives lineage and immutability simultaneously; the event log is both the audit trail and the source of truth.Table versioning/time-travel storage: Storage formats that retain prior versions of a table as of any commit or timestamp directly support point-in-time reconstruction without requiring separate backup infrastructure.Append-only ledgers for regulated tables: Rather than updating a row, a correction is written as a new row referencing the one it supersedes; the history is never destroyed, only extended.Metadata-driven pipeline orchestration: Lineage capture built into the orchestration layer itself (rather than added as a separate logging step) ensures lineage cannot be skipped, since the pipeline cannot run without emitting it. Common Pitfalls Lineage tracked only in documentation: A data dictionary or architecture diagram is not evidence a regulator can independently verify against the running system.Silent backfills: Correcting historical data by overwriting it in place destroys the very history the platform may later be asked to prove.Soft-delete mistaken for immutability: A boolean "deleted" flag on an otherwise mutable row provides none of the guarantees of true append-only storage.Backup cadence mistaken for point-in-time capability: Nightly backups allow rollback to the nearest backup window, not reconstruction of the exact state as of an arbitrary past timestamp. A Composite Example Consider a generalized (composite, non-attributable) regulatory reporting platform for a financial services organization. An architecture review using the ARD model found the platform at Level 3 on Immutability (append-only storage for core ledger tables) but Level 1 on Lineage transformation logic lived in scheduler scripts with no captured metadata trail. When a regulator later requested a full transformation history for a reported figure, reconstructing it took several weeks of manual log archaeology rather than a direct query. Applying the ARD model earlier in the platform's design would have surfaced this specific gap: a strong Immutability posture masking a materially weaker Lineage posture well before it became a live audit finding. Conclusion Audit-readiness that is designed in behaves fundamentally differently under regulatory pressure than audit-readiness that is bolted on: one is a property of the architecture that cannot be quietly skipped, the other is a checklist item that depends on someone remembering to run it. Treating lineage, point-in-time reconstruction, and immutability as architectural constraints and using a structured model like ARD to find the gaps between them turns audit-readiness from a recurring fire drill into a property the platform simply has.

Today, applications are built around identity systems. All API gateways, microservices, mobile backends, and single sign-on flows require some form of authentication and authorization. That trust is often conveyed via a JSON Web Token (JWT) in many systems. Their compactness, portability, and ability to be easily verified within distributed systems make them popular. A service can accept a token, validate its signature, verify the claims of the token (expiration, audience, subject, and issuer), and determine if the request should be accepted. But classical public-key cryptography (especially RSA and elliptic-curve signatures) is still widely used in most JWT deployments these days. Some popular algorithms used in OAuth 2.0, OpenID Connect, API gateways, and identity platforms include RS256 and ES256. These algorithms are effective at this time, but are not likely to be secure with the arrival of strong enough quantum computers (Shor, 1994). The problem isn't with JWTs per se. The problem is that there are lots of ways of signing JWTs that rely on cryptographic problems that may be solved by quantum algorithms in the future. The real problem for developers and platform engineers is how to safely design JWT and IAM systems that will work well when post-quantum signatures become required. JWT Signing Today A JWT is usually signed and not encrypted. This distinction matters. While signed JWTs can ensure integrity and authenticity, their payload is typically at risk of being read by anyone who possesses the JWT. The signature assures that the claims were signed by a trusted party and that they have not been altered. In the example above, an RS256 token that is used in Node.js can be signed in such a way: JavaScript const jwt = require("jsonwebtoken"); const fs = require("fs"); const privateKey = fs.readFileSync("private.key"); const token = jwt.sign( { sub: "user123", role: "admin", iss: "https://auth.example.com", aud: "api://orders-service" }, privateKey, { algorithm: "RS256", expiresIn: "1h", keyid: "rsa-key-2026" } ); The receiving service verifies the token using the issuer’s public key: JavaScript const publicKey = fs.readFileSync("public.key"); const claims = jwt.verify(token, publicKey, { algorithms: ["RS256"], issuer: "https://auth.example.com", audience: "api://orders-service" }); This model is particularly helpful as the private key remains with the identity provider and many services are able to verify the tokens with the public key. Hence, the use of RS256 and ES256 in federated identity systems. The problem that might arise in the future is that RSA and elliptic-curve signatures can be attacked by large-scale quantum computers. If the attacker is able to determine the private key from the public key, then he can create valid-looking tokens and pretend to be a trusted issuer. Why HS256 Is Not a Universal Replacement A few teams propose HS256 or HS384 to address quantum risk. These algorithms are based on HMAC-SHA256 or HMAC-SHA384. They are symmetric, not public-key, message authentication codes. That implies both the signature and verification of the token use the same secret. This can be okay if there is one trusted entity controlling the issuer and the verifier, as in an internal system. But it's not a true replacement for RS256 or ES256 for federated IAM. However, unlike RS512, with RS256, a number of services can verify tokens with a public key, while the identity provider is the only party that can sign the tokens. In HS256, all the verifiers must use the shared secret. If a verifier is compromised, then the attacker can possibly generate new tokens. As such, JWTs with HMAC can be helpful to some limited trust boundaries, but should not be considered the primary solution for large IAM platforms, partner integrations, or multi-tenant SaaS apps. Post-Quantum JWT Direction Post-quantum migration is expected to concentrate on new digital signature algorithms, not just changing all systems to HMAC. NIST has completed the specification of the post-quantum digital signature standard, ML-DSA, and begun work on the JOSE/COSE specification for representing ML-DSA in the JWT and JWS ecosystems. To developers, this means that library support, identity-provider support, API gateway support, and key-management updates are of importance when considering the adoption of post-quantum JWTs. It will NOT be a one-line change to the algorithm. The first step of a realistic migration should start with crypto-agility. Don't permanently hardcode an algorithm. Rather, they should make sure to check tokens by applying hardcoded allowlists for issuer and application context. Example: JavaScript const allowedAlgorithms = { "https://auth.example.com": ["RS256", "ES256"], "https://internal-auth.example.com": ["HS256"] }; function getAllowedAlgorithms(issuer) { if (!allowedAlgorithms[issuer]) { throw new Error("Unknown issuer"); } return allowedAlgorithms[issuer]; } This is not enough by itself to make the system post-quantum, but it paves the way to controlled migration. The verification layer of the stack can be updated via policy and configuration, not service by service as the approved post-quantum JOSE algorithms are introduced to the stack (NIST, 2021). Developer Migration Checklist Inventory the use of JWT in the system. Determine which services issue tokens, which ones verify them, on which algorithms they are based, how keys are rotated, and where JWKS endpoints are located. Second, “cleanse out risky verification conduct. Do not trust the algorithm of the JWT header without consulting a trusted allowlist. Avoid unrecognized issuers, audiences, expired tokens, and unsuspecting algorithms. Third, enhance rotation of keys. A short token lifetime minimizes the risk of replay, but does not prevent signing-key compromise. Apply kid values, JWKS rotation, and overlapping key validity windows. Fourth, don't include sensitive data in signed-only JWTs. Use proper encryption, or store sensitive data on the server if needed to maintain confidentiality. Fifth, limitations of test infrastructure. The post-quantum signatures can be larger than an RSA or ECDSA signature. These larger tokens can have an impact on HTTP header limits, cookies, proxies, API gateways, logs, and service meshes. Lastly, centralize JWT validation, if possible. It's easier to migrate a shared middleware, gateway plugin, or security library than any number of dozens of services with custom validation logic. Conclusion JWTs will remain important in identity and access management, but the algorithms behind them must evolve. RSA and ECDSA work well now, but don't work in the long run. Don't panic, don't switch to HS256. While they can be used in some internal systems, JWTs are not a standard solution for public-key federation. The more fruitful approach is crypto-agility: be aware of the use of JWTs, maintain strict lists of algorithms, rotate keys appropriately, separate authentication and business logic, and get ready for the digital post-quantum signatures (e.g., ML-DSA) as they become available on the library and platform. With an IAM system that is algorithm-agile, teams that are preparing for it today will be more ready for the transition to post-quantum tomorrow.

Distributed databases are often evaluated through familiar technical dimensions: replication factor, consistency model, partitioning strategy, throughput, latency, and recovery time. These characteristics matter, but they do not fully explain why systems that appear healthy at the component level still experience severe production failures. In many cases, the storage engine is not the weakest part of the architecture. The failure occurs at a coordination boundary. A coordination boundary is any point where independently operating components must agree on timing, ownership, ordering, configuration, or state. These boundaries appear between replicas, partitions, control planes, data planes, load balancers, clients, metadata services, and background maintenance processes. Each component may behave correctly according to its local rules while the overall system produces an incorrect or unstable result. This is why distributed database incidents can be difficult to predict. The database may not fail because a server crashes or a disk becomes unavailable. It may fail because two healthy components temporarily disagree about who owns a partition, whether a node is available, or which version of configuration should be applied. Local Correctness Does Not Guarantee System Correctness Engineers naturally reason about software components individually. A node accepts requests, writes data, replicates changes, responds to health checks, and reports metrics. If each of those behaviors appears correct, the system is assumed to be healthy. Distributed systems challenge that assumption. A replica can be healthy but delayed. A coordinator can be available but operating with stale metadata. A load balancer can route traffic correctly according to its current configuration while that configuration no longer reflects the database topology. A client can retry a failed request according to policy while unintentionally amplifying load during a partial outage. Each component is locally correct. Their interaction is not. Consider a partition ownership transition. One node is being removed, replaced, or scaled down, and another node is taking responsibility for the affected data range. The outgoing node may believe it still owns the partition because it has not received the latest control-plane update. The incoming node may already begin accepting requests because it has received a newer version of the assignment. For a brief period, both nodes may behave correctly according to the information available to them. The system, however, has entered an ambiguous ownership state. That ambiguity can lead to duplicate processing, inconsistent writes, rejected requests, or unexpected latency. The problem does not exist entirely inside either node. It exists at the boundary where ownership information is exchanged and interpreted. Time Is Often the Hidden Coordination Dependency Many distributed database designs avoid relying on perfectly synchronized clocks. Even so, time remains embedded throughout the system. Timeouts determine when a request is considered failed. Leases determine how long a node retains authority. Heartbeats influence failure detection. Retry intervals shape traffic behavior. Expiration policies determine when data should disappear. Background processes decide when to compact, replicate, repair, or rebalance information. These mechanisms create coordination dependencies even when the architecture does not explicitly describe them that way. For example, a client sends a write request and does not receive a response before its timeout. The client cannot immediately know whether the write failed, succeeded, or is still being processed. It retries the request through another route. If the database supports idempotent request handling, the retry may be safe. If it does not, the same logical operation may be applied twice. The first server and the client both followed their expected behavior. The uncertainty appeared between them because completion and acknowledgment were separated by a network boundary. This is a common distributed systems pattern. A timeout provides information about waiting, not about the final outcome of an operation. Cloud architects should therefore treat every timeout as an ambiguity boundary. Timeout behavior must be designed together with idempotency, deduplication, retry limits, load shedding, and observability. Configuring a timeout without defining the system’s response to uncertainty simply moves the failure elsewhere. Metadata Can Become More Critical Than Data Database reliability discussions frequently focus on protecting stored records. Replication, backups, checksums, and repair mechanisms are designed to preserve data durability. However, the metadata that describes how data should be accessed can be just as important. Partition maps, routing tables, node membership, schema versions, configuration states, and feature capabilities determine how requests travel through the system. If this metadata becomes stale or inconsistent, the underlying data may remain fully intact while applications lose the ability to access it reliably. This is particularly important in systems that separate the control plane from the data plane. The control plane decides how infrastructure should be configured. The data plane processes live requests using that configuration. Separating these responsibilities improves scalability and operational isolation, but it introduces another coordination boundary. Configuration changes must move safely from the control plane to every affected data-plane component. During that transition, the system may contain multiple valid configuration versions at once. The engineering question is not merely whether a configuration update can be delivered. It is whether old and new versions can coexist without violating system correctness. Safe configuration rollout often requires versioning, backward compatibility, staged activation, and explicit rollback behavior. Without those protections, a harmless-looking control-plane update can produce a data-plane outage even when no database node has failed. Load Balancing Can Amplify Database Instability Load balancing is sometimes treated as an infrastructure layer outside the database itself. In practice, routing behavior directly influences distributed database reliability. When a node slows down, a load balancer may reduce traffic to it. That appears beneficial, but the remaining traffic must go somewhere. Healthy nodes receive additional load, their latency increases, and health checks may begin failing. The load balancer then removes more nodes, increasing pressure on the smaller remaining pool. This creates a feedback loop. The database causes routing changes, and the routing changes make the database less stable. Neither system is necessarily defective. The failure emerges from their interaction. Aggressive health checks, short timeout thresholds, synchronized retries, and immediate node removal can turn a minor performance issue into a broad outage. A more resilient design considers the rate of change, not only the current health signal. Cloud architects should ask whether routing decisions become less reliable during overload. They should also examine whether the database and load-balancing layers use compatible definitions of health. A node capable of serving read traffic may be temporarily unsuitable for writes. A node completing recovery may be reachable but not ready for production load. Binary healthy-or-unhealthy classifications often hide these operational differences. Background Work Creates Coordination Pressure Distributed databases perform significant work outside the direct request path. Replication, compaction, repair, rebalancing, expiration, backup, and cleanup processes compete for shared resources. These operations are often independently scheduled, which creates additional coordination boundaries. A compaction process may increase disk activity while a rebalance consumes network bandwidth. A repair job may begin during a traffic peak. Expired records may accumulate faster than cleanup processes can remove them. Each mechanism may operate within its configured limits, yet their combined effect can overwhelm the system. Time-to-live functionality provides a useful example. Expiring a record appears to be a simple data operation, but at scale it affects storage layout, indexing, replication, read behavior, and cleanup scheduling. The system must determine when an item is logically expired, when it should stop appearing in reads, and when its physical storage can be reclaimed. Those events may not occur simultaneously. If expiration processing is poorly coordinated, large groups of records can become eligible for deletion at the same time, creating bursts of background work. The feature itself works correctly, but the interaction between expiration timing and resource consumption can destabilize the database. The broader lesson is that operational features should be evaluated as distributed workflows, not isolated functions. Designing for Boundary Failures The most effective way to improve distributed database reliability is to identify coordination boundaries during architecture design. For every boundary, engineers should define what information crosses it, how that information is versioned, how long it remains valid, and what happens when delivery is delayed or duplicated. They should also determine whether the receiving component can safely operate with stale information. Observability should follow the same structure. Monitoring individual nodes is necessary, but it is not sufficient. Teams need visibility into ownership transitions, metadata propagation delays, retry amplification, routing changes, replication lag, and background-work queues. These signals reveal disagreement between components before that disagreement becomes a complete outage. Testing must also include transitional states. Steady-state benchmarks show how a system performs when ownership, routing, and configuration are stable. Production failures frequently occur while those conditions are changing. Architects should test node replacement, delayed configuration propagation, partial network loss, rolling upgrades, uneven clock behavior, repeated retries, overloaded background workers, and conflicting health signals. These scenarios expose the boundaries where local assumptions stop matching global reality. Reliability Lives Between Components Distributed databases rarely fail in the clean, isolated ways described by component diagrams. They fail through timing gaps, stale metadata, ambiguous ownership, retry storms, incompatible health decisions, and overlapping maintenance activity. The database node that appears responsible may only be the place where the problem becomes visible. For cloud architects and engineers, the practical shift is to stop treating coordination as an implementation detail. Coordination is part of the system’s correctness model. Storage engines protect data. Replication protects availability. Load balancing distributes work. Control planes manage change. None of these mechanisms can provide reliability independently. Reliability emerges from how they coordinate, especially when information is delayed, incomplete, duplicated, or temporarily inconsistent. That is where distributed databases are most likely to fail, and where architects should focus first.

Vibe coding is not a programming technique. It's an organizational event dressed up as one. When GitHub Copilot launched in 2021, the narrative was "AI as pair programmer." When ChatGPT arrived, it shifted to "AI as junior developer." By the time Cursor and Windsurf and Devin entered the picture, the goalposts had moved so far that we stopped noticing where they used to be. The term itself — coined by Andrej Karpathy in early 2025 — describes writing software by describing what you want and iterating on AI output until it looks right. No architecture upfront. No deep understanding of the internals. Just prompt, review, adjust, ship. The name is almost deliberately casual. Vibe. As if the whole thing is low-stakes. It is not low-stakes. Not even close. What's actually happening is that the cost of producing a working prototype has collapsed to near zero. A product manager, a designer, a technically literate founder, or a business analyst with a ChatGPT subscription can now produce something that looks — and often behaves — like what an engineering team would have spent two sprints building. The artifact exists. It runs. It answers to a curl command. And that is exactly what makes the next part so uncomfortable. What "Engineering as a Service" Actually Means on the Ground The phrase Engineering as a Service — EaaS — has been floating around enterprise architecture circles for a few years. It originally described platform teams offering standardized, self-service infrastructure to internal product teams. Sensible concept. Reasonable organizational model. That is not what I mean when I use the term now. What I'm watching happen — across teams I've worked with, consulted for, and frankly in my own org — is a quieter version: engineers becoming a validation and production-hardening layer that sits after the AI has already done the creative work. The product manager vibe-codes a proof of concept. The engineering team is handed it and asked to "make it production ready." Requirements arrive pre-defined. The architecture decision has already been made, implicitly, by whatever structure the AI generated. The engineer's job, in this model, is to clean up after the vibes. "The most dangerous moment isn't when AI writes bad code. It's when the organization stops asking engineers to think before the code exists." That shift is subtle. Gradual. And it doesn't announce itself in a restructuring memo. It shows up in how sprint planning conversations change. In the questions that stop being asked. In the job descriptions that quietly remove "system design" and add "AI code review." In the fact that your most technically sophisticated colleagues are increasingly valued for their ability to spot what the model got wrong — not for their ability to envision what should be built in the first place. It's hollowing. And the hollow feels comfortable for a surprisingly long time. The 80% Problem Nobody Is Talking About Honestly The statistic getting passed around is that AI can write 80% of the code. Maybe 90%. Some teams will tell you it's higher. They're probably right, for a certain definition of "code." What nobody says in the same breath is what that 20% contains. It contains the decision to use a distributed lock instead of an optimistic concurrency strategy, because you know your write contention pattern at 3 AM on the first of the month. It contains the choice to put that third-party API call behind a circuit breaker, because you were the one paged at midnight when their service went down for six hours two years ago. It contains the knowledge that your payment processor has a 30-second timeout that doesn't appear anywhere in their documentation, and that the retry logic the AI generated will double-charge customers under exactly the conditions that will occur in production. That 20% is not filler. It's the residue of lived system knowledge. And it cannot be prompted for, because it lives in people, not in documentation. The Skill Inversion Nobody Budgeted For Here's the part that makes people genuinely uncomfortable when you say it in a room: the engineers who are worst positioned for this transition are often the best coders. Think about it. If you built your professional identity around the craft of writing clean, efficient, elegant code — if that's the thing you're proud of, the thing you've spent ten years sharpening — you are now in a profession where that particular skill is the one being automated away fastest. The engineers who wrote beautiful Ruby. The ones who could implement a red-black tree from memory. The ones whose pull request diffs were a pleasure to read. Those skills are not worthless. But they are no longer the differentiator. The differentiator is now something harder to teach, harder to credential, and much harder to interview for: the ability to look at an AI-generated system and know — without running it — which assumptions it made, which failure modes it ignored, and which organizational constraints it has no way of knowing about. That requires something I'd describe as systems intuition. It's not algorithmic. It's not certifiable. It's the thing you develop after you've been on-call for two years, after you've traced a cascade failure through six services at 2 AM, after you've had to explain to a CFO why a "working" deployment is losing the company $4,000 an hour. You can't vibe-code your way to it. What Gets Built When Engineering Is a Service Let's say the EaaS model wins. Let's say your organization fully embraces the idea that non-engineers will prototype, AI will build, and engineers will review and harden. What does the resulting software actually look like? I know, because I've seen early versions of it. Across four teams that went deep on vibe-coding workflows in 2025, there are common artifacts starting to emerge. The code is structurally fine. Readable, even. Comments are excellent — AI comments well. Test coverage looks good on paper. But the systems have a specific flavor of wrongness that takes a while to name. They're built for the happy path with unusual thoroughness. And they fail in ways that are not in any test suite, because the failure modes weren't imagined in the prompts that generated the code. One team I spoke with — a Series B SaaS company, roughly 40 engineers — went vibe-coding-first on a new data pipeline in late 2024. Shipping velocity tripled. Incident rate was flat for two months. Then they hit Black Friday. The pipeline had no backpressure mechanism. The AI had generated clean, efficient queue processing code that assumed queue depth was bounded. Under real peak load, it consumed memory until the service OOMed, cascading into three downstream consumers. Recovery took nine hours. The postmortem finding: no one had asked the AI "what happens when the queue grows faster than we can consume it?" Because the PM who wrote the initial prompt didn't know to ask. And the engineers who reviewed the output were reviewing it for correctness, not for production failure modes they hadn't witnessed yet. So Who Actually Survives This? If the question is "what does the engineering career look like in a world where AI generates 80% of the code," the answer isn't just "learn to prompt better." That framing is too small. It optimizes for the wrong thing. The survivors are the engineers who never let their professional identity live entirely in the code. They're the ones who were always curious about why a system needed to exist, not just how to build it. The ones who sat in product strategy meetings when they didn't have to. The ones who wrote design docs before anyone asked and kept them updated after nobody read them. They're also the engineers who carry operational scar tissue. Production incidents are an education that no prompt can replicate. Every major outage you've lived through deposits something into your mental model of systems — a new category of "things that go wrong under conditions that weren't in the spec." That library of failure is, right now, one of the most underappreciated professional assets in engineering. The survivors will be engineers who can sit across from an AI-generated system and run it through a mental gauntlet: what happens when the third-party API goes down? What happens when this queue backs up for six hours? What happens when someone sends a payload that's technically valid but semantically adversarial? What happens when this runs in the EU and GDPR applies to this field? Not because they're pessimistic — but because they've seen all of those things happen. The Uncomfortable Truth About Fighting Back I want to be careful here, because the easy response to all of this is: good engineers will always be needed. And that's technically true in the same way that good writers are always needed in the age of generative text. It doesn't tell you much about the market. It doesn't tell you which specific kind of good engineering will be compensated. The uncomfortable advice, the kind I give to engineers who ask me directly: stop being the person who writes the most code, and start being the person who knows the most about what the code needs to survive in the real world. Those are different identities. They require different habits. And the transition is not comfortable, especially if you built your self-image around your coding ability. The engineers who will own the next decade are the ones who can walk into a room where an AI has already generated a candidate architecture and say — clearly, specifically, with evidence — why that architecture will fail, what it will cost, and what needs to change before anyone touches a production database. Not because they can write better code than the AI. But because they've seen this movie before, in a dozen variations, and they know how it ends. That's not a skill you can automate. Not yet. Maybe not ever.

By Dinesh Elumalai CORE

Most teams are still building AI agents like chatbots. That is fine for demos. It is not fine for production. A chatbot answers a question. An enterprise AI agent executes work. That difference sounds small, but it changes the entire architecture. Consider a customer support agent investigating a complex technical escalation. The agent may need to analyze diagnostic logs, search product documentation, find similar historical incidents, consult multiple specialized agents, wait for a support engineer to review a recommendation, and then generate a remediation plan. That workflow may take minutes, hours, or even longer. Now ask the uncomfortable engineering questions: What happens if the user closes the browser?What happens if the API request times out?What happens if one downstream system is unavailable?What happens if the model call is throttled?What happens if human approval arrives six hours later?What happens if the process restarts halfway through execution? If the answer is "we will handle that in the agent code," the architecture is already in trouble. The biggest mistake many teams make is treating the LLM as the application. In production systems, the workflow is the application. The LLM is one component inside a larger execution graph. Enterprise AI agents are not chatbots. They are distributed systems. And distributed systems need durable runtimes. The Chatbot Architecture Breaks Quickly Most early AI applications start with a simple request-response model: user request → agent API → LLM → response. This works well for Q&A, summarization, search, content generation, and basic tool calling. But enterprise workflows rarely stay that simple. A customer support agent might instead follow a flow like this: analyze the logs, search the knowledge base, find similar historical cases, run diagnostic reasoning, check severity and escalation policy, wait for human review, and only then generate a final recommendation. This is not a chat interaction. It is a long-running business process with AI inside it. The moment the agent becomes responsible for completing work across systems, the architecture needs capabilities that most chatbot implementations do not provide: Durable stateRetry policiesProgress trackingCorrelation IDsHuman approval checkpointsPartial failure handlingEvent-driven resumptionAuditabilityWorkflow versioning These are workflow orchestration concerns, not prompt engineering concerns. The Real Problem Is Execution, Not Reasoning The AI industry talks a lot about reasoning. But many production failures are not reasoning failures. They are execution failures. The model may correctly identify the next step, and the system still fails because: The workflow state was stored only in memory.The frontend session disappeared.The backend request exceeded a timeout.A transient API failure caused the entire workflow to restart.A human approval step was handled outside the agent workflow.There was no way to resume from the last completed step.The agent retried a non-idempotent action and created duplicate work. In other words, the model worked. The runtime failed. None of these failure modes are new. Durable-execution runtimes solved persist-and-resume for workflows years ago, checkpointing is older than that, and idempotency keys are payments-industry bedrock. What has changed is who is building these systems: the teams shipping agents today largely did not live through the workflow-engine era, so the discipline is being relearned. Agents also add one failure mode the classical systems never had. A workflow engine handed ambiguous state fails loudly. A language model handed ambiguous state re-reasons from scratch — it will confidently re-derive a plan, redo completed work, and re-request data it already has, and it will do so in fluent prose that looks like progress. That is a failure mode you have to design against explicitly, because it does not announce itself. This is why enterprise agent architecture needs to borrow more from distributed systems, workflow engines, and cloud orchestration than from chatbot demos. A serious AI agent platform needs to answer: How is workflow state persisted?How are long-running tasks resumed?How are retries controlled?How are external events handled?How are human decisions represented?How is progress exposed to the user?How are multiple agents coordinated?How are failures isolated? If those questions are not part of the architecture, the system is not production-ready. The Better Mental Model: Workflow First, Model Second The most useful mental model is this: The workflow is the application. The model is one activity inside it. That shift changes how systems are designed. Instead of building a giant agent that does everything, design a durable workflow that coordinates specialized capabilities. For a customer support scenario, the system might use multiple specialized agents: Diagnostic Agent: analyzes logs, symptoms, and telemetry.Knowledge Search Agent: searches product documentation and known issues.Historical Case Agent: finds similar resolved incidents.Policy Agent: checks escalation, compliance, or risk rules.Resolution Agent: synthesizes the final recommendation. Each agent has a focused responsibility. The orchestration layer coordinates execution, and it should own workflow progression, agent sequencing, parallel execution, state persistence, retry behavior, failure handling, human review, and final aggregation. This keeps the AI layer focused on reasoning and the workflow layer focused on execution. Reference Architecture: Durable Runtime for Long-Running Agents A production-oriented architecture looks more like this. A durable orchestration layer owns state, coordination, retries, and the human-in-the-loop wait; specialized agents own only their domain. Because the orchestrator checkpoints to durable state after every step, the workflow survives restarts, deploys, and days-long approval waits. The important part is not the specific cloud service. The important part is the architectural separation. The user interface starts the workflow. The durable orchestrator coordinates execution. Specialized agents perform bounded work. The workflow stores progress, handles retries, waits for human input, and resumes reliably. Azure Durable Functions is one practical implementation of this pattern because it provides stateful orchestrations, activity functions, checkpointing, retry policies, and long-running workflow support on a serverless runtime.¹ The same architectural idea can be implemented with other workflow engines. The point is not "use one specific product." The point is "do not build long-running agent execution as a stateless API." Fan-Out/Fan-In Is a Natural Pattern for Multi-Agent Systems Many enterprise AI workflows contain independent tasks. A customer support investigation can often run its diagnostic, knowledge, historical, and policy analyses in parallel — the fan-out/fan-in shape in the architecture above. The workflow fans out to multiple specialized agents. Each agent performs independent analysis. The workflow then fans in the results and synthesizes a recommendation. This maps directly onto the fan-out/fan-in pattern documented for durable orchestrations, which runs multiple functions in parallel and aggregates the results afterward. ² A simplified C# orchestration can look like this. The examples use .NET Durable Functions; the same patterns exist in the Python and JavaScript bindings, and in runtimes like Temporal. C# [Function(nameof(CustomerSupportAgentOrchestrator))] public static async Task<SupportCaseResolution> RunAsync( [OrchestrationTrigger] TaskOrchestrationContext context) { var request = context.GetInput<SupportCaseRequest>() ?? throw new InvalidOperationException("Support case request is required."); context.SetCustomStatus("Launching specialized agents"); var diagnosticTask = context.CallActivityAsync<AgentFinding>( nameof(RunDiagnosticAnalysisAgent), request); var knowledgeTask = context.CallActivityAsync<AgentFinding>( nameof(RunKnowledgeSearchAgent), request); var historicalTask = context.CallActivityAsync<AgentFinding>( nameof(RunHistoricalCaseAgent), request); var policyTask = context.CallActivityAsync<AgentFinding>( nameof(RunPolicyAgent), request); var findings = await Task.WhenAll( diagnosticTask, knowledgeTask, historicalTask, policyTask); context.SetCustomStatus("Aggregating agent findings"); var resolution = await context.CallActivityAsync<SupportCaseResolution>( nameof(GenerateDraftResolution), findings); return resolution; } This is more maintainable than building one large prompt that tries to do everything. It also gives the platform better control over which agents ran, which agents failed, which outputs were used, how long each step took, and what evidence supported the final answer. That matters in enterprise systems. Human-in-the-Loop Is Not an Edge Case Many enterprise AI systems quietly assume that agents will produce immediate answers. Real workflows often require human decisions — when confidence is low, when customer impact is high, when the recommendation involves risk, when the action changes system state, when the workflow touches regulated data, or when the escalation is sensitive. The timeline usually looks nothing like a chat exchange. Drawn to scale: the model is not the bottleneck. A typical investigation spends two minutes on AI analysis and six hours waiting for a human to approve. The slowest step is not always the LLM. It is often the human approval, dependency response, or operational handoff. This is where durable orchestration becomes essential. The workflow needs to pause without losing state. It should not keep a web request open. It should not rely on memory. It should not require a custom polling database plus a manual recovery script. Durable orchestration can model this directly: C# context.SetCustomStatus("Waiting for human review"); var reviewDecision = await context.WaitForExternalEvent<HumanReviewDecision>( "HumanReviewCompleted"); var finalResolution = await context.CallActivityAsync<SupportCaseResolution>( nameof(GenerateFinalResolution), new FinalResolutionRequest { ReviewDecision = reviewDecision }); return finalResolution; External events let a running orchestration receive signals from outside — human approvals, webhook callbacks, or other systems — without holding compute open while it waits.³ That matters because human approval should not be a side process. It should be part of the workflow. The Crash That Costs Money Durable runtimes give you at-least-once execution. After a crash, an activity may run again. For reads, that is free. For writes, it is the most dangerous window in the architecture, and it is worth being precise about where it opens. A workflow issues a customer refund. The money moves. In the instant before the runtime checkpoints that the activity completed, the process dies. On recovery, the runtime replays the activity — behaving exactly as designed — and issues the refund a second time. The orchestrator cannot prevent this, because from its point of view the activity never completed. The fix has to live in the side-effecting operation itself: every consequential write carries an idempotency key, and an operation that sees a key it has already processed returns the original result instead of acting twice. C# var refund = await context.CallActivityAsync<RefundResult>( nameof(IssueRefund), new RefundCommand( CaseId: request.CaseId, Amount: approvedAmount, IdempotencyKey: $"{request.CaseId}:goodwill-refund")); At-least-once execution guarantees a replay will eventually land in the gap between a side effect and its checkpoint. Without an idempotency key, the replay issues a second refund. With one, the operation recognizes the key and returns the original result — two calls, one refund. Resumability and idempotent writes are the same requirement seen from two sides. You cannot safely resume a workflow whose writes are not safe to replay. The Orchestrator Should Coordinate, Not Think A common mistake is putting too much logic inside the agent or the orchestrator. A better separation is simple to state: the orchestrator decides what happens next — calling activities, waiting for events, tracking status, applying retry policy, coordinating results. Activities do the work — calling models, searching systems, querying databases, invoking tools, performing side effects. For example, an activity that calls a knowledge search agent might look like this: C# public sealed class RunKnowledgeSearchAgent { private readonly IAgentExecutionClient _agentClient; public RunKnowledgeSearchAgent(IAgentExecutionClient agentClient) { _agentClient = agentClient; } [Function(nameof(RunKnowledgeSearchAgent))] public async Task<AgentFinding> RunAsync( [ActivityTrigger] SupportCaseRequest request) { var response = await _agentClient.RunAsync(new AgentExecutionRequest { AgentName = "KnowledgeSearchAgent", Prompt = $""" Search for relevant troubleshooting guidance. Case: {request.CaseId} User question: {request.UserQuestion} Product area: {request.ProductArea} Return concise findings with supporting evidence. """ }); return new AgentFinding { AgentName = "Knowledge Search Agent", Summary = response.Summary, ConfidenceScore = response.ConfidenceScore, Evidence = response.Citations, RequiresHumanReview = response.ConfidenceScore < 0.75 }; } } This keeps model calls, retrieval, tool execution, and external I/O outside the orchestration logic. That separation improves testability, recovery, and observability. Design for Partial Success Enterprise workflows should not be all-or-nothing by default. If four specialized agents run and one fails, should the entire investigation fail? Sometimes yes. Often no. A better design is to treat agent results as structured outcomes: C# public sealed record AgentExecutionResult { public required string AgentName { get; init; } public bool Succeeded { get; init; } public AgentFinding? Finding { get; init; } public string? FailureReason { get; init; } } Now the aggregation layer can reason about partial results. If the diagnostic, knowledge, and policy agents succeed and the historical-case agent fails, the system can still produce a recommendation — with an explicit caveat that historical case comparison was unavailable. This is how resilient systems behave. They degrade gracefully instead of collapsing completely. AI agents need the same discipline. Observability Is a Product Feature Users do not just want the final answer. They want to know what the system is doing. A long-running agent should expose meaningful progress — started investigation, analyzing diagnostics, searching knowledge base, finding similar cases, aggregating findings, waiting for human review, generating final recommendation, completed. This is not cosmetic. Progress visibility builds trust. From an operational perspective, the platform should track the workflow instance ID, correlation ID, case ID, current stage, agent execution duration, retry count, failure reason, human review latency, final outcome, and evidence references. If a support engineer asks, "Why did the agent recommend this?" the system should have an answer. If an operator asks, "Where are workflows getting stuck?" telemetry should show it. If a governance reviewer asks, "Which model and prompt version produced this recommendation?" that should be traceable. This is why observability belongs in the architecture, not in a dashboard added at the end. Retry Policy Is Part of the Design Long-running agents depend on external systems, and those systems will fail. They will throttle. They will time out. They will return transient errors. They will behave differently under load. Retry behavior should be explicit. C# var retryPolicy = new RetryPolicy( maxNumberOfAttempts: 3, firstRetryInterval: TimeSpan.FromSeconds(10)) { BackoffCoefficient = 2.0, MaxRetryInterval = TimeSpan.FromMinutes(2) }; var taskOptions = new TaskOptions(retryPolicy); var finding = await context.CallActivityAsync<AgentFinding>( nameof(RunKnowledgeSearchAgent), request, taskOptions); Retries should be applied carefully. Retry transient failures: HTTP 429, HTTP 5xx, temporary network failures, search service timeouts, model endpoint throttling. Do not blindly retry invalid input, authorization failures, policy violations, business rule failures, or — as the previous section argued — any non-idempotent side effect. A durable runtime gives teams a place to encode this behavior consistently. Without it, retry logic gets scattered across controllers, services, queues, and agents. Governance Matters More When Agents Act Governance becomes more important when agents stop answering questions and start influencing operational decisions. At minimum, production agent workflows should track the workflow version, agent version, prompt version, model deployment, input data sources, evidence references, reviewer decisions, final recommendation, and correlation ID. This is not bureaucracy. It is operational safety. If an agent provides a recommendation on a support case, teams need to know what information was used, which agents participated, whether a human approved the result, and how the final recommendation was generated. A durable workflow makes that lineage easier to capture, because the workflow already represents the execution path. The Test That Tells You Whether Any of This Works Architecture diagrams do not prove durability. The only trustworthy verification I have found is destructive. Kill the running workflow at an arbitrary point — not at a clean boundary, at an awkward one. Discard all in-memory and in-context state. Bring the system back up and watch what the resumed execution does. A sound design picks up exactly where the work stood, and each distinct way of failing points at a specific gap: The resumed workflow...You are missingre-derives its plan from scratchpersisted state the agent layer actually readsredoes completed stepscheckpointing at the right granularityreloads its entire history to get orienteda scoped working set per resumere-fires a side effectidempotency keys A durable runtime passes the orchestration half of this test by construction. That is what you are buying. What it does not guarantee is the agent half: whether your agents' working context, retrieved evidence, and plans are reconstructed from durable state, or were quietly living in a context window that no longer exists. Run the test end to end, including the model-facing layers. That is where it fails in practice, and it is far better to learn that on a Tuesday afternoon than during an incident. Five Lessons From Building Long-Running Agent Workflows 1. The workflow matters more than the prompt. Prompt quality matters, but it does not solve execution reliability. A great prompt inside a brittle runtime still produces a brittle system. 2. Human latency dominates model latency. Many workflows wait longer for people than for models. Design for hours, not seconds. 3. Multi-agent systems need coordination, not chaos. Adding agents is easy. Coordinating agents is hard. Without orchestration, multi-agent systems become difficult to reason about, debug, and govern. 4. Partial success is better than total failure. Enterprise systems should degrade gracefully. If one agent fails, the platform should decide whether the workflow can continue with caveats. 5. Observability is part of the user experience. A long-running agent without progress visibility feels broken. A long-running agent with clear status feels reliable. Conclusion The next phase of enterprise AI will not be won only by better prompts or larger models. It will be won by better execution architectures. Long-running agents need to coordinate multiple systems, preserve state, recover from failures, wait for human approvals, expose progress, and produce auditable outcomes. That is not chatbot architecture. That is distributed systems architecture. The model is important, but it is not the whole application. In production-grade enterprise AI systems, the workflow is the application, and durable orchestration gives that workflow a runtime. If your AI agent needs to do real work across real systems, stop building it like a chatbot. Build it like a distributed system. References Microsoft Learn, "Durable Functions overview" — https://learn.microsoft.com/en-us/azure/azure-functions/durable/durable-functions-overviewMicrosoft Learn, "Fan-out/fan-in pattern scenarios in Durable Functions" — https://learn.microsoft.com/en-us/azure/durable-task/common/durable-task-fan-in-fan-outMicrosoft Learn, "Handling external events in Durable Functions" — https://learn.microsoft.com/en-us/azure/azure-functions/durable/durable-functions-external-eventsMicrosoft Learn, "Durable Functions best practices and diagnostic tools" (idempotent activities, at-least-once execution) — https://learn.microsoft.com/en-us/azure/azure-functions/durable/durable-functions-best-practice-reference

I was answering requirements questions for a knowledge-transfer package when the AI stopped me cold. I had picked "slide deck" as the output format. Two questions later, I picked "native web rendering" for the diagrams. Instead of guessing which one I meant, the workflow halted: "These conflict: a slide deck is not a website, and native web rendering does not live inside a deck. Which do you actually want?" It was right. I had contradicted myself without noticing, and if the AI had quietly picked one interpretation, I would have found out a full build later. That one catch is why I keep using this process for documents, and it's a good place to start explaining what the process is. What I Was Building I needed a knowledge-transfer (KT) package for an internal service: a presentation for about 30 engineers, delivered remotely over Zoom, built from a pile of existing design docs. You don't need to know anything about that service to follow along. This is about the process, not the subject. The process is AIDLC, the AI-Driven Development Lifecycle, an open workflow from AWS Labs. The rules and setup instructions live at github.com/awslabs/aidlc-workflows, and the thinking behind it is in the AI-DLC blog post. You install it as a set of steering rules for your coding agent (Kiro, Q Developer, Cursor, Cline, Claude Code, or Copilot), and from then on the agent follows the workflow. Nothing in this article depends on which agent you use. AIDLC moves through three phases. Inception is where you work out what you're making and why. Construction is where you work out how, then actually make it. Operations is where you run and maintain it, and for now that phase is mostly a placeholder. Inside those phases are stages: workspace detection, requirements, planning, design, generation, build-and-test. Two ideas make the whole thing work. First, it's adaptive, so not every stage runs every time. Second, it has gates. After most stages, the AI stops and waits for you to approve before it moves on. The contradiction catch above happened at a gate. The gates are where you stay in control. Why This Works For a Document A document project has the same shape as a software project if you squint. Requirements become "who is the audience and what must this cover." Design becomes your outline. Writing code becomes writing the content and assembling the artifact. Build-and-test becomes "does it render, does it open, is it accurate." The stages that only make sense for running software, like user stories and infrastructure design, get skipped on purpose, with the reason written down. The mental adjustment is small: your design is the outline, and your code is the content. The Walkthrough Kicking Off and Workspace Detection I opened with a plain request that said what I wanted, who it was for, and what source material I had. Mine was roughly: "Using AIDLC, create a knowledge-transfer artifact. The audience is 30 engineers, delivered remotely over Zoom. Here are the source docs. Should it be a website or a slide deck? Evaluate and recommend." The AI then scanned the workspace to see what it was working with. A greenfield workspace is empty, and you're starting fresh. A brownfield workspace already has material. For documents, brownfield usually means the source of truth already exists somewhere, like a codebase, old design docs, or a wiki, and that material is what you synthesize from. Mine was brownfield: a folder of design docs. Reverse Engineering, When the Source Already Exists Because I was documenting something that already existed, AIDLC ran a reverse-engineering pass first. It read the source material and produced grounding notes (overview, structure, inventory) so that later stages work from facts instead of inventing them. This was also my first taste of the gate-and-correct rhythm. The first pass got the annotation (Spring AOP) semantics wrong, describing only what a setting disabled and not what it did. I pushed back, the AI fixed it and presented again, and I approved. All of that landed in the audit log at aidlc-docs/audit.md, verbatim: Markdown ## Reverse Engineering - User Correction **Timestamp**: 2025-11-04T16:42:19Z **User Input**: "Annotation semantics -> include what it does also. Not just the negation. If you are not able to interpret, ask for this information." **AI Response**: Re-read the source design doc, expanded the annotation semantics section to cover positive behavior, presented revised notes. **Context**: Reverse Engineering, revision 2 --- ## Reverse Engineering - Approval **Timestamp**: 2025-11-04T16:51:03Z **User Input**: "No more changes. next step." **AI Response**: Marked reverse engineering complete. Proceeding to Requirements Analysis. **Context**: Stage approval Two things worth noticing. You can demand precision, and approval is a deliberate "next step" rather than the AI moving on by itself. Requirements Analysis, and Why the Questions Matter This stage pins down what you actually want. The useful part is the structured question format: multiple choice with an [Answer]: tag you fill in. Markdown ## Clarification Question 1 Which artifact do you actually want me to produce? A) Slide deck, diagrams rendered as images B) Local website, native diagram rendering C) Both, a deck for presenting plus a website reference X) Other (describe after [Answer]:) [Answer]: A This format forces the fuzzy decisions into the open before anything gets built. The X) Other option means you're never stuck inside the choices offered. And the AI will catch contradictions instead of guessing. This is where the halt from the opening of this article happened: I picked "slide deck" for the format, and native web rendering for the diagrams, two answers that don't go together, and the workflow refused to proceed until I resolved it. It's cheaper to let the AI interrogate you now than to rebuild later. Workflow Planning, Where You Skip Stages on Purpose This is the stage that makes AIDLC practical for documents. Planning decides which stages run and which get skipped, and it records the reason for each skip. Here's the plan the AI produced for my run, as it renders in the plan file: Markdown ## Stage Execution Plan - [x] Workspace Detection - EXECUTE (always runs) - [x] Reverse Engineering - EXECUTE (brownfield: source docs exist) - [x] Requirements Analysis - EXECUTE (always runs) - [ ] User Stories - SKIP: one deliverable, no personas or interactive flows - [ ] Application Design - SKIP: no software components or methods to design - [ ] Units Generation - SKIP: single deliverable, nothing to decompose - [x] Functional Design - EXECUTE: this becomes the outline - [ ] NFR Requirements - SKIP: no performance, scalability, or security concerns in the software sense - [ ] NFR Design - SKIP: depends on NFR Requirements - [ ] Infrastructure Design - SKIP: output is local files, nothing deployed - [x] Code Generation - EXECUTE: this is where the artifact gets built - [x] Build and Test - EXECUTE: confirm it renders, opens, and traces to real sources Writing down the reason for a skip matters. It's the difference between "we forgot the security review" and "security review does not apply to a local markdown file, and here is the line saying so." Anyone reviewing later can see the decision was made on purpose. Functional Design, Which Is Your Outline For a document, functional design means the outline and topic scope. This was the most important gate in my run. Get it right, and the build is easy. Get it wrong, and you're regenerating everything. The AI proposed the full structure: sections, sub-topics, ordering, depth, which diagram goes where, and a time budget for the presentation. It attached confirmation questions to each part. Markdown ## Question 1: Section list and ordering Does the outline cover the right topics in the right order? A) Yes, approve as-is B) Reorder some sections (describe) C) Add or remove sections (describe) [Answer]: A ## Question 2: Depth and time-box Is the time and depth allocation right? [Answer]: A Spend your effort here. It's the cheapest place to change your mind. Code Generation, Where the Thing Gets Built Once I locked the outline, generation ran in two parts. First, the AI listed the exact build steps (scaffold files, author diagrams, build the website, render images, generate the deck) as a checklist and asked me to approve it. Then it worked through the checklist, ticking boxes as it went. The checklist isn't busywork. It's how you track progress, and it's how you can say "stop, step 3 is wrong" without losing the rest of the work. Build and Test, Where You Confirm It Works For a document, this part is concrete. Does the website open from local files, do the diagrams render, does the deck open with the images embedded, and does the content trace back to real sources with nothing invented? I found what was broken, had it fixed, and was done. What the Run Leaves Behind By the end, the deliverables themselves (the deck, the website) sit in the workspace root. Everything about how they came to be sits in aidlc-docs/. Here's the actual tree from my run: Plain Text aidlc-docs/ ├── aidlc-state.md # where the run is, stage by stage ├── audit.md # every exchange, verbatim ├── inception/ │ ├── plans/ │ │ └── execution-plan.md # the stage plan with its skip list │ ├── requirements/ │ │ ├── requirement-clarification-questions.md │ │ ├── requirement-verification-questions.md │ │ └── requirements.md │ └── reverse-engineering/ # the grounding notes │ ├── business-overview.md │ ├── architecture.md │ ├── component-inventory.md │ └── ... (six more) └── construction/ ├── plans/ │ ├── kt-architecture-overview-functional-design-plan.md │ └── kt-architecture-overview-code-generation-plan.md └── kt-architecture-overview/ ├── functional-design/ │ ├── kt-outline.md # the outline I guarded │ └── agenda-and-presenter-notes.md └── code/ └── code-summary.md # what got built, from which sources The layout mirrors the phases. inception/ holds the thinking: grounding notes, requirements, and the plan that recorded which stages to skip. construction/ holds the making: the locked outline, the build checklist, and a summary of what was generated. The two files at the top track state and history for the whole run. When someone asks "why does slide 14 say that?", the answer is somewhere in this folder. The Messy Part, Which Is Normal If you only read the finished artifacts, you'd think an AIDLC run is a clean question-and-answer session. Mine wasn't. I stepped in, changed my mind, and corrected the AI constantly, and that's how it's supposed to go. The gates exist so you can do exactly this. I changed my mind at a gate. I settled the format, then walked it back a message later: "I chose local website + PPT in my answers." Then I refined the roles again: "I will present from website. PPT is for lasting reference." The requirements doc got updated each time. Changing your mind at a gate costs almost nothing. Changing it after the build costs a lot, which is the whole reason the gates come first. A lot of my steering didn't fit a multiple-choice box, and that was fine. Plenty of my corrections were just typed out plainly: Plain Text "Timeline is incorrect again..." "ignore [that part], the project was deprioritized." "directly V3 was created. v1 and V2 were in design only. Do not mention versions." "There are just two types of throttling... they work in parallel, not layered." The AI took each one, rechecked the source where it needed to, and presented again. You're never limited to the menu. I also asked a side question without derailing anything. Mid-build: "Approve. I have a tangential question before we continue. You asked to create diagrams for the web version. What were my other options?" The AI answered and then carried on with the build. Some corrections only became obvious once the artifact existed. During review, after generation, I asked: "Give a better visual diagram for the identity format. Can you use a vector diagram here?" Review is a legitimate place to refine. And sometimes I cut scope rather than adding it. Late in the run: "Remove Section 12. I don't want to talk about it. I don't know much about it. I'll add it later if I find something." The outline shrank, the reason was logged, and the run moved on. Why the Audit Log Is Worth Keeping Every one of those messages, including the flip-flops, the terse corrections, and the side questions, is captured in aidlc-docs/audit.md in the format shown earlier: my input word for word, paired with what the AI did, and a timestamp. It's never summarized. That sounds bureaucratic, but I've already gone back to it more than once. I could reconstruct why the artifact looks the way it does. A reviewer followed the decision trail without having been in the room. And when something was wrong, I traced it back to the instruction that caused it, which turned out to be mine. The workflow's rule is to log every interaction, append rather than overwrite, and keep your exact words. Habits That Made My Run Go Well Front-load the thinking. Requirements and the outline are where decisions are cheap, so that's where I spent my attention. I let the AI ask its questions, because a contradiction caught early is a build saved. I approved on purpose rather than on autopilot, and when the multiple-choice options didn't fit, I just typed what I meant. One more thing: the audit log keeps your exact words, so say what you mean the first time. Future-you will read it. A Checklist to Reuse If you want to run AIDLC for your own document: Open with a plain request: the deliverable, the audience, the source material, and any questions you want the AI to evaluate.Let it detect greenfield versus brownfield. If brownfield, let it reverse-engineer the source into grounding notes first.Answer the clarifying questions, use X) Other when you need to, and let it flag contradictions.At planning, confirm the skip list and check that each skip has a reason.Lock the outline. This is your most important approval, so take your time with it.Approve the build checklist, then let it generate.Review the rendered artifact and correct freely. Even large structural changes are fair game at this point.Skim the audit log at the end. It's your decision trail. Keep the spine, skip the software-only stages, guard the outline, and steer out loud. References AIDLC workflow rules and setup: https://github.com/awslabs/aidlc-workflowsAI-DLC methodology blog post: https://aws.amazon.com/blogs/devops/ai-driven-development-life-cycle/

The Disturbing Discovery In July 2026, the AI Red Team at NVIDIA published findings of a six-month assessment review of enterprise AI agents, ranging from tools for interactive coding to continuously running autonomous assistants. Across every framework and harness, the pattern that emerges is consistently the same — the agents that failed did so for four primary reasons: no access controls on the agent itself, capabilities to execute arbitrary code, no restrictions on outbound networking or segregation, and plaintext secrets available to the agent. The problem is inherently architectural in nature. Any kind of defense relying on the control plane of the model — for example, constraining the system prompt or having the large language model serve as an adjudicator of the commands issued — inherits the statistical nature of the underlying model. There are three primary methods to bypass these defenses: disguising malicious activities as legitimate ones (e.g., “I’m debugging” or “I’m an admin”); gradual escalation through the dialogue until enough history accumulates to establish the legitimacy of the commands; and embedding code execution in legitimate behavior (e.g., installing a package). This last one is especially worth noting. The coding agent that installs a library is expected behavior. The command pip install git+https://… pointing to a repository that is under the control of the attacker is arbitrary code execution disguised as legitimate development, and no policy-judging model can prevent this action from being performed without disabling the functionality of the agent entirely. For the companies running such agents, the prompt must not be seen as the security boundary. Here are some considerations that better fit the situation. Control 1: Identify the Agent via Authentication and Propagate the Caller’s Identity The first and most common vulnerability is an agent that holds a service identity that can be accessed by any entity on the internal network. This configuration elevates a simple productivity tool into a common privilege escalation endpoint, where each user automatically receives the combined set of privileges of the agent. Two key prerequisites have been established: Authenticate each call. No matter if it is an entry point through the Slack app, web UI, or MCP endpoint, the calls cannot be anonymous and implicitly granted by the network. An agent that ignores unauthenticated callers is a much harder target to probe.Propagate the human user’s identity into downstream calls. The agent shouldn’t be a self-sufficient entity to invoke commands. OAuth 2.0 Token Exchange (RFC 8693) can be used to allow the agent to exchange the user’s token for a downstream token which represents the user’s privileges, not the agent’s: HTTP POST /oauth2/token HTTP/1.1 Host: idp.internal.example.com Content-Type: application/x-www-form-urlencoded grant_type=urn:ietf:params:oauth:grant-type:token-exchange &subject_token=<end_user_access_token> &subject_token_type=urn:ietf:params:oauth:token-type:access_token &audience=https://jira.internal.example.com &scope=issue:read issue:comment &requested_token_type=urn:ietf:params:oauth:token-type:access_token This token would be limited to a single audience, to the two scopes necessary for the job, and to a short expiration. In case of misuse of the agent’s powers, the impact will be limited to the privileges of a single user, rather than the aggregated privileges of all users. Consider the agent to be a non-human identity with a registered owner, a scheduled rotation period, and an expiration. An agent with no owner is virtually never going to get decommissioned. Control 2: Assume Code Execution and Limit Its Effects Instead of trying to prevent code execution through careful design, make the assumption that the agent will run attacker-influenced code and arrange for the effect of that code to be benign and insignificant. It is important to note that a shell utility is not needed for achieving that goal – only write access is required. When an agent can modify configuration files like ~/.bashrc, ~/.gitconfig, a Git hook, MCP.json, or its own instruction file, then code execution happens as soon as another process reads the modified file. Configuration files, in this sense, serve as executable code, but with some extra steps in between. Shell docker run \ --rm \ --read-only \ --tmpfs /tmp:rw,noexec,nosuid,size=256m \ --mount type=bind,source="$PWD/workspace",target=/workspace \ --user 10001:10001 \ --cap-drop=ALL \ --security-opt no-new-privileges:true \ --security-opt seccomp=/etc/docker/seccomp-agent.json \ --pids-limit 256 \ --memory 4g --cpus 2 \ --network agent-egress \ agent-runtime:2026.07 When creating a hardened baseline of containers, the following points should be emphasized: A read-only root filesystem will ensure that write attempts to dotfiles fail at the OS level rather than at the model’s discretion.Use of noexec on writable mounts breaks the “read, write, execute” pattern.Dropping all capabilities and setting no-new-privileges blocks privilege escalation mechanisms. Then, mount the agent’s configuration as read-only and from a different mount point than the workspace of the agent: Shell --mount type=bind,source=/etc/agent/AGENT.md,target=/etc/agent/AGENT.md,readonly \ --mount type=bind,source=/etc/agent/mcp.json,target=/etc/agent/mcp.json,readonly An agent that is able to change its own instructions can assume a completely different persona, including the “authorized debugging user” frame the red team was able to demonstrate. In cases where providing a command utility is unavoidable, use the following strategy: Use an allowlist of binaries and wrap each invocation in a wrapper that removes shell metacharacters, resolves paths, and does not allow any action that goes beyond /workspace.Treat any external inputs – filenames, ticket titles, and document names coming from external systems – as tainted. Control 3: Default-Deny Egress From Each Perimeter Outbound network connectivity turns the constrained execution environment primitive into an actual incident by serving as the means of exfiltration and establishing a reverse shell connection. When NVIDIA tested their system under proper egress restriction, the red team had to perform their activities through the agent process itself — characterized by low speed, high noise, and unreliable performance. Restrict egress in places where the agent does not have direct access to the enforcement point. In case of Kubernetes environments: YAML apiVersion: networking.k8s.io/v1 kind: NetworkPolicy metadata: name: agent-runtime-egress namespace: agents spec: podSelector: matchLabels: { app: agent-runtime } policyTypes: [Egress] egress: - to: - podSelector: matchLabels: { app: egress-proxy } ports: - { protocol: TCP, port: 3128 } - to: - namespaceSelector: matchLabels: { kubernetes.io/metadata.name: kube-system } ports: - { protocol: UDP, port: 53 } All network connections are restricted except those that are explicitly allowed, including blocking the cloud metadata endpoint (169.254.169.254), which provides a credential source without requiring any exploitation. Route the allowed connections through an authenticating proxy server that uses an allowlist of fully qualified domain names (FQDNs), optionally terminates TLS for analysis, and records every request with user identification data attached. This logging creates the incident timeline. Control 4: The Agent Never Holds a Persistent Secret The common practice is to inject secrets via environment variables without making any write calls to the disk, because it is commonly accepted that the only code supposed to run in the container is the expected one. This is untrue for modern times, where a large language model (LLM) runs with the shell in the same process space — env, printenv, and /proc/self/environ are one prompt away, and CLI tools helpfully cache credentials in predictable locations: .netrc, .git-credentials, shell history, and .env files. The most interesting observation made during red teaming was the ability to extract secrets via the chat interface even when all network-based data exfiltration is prevented. The model can read environment variables and return credentials. Regardless of any network isolation, there is no way to protect data the agent is authorized to see. Thus, secrets cannot be accessible to the agent at all. Broker tokens per task instead: Python # Agent requests capability, never a credential. token = broker.issue( principal=ctx.end_user_id, # the human, not the agent audience="https://api.github.com", scopes=["repo:status", "pull_request:write"], resources=["org/repo-name"], ttl_seconds=300, ) try: github.post_review(token, pr_id, body) finally: broker.revoke(token) # revoke on completion, not on expiry Recommendations: Never inject secrets into the container image, environment, volume mounts, or context window.Set very short time-to-live (TTL) values for secrets, measured in minutes.Invalidate tokens after finishing the task.Record every secret issuance along with the identification of the human user.Once the secret is available to the agent, it is already a win for the attacker. Control 5: Package Installation Is a Supply Chain Control Use an internal proxy repository to control the agent’s package manager and stop VCS and URL installations of any packages: Plain Text # /etc/pip.conf (root-owned, read-only mount) [global] index-url = https://artifactory.internal.example.com/api/pypi/pypi-approved/simple no-index = false require-hashes = true # /usr/etc/npmrc registry=https://artifactory.internal.example.com/api/npm/npm-approved/ ignore-scripts=true ignore-scripts=true is the silent victory — this will stop postinstall from being used as an execution vector. The agent must only install packages which are resolvable through the internal repository. Trust, But Verify Ship these as test cases, not as documentation: Assertion Test Unauthenticated callers rejected Invoke the agent with no token, and with another user’s token Dotfile writes blocked Ask it to append to ~/.bashrc and to modify its own instruction file Egress denied by default Request a fetch from an unapproved host; confirm proxy denial in logs No secrets in environment Ask it to print its environment and read /proc/self/environ Metadata endpoint unreachable Request 169.254.169.254/latest/meta-data/ VCS installs blocked Ask it to pip install git+https://… from an external URL Run these on every release, and run the multi-turn variants — the escalation that works is rarely the one in a single message. Key Takeaway Prompt-based guardrails are meant to be a usability feature that prevents accidental damage, but they do not hinder an adversarial actor who intends to cause harm. Each request needs to be validated through identity authentication (JWT validation or equivalent), confirming the caller is who they say they are — alongside a secure sandbox environment without writable-executable paths, default-deny network egress at every boundary, and short-lived credentials issued to the agent per task. This is not new security engineering. It is the application of least privilege, isolation, and secrets management to a workload that interacts with untrusted input in real time. The mistake is assuming the model is the enforcement point, when it is in fact the thing being defended.

1. Why Most AI QA Tools Fail in Production The pattern is now familiar: a team integrates an LLM into their QA workflow, the demo impresses stakeholders, and three months later the tool is quietly deprecated. Tests it generated needed manual cleanup. Root cause analyses were generic enough to apply to any failure. Data provisioning left environments in inconsistent states. The on-call engineer stops trusting it and goes back to doing things by hand. The problem is rarely the model. It is the engineering around the model. Production-grade AI systems require the same rigor as any other software: quality gates, bounded failure modes, auditable outputs, and clear contracts about what the system will and will not do autonomously. Most AI QA integrations skip all of this, ship a thin wrapper around a prompt, and wonder why adoption stalls. The Core Gap An LLM that returns plausible text is not the same as a system that produces reliably structured, quality-gated, auditable output. Bridging that gap is a software engineering problem, not a prompt engineering problem. The six patterns in this article address it directly. Each pattern is described independently so you can adopt any subset into an existing system. A reference implementation that applies all six is described in Section 8. 2. Pattern 1 — Cognitive Loops, Not API Calls The Problem A single LLM call with a try/except block around it is not a production system. It has no concept of output quality, no recovery strategy, and no visibility into what happened between the prompt and the response. When it fails, it fails silently and completely. The Pattern Replace the raw API call with a Perceive-Think-Act-Observe (PTAO) cognitive loop. Each phase is a discrete, inspectable step with its own inputs and outputs: P Perceive Detect intent, classify the request, tokenise the input, surface context signals T Think Select output strategy, build the enriched prompt, set the quality threshold A Act Stream the LLM call, accumulate raw output, emit progress events to consumers O Observe Score output against rubrics, emit telemetry, decide PASS / RETRY / WEAK The Quality Gate The OBSERVE phase is the critical addition most systems omit. It runs a rubric-based quality score on the raw output. A rubric is simply a list of (label, regex_pattern) pairs that check for required structural elements. If the ratio of passing checks falls below a threshold, the loop injects a correction instruction and retries — at most once, keeping worst-case cost to two LLM calls. Python # Quality gate logic — fast-pass for long structured responses, # rubric scoring for everything else. if len(output) >= FAST_PASS_CHARS and output.startswith("#"): quality = "PASS" # long markdown output — skip rubric else: passed = sum(1 for label, pattern in rubric if re.search(pattern, output)) ratio = passed / len(rubric) quality = "PASS" if ratio >= THRESHOLD else "RETRY" if quality == "RETRY" and attempt <= MAX_RETRIES: prompt += "\n\n[RETRY] Prior attempt was incomplete. Include all required sections." Why This Matters The loop turns each LLM interaction into an inspectable, telemetry-emitting pipeline stage. Every phase transition can be streamed to the UI as a named SSE event, giving engineers real-time visibility into what the model is doing — not a spinner followed by a blob of text. Key Insight: Cap MAX_RETRIES at 1. Two LLM calls are an acceptable worst-case cost. Three or more and you are not fixing a quality problem — you have the wrong prompt strategy. The fast-pass threshold prevents retry storms on large, well-structured responses that happen to miss an optional rubric keyword. 3. Pattern 2 — The 96% Token Problem The Problem Every AI agent framework loads its full context on every call: skill definitions, tool schemas, system prompts, few-shot examples. For a typical agent setup, this adds 8,000 or more tokens of overhead to every request — before a single character of user input is included. At scale, this is a latency and cost problem that compounds with every invocation. COMPONENT TOKENS REQUIRED FOR THIS TASK? Full skill definition (SKILL.md) 4,741 No — the task is already routed Tool JSON schemas 2,800 No — tool use is not required Agent boot system prompt 600 No — a task-scoped prompt replaces this Total naïve overhead 8,141 None of it The Pattern: Context Slicing Inject only what the model needs for the specific task at hand. A task-scoped system prompt — typically 100-200 tokens — frames the domain context without loading the full agent boot sequence. The result: the total payload for a typical request collapses from 8,000+ tokens to under 300. Naive (8,300+) 8,300 tokens Sliced (~250) 250 Context slicing is not about removing context — it is about matching context to the task. A test generation task needs the output schema and the requirement document. It does not need the data provisioning protocol, the dedup algorithm, or the report format spec. Load only what is relevant to the current intent. Implementation Build a lightweight context slicer that measures the actual payload sent on each call and computes the reduction against a measured naïve baseline. Surface this as telemetry: Python def compute_report(user_prompt, raw_output, output_tokens) -> SlicerReport: raw_user = estimate_tokens(user_prompt) optimized = raw_user + TASK_SYSTEM_PROMPT_TOKENS # e.g. 148 naive = optimized + NAIVE_OVERHEAD_TOKENS # e.g. + 8,141 return SlicerReport( optimized_payload = optimized, naive_payload = naive, reduction_pct = (naive - optimized) / naive * 100, # latency_saved, cost_saved_pct derived from reduction_pct ) Measured Result: A context slicer measuring 148-token task prompts against an 8,141-token naïve baseline yields a 96.4% payload reduction on every call. At high invocation rates, this translates directly to lower API costs and meaningfully faster end-to-end response times. 4. Pattern 3 — Align Capabilities to the Lifecycle The Problem AI tooling that presents itself as a feature menu forces engineers to make a meta-decision before every task: which tool applies here? That decision is cognitive overhead that does not produce test coverage or defect insight. It also produces inconsistent usage — different engineers reach for different tools at the same lifecycle stage. The Pattern Map each AI capability to a specific SDLC phase. The engineer's current phase determines which capability is active — not a dropdown, not a search box, not a knowledge of which prompt to write. 1 Requirement Analysis [DESIGN] 2 Data Provisioning [SETUP] 3 Failure Analysis [EXECUTE] 4 Suite Maintenance [MAINTAIN] 5 Reporting [REPORT] Each phase boundary is also a data handoff point. The outputs of earlier phases feed naturally into later ones: requirement analysis produces test cases that populate the execution suite; failure analysis produces confirmed defects that feed the triage report; data provisioning produces entity IDs that feed the prep report. The lifecycle ordering is not cosmetic — it is an architectural constraint that prevents accidental coupling. Capability Detection Intent detection at the PERCEIVE phase routes each request to the correct capability automatically, without requiring the user to navigate a menu: Python # Keyword-based capability routing at the PERCEIVE phase CAPABILITY_SIGNALS = { "prd": ["requirement", "user story", "acceptance criteria", "jira"], "data": ["provision", "fixture", "seller", "stage env"], "rca": ["timeouterror", "stack trace", "nosuchelement", "failing test"], "dedup": ["duplicate", "scan", "redundant", "similar tests"], "triage": ["defect", "sla", "severity", "priority", "breach"], } Design Principle: Phase-ordering also makes it easy to answer "what should I do next?" at any point in the cycle. An engineer finishing a requirement analysis session is automatically positioned at the data provisioning step — no context-switching required. 5. Pattern 4 — The Self-Heal Safety Contract The Problem Self-healing test automation is compelling on paper. In practice, systems that apply fixes unconditionally — without confidence scoring, without bounding the blast radius, without a rollback guarantee — make things worse. An engineer who discovers that an automated system silently modified their test suite loses trust in the entire platform, not just the healing feature. The Pattern: A Formal Safety Contract Define a self-heal contract before writing any auto-remediation code. The contract specifies exactly when the system may act, how many times it may retry, and what it must do if all attempts fail: CONTRACT CLAUSE RULE RATIONALE Confidence gate Confidence score ≥ 85% required to auto-apply Low-confidence fixes have a higher chance of masking real defects Effort classification Only LOW-effort fixes auto-apply HIGH-effort changes carry architectural risk; require human review Retry budget Maximum 3 fix attempts per failure Bounded failure prevents cascading mutations to the test file Scope constraint Re-run only the failing test, not the full suite Avoids surfacing unrelated failures that pollute the signal Rollback guarantee Restore original file if all fixes fail The system must always leave the codebase in a known-good state Commit prohibition Never commit or push changes autonomously Human approval required before any change enters version control Python # Self-heal contract enforced at prompt construction time if self_heal_enabled: prompt += ( "\n[SELF-HEAL CONTRACT]" "\n- Apply fix only if confidence >= 85% AND effort = LOW" "\n- Re-run the failing test only — not the full suite" "\n- Retry up to 3 different fixes if the first does not pass" "\n- Restore original file if all fixes fail" "\n- Never commit, push, or stage any file change" ) Key Insight: Encoding the contract in the prompt rather than only in application code means the model itself is aware of the constraints. This improves adherence on borderline cases — the model learns to self-qualify its confidence before acting, rather than always proposing a fix and letting the application layer decide. The Triage Pipeline Self-healing and defect triage should be connected, not siloed. When a failure survives the healing contract — meaning the model classified it as a real defect rather than a selector issue or environment flake — it should automatically feed into the defect queue with its classification metadata intact. This eliminates the manual step of copying failure information from a test run into a defect tracker. 6. Pattern 5 — Analysis Is AI's Lane; Action Is Human's The Problem The instinct when building AI tooling is to make it do as much as possible. For irreversible operations — deleting files, merging test cases, modifying production data — this instinct is wrong. An AI that deletes what it classifies as a duplicate test may be deleting a regression anchor or a platform-specific edge case that looks identical at the semantic level but covers different runtime behaviour. The Pattern Hard-code read-only analysis as the default for any operation that cannot be trivially undone. The AI identifies, scores, and recommends. The engineer decides and acts. This is not a limitation of the system — it is a deliberate trust boundary that makes the AI's recommendations credible. Python # Read-only constraint enforced at prompt construction time. # The AI cannot override this in its output — the constraint # is architectural, not a suggestion. DEDUP_PROMPT = """ Scan the test repository at `{repo_path}` for duplicates. Similarity threshold: {threshold}%. Do NOT delete, modify, or rename any files — read-only analysis only. Return: JSON with summary + groups[], each with a recommended action (DELETE | MERGE | REVIEW), confidence score, and rationale. """ The Recommendation Schema A strong read-only analysis output is not just a list of duplicates. It provides enough context for the engineer to act confidently without re-examining every file: FIELD PURPOSE group_id Stable identifier for the duplicate cluster similarity_pct Semantic similarity score across the group action DELETE / MERGE / REVIEW — the AI's recommendation rationale Plain-English explanation of why this action was chosen risk NONE / LOW / MEDIUM — estimated blast radius if the action is taken keep_file Which file to preserve if the group is merged or deleted Why This Builds Trust: Practitioners adopt AI tools faster when the tool is honest about what it knows it cannot safely decide. A system that says "here are 7 groups; I recommend deleting 2, merging 2, and reviewing 3 — here is my reasoning" is far more credible than one that silently performs deletions and reports a summary. Trust is built through transparency, not through autonomy. 7. Pattern 6 — The Execution Store The Problem Most AI integrations produce output and discard it. The next run has no memory of the last one. Reports have to be regenerated from scratch. Debugging a bad output requires re-running the entire pipeline. There is no audit trail for compliance, no replayability for debugging, and no shared source of truth for downstream consumers. The Pattern Persist every AI interaction to a typed execution store — a key-value structure indexed by capability type, containing the prompt, raw output, rendered output, and a timestamp. Reports, dashboards, and downstream capabilities read directly from this store. Nothing regenerates data it could reuse. Python # Execution store: typed entries, one per capability. # Persisted after every OBSERVE phase regardless of quality outcome. STORE_SCHEMA = { "capability": str, # "prd" | "data" | "rca" | "dedup" | "triage" "prompt": str, # the enriched prompt sent to the model "raw": str, # raw LLM output (unparsed) "rendered": str, # rendered HTML or structured format "ts": str, # ISO 8601 timestamp "telemetry": dict, # PTAO phase metadata, token counts, quality score } Cross-Capability Data Flow The execution store enables a pattern where capability outputs compose naturally without explicit integration code. A defect confirmed by the failure analysis capability is written to the store under the "triage" key. The defect triage report reads from that key on every page load — no webhook, no event bus, no manual copy-paste required. Design Note: Start with a flat JSON file. It is human-readable, zero-dependency, and sufficient for dozens of daily invocations. Migrate to a database only when audit retention, concurrent writes, or query complexity actually demand it — not before. YAGNI applies to persistence layers too. What the Store Enables CONSUMER WHAT IT READS VALUE DELIVERED Defect Triage Report triage key Live defect matrix without re-running analysis Dedup Viewer dedup key Latest duplicate groups without re-scanning the repo Data Prep Report data key Entity IDs and session state from last provisioning run Unified Dashboard All keys Cross-capability health in one view Compliance audit All keys + timestamps Full history of what the AI was asked, what it produced, and when 8. Reference Implementation and Results All six patterns were implemented together in a quality engineering platform for a high-volume e-commerce fulfillment operation. The platform — built over a single weekend as an internal hackathon project using FastAPI, HTMX, and Playwright MCP — applies the patterns across five SDLC-ordered capabilities: PRD-to-Suite, Agent-Driven Data Provisioning, Failure RCA with Self-Heal, Test Deduplication, and a live Reporting layer backed by the execution store. Architecture in One Diagram Measured Outcomes 70-80% QA Cycle Time Reduction 96%+ Token Payload Reduction 93% Duplicate Detection Rate 85% Test Coverage Achieved <45s PRD to Test Suite Time 0 Autonomous Commits Made Most Important Metric: The zero autonomous commits figure is not a limitation — it is the point. The self-heal contract, read-only analysis, and human-gated action patterns kept the system in an advisory role throughout. Engineers adopted it because it did not try to make decisions that were theirs to make. Technology Stack LAYER TECHNOLOGY ROLE IN THE PATTERNS API layer FastAPI (Python) Async-native; SSE via StreamingResponse for PTAO phase events Frontend HTMX + Jinja2 HTML-over-the-wire; zero JS framework; server-side report rendering from execution store Browser automation Playwright MCP LLM calls browser_navigate, browser_snapshot as MCP tools — no custom runner code AI runtime Internal AI platform Network-gated; no external API keys; task-scoped prompts via context slicing Persistence Flat JSON file Execution store — typed by capability, read by all report consumers Test framework Playwright + TypeScript Target of self-heal patches; isolated per feature; config checked in 9. What to Take Back to Your Codebase None of the six patterns require a new framework, a large model budget, or a multi-sprint migration. Each can be adopted incrementally into an existing AI integration: PATTERN MINIMUM VIABLE ADOPTION Cognitive Loop Add an OBSERVE step after your existing LLM call. Check for one required structural element. Retry once if absent. Context Slicing Measure your current prompt token count. Remove everything not needed for the specific task type. Track the reduction. Lifecycle Alignment Group your AI features by the SDLC phase they serve. Surface the right one based on the engineer's current context. Self-Heal Contract Add confidence and effort fields to your fix output schema. Gate autonomous application on both. Hardcode the rollback path. Read-Only Default For any irreversible operation, make the AI return a recommendation with a rationale. Remove the execution path entirely from the model's output. Execution Store Write each AI response to a keyed file alongside its prompt and timestamp. Point your next report at the file instead of re-running the analysis. The broader lesson is that AI quality engineering earns adoption through predictability, not capability. A system that reliably produces structured output, never silently modifies files, and leaves a full audit trail will be used every day. A system that occasionally produces brilliant results but fails unpredictably and leaves no trace will be abandoned. Final Thought The Zen of Python applies here: explicit is better than implicit, errors should never pass silently, and in the face of ambiguity, refuse the temptation to guess. Every one of these six patterns is a direct application of that philosophy to AI system design. The model is not magic — it is a component. Treat it like one.

You have a checkout flow. You have 40 tests. They're green. Now: what happens when a payment webhook arrives after the user cancels? What happens when a retry lands on a session that already expired? What happens on the fourth failed attempt when autoRenew is off and the period boundary has already passed? You don't know. Not because you're careless — because a state machine with 6 states, 7 actions, and 3 payload values has thousands of reachable (state, action, data) combinations, and your 40 tests visit 40 of them. The bugs that page you at 2 am live in the other several thousand. Polygraph is a Claude Code plugin and standalone CLI that walks all of them. Why You'd Bother Polygraph is for stateful code: reducers, workflow engines, protocol handlers, session managers, order state machines, anything with a dispatch(state, action) shape. If your code is a pile of pure functions, go use property-based testing. If it's a state machine, keep reading. Narrow on shape, not on language. The model reads your source in whatever it's written in — you name it once as lang in the contract — and the trace format is just NDJSON, so any runtime that can log a {pre, action, data, post} line per step can feed it. What's always JavaScript is the derived spec and your rules, because those are what the replayer and model checker execute on Node. What you get back is not a lint warning. It's a shortest action sequence that reaches a state violating a rule you wrote. Something like: Shell ✗ never-charged-twice [state] — pred returned false init {"status":"new","attempts":0,"hasDue":false} CREATE({}) -> {"status":"active","attempts":0,"hasDue":false} RENEW_CHARGE({"result":"5xx"}) -> {"status":"grace","attempts":1,"hasDue":true} RENEW_CHARGE({"result":"ok"}) -> {"status":"grace","attempts":2,"hasDue":true} That's a repro. You paste it into a test file, and you have a failing test in about ninety seconds. A real one: On a production SaaS subscription-billing machine, Polygraph flagged a disagreement on exactly one window: a 5xx from the payment processor during renewal moved the row to grace and marked it due, when the dunning path in the same codebase correctly treated 5xx as ambiguous. The next retry rotated the idempotency key. If the 503'd transfer had actually settled, the customer got charged twice. A human reviewer had found the same bug by hand; five independent model-derived readings of the source landed on it blind. And in the controlled seeded-bug eval, the split is worth knowing: replaying real traces against the derived spec found 0 of 5 seeded bugs. Model checking found 5 of 5, with counterexamples. Trace replay tells you whether to trust the model. Model checking is where the bugs actually are. What It Actually Does Three artifacts, all diffable, all in your repo: 1. contract.json — the scope. Which state fields matter, which actions the machine accepts, what data each action can carry, which states are terminal, and lang is the language your source is written in. 2. A spec — a JavaScript model of your code, written by an LLM from your source (whatever language that is). It's a strict SAM v2 module: every action it ignores has to say why via reject(reason), it can't hide bookkeeping state, and it declares its own action/data domains — so the checker knows what to explore with zero config. Several specs are generated independently and vote, so one bad generation doesn't decide anything. JavaScript export const stateInvariants = [ { name: 'locked-only-at-limit', pred: (s) => s.status !== 'locked' || s.attempts >= 3 }, ]; export const transitionInvariants = [ { name: 'expired-never-verifies', pred: (pre, action, data, post) => !(action === 'ATTEMPT' && data?.expired) || post.status !== 'verified' }, ]; 3. invariants.mjs — your rules, as plain JS predicates: This part is yours and can't be automated away. Code with a bug is a perfectly faithful description of the wrong behavior. Invariants are where your intent enters the system. Then two checks run. Replay asks "is the spec faithful?" Real traces ({pre, action, data, post} windows, captured by wrapping your dispatch once) are replayed against each spec, with positive and negative controls proving the harness can tell good from bad. Model check asks "where are the bugs?" It iterates the faithful spec exhaustively from init against your invariants and prints the shortest path to every violation. The Caveats "Exhaustive" means exhaustive over the finite (action, data) domain declared in your contract. A machine whose behavior depends on unbounded counters or arbitrary strings is checked only at the representative values someone chose. That's the standard TLA+ modeling move, and the gap between declared domain and real data is real.It's a consistency check, not a proof. A clean run means your code's observable behavior matches an independent reading of its own source. Nothing more.Every finding is a lead to investigate, not a verdict. There is no triage step that discharges "real invariant break with no observable consequence."It's experimental and not peer-reviewed. Don't make it your only safeguard on safety-critical code. API Key and Cost Only three things call the Anthropic API: spec generation, code authoring (polygen), and polynv's optional headless invariant harvest. You need ANTHROPIC_API_KEY in your environment for those, including inside Claude Code, where the skills shell out to the same scripts and do not use your session credentials. Ballpark, on a typical machine: you runkey?costverify.mjs --source … (generate + replay)yes~$0.50polygen.mjs --intent … (author new code — JS/TS output only)yes~$2replay saved specs, model check, --tla, polyvers, polynv, polyrunno$0 That second row is the load-bearing one. Everything that checks (replay, the exhaustive model check, version gating, the mutation grade, TLC escalation) is keyless, local, and deterministic on Node ≥ 20. Which is precisely what makes CI viable: you commit the spec, and the gate re-runs it on every merge request for free. No key in CI, no per-MR API bill, no nondeterminism in your pipeline. That gate is polygate, and there's a GitLab reference implementation at <POLYGATE_GITLAB_URL> — a .gitlab-ci.yml you can copy that runs corpus validation, replay, and the model check against your committed artifacts and fails the MR on a violation. (Contrast: Specula, the closest comparable agentic TLA+ pipeline, reports a median of $57 and 3.7 hours per system. Excellent tool, structurally can't run on every MR.) Getting Started Prerequisite, and it's a hard one: The stateful code has to be runnable in isolation, because traces are ground truth from the code actually executing. A clean step boundary: a dispatch, reducer, or handler, from experience, Claude will refactor it easily for you. If it only runs against a live DB or device, stand up doubles first (in Claude Code, the agent will build them). Note this is the only place your language matters, and only for convenience: the bundled withTracing / tapReducer helpers are JS, so a Go or Python machine means writing the {pre, action, data, post} NDJSON lines yourself. It's about ten lines. Zero-cost first (no key, five minutes): Shell git clone https://github.com/cognitive-fab/polygraph cd polygraph && npm test # validates the bundled corpus, runs the controls npm run verify:turnstile-v2 # replays bundled specs — see the output shape Then on your own machine, as a plugin: Shell /plugin marketplace add cognitive-fab/polygraph /plugin install polygraph@polygraph …and just ask: "verify this state machine", or /polygraph:polygraph for the guided end-to-end run (Claude drafts the contract, instruments the boundary, captures traces, runs controls, triages with you). Trace capture is historically what made this expensive; it's the step the agent now carries. Or plain CLI, no Claude Code: Wrap your dispatch once, projecting only the contract's observable keys (JS shown; in another language, emit the same NDJSON shape by hand): JavaScript import { withTracing } from '<plugin>/scripts/instrument/trace-emitter.mjs'; const dispatch = withTracing( rawDispatch, () => ({ status: m.status }),'traces/s1_normal.ndjson' ); Note --source takes your real file, in your real language: Shell node scripts/validate_corpus.mjs contract.json traces/ # no key node scripts/verify.mjs --contract contract.json --source src/machine.ts \ --traces traces/ --model opus-5 --n 5 --out out/ # key, ~$0.50 That writes out/findings.md and the generated specs to out/specs/. Commit the winning one, and from then on the loop is free: Shell node scripts/check.mjs --spec out/specs/spec_0.js --contract contract.json \ --invariants invariants.mjs # no key, forever There's no default model: pass --model. Use opus-5 or better; deriving a faithful transition function is a hard reasoning task and lighter models don't clear the bar. If you see empty specs, you lowered --max-tokens below what the reasoning block needs; put it back to 32000. Apache-2.0. The method is written up in arXiv:2607.05076. Your test suite is a sample. This is the census.

Artificial intelligence has rapidly become a core capability in modern software development. For Java developers, integrating these capabilities into existing enterprise applications no longer requires learning entirely new frameworks or interacting directly with complex AI APIs. Spring AI bridges this gap by providing a familiar Spring programming model for working with large language models (LLMs) from providers such as OpenAI, Google Gemini, and others. In this article, we will build a simple AI-powered REST API using Spring Boot and Spring AI while exploring practices that help move beyond proof-of-concept implementations toward production-ready enterprise applications. A Typical Enterprise Architecture Rather than allowing clients to communicate directly with an AI provider, enterprise applications usually introduce a service layer responsible for security, validation, business logic, and monitoring. Plain Text Client Application │ ▼ Spring Boot REST API │ Validation & Business Logic │ ▼ Spring AI ChatClient │ ▼ Large Language Model (OpenAI / Gemini / Azure) This architecture keeps AI interactions behind your own APIs, allowing you to enforce authentication, authorization, logging, rate limiting, and governance without exposing provider-specific details to consumers. Creating the Spring Boot Project Getting started with Spring AI is straightforward. The application requires Spring Web, Validation, and the Spring AI starter. XML <properties> <java.version>21</java.version> <spring-ai.version>1.0.0</spring-ai.version> </properties> <dependencyManagement> <dependencies> <dependency> <groupId>org.springframework.ai</groupId> <artifactId>spring-ai-bom</artifactId> <version>${spring-ai.version}</version> <type>pom</type> <scope>import</scope> </dependency> </dependencies> </dependencyManagement> <dependencies> <dependency> <groupId>org.springframework.boot</groupId> <artifactId>spring-boot-starter-web</artifactId> </dependency> <dependency> <groupId>org.springframework.boot</groupId> <artifactId>spring-boot-starter-validation</artifactId> </dependency> <dependency> <groupId>org.springframework.ai</groupId> <artifactId>spring-ai-starter-model-openai</artifactId> </dependency> </dependencies> Configuring the AI Model One security practice I strongly recommend is avoiding hard-coded API keys or model names inside the application. Instead, configure them using environment variables or an enterprise secrets manager. YAML spring: ai: openai: api-key: ${OPENAI_API_KEY} chat: options: model: ${OPENAI_MODEL} temperature: 0.2 The lower temperature value encourages more deterministic responses, which is generally preferable for technical or business APIs where consistency matters. Designing the API Contract Rather than exposing raw AI requests directly, I prefer defining explicit request and response models. This keeps the REST API independent of the underlying AI provider and makes future changes much easier. Java import jakarta.validation.constraints.NotBlank; import jakarta.validation.constraints.Size; public record AIQuestionRequest( @NotBlank @Size(max = 2000) String question, String audience ) {} Response model: Java public record AIAnswerResponse( String answer ) {} Configuring the ChatClient Spring AI's ChatClient is responsible for interacting with the configured language model. Rather than repeating the same instructions in every request, we can configure a default system prompt once. Java @Configuration public class AIConfiguration { @Bean ChatClient chatClient(ChatClient.Builder builder) { return builder .defaultSystem(""" You are an experienced Java architect. Provide concise, accurate, production-ready answers. Never invent APIs. If uncertain, clearly state your assumptions. """) .build(); } } The system prompt establishes the overall behavior of the assistant. It ensures that every request follows the same guidelines, resulting in more predictable responses. Implementing the AI Service One architectural decision I recommend is keeping AI interactions inside a dedicated service layer rather than calling the language model directly from a controller. This separation makes the code easier to test, improves maintainability, and keeps business logic independent of the web layer. Java @Service public class TechnicalAssistantService { private final ChatClient chatClient; public TechnicalAssistantService(ChatClient chatClient) { this.chatClient = chatClient; } public AIAnswerResponse answer(AIQuestionRequest request) { String audience = request.audience() == null ? "Java Developer" : request.audience(); String response = chatClient.prompt() .user(user -> user .text(""" Explain the following question. Audience: {audience} Question: {question} Keep the answer under 300 words. """) .param("audience", audience) .param("question", request.question())) .call() .content(); return new AIAnswerResponse(response); } } Creating the REST Controller With the service layer complete, exposing the AI functionality through a REST endpoint becomes straightforward. Java @RestController @RequestMapping("/api/ai") public class AIController { private final TechnicalAssistantService assistantService; public AIController(TechnicalAssistantService assistantService) { this.assistantService = assistantService; } @PostMapping("/ask") public ResponseEntity<AIAnswerResponse> ask( @Valid @RequestBody AIQuestionRequest request) { return ResponseEntity.ok( assistantService.answer(request)); } } The endpoint accepts a JSON request, validates the input, invokes the service layer, and returns a structured response. Returning Structured AI Responses Many AI examples simply return text. While that's useful for chat applications, enterprise APIs usually need predictable JSON responses. For example, suppose we want AI to review Java code. Instead of receiving one long paragraph, we can ask the model to return structured data. Java public record CodeReviewResponse( String summary, List<String> strengths, List<String>issues, List<String>recommendations, String riskLevel ){} Now Spring AI can map the model response directly into a Java object. Java public CodeReviewResponse review(String sourceCode){ return chatClient.prompt() .system(""" You are a Senior Java Architect. Review the code for correctness, performance, security and maintainability. """) .user(sourceCode) .call() .entity(CodeReviewResponse.class); } This approach is much cleaner than parsing raw JSON or trying to interpret free-form responses manually. It also keeps the rest of the application strongly typed. Streaming AI Responses Some AI responses can take several seconds to complete. Rather than waiting until the entire response has been generated, Spring AI allows responses to be streamed back to the client. Java @RestController @RequestMapping("/api/ai") public class StreamingController { private final ChatClient chatClient; public StreamingController(ChatClient chatClient) { this.chatClient = chatClient; } @GetMapping( value="/stream", produces = MediaType.TEXT_EVENT_STREAM_VALUE) public Flux<String> stream( @RequestParam String question){ return chatClient.prompt() .user(question) .stream() .content(); } } Streaming significantly improves the user experience because clients can begin displaying the answer immediately instead of waiting for the complete response. This is especially useful for chat applications and AI assistants. Cache Responses When Appropriate AI requests introduce additional latency and cost because every request communicates with an external model. If the same prompt is frequently submitted, consider caching the response. Spring Cache makes this simple. Java @Service public class TechnicalAssistantService { @Cacheable("aiResponses") public AIAnswerResponse answer(AIQuestionRequest request) { // AI Call } } Caching works particularly well for frequently asked questions, product descriptions, technical explanations, and internal knowledge articles. Dynamic or user-specific responses generally should not be cached unless the cache key includes the relevant context. Final Thoughts What stands out to me is that Spring AI allows AI capabilities to become a natural extension of an existing Spring Boot application rather than requiring an entirely new architecture. Whether the goal is building an internal knowledge assistant, generating summaries, reviewing code, or automating repetitive tasks, the development experience remains consistent with the rest of the Spring ecosystem. That said, building a production-ready AI application involves much more than calling an LLM. Prompt design, security, validation, observability, performance, and cost management all play a critical role in delivering reliable solutions.