Understanding System Observability

If your agent runs for 10 minutes, you need to know what happened at minute 3. High-performing teams don’t just log outputs. They trace steps. For long-running agents, you need: 🔍 Step-level execution logs 🧠 Intermediate reasoning checkpoints 🛠 Tool invocation metadata 📊 Token consumption visibility ⏱ Latency per action Without tracing: 1. You can’t debug hallucinations. 2. You can’t explain decisions. 3. You can’t detect drift. 4. You can’t prove compliance. Observability turns agents from magic into machinery. If your only metric is “final output quality,” you’re blind to systemic fragility. Would you ship a distributed system without tracing? Then why ship agents without it? #AIEngineering #Observability #AIOps #AgentSystems #Tracing #ProductionAI #SystemReliability #ModelMonitoring #LLMOps #EnterpriseAI

Imagine you’re a data engineer. It’s 3 AM on a Friday. You’re home, asleep, but back in the office, your data pipeline is busy. And tonight, a bug sneaks into production. Just a tiny change, a single wrong script runs. Nobody notices at first (well, cause they’re busy on the weekend) Suddenly, fake transactions start landing in your main tables. Customer data gets mixed up. Dashboards shift, and nobody knows why. Years ago, this would have been a nightmare. By Monday morning, you’d be scrambling to guess what happened and where the mess began. But tonight is different,  Because every step your data takes is recorded. Your system has data lineage. It’s like having security cameras for your entire pipeline. Every row knows where it came from, every script leaves a footprint, and every transformation is logged. So when you wake up and check the dashboard, you see the story: ↬ What script ran ↬ When it started ↬ Which tables it touched ↬ Where the wrong values spread You hit rewind, isolate the problem, and fix only what needs fixing. And as a result, there will be no mass panic or engineers searching endlessly.  You can get answers even at 3 AM! This is the power of data lineage and observability: That’s how you sleep well as a data engineer. That’s how you build pipelines you can trust. – P.S: Did you learn something new with this post? Would you want more posts like this?

Most systems detect node or master failures using simple polling, and while this approach sounds straightforward, it has an interesting reliability issue... The typical approach is to observe a node directly. This usually means pinging it, checking if a port is open, or running a lightweight query to confirm it is alive. On paper, this seems fine, but all of these methods share the same weakness - what if the observer itself is wrong? In a distributed setup, network glitches are normal. Temporary packet loss, routing hiccups, or partial network partitions can easily make a healthy node appear unreachable to the observer. The usual way to deal with this is to retry multiple times and declare failure after the n-th consecutive failure. This creates a classic tradeoff. If n is small (or polling happens frequently), failure detection becomes fast, but false positives increase. A short-lived network blip can trigger an unnecessary failover, which can sometimes be more disruptive than the original issue. If n is large (or polling intervals are longer), false positives decrease, but real failures take longer to detect. That delay directly increases downtime. But there is a more reliable way to think about this problem when you already have a cluster of nodes available. Instead of relying on a single observer repeatedly polling a target node, you can allow multiple nodes in the cluster to independently perform health checks. The system then treats a node as failed only when a majority of observers agree that the node is unreachable. This consensus-based approach reduces the risk of false positives caused by network partitioning. Even if one observer loses connectivity, the rest of the cluster can still provide an accurate view of system health. Consensus is costly, so this approach is not the most cost-efficient. However, it can be very useful if your system is large enough and distributed across multiple geographies.

It takes 10 minutes to fix a crash. It takes 3 days to find a silent data quality error. Most data architectures fail quietly. They don't break on launch day. They break on day 90, when nobody remembers the decision that caused it. Here’s what that looks like in practice: INGESTION ✕ Pull everything, filter later ✓ Validate at the edge Bad data is cheapest to kill at entry. Let it in and it travels everywhere. ✕ No schema contract with the source ✓ Agree on types and nullability upfront Upstream changes without a contract = your problem, not theirs. STORAGE ✕ One giant table, query it all ✓ Partition by how the data is actually read Wrong partitioning doesn’t error. It just costs you forever. ✕ Mix raw and transformed in the same layer ✓ Separate raw, cleaned, and serving You will always need to reprocess. Design for it. TRANSFORMATION ✕ Transform then validate ✓ Validate then transform You can’t trust output built on dirty input. ✕ Logic buried inside SQL joins ✓ Explicit, tested, documented If only one person understands it, it’s already a liability. ORCHESTRATION ✕ Trigger jobs on a schedule ✓ Trigger on data arrival and completeness Schedules don’t know if the data actually showed up. ✕ No dependency mapping ✓ Every pipeline knows what it needs before it runs Silent upstream failure + blind downstream trigger = corrupted output, zero alerts. OBSERVABILITY ✕ Alert only when the pipeline crashes ✓ Alert when data behaves unexpectedly A crash is obvious. Quietly wrong data isn’t. GOVERNANCE ✕ Give access on request, document once ✓ Define ownership, lineage, and living docs When something breaks, lineage is the difference between 10 minutes and 3 days.. Most engineers optimize what’s visible. Great architects design for what breaks. Before your next diagram, ask: What hidden failure am I introducing today? 💡 Save this for your next design review. 🔖Tag an engineer who needs to see it. #data #engineering #systemdesign #cloud #intellingence #business #growth

Roughly seven months ago there was a noticeable shift in how engineers at Resolve AI worked, and no I'm not talking about AI coding. What I noticed is that they stopped querying observability tools directly. Instead of opening dashboards or crafting queries manually, they started routing investigations through Resolve. It wasn't a top-down decision, engineers just gradually stopped going direct once the agents were consistently getting them accurate answers faster. I talked about this recently with Tom Wilkie, Manoj Acharya, and Cyril TOVENA on the Grafana Labs Big Tent Podcast. Every major observability platform was designed for human operators, from the query languages to the interfaces. All of it assumes a human is on the other end, running a handful of queries during an incident. Agents don't work that way. They query constantly, pull from multiple systems simultaneously, and need API throughput most platforms weren't architected to handle. When they hit a bottleneck, they don't wait. They route around it. There's a practical question here for anyone running production systems at scale. What's the API throughput ceiling before you get rate-limited? What happens to your bill when query volume goes up by an order of magnitude? Can an agent traverse metrics, logs, and traces in a single investigation without hitting access gaps between tools? The organizations moving quickest on this are evaluating observability vendors not on dashboard quality or ingestion pricing, but on whether the platform is ready to be operated by agents as the primary interface. Lastly, the old argument in observability has always been that consolidation wins (i.e., all data on a single platform, reduced tool sprawl), but I think agents actually reverse that logic. If an agent can query five specialized systems and synthesize results faster than a human can navigate one general-purpose platform, the case for specialized tooling gets stronger, not weaker. The glue between systems isn't a human anymore. It's the agent. Listen to the full episode below in the comments.

Another SRE anti-pattern stems from not having adequate observability which is the practice of understanding how systems behave by collecting and analyzing data from various sources. Without adequate observability, SREs and engineering teams are essentially flying blind, making it difficult to identify, diagnose, and resolve issues effectively. Some of the problems and consequences associated with inadequate observability can be: - Increased Mean Time to Detection (MTTD): With inadequate observability, it takes longer to detect issues in your system. This can lead to increased downtime and negatively impact user experience. - Increased Mean Time to Resolution (MTTR): Once you detect a problem, troubleshooting becomes more challenging without proper observability tools and data. This results in longer downtime and more significant disruptions. - Difficulty in Root Cause Analysis: Without comprehensive data on system performance, it's hard to pinpoint the root causes of incidents. This can lead to "fixing symptoms" rather than addressing underlying issues, leading to recurring problems. - Inefficient Capacity Planning: Inadequate observability can hinder your ability to monitor resource utilization and plan for scaling. This may result in overprovisioning or underprovisioning resources, both of which can be costly. - Limited Understanding of User Behavior: Observability isn't just about monitoring system internals; it also includes understanding user interactions. Without this knowledge, it's challenging to optimize your system for user needs and preferences. What are some of the practices and tools that SREs can use? - Logging: Implement structured logging and ensure that logs are collected, centralized, and easily searchable. Use logging toolings like Elasticsearch, Fluentd, or Loki. - Metrics: Define relevant metrics for your system and collect them using tools like Prometheus or InfluxDB. - Distributed Tracing: Implement distributed tracing to track requests as they traverse various services. Tools like Jaeger and OpenTelemetry can help you gain insights into service dependencies and latency issues. - Event Tracking: Capture important events and errors in your system using event tracking systems like Kafka or RabbitMQ. - Monitoring and Alerting: Set up monitoring and alerting systems that can notify you of critical issues in real time. Tools like Grafana or Prometheus help in this regard. - Anomaly Detection: Consider implementing anomaly detection techniques to automatically identify unusual behavior in your system. - User Analytics: Collect data on user behavior and interactions to better understand user needs and improve the user experience. By investing in observability, teams can proactively identify and address issues, improve system reliability, and provide a better overall user experience. It's a fundamental aspect of SRE principles and practices.

🚀 Building Observable Infrastructure: Why Automation + Instrumentation = Production Excellence and Customer Success After building our platform's infrastructure and application automation pipeline, I wanted to share why combining Infrastructure as Code with deep observability isn't optional—it's foundational as shown in screenshots implemented on Google Cloud. The Challenge: Manual infrastructure provisioning and application onboarding creates consistency gaps, slow deployments, and zero visibility into what's actually happening in production. When something breaks at 3 AM, you're debugging blind. The Solution: Modular Terraform + OpenTelemetry from Day One with our approach centered on three principles: 1️⃣ Modular, Well architected Terraform modules as reusable building blocks. Each service (Argo CD, Rollouts, Sonar, Tempo) gets its own module. This means: 1. Consistent deployment patterns across environments 2. Version-controlled infrastructure state 3. Self-service onboarding for dev teams 2️⃣ OpenTelemetry Instrumentation of every application during onboarding as a minimum specification. This allows capturing: 1. Distributed traces across our apps / services / nodes (Graph) 2. Golden signals (latency, traffic, errors, saturation) 3. Custom business metrics that matter. 3️⃣ Single Pane of Glass Observability Our Grafana dashboards aggregate everything: service health, trace data, build pipelines, resource utilization. When an alert fires, we have context immediately—not 50 tabs of different tools. Real Impact: → Application onboarding dropped from days to hours → Mean time to resolution decreased by 60%+ (actual trace data > guessing) → nfrastructure drift: eliminated through automated state management → Dev teams can self-service without waiting on platform engineering Key Learnings: → Modular Terraform requires discipline up front but pays dividends at scale. → OpenTelemetry context propagation consistent across your stack. → Dashboards should tell a story by organising by user journey. → Automation without observability is just faster failure. You need both. The Technical Stack: → Terraform for infrastructure provisioning → ArgoCD for GitOps-based deployments → OpenTelemetry for distributed tracing and metrics → Tempo for trace storage → Grafana for unified visualisation The screenshot shows our command center : → Active services → Full trace visibility → Automated deployments with comprehensive health monitoring. Bottom line: Modern platform engineering isn't about choosing between automation OR observability. It's about building systems where both are inherent to the architecture. When infrastructure is code and telemetry is built-in, you get reliability, velocity, and visibility in one package. Curious how others are approaching this? What's your observability strategy look like in automated environments? #DevOps #PlatformEngineering #Observability #InfrastructureAsCode #OpenTelemetry #SRE #CloudNative

If you can't see what an agent does, you can't improve it, you can't debug it, and you can't trust it. It's crazy how many teams are building agents with no way to understand what they're doing. Literally ZERO observability. This is probably one of the first questions I ask every new team I meet: Can you show me the traces of a few executions of your agents? Nada. Zero. Nilch. Large language models make bad decisions all the time. Agents fail, and you won't realize it until somebody complains. At a minimum, every agent you build should produce traces showing the full request flow, latency analysis, and system-level performance metrics. This alone will surface 80% of operational issues. But ideally, you can do something much better and capture all of the following: • Model interactions • Token usage • Timing and performance metadata • Event execution If you want reliable agents, Observability is not optional.

Distributed Tracing and Observability at scale - Your infra is on fire. You have logs, metrics, and traces. And you still can't find the bug. Here's why — and how to fix it. ────────────────────────── Most teams instrument all three signals but never connect them. ♦️ Logs tell you what happened. ♦️Metrics tell you how bad it got. ♦️Traces tell you exactly why — and in which service. But only if you do this one thing: → Embed trace_id in every log line. → Tag every metric with the service that owns it. → Propagate context across every async boundary. Without this, you're debugging three disconnected puzzles instead of one picture. ────────────────────────── The GIF above shows what it looks like when it actually works: ↳ A request enters your API gateway ↳ Flows through auth, order, payment services ↳ Slows down at postgres (233ms — the bottleneck) ↳ Your trace waterfall catches it in seconds ↳ Your logs and metrics are already correlated to the same trace_id You go from "something is slow" to "postgres query at line 47" in under a minute. That's the difference between observability and just collecting data. ────────────────────────── The stack that makes this possible: → OpenTelemetry for vendor-neutral instrumentation → Tail sampling to keep errors, drop noise → Grafana Tempo or Datadog APM for the waterfall → Structured logs with trace context baked in I wrote a deep-dive on this — full code, sampling config, and the common mistakes that break your traces silently. Link in comments. ───────────────────────── #SystemDesign #Observability #DistributedSystems #OpenTelemetry #SoftwareEngineering

Most engineers obsess over which model to use. GPT-4o or Claude. Benchmarks. Latency. Cost. They ship. The system breaks. And they have no idea why. Because model choice is a one-time decision. 𝗢𝗯𝘀𝗲𝗿𝘃𝗮𝗯𝗶𝗹𝗶𝘁𝘆 is what tells you what is actually happening after that. Without traces, 𝘆𝗼𝘂 𝗮𝗿𝗲 𝗴𝘂𝗲𝘀𝘀𝗶𝗻𝗴. You do not know: • if retrieval is returning the wrong chunks • if the model is ignoring context • if certain queries are quietly failing You just see bad outputs. I have seen teams spend days swapping models trying to fix this. The issue is almost never the model. One team I worked with had a broken chunking strategy. Sentences were getting split mid-thought. The model never even saw complete context. With proper tracing, we found it in 20 minutes. That is the difference. Model upgrades change behavior. Sometimes they fix things. Sometimes they just move the failure somewhere else. Without evals and traces, you cannot tell which one happened. The teams that move fastest in production are not the ones picking the “best” model. They are the ones who can: • trace a request end-to-end • inspect retrieval outputs • evaluate whether responses are actually grounded 𝗧𝗵𝗶𝘀 𝗶𝘀 𝗮 𝘀𝘆𝘀𝘁𝗲𝗺. 𝗡𝗼𝘁 𝗮 𝗴𝘂𝗲𝘀𝘀. Observability is not overhead. It is your ability to improve anything after you ship. Pick your model. Then instrument everything. If you are building with LLMs and you do not have this setup yet, 𝘆𝗼𝘂 𝗮𝗿𝗲 𝗱𝗲𝗯𝘂𝗴𝗴𝗶𝗻𝗴 𝗯𝗹𝗶𝗻𝗱. I break this down a lot more in my content.