It's 2 a.m. and a pod is in CrashLoopBackOff. Whoever is on call opens five terminals and starts the usual routine: describe, logs, get events, check the deployment, check the config map, check the node, then do it again for the service upstream. Usually the answer is somewhere in that pile. The hard part is getting to it.
KubeTrace started from a simple question: what if an agent ran that routine for you, quickly and the same way every time, without being allowed to break anything?
Put the constraint first
Most "AI for ops" demos let the model run kubectl apply. That looks great in a video and is unacceptable in a regulated production cluster. No serious team will give a language model write access to the cluster that runs its payments system.
So the first decision came before any model choice: the agent can look, but it can never touch. Every path into a cluster is read-only. There's no clever prompt that says "please don't delete things". Nothing in the system can delete anything in the first place.
That sounds limiting. In practice it's what made the product usable at all. Once mutation is off the table, the security conversation goes from "absolutely not" to "when can we try it?"
The agent diagnoses. The operator decides. That split is the whole product.
The stack at a glance
| Layer | What I used |
|---|---|
| Agent reasoning | Python, LangGraph, LLMs via an OpenAI-compatible gateway |
| Tool access | Model Context Protocol (MCP) |
| Cluster access | Kubernetes API, read-only |
| Knowledge | RAG over Qdrant with local embeddings |
| Real-time UI | TypeScript front end, streamed over WebSockets |
| State | Redis |
I'm keeping the internals at this level on purpose. The ideas carry over to other systems, and the implementation details belong to the product.
Why MCP
The first prototype hard-coded every integration: a function for events, another for logs, another for metrics. Each new data source meant touching the agent, the prompts and the tests at the same time.
Moving tool access behind MCP turned that inside out. The agent talks to one protocol. Each capability is a server that describes itself. Adding a new kind of evidence means adding a server, not rewriting the agent. It also gave me one place to enforce the read-only boundary and audit every call the agent made.
It isn't free. Designing good tool schemas takes real work, and every call goes through an extra hop. Even so, I'd make the same choice again.
Correlation is the real job
Fetching data is the easy part. Real incidents rarely live in one log stream. They show up as an event in one namespace, a restart in another and a resource that quietly ran out somewhere else.
The agent's job is to correlate. It pulls events, logs and resource state together, forms a hypothesis, collects more evidence for or against it, and repeats until the story holds up. LangGraph fits this well because the investigation is a loop with explicit state, not a single prompt.
Next to the live investigation, KubeTrace runs continuous validation suites across security, networking, performance, storage and APIs, so drift gets caught before it turns into an incident. Each run produces a report a compliance team can actually use.
What I took away
- A safety constraint can be a product feature. "Read-only" was the line that got KubeTrace into production clusters.
- Pick the protocol before you pile up tools. MCP saved me from a long tail of one-off integrations.
- Agents need loops, not longer prompts. A good root cause comes from hypothesis → evidence → revision, not from cramming the whole cluster into a context window.
KubeTrace is live at kubetrace.net. The architecture is in the case study.
All posts