case study · 2025
KubeTrace
An AI-driven Kubernetes testing, security, and reliability platform — the entire platform, built solo. It continuously validates clusters for security, reliability, and configuration drift — and pinpoints root cause in seconds.
Hours → seconds
Root-cause time
5 automated suites
Test domains
Read-only, real-time
Cluster access
What needed solving
Kubernetes clusters fail in ways that are hard to trace. Teams running regulated production workloads had no continuous way to validate security posture, catch configuration drift, or find the root cause of an incident without an expert manually walking the cluster — a process that took hours while services stayed degraded.
How it fits together
A product interface sits on a Python platform that orchestrates two engines: an AI root-cause agent and a set of automated test suites. The agent reaches cluster tooling only through MCP, and every path into a cluster is read-only, so investigations can run against live regulated production without any risk of changing it.
Product
Web interface
TypeScript
Platform
Orchestrator
Python · pipelines
Report builder
compliance output
Engines
Root-cause agent
events·logs·state
Test suites
5 domains
Access
MCP client + tools
one interface
Cluster connect
real-time
Infra
Kubernetes
multi-cluster
// stack
The calls that shaped it
MCP as the integration layer
Agents reach every cluster tool through one consistent interface, instead of a hard-coded integration per tool. Adding a tool means adding an MCP server, not changing the agent.
trade-off · Every tool needs a carefully designed schema the model can use well, and each call takes an extra hop through the protocol.
Read-only by construction
Regulated production clusters can be investigated live with no risk of mutation. That's what made it possible to point the platform at real incidents at all.
trade-off · The platform diagnoses but never remediates. Fixes stay with the operator.
Correlate signals across namespaces and clusters
Real failures surface as a mix of events, logs and resource state spread over several namespaces. The agent correlates all three rather than reading one log stream.
trade-off · Reading everything blows up context and latency, so the agent works as a LangGraph loop: it forms a hypothesis and fetches only the evidence that tests it.
Validation as continuous test suites
Security, networking, performance, storage and API checks run as automated suites, so drift is caught continuously and every run produces a compliance-ready report.
trade-off · Suites add read load to the cluster, so they run on a schedule rather than continuously hammering the API server.
Where it landed
KubeTrace runs in production at kubetrace.net, cutting root-cause investigation from a manual expert-hours exercise to a matter of seconds, with continuous validation running against multi-cluster regulated environments. Every part of the platform — architecture, backend, AI agents and interface — was designed and built by one engineer, solo.
Hours → seconds
Root-cause investigation
5 automated suites
Validation domains
Zero, by design
Writes to production clusters
One engineer, solo
Built by
What I'd do differently
- →I'd put MCP in from day one. The early hard-coded integrations had to be rewritten once the tool count grew, and every one of them was a place the read-only boundary had to be re-checked.
- →In production the hard part was rarely fetching data. It was deciding which evidence mattered. Structuring the agent as an explicit hypothesis-and-evidence loop did more for answer quality than any prompt change.
- →To another engineer building an MCP agent: design the safety boundary into the tools, not the prompt. If a tool can't mutate anything, no prompt injection can make it.
// also offered as freelance services
// next step
Hiring for this kind of work?
I'm open to full-time AI Engineer roles. Grab my resume, or email me and I'll walk you through how this was built.