One of the world's largest retailers · productionProduction
AI SRE: LLM-driven incident correlation
A production system that consolidates correlated alerts into a single incident, generates a root-cause summary, and produces a virtual runbook grounded in the customer telemetry.
OpenObserveClaudeAnthropic SDKOpenTelemetryKubernetes
architecturescroll to pan →
Problem
One failure fires dozens or hundreds of alerts across logs, metrics, and traces. Engineers burn hours reconstructing what actually happened before they can act. Alert storms bury the signal.
My role
I co-developed the system (partial build) and lead customer deployments. For self-hosted customers we deploy into their infrastructure. For cloud we operate it managed.
What shipped
- +Correlated alerts consolidated into a single incident
- +Root-cause summary generated per incident
- +Virtual runbook grounded in the customer telemetry
- +Delivery: discovery on the alerting landscape, telemetry pipeline integration, per-environment correlation tuning
Outcome
- →In production at one of the world's largest retailers and at partner-managed environments
- →Output trust validated with customer teams before it enters on-call workflows