An AI Agent Found a WireGuard Bug in GKE: A Blueprint for Agent-Led Triage on Your Own Fleet

It was a Sunday, users were seeing errors that made no sense, and the engineer on point did something he had never done in an incident before: he pointed an AI agent at millions of log lines and started asking questions. Minutes later the agent surfaced the thread that unraveled the whole thing — a Kubernetes networking daemon crashing roughly once an hour, panicking inside its WireGuard encryption module. That incident, written up by Lovable infrastructure engineer Sascha Eglau in April 2026, is the clearest real-world demo yet of what agent-led triage actually looks like: not an agent replacing on-call, but an agent doing first-response evidence gathering fast enough that humans spend the incident reasoning instead of grepping.

Here is the full trail, with the agent-found and human-found steps marked honestly — because the split between the two is the entire point.

The agent cracked the case open. Humans closed it. If you run a self-hosted fleet, that division of labor is a blueprint worth copying — with one hard boundary in place first.

Two failures stacked, not one​

Lovable's platform creates more than 50 sandboxes per second at peak hours. At that churn rate, anything that slows pod networking startup is immediately user-facing — and something was wobbling. The symptoms had no obvious pattern, which Eglau calls "the worst kind of pattern."

The agent's contribution came first and was decisive: pointed at ClickHouse logs, it surfaced that the anetd pods — Google's Cilium-based networking daemon for GKE — were crash-looping at roughly one restart per hour per pod. Digging into the crash dumps revealed a concurrent map-access panic: multiple goroutines reading and writing the same Go map without locking.

The critical detail was where: inside the WireGuard module of anetd. WireGuard itself is an open-source protocol Google does not own — but Google owns the integration code that tracks WireGuard connections inside anetd, and that is where the unsynchronized map lived. The bug was in Google's code, which meant the fix needed Google.

On the Sunday call, Google's recommendation was pragmatic: disable transparent node-to-node encryption entirely, bypassing the buggy module. The tradeoff was explicit — less encryption between nodes, but the cluster already ran on Google's private network, and stable beats perfect during an active incident. They rolled it out, restarted anetd, and the crashes stopped. For about four hours, the team thought it was over.

Then layer two surfaced. Random connection failures to Valkey, the in-memory data store. CPU climbing, node count doubled, errors continuing. This is where a second engineer, Erik, reasoned that unchanged application code plus a networking change meant the problem was deeper in the stack. He ran tcpdump on nodes and found the smoking gun in Wireshark: Destination unreachable (Fragmentation needed).

The mechanism: with WireGuard enabled, the cluster used an MTU of 1420 bytes to leave room for encryption overhead (WireGuard adds roughly 60 bytes of IP + UDP + crypto headers to each packet). With encryption disabled, nodes should have moved to the full 1500-byte Ethernet MTU — but nodes that had not been restarted yet were still configured for 1420. Traffic between mismatched nodes fragmented or died depending on which node each pod happened to land on, which is exactly why Valkey connections failed mysteriously and intermittently. The fix was a full node reroll for a consistent MTU. Google, for its part, recognized the anetd bug from the evidence and has since patched it — Lovable's extreme pod churn had surfaced something Google's own testing had not caught.

Eglau's summary line is the thesis of this post: distributed systems rarely fail in just one layer. Fixing layer one revealed layer two hiding underneath.

What the agent actually did — and what it did not​

This is the part most retellings get wrong, so let us be precise. The agent did not diagnose a WireGuard bug. It did something narrower and, for on-call purposes, more valuable: it converted millions of log lines into one suspicious, checkable fact.

Every agent action in this incident was read-only evidence gathering: querying logs, counting restarts, surfacing a pattern no human had time to grep for. Every mutation — disabling encryption, restarting daemons, rerolling nodes — was a human decision, and the biggest one (turning off node-to-node encryption) was an explicit security tradeoff no agent should make unilaterally.

That split held up because this incident predates agents having write access at all. But the industry is already moving past read-only demos: Elastic shipped an agentic Kubernetes investigation workflow with MCP-based observability skills in June 2026, HolmesGPT now pairs diagnosis tools with approval-gated remediation servers, and a small ecosystem of open-source triage agents (Hivemind, SentinelOps, k8s-incident-triage) dispatches LLM agents straight from alerts. The question for your fleet is no longer whether an agent does first response — it is where you draw the line between investigating and acting.

The trust boundary: read-only evidence vs mutating the network​

Here is the matrix worth stealing. Tier 1 actions an agent may take autonomously, any time, on any alert. Tier 2 actions require an explicit human approval — a Slack button, a PR merge, a voice confirmation — every single time, no matter how confident the agent sounds.

Three implementation notes make this boundary real instead of aspirational:

RBAC is the fence, not the prompt. System instructions asking an agent to behave are not access control. The agent needs its own dedicated ServiceAccount bound to a read-only ClusterRole — get and list on the resources it triages, nothing else — so that even a prompt-injected or hallucinating agent cannot mutate the cluster. If someone bypasses the tool layer entirely, RBAC still says no. Several Kubernetes MCP servers now ship exactly this posture: GiantSwarm's ships a readonly RBAC profile, and the kubeleash project gates every destructive action per kube context even when pointed at an over-privileged kubeconfig.

Secrets must be unreadable, not just unlogged. An agent that can read logs can read whatever applications log — including accidentally logged credentials. Mask secret values at the tool layer (--mask-secrets style redaction) and deny the agent's ServiceAccount read access to Secret objects. Both layers, because either one alone has a bypass story.

The agent is an actor in the audit log, not an invisible helper. Attribute every agent API call to its own identity (its ServiceAccount, an Impersonate-Extra-agent header, or equivalent) so the audit trail answers "who restarted that DaemonSet — the human or the triage bot?" HolmesGPT's remediation server design does this explicitly: read-only tools return success freely, while run_kubectl_command returns APPROVAL_REQUIRED until a human signs off. That asymmetry belongs in your audit log too.

Notice what this boundary would have changed about the Lovable incident: nothing — and that is the point. The agent's actual contribution (log queries, restart counts) sits entirely in Tier 1. The boundary does not slow down the thing that worked; it fences off the things that could have gone wrong if the agent had been allowed to "help" further, like restarting anetd fleet-wide or toggling encryption itself.

A first-response runbook you can build this week​

You do not need a platform team to get most of this value. The Lovable setup was, at its core, an agent with log access and good questions. Here is the minimal version for a self-hosted fleet:

1. Give one agent read-only cluster access. Dedicated ServiceAccount, read-only ClusterRole, no Secret reads. Wire it to your log store (ClickHouse, Loki, Elasticsearch — wherever kubectl logs output lands) plus metrics and events.
2. Wire alerts to investigations, not just pages. When an alert fires, the agent's job is to produce an evidence bundle within minutes: affected pods/nodes, restart counts, recent events, log excerpts, and a ranked list of hypotheses with links to the raw data. The human still gets paged — but pages now arrive with a brief attached.
3. Add one approval gate before any mutation. Start with a single choke point: the agent may propose remediation commands, but executing anything beyond Tier 1 requires a human click. Tools like HolmesGPT's approval-required tools or a simple Slack-approval webhook implement this in an afternoon.
4. Audit and review for a month before widening. Log every agent action under the agent's identity. Review weekly: which hypotheses were right, which Tier 1 queries were most useful, and whether any Tier 2 proposal would have been wrong. Widen autonomy only with receipts.
5. Rehearse the layered failure. Lovable's incident had two layers, and the second one only appeared after "fixing" the first. Your runbook should require a monitoring hold after any remediation — watch the dashboards for what emerges next instead of declaring victory when the first symptom clears.

That last item is the cheapest lesson in the whole story and the easiest to skip. The team logged off after four quiet hours. Then Slack lit up again.

Delegate the triage, keep the trigger​

The honest moral of Lovable's Sunday is not "agents can debug Kubernetes networking." It is narrower and more useful: an agent with log access found in minutes a pattern — one crash per hour, buried in millions of lines — that humans might have chased for hours, and everything after that was human judgment: reading dumps, pushing back on a vendor, capturing packets, rerolling nodes. Eglau's verdict ("I haven't gone back") is about investigation leverage, not autonomous remediation.

That is exactly the shape agent-operations should take on a fleet you own: agents as tireless Tier 1 investigators with read-only access and their own audit identity, humans holding every trigger that changes production state. Build the read-only tier this week, add the approval gate before you need it, and the next time your CNI starts crash-looping at one restart per hour, the page you get will already name the suspect.

Bex.co is the open-source, AI-native Render alternative — push a git repo, get a running HTTPS service on machines you own, with agents as first-class operators. Star the repo on GitHub or deploy your first app today.

Sources​

• Sascha Eglau, "A Bug Hunt in Our Kubernetes Cluster," Lovable blog, April 2026 — lovable.dev/blog/hunting-networking-bugs-in-kubernetes
• Last Week in Cloud Native, 2026 week 19 articles; Hacker News discussion, May 1 2026
• Elastic, "agentic Kubernetes investigation workflow and MCP-based observability skills," June 2026
• HolmesGPT kubernetes-remediation MCP server spec (approval-required tools design)
• Cilium transparent encryption docs (WireGuard mode, MTU considerations)