I work in infra. My day-to-day is Terraform, node pools, and YAML. I figured this would be a relaxed conversation. It was not. I ended up asking for questions to be repeated more than once, not because they were unclear, but because I genuinely needed time to think.

I am writing this because the reflection itself was worth it, whatever comes next.

The Cost Reduction That Took Down Production

One story I kept coming back to: a node pool migration I ran for cost reduction. I moved from an expensive node pool to a cheaper one. I knew there was risk. I moved forward anyway.

Production went down. Nearly an entire site.

My initial framing of this story was almost defensive: the company got cost savings, and I got experience handling a real incident. But when I tried to actually say that out loud, I heard how it sounded. Like I was using a small saving to justify a downtime I could have prevented.

The real problem was not that I moved fast. It was that I had no rollback plan. No communication to stakeholders about what might go wrong. I moved because I was confident in myself, not because I had done the math.

If you cannot explain what changed in your process after an incident, the story is not done yet. The incident itself is not proof of growth. What you did differently afterward is.

From “Help Me Set Up MCP” to Something Company-Wide

A lighter story, but with broader reach.

My lead asked for help integrating an MCP server into his Claude Code setup. The scope was personal, just for him.

When I looked at it more carefully, I saw something bigger. The same setup, extended properly, could give non-technical people across the company access to infra and internal tools through natural language in chat, without needing a technical background or waiting on the infra team. So I took it further and integrated it into Google Chat, the tool everyone already uses.

What I took from this: sometimes the gap between a junior and senior move is not about the size of the scope you are handed. It is how far you see from a small one.

There was also a part of this I had to reframe. For something that touched workflows outside my team, I brought my manager in to handle stakeholder communication. My first instinct was to frame that as “coming from me carries less weight.” That framing is off. The better read: it was a strategic choice. I drove the technical side and ran the demo. My manager helped position it at the right level for a broader audience. Different framing, different meaning.

Disagreeing With Enough Data to Back It Up

The team was planning to migrate from Nginx to HAProxy for ingress. I disagreed. HAProxy uses a single config for all services, so if something breaks, everything breaks. The risk was too high for a decision that, in my view, had been researched too quickly without a proper impact assessment.

I brought my concern to the team lead with a trade-off analysis attached, not just a rejection. And I offered an alternative: stay on Nginx and look for a better solution in parallel.

Disagreement is fine. What makes it constructive is bringing data and an alternative. Without those, it is just friction.

The Question That Made Me Go Quiet

A scenario: a small health clinic with self-managed data gets acquired by a large hospital group. They want everything, including patient records, moved to the cloud. There are regulatory concerns around medical data. Nobody on your team has experience in this domain. What do you do?

I answered with a standard approach: assess first, map all services and data, identify what can and cannot move, then execute.

Then the follow-up came: “What would you not touch?”

I went quiet. Asked for it to be repeated. Genuinely needed a moment. I have not worked in a regulated industry, so I was reasoning from principles, not experience. My answer: I would not touch the business processes. Those should stay unchanged. Only where the workloads and data live should move.

Looking back, that answer was not wrong, just too general. What the question was probably pointing at: data with hard regulatory constraints, the most critical and least reversible core systems, or the compliance sign-off that has to happen before anything sensitive gets touched.

But I answered from what I actually know. I would rather give an honest answer from a limited frame than a polished one I do not actually stand behind.

The Shortcut I Was Afraid to Admit

There was a technical task where I had to provision a database service via Infrastructure as Code. When asked what my biggest sacrifice was, I gave the safe answer: time and planning.

The honest answer: I took a shortcut in that repo. I knew it was not best practice. I did it deliberately to hit the deadline.

I was afraid to say it. I thought it would count against me.

I think that was wrong. Acknowledging that you knowingly took a shortcut as a deliberate trade-off for a specific constraint is a stronger signal than a generic answer. It shows you understood the risk. It also gives you a natural opening to explain what the production version would look like.

If you can explain why you made the shortcut and what you would do differently in production, that is judgment. Hiding it is defensiveness.

Vendor Speed as a Strength That Can Become a Habit

Most of my career has been at vendors. In that world you learn to move fast. Decisions get made on pattern recognition: “this usually works,” “that should be fine.” There is rarely room for long research cycles.

That skill has real value. But I have started to notice it becoming a default even in situations that deserve more time, more careful thinking about what cannot be undone.

The node pool incident is the clearest example. I moved fast, took a trade-off, and did not think hard enough about the downstream impact.

I want to build the other muscle: the one that slows down on purpose, structures the decision before touching anything, and thinks through reversibility first. That one has not had much room to grow.

Near the end of the conversation I asked directly: if my answer to the clinic scenario was off, what would a better one look like? I was worried that would make me look unsure of myself. It did not. It turned into an actual discussion instead of a one-sided evaluation.

What Stayed With Me

An incident story without “and here is what changed afterward” is an unfinished story. The failure is not the point. The process change is.

Disagreement lands when it comes with data and an alternative. Without those, it is just resistance.

Admitting a shortcut is not a weakness, as long as you can explain the trade-off and what production would look like.

Speed is an asset. It becomes a liability when it is the only mode you have.

And it is fine to say “I have not been in that situation, but reasoning from first principles, here is how I would think about it.” That is more useful than a rehearsed answer you do not fully believe.

Sometimes the most valuable thing a hard conversation does is force you to find the gaps in your own thinking. This one did that.

Then someone asked: can I just ask the bot a question? Not a slash command. Just a question, in natural language, inside a Google Chat thread, and get a useful answer back.

That broke what I had.

What I wanted was specific: manage our Kubernetes cluster from anywhere, including from a phone. No terminal, no SSH, no context-switching to a laptop. Type a message in Google Chat, get a real answer, and keep the conversation going across turns in the same thread.

Here is what an actual session looks like:

From listing namespaces to rolling out a deployment to reading pod logs, all in one thread, no terminal needed.

Getting there took more than adding an API key.

Why Not Just Call the API Directly

The obvious move is to call the Anthropic API from the bot. Add a route, drop in an API key, call completions, return the response. Half an hour of work.

I tried that first and hit three problems pretty fast.

Session continuity. Google Chat threads are persistent. A conversation might span hours, with different people jumping in. If every message is a fresh API call with no memory of what came before, you basically have a stateless chatbot that forgets everything. I needed conversation state tied to a thread somehow.

Context injection. The bot already knew things about our infra: namespaces, services, common failure patterns. I wanted Claude to have that context without duplicating it across every caller. Clean injection at request time, not copy-pasted system prompts in three different codebases.

Duplication. I also maintain a Discord bot with a similar purpose. Every time I wanted to change the model or tweak the system prompt, I was doing it in two places. That gets old fast.

So I put a service in the middle.

What the Wrapper Does

The wrapper is a separate service (TypeScript, Fastify, PostgreSQL) sitting between the bots and Claude. Any client can call it without knowing what is running underneath.

The core endpoint is /projects/:name/run:

POST /projects/sre-bot/run
X-Api-Key: <project-key>

{
  "prompt": "What's using the most memory in the staging namespace?",
  "session_id": "spaces/XYZ/threads/ABC",
  "timeout": 270
}

Response:

{
  "output": "The top memory consumers in staging are...",
  "session_id": "spaces/XYZ/threads/ABC",
  "memory_id": 4821,
  "duration_ms": 8340
}

Under the hood, a few things happen that the bot never needs to think about.

Each project has a stored system prompt plus an optional set of skills, which are reusable prompt fragments injected at runtime. The bot project’s system prompt has context about our Kubernetes setup, naming conventions, and what actions are safe to suggest versus what needs explicit confirmation before running.

For session continuity, the wrapper runs Claude Code CLI as a subprocess using --resume <session_id>. The session ID from the first run becomes the input for the next call. The bot just passes back whatever session ID it last received.

The wrapper also supports two runners: Claude and OpenCode. The project config picks which one. From the bot’s side, the API call looks identical either way, so switching backends is a config change, not a code change.

Every prompt and response gets stored in a memory table, tied to project and session. That has already come in handy more than once when something unexpected came back and I needed to trace what happened.

How the Bot Actually Uses It

The bot code is intentionally thin. When a user @mentions the bot or sends a DM without a slash command, the message hits a handler that does exactly three things:

session_id = _load_session_id(thread_name)
reply_text, new_session_id = _call_ai(user_text, session_id)
_save_session_id(thread_name, space_name, new_session_id)

That is the entire integration. Load the session, call the wrapper, save the new session ID. Conversation state lives inside the Claude Code session the wrapper manages, plus a thread-to-session mapping in the bot’s own PostgreSQL database.

Session Continuity in Practice

Sessions are keyed to the Google Chat thread name, which looks like spaces/SPACE_ID/threads/THREAD_ID. Each thread gets its own conversation, isolated from everything else. When someone comes back to a thread two hours later with a follow-up question, the wrapper resumes the same Claude Code session with full context of what was discussed.

The Discord bot does the same thing, just without delegating to the wrapper. There, the bot manages the session ID itself. Here, it hands that off to the wrapper and just stores a key.

The cost is one extra database roundtrip per message. In practice that is under two milliseconds, completely invisible when the response itself takes 5 to 10 seconds. Worth it to keep the bot code clean.

Thread Summarization

One feature that validated the wrapper approach: thread summarization. If someone writes “summarize” or “ringkas” in a message, instead of sending that literally to Claude, the bot fetches the full thread history from the Google Chat API, builds it into a transcript, and sends that as the prompt.

if _is_summarize_request(user_text):
    messages = list_thread_messages(space_name, thread_name)
    prompt = _build_summarize_prompt(messages)
    reply_text, _ = _call_ai(prompt, session_id=None)

The session_id=None is intentional. A summary is a one-shot thing with no prior context needed, so the wrapper starts fresh. If I had hardwired the Anthropic API into the bot, this would work the same way. But with the wrapper, the summarization prompt and model choice live at the project level. Changing them does not touch the bot.

Why Claude Code CLI Instead of the SDK

People ask this when I walk through the setup. Why run the CLI as a subprocess rather than just using the SDK?

Claude Code already handles the things I would have to build from scratch: multi-turn conversation state, tool use, context window management, and session continuity with --resume. When I hand it a prompt via subprocess, I get something that maintains a coherent conversation across calls. The --output-format json flag keeps the output structured and predictable.

For an internal ops tool, that simplicity is worth more than the control you get from direct API calls. I do not need custom tool definitions or fine-grained context management. I need something that can answer infrastructure questions and occasionally suggest a command that needs a confirmation step before it runs. Claude Code handles that without extra work on my end.

The real trade-off is that the wrapper depends on Claude Code being installed and authenticated on the host. That would be a deal-breaker for a customer-facing product. For an internal tool on a server I already own, it is fine.

The Full Picture

Here is how everything fits together:

Each Google Chat thread maps to one Claude Code session. The session ID travels from the wrapper to the bot, gets stored in PostgreSQL, and comes back on the next message. The thread is the conversation boundary, persistent and isolated per channel.

The MCP servers are where the actual work happens. The kubectl MCP server is how listing namespaces, rolling out deployments, and reading pod logs actually gets done. Claude figures out what to run, the MCP server executes it, and the result feeds back into the next turn as context.

The Wrapper as Infrastructure

What makes this setup worth maintaining is that the wrapper is real infrastructure now. It has its own deployment, its own database, its own API keys scoped per project. Adding a new bot means creating a project, writing a system prompt, and pointing at /projects/:name/run. The AI layer is shared. Project isolation keeps contexts separate.

The Google Chat bot is one project. The Discord bot is another. A Slack or Telegram integration would each get their own project with their own system prompt and session space.

That keeps each bot small. They do not carry any AI logic. Each one is just an interface: take input, call the wrapper, return the output.

There is also a cost angle worth mentioning. Claude Code runs on a subscription, not per-token billing. For the usage pattern of an internal ops bot (a few multi-turn conversations per day, each with tool calls and some reasoning in between), per-token API costs accumulate faster than a flat subscription. With the wrapper, multiple bots share one Claude Code instance. Ten bots pay once, not per token per bot.

I did not design this as a platform from the start. I built the wrapper when I got tired of copying session management code into a second bot. Whether it stays as is or grows into something bigger depends on how many more bots end up pointing at it.

That sentence took a while to sit right. For the first few months after the team adopted AI coding assistants into the workflow, everything felt like a win. Pull requests were moving faster. Features that used to take three days were done in one. Someone on the team joked that we had finally become a “10x team.” And statistically, the numbers backed it up. We went from pushing to production roughly once a day to doing it three times.

Nobody stopped to ask whether our security posture had kept up.

The Problem Nobody Named

Here is the thing about velocity: it does not come with a warning label. When your team starts shipping faster, it feels like progress, because it is. But it also means more code, more dependencies, more configuration files, more credentials, all moving at a pace that humans were not originally designed to review carefully.

Three things broke quietly before we noticed.

The first was secret leakage. Not a dramatic breach, just a developer committing a test config file with a database password baked in. Easy to do when you are moving fast. The password was rotated before anything happened, but only because someone caught it in a peer review two days later. That gap of two days is the problem.

The second was harder to see. Code written with AI assistance is often confidently written. It looks correct. It passes the eye test. But it does not understand your internal security policies, your compliance requirements, or the specific threat model of your system. A function that handles user input might be technically functional and still be missing input validation. A Dockerfile might work perfectly and still be running as root. The code does what it was asked to do. It just was not asked the right questions.

The third was dependency creep. New libraries get suggested, developers install them, and nobody checks CVEs. Not because engineers are careless, but because at 3x velocity, the moment you finish reviewing one new dependency someone else has already merged three more. The surface area of your supply chain quietly expands faster than any human can track manually.

The DevOps Question Changed

For most of the last decade, the central DevOps question was: how do we ship faster? Shorter feedback loops. Smaller batches. Continuous deployment. Kill the release bottleneck. The whole discipline was organized around removing friction between code and production.

That question got answered. Faster CI runners, better tooling, leaner review processes. And then AI coding assistants came along and compressed what remained. Suddenly the answer was not just “faster” but “much faster, with fewer people.” The friction mostly disappeared.

Now we are sitting with a follow-up question the original DevOps playbook never fully addressed: how do we ship fast safely?

That is not a security team question anymore. It is a DevOps question. Because the bottleneck is no longer deployment frequency. It is the gap between how fast code is being written and how thoroughly it is being checked. If your pipeline was designed around a team that shipped once a day, it was probably fine for engineers to eyeball dependencies and scan configs manually. At 3x velocity with that approach, things start slipping through.

DevSecOps used to feel like a compliance exercise, something you bolted on before an audit. What it actually is, when done right, is the engineering discipline that makes high velocity survivable. The question is not whether to add security to your pipeline. It is how to add it without creating the new bottleneck you spent years trying to remove.

What the Pipeline Needs to Look Like

The answer is not to slow down. The answer is to build a pre-check zone that runs before your code ever touches CI, a set of automated gates that catch the obvious things so humans can focus on the non-obvious ones.

The flow looks like this:

local commit → pre-commit hooks → push → SAST → SCA → PR gate → deploy

Every stage has a job. Pre-commit hooks are your first line: fast, local, zero infrastructure required. SAST catches code-level vulnerabilities before merge. SCA scans your dependency tree for known CVEs. The PR gate is where everything must be green before a human review even begins. Only then does code move toward deployment.

The critical thing is that this is not optional at any stage. A pipeline that only runs SAST in CI but skips pre-commit hooks gives developers an 8-minute feedback loop instead of a 30-second one. That difference changes behavior. People start ignoring the checks.

The Tool Stack

Here is what a practical setup looks like, split into what I consider non-negotiable and what you can add later:

One thing worth calling out: gitleaks needs to run in two places. Local pre-commit catches secrets before they are ever committed. CI catches anything that slipped through, maybe someone bypassed the hook, maybe they pushed from a machine without the hooks installed. Defense in depth is not paranoia, it is just engineering.

Semgrep is worth spending time on because it is configurable. The default rules cover a lot of ground, but you can write custom rules for your codebase. Things like “never call this internal function without this wrapper” or “always validate input before passing to this library.” That is where the real value shows up at scale.

What Comes Next

This is the architecture and the philosophy, the argument for why shifting security left is not just a buzzword but an operational necessity when your team is shipping at high velocity.

In the next article, I will set all of this up from scratch: gitleaks, semgrep, trivy, the whole pipeline, on Jenkins running in Kubernetes. Every config file, every hook, every CI stage. The kind of walkthrough you can actually follow, not just reference.

Because knowing what to build is step one. Building it is the part that matters.

The standard assumption baked into that responsibility is that you will be at your desk. That you have a terminal open, or can get one open quickly. That your laptop is somewhere nearby, that you can SSH into things, run commands, check logs, and do the actual work.

Most of the time that is true. And then sometimes you are on a commute, or standing in a queue, or halfway through a trip with your laptop sitting at home, and a notification arrives telling you something is down.

That is the problem I was actually trying to solve. Not “how do I automate Kubernetes operations” in the abstract sense, but a much more specific and uncomfortable question: what do I do when I genuinely cannot open a laptop?

Why Discord

The answer I landed on was a Discord bot. Not because it was the most technically interesting choice, but because Discord is already on my phone, the app is solid, and I know it well enough that using it under pressure doesn’t add cognitive load on top of the incident itself.

The threading model turned out to matter more than I expected. Each request from the /ask command spawns its own thread. That thread becomes the workspace for that specific issue, a contained conversation where the bot can ask clarifying questions, present findings, request confirmation before running anything destructive, and show output from commands. It maps naturally onto how debugging actually works, which is iterative and back and forth, not a single command and a single result.

Telegram would work just as well in principle. I know the API, the bot ecosystem is mature, and the mobile experience is comparable. The honest reason I picked Discord is that it is where my expertise already lives. Telegram is not off the table as a future interface, just not the one I started with.

The goal was simple: something I could operate with one hand, on a bad connection, without needing to context switch into a full terminal session. The interface just needed to be good enough to not be the bottleneck during an incident.

The Day demo-ui Went Pending

The first real test came from a report that one of the demo services was inaccessible. The deployment target was Kubernetes, namespace demo, and the symptom was that the UI was simply not responding. No more detail than that.

The kind of thing that, a few months ago, would have sent me looking for the nearest desk.

Instead, I opened Discord, used /ask, typed the request into the modal, and submitted it.

The bot asked for approval to SSH into the Kubernetes node and run kubectl get pods -n demo -o wide. That approval gate is intentional. The bot will not execute shell commands automatically on the first turn. It presents what it wants to run and waits for a yes. Once confirmed, it ran the command and surfaced the result directly in the thread.

The output told the story immediately. demo-ui-fc8b87b7c-lmtwb was sitting at 0/1 Pending, no node assigned, no IP. Every other service in the namespace was running normally. The scheduler had decided it could not place this pod, and the logs inside the container were irrelevant because the container had never started.

The bot’s interpretation was accurate: no node assigned and no IP means the scheduler cannot place the pod. The next step was to check why.

The Root Cause Was a Typo

The bot described the pod, checked its events, and found nothing immediately obvious. It then looked at the PersistentVolumeClaim attached to the deployment.

That is where the problem was.

The PVC named demo-ui-broken-pvc had storageClassName: csi-hostpath-sc-typo. The correct StorageClass on the cluster is csi-hostpath-sc. One extra word, appended to the end of the name, and the scheduler had no idea what to do with the volume claim. The PVC stayed pending. The pod stayed pending. The service stayed down.

Before touching anything, the bot laid out its fix plan:

1. Delete demo-ui-broken-pvc, which has the typo in the StorageClass name
2. Recreate it with the correct StorageClass csi-hostpath-sc
3. The pod will reschedule automatically once the PVC is bound

Then it asked for confirmation. Not as a formality. The delete and recreate of a PVC is a meaningful operation and the bot treats it that way. The message ended with “Type yes to proceed or no to cancel.”

That behavior is by design. The system prompt instructs the bot to confirm before anything risky, and deleting storage claims in a Kubernetes namespace qualifies. Getting confirmation wrong in that direction, asking too often, is recoverable. Getting it wrong in the other direction is not.

Fixed

After the confirmation, the bot executed the sequence.

The PVC was deleted and recreated with the correct StorageClass. The bot then watched the pod status as it transitioned from Pending to ContainerCreating and then to Running. The final message in the thread: “demo-ui is back up and running.”

Root cause in the thread summary: PVC demo-ui-broken-pvc had a typo in the StorageClass name, csi-hostpath-sc-typo instead of csi-hostpath-sc, preventing the pod from being scheduled. Fix applied by deleting and recreating the PVC with the correct name. Pod scheduled and started cleanly.

From the first message to resolution, the entire conversation fit in a single Discord thread. I did not open a terminal. I did not SSH into anything manually. I confirmed two things: the initial command approval and the fix.

How It Actually Works

The tool is around 330 lines of Python spread across a handful of modules. There is no framework doing the heavy lifting. The architecture is straightforward enough to explain in a paragraph.

When a message arrives in a bot-managed thread, the handler builds a prompt by combining a system prompt, an infra runbook with cluster details like node addresses, namespace lists, and common command patterns, and the user’s request. That combined prompt gets passed to the Claude Code CLI via subprocess.

cmd = ["claude", "-p", prompt, "--output-format", "json"]
if session_id and not is_new:
    cmd += ["--resume", session_id]

The response comes back as JSON. The bot parses it and looks for structured action tags in the text.

pattern = r'<action type="(\w+)">(.*?)</action>'

Three action types exist: execute for shell commands, confirm for anything destructive, and input for when the bot needs more information from the user. If none of those tags are present, the response is treated as a plain message and sent directly to the thread. If an execute tag is found, the shell command inside it runs, the output gets captured, and the next turn passes that output back to Claude as context. That loop runs up to five times before the bot gives up and asks for manual intervention.

Session continuity works through Claude Code’s --resume flag. The first turn starts a new session and gets back a session ID. Every subsequent message in the same thread resumes that session. The conversation state lives in Claude Code, not in the bot, which keeps the session management trivially simple.

The Wrapper Approach

The natural question when building something like this is whether to call the Anthropic API directly. It is more control, more flexibility, no dependency on a local binary.

The reason I went the other way is that Claude Code already handles a lot of things that would otherwise require careful implementation. Tool use, context management, prompt formatting, output parsing, the conversation state across turns. When you call the CLI and pass it a prompt, you get a reasoning engine that already knows how to think through multi-step problems. The --output-format json flag makes the output structured and predictable. The --resume flag handles session continuity. None of that requires writing a single line of scaffolding.

The trade-off is real. The bot depends on Claude Code being installed and authenticated on the host. You cannot deploy this to a serverless environment or a container that does not have the CLI. The behavior is also less directly controllable than crafting API calls with explicit tool definitions and system prompts baked into the request.

For an internal ops tool running on a machine I already control, those trade-offs are worth it. The complexity that the wrapper approach avoids is more than the flexibility it gives up, at this scale and for this use case. If this ever needed to handle hundreds of concurrent users or run in a managed cloud environment, the calculus would be different. But it does not, and that is fine.

Small by Design

The 330 lines is not an accident or a sign that the tool is incomplete. It is the result of keeping the scope narrow on purpose.

The bot does not manage deployments. It does not have opinions about your CI pipeline. It does not generate Helm charts or suggest infrastructure improvements or send reports. It answers questions, runs commands you approve, and reports back. That is the entire scope.

Part of why the Claude Code wrapper approach works here is precisely because the intelligence does not need to live in the bot. The bot’s job is to be the interface: take the request, build the context, route the command, return the result. The reasoning, the investigation strategy, the decision about what to look at next, that all lives in Claude. The 330 lines are the plumbing. The thinking is elsewhere.

There is also something honest about keeping it small. A tool you understand completely is a tool you trust in the middle of an incident. When something goes wrong with the bot itself, and it will eventually, I can read the entire codebase in twenty minutes and understand exactly where the failure is. That is not a property you get from a platform.

The original problem was not “I need a powerful automation system.” It was “I need to be able to troubleshoot something from my phone when I am not at a desk.” The tool that solves that problem is one that is always available, easy to invoke, safe enough to use under pressure, and small enough to understand when it misbehaves.

A StorageClass typo kept a service down. A Discord message fixed it. The laptop stayed at home.

The engineering team pulled up the logs. Order service: reservation created. Room service: room assigned. Payment service: charge processed. Everything green. Every service reporting success. But somehow, users were checking into rooms they hadn’t actually paid for. The money wasn’t there.

Nobody could explain it. Not because the system wasn’t logging, it was. Not because there was no observability stack, there was. Grafana was deployed. Loki was ingesting logs from every service. Tempo was ready for traces. The team had spent two sprints setting it all up and had proudly declared themselves production-ready.

And when the moment of truth came, they were just as blind as before.

Everything logged. Nothing traced. And somewhere in the gap between those two things, money disappeared.

The Night Finance Asked a Question Nobody Could Answer

Here’s what the investigation actually looked like.

The booking flow spanned three services. A user would place an order, which triggered a room reservation in the inventory service, followed by a payment charge in the payment service. If payment failed, the reservation should roll back. Clean saga pattern, well-designed on paper.

When the team started digging, the logs were technically there. reservation_id: abc-123 in the order service. room_id: room-42 in the inventory service. transaction_id: txn-789 in the payment service. But here’s the problem: those three IDs had no relationship to each other in the log system. There was no way to answer the simplest question in debugging: did these three log entries come from the same user request?

What the team actually had was three independent event streams that happened to contain similar timestamps. They couldn’t tell if txn-789 was the payment attempt for reservation abc-123, or for a completely different booking. They couldn’t reconstruct what happened for any specific user. They couldn’t see whether the rollback ran or silently failed. The fraud detection logic had triggered on certain transactions, causing payment to return an error, but the rollback that should have followed never executed cleanly, and nobody could trace why.

The logs told them what happened inside each box. Nobody had built anything to tell them what happened between the boxes.

Logs Are Not Traces

This is the distinction that most teams learn too late, usually in a conversation with finance.

A log entry is a record of something that happened inside one service at one point in time. It’s valuable. It’s necessary. But it’s inherently local, it has no awareness of the larger request it’s part of.

A trace is a record of a single request’s journey across every service it touched, with timing at each step, parent-child relationships between operations, and a single ID that ties everything together. When a user places an order, a distributed trace shows you the HTTP call that came in, the inventory check that happened as a result, the payment attempt that followed, and the rollback that triggered when payment failed, all as one unified story, not three separate entries in three separate log streams.

The difference matters most when things go wrong in between services. Log-based debugging forces you to manually correlate events using timestamps and IDs that your own developers happened to include in log messages. Trace-based debugging shows you the exact path, the exact failure point, and the exact moment the saga broke down.

One is archaeology. The other is a flight recorder.

The Three-Piece Puzzle

Here’s where teams get burned, and it’s almost never obvious until you’re staring at Tempo wondering why every service has its own separate trace.

OpenTelemetry distributed tracing requires three components to be present simultaneously. Not two. All three. And the frustrating part is that missing any one of them produces no error, no warning, and no indication that anything is wrong. Your metrics still look fine. Your logs still come in. Your traces still appear in Tempo. They’re just all disconnected from each other.

The propagator. This is what teaches OpenTelemetry how to read and write trace context across service boundaries. Without it, every service starts a brand new trace when it receives a request, because it doesn’t know how to look for an existing trace ID in the incoming headers.

otel.SetTextMapPropagator(propagation.NewCompositeTextMapPropagator(
    propagation.TraceContext{},
    propagation.Baggage{},
))

The instrumented HTTP transport. When your service makes an outbound HTTP call to another service, the trace context lives in the Go context.Context attached to the request. Without an instrumented transport, nothing reads that context and injects the traceparent header into the outgoing request. The downstream service receives the call with no trace context, so it starts a new trace.

http.Client{Transport: otelhttp.NewTransport(http.DefaultTransport)}

The server middleware. When your service receives an inbound HTTP request, it needs to extract the trace context from the traceparent header and attach it to the request context. Without middleware, your handlers have no trace context, so any spans they create float in isolation, disconnected from the upstream caller.

r.Use(otelgin.Middleware("order-svc"))

Three pieces. All required. The propagator defines the language. The transport writes it into outbound requests. The middleware reads it from inbound requests. Remove any one of them and you get the worst possible failure mode for an observability tool: it runs silently, produces data, and gives you false confidence that everything is working.

The hotel booking team had deployed OpenTelemetry across all three services. But the payment service was added later by a different developer who didn’t know about the propagator setup. Two services spoke the same tracing language. One didn’t. The result was three islands of telemetry with no bridges between them.

If you want to see these three components wired together end-to-end, I put together a working demo at github.com/vourteen14/grafana-tempo that shows exactly this pattern across three Go services connected to Grafana, Loki, and Tempo. You can reproduce the broken and working state by pulling one of the three components out and watching the traces fall apart.

What Google’s W3C Bet Tells Us

The propagator mechanism isn’t arbitrary. It implements the W3C Trace Context specification, which became an internet standard precisely because distributed tracing kept breaking when different frameworks and vendors used incompatible header formats.

Before standardization, a request leaving a Java service would carry a Zipkin-format trace header. The Python service receiving it wouldn’t recognize the format, would start a fresh trace, and the correlation was lost. Different vendors, different formats, different incompatibilities. Google, Microsoft, and major observability vendors pushed for a single standard because they’d all watched their customers lose trace correlation at service boundaries.

The traceparent header is what that standardization produced. Every compliant instrumentation library knows how to read it and write it. But “compliant” requires that you explicitly set up the propagator. It doesn’t happen automatically. And because the failure is silent, teams deploy distributed tracing thinking it’s working, run it in production for months, and only discover the broken linkage when they need to debug something specific.

Google’s own Cloud Trace documentation leads with propagation setup. Not metrics. Not dashboards. Propagation. Because without it, nothing else matters.

What Actually Happened to the Money

Once the hotel booking team correctly configured all three components and redeployed, the answer was visible in sixty seconds.

The unified trace showed the full story. An order comes in. Inventory reserves a room, that span is there, with the reservation_id attached as an attribute. Payment service attempts to charge the card, that span is there too, marked red, with a fraud check event that shows the transaction amount exceeded the card’s fraud limit. Payment returns an error. Back in the order service, the rollback logic runs, but there’s a span marked red there as well, with an event that says release_reservation_failed: context deadline exceeded.

The rollback was timing out. The room was already assigned. Payment had failed. But the reservation release call was exceeding its timeout, returning silently, and leaving the room in a reserved state. Users could check in because the room showed as reserved in the system. Finance noticed because the payment never cleared.

The bug wasn’t exotic. A missing timeout configuration in the HTTP client used for the rollback call. But finding it without the distributed trace would have taken days, if it was found at all. The logs had all the individual events. None of them could tell you they were part of the same request, or that the rollback that should have followed a payment failure had silently timed out instead.

The False Confidence Problem

There’s a specific kind of blind spot that observability tooling creates when it’s partially broken. It’s worse than having no tooling at all.

When you have nothing, you know you’re flying blind. You build careful logging, you add manual correlation IDs, you trace things through by hand. It’s slow and painful, but you’re aware of your limitations.

When you have a partially broken observability stack, you believe you’re covered. You show stakeholders the Grafana dashboard. You point at the traces in Tempo. You explain that every request is being tracked. You have the confidence of someone who has done the work, except the work has a gap in it that you can’t see, because broken tracing doesn’t announce itself.

This is what makes the three-piece puzzle dangerous. It’s not a misconfiguration that causes errors. It’s a misconfiguration that causes gaps, and gaps are invisible until the exact moment you need the data that should have been in them.

We’ve gotten good at building observability infrastructure. Helm charts, managed collectors, pre-built Grafana dashboards, cloud-native tracing backends. The tooling has never been more accessible. But accessibility makes it easy to go through the motions of observability without achieving it. A deployed OpenTelemetry collector isn’t the same thing as distributed tracing that works. A Tempo datasource in Grafana isn’t the same thing as traces that are connected across your services.

The hotel booking system had every component in place. Prometheus scraping metrics. Loki aggregating logs. Tempo storing traces. OTEL Collector routing everything to the right backend. The team had done everything right in terms of infrastructure, and nothing right in terms of the three lines that actually make distributed tracing work.

Real observability isn’t a stack you deploy. It’s a property your system either has or doesn’t, and the gap between the two can be as small as a missing propagator configuration.

The money disappears quietly. The traces stay disconnected. And the dashboards stay green the whole time.

Then the principal backend left.

Not immediately, not in a single dramatic incident, but gradually. The kind of collapse that you only recognize clearly in hindsight. Standards started becoming suggestions, and suggestions started becoming optional.

What It Looked Like Before

When a strong architectural authority is present, a lot of things happen quietly and without much ceremony.

New developers get onboarded into a set of norms that already exist. Naming conventions are established and followed because someone is paying attention. Database changes go through a review process, or at minimum, someone is asking the right questions before anything lands in production. The architecture decisions are made by someone whose job it is to think about the system as a whole, not just the feature they are currently building.

This is not magic. It is the result of someone holding the line on standards consistently enough that the standards start to hold themselves.

Teams under this kind of structure tend to have a certain predictability. The DevOps side of the house can plan around it. Infrastructure tooling is built against known patterns. Migrations are manageable. The gap between development and production environments is narrow, and when it is not, someone notices quickly.

It is easy to take this for granted when it is working.

What It Looks Like After

When that person leaves, the vacuum does not stay empty for long. It gets filled, just not with the same thing.

Individual developers make local decisions. Not out of malice, but because there is no one coordinating the global view. One developer creates a table using snake_case because that is the convention in the language they came from. Another uses camelCase because that is what they are used to. There is no one to say that both of them are wrong, not because of their choice specifically, but because divergence itself is the problem.

Database access becomes informal. Tables get created directly against production-equivalent environments without a clear migration path. Imports happen because someone needs the data right now and the process for doing it properly is unclear or absent. Each individual action is defensible on its own terms. The accumulation is not.

The schema starts to drift. And then the environments start to drift.

This is where it gets expensive.

The Dev-Prod Gap

At some point, the development database and the production database stop being the same thing in different environments. They become different things that happen to share a name.

Column data types diverge. A field that was created as VARCHAR in one environment exists as TEXT in another. An integer column in development was backfilled differently in production. A foreign key constraint exists in one place and not the other.

The result is a category of bugs that are genuinely difficult to reason about. Something works correctly in development, passes whatever testing exists, and then behaves differently in production, not because the code is wrong, but because the ground the code is running on is different.

This is the kind of issue that should be caught long before it reaches production. In an environment with proper migration practices and schema versioning, it is caught. In an environment where the database has been managed informally by multiple developers over time, it is a recurring surprise.

The DevOps team, who is responsible for managing these environments, is now operating on infrastructure they cannot fully trust. Every deployment carries a margin of unknown risk. The work of debugging shifts from the development team, where it belongs, to operations, where the blast radius is larger.

The Real Problem Is Structural

It is tempting to frame this as a people problem. The developers who created inconsistent naming conventions are not bad engineers. The developers who ran informal database imports were not trying to cause problems. They were filling a vacuum the best way they knew how.

The problem is not the people. The problem is the absence of the structure that coordinates the people.

A principal engineer or solution architect is not just a senior developer who codes well. The role carries a specific function in a team’s organizational design: owning the standards, defining the boundaries, making decisions that affect the system as a whole, and critically, being the reason those decisions are consistent over time.

When that role is absent, every developer becomes an implicit architect for the parts of the system they touch. Most developers are not set up to do that job. Not because they lack skill, but because they are optimizing for their immediate task, not for the coherence of the whole system. That is not a failure on their part. It is a predictable outcome of removing the role that was doing that work.

Why It Compounds

The dangerous property of this kind of decay is that it accelerates.

Early inconsistencies are small. A naming convention here, an undocumented table there. The cost of fixing them is low. But they do not get fixed, because there is no one whose job it is to notice them and push back. So they stay, and new inconsistencies are added on top of them.

As the codebase and schema grow, the cost of correcting the foundational issues grows with them. A team that could have spent a few hours standardizing a naming convention six months ago is now looking at a multi-week migration effort that requires coordination across multiple services and careful handling of production data.

The issues that work fine in development but break in production are not isolated incidents. They are symptoms of an environment where the feedback loops that should catch these problems have degraded. They will keep happening, in different forms, until the structural issue is addressed.

What This Means for Engineering Managers

If you are running an engineering team and you lost your principal engineer or SA, the technical work did not stop needing architectural oversight just because the person providing it is gone.

The question is not whether your team is capable of continuing to ship features. They probably are. The question is whether the decisions being made at the level of system design, conventions, and cross-service boundaries are being made by anyone with a view of the whole.

If the answer is no, the work is still getting done. It is just getting done in a way that will be progressively harder to understand, maintain, and operate.

That cost does not show up immediately on a sprint board. It shows up months later, in production incidents that should not have happened, in onboarding time for new engineers who cannot understand why the schema looks the way it does, and in the infrastructure team spending their time debugging environment drift instead of improving reliability.

What This Means for Developers

If you are a developer on a team without a clear architectural authority, you are probably making more system-level decisions than your role is designed to handle. That is not a criticism. It is a situation you have been placed in.

The decisions you make today about naming, schema design, and how you interact with shared infrastructure will be the baseline that everyone else inherits. The shortcuts that seem harmless in isolation become the patterns that define how the system works.

It is worth asking, before every database change or schema addition, whether someone with a view of the whole system has had a chance to look at it. If there is no one in that role, that is the real problem to surface, not to your codebase, but to whoever is responsible for the engineering organization.

The standards that no one is holding will not hold themselves.

The logs showed a clean 308 redirect. HAProxy was adding a trailing slash to our OAuth callback path. One character, silently appended, was enough to invalidate the entire flow.

What made it worse was that we had not touched the Ingress responsible for that route. No config changes, no deployments to that service, nothing. The Ingress looked exactly as it should.

What Actually Happened

OAuth flow depends on exact URL matching. Google redirects the user back to a pre-registered callback URL. If anything changes that URL, even a trailing slash, the provider rejects it.

Our flow was:

1. User clicks “Login with Google”
2. Google redirects back to /api/auth/callback/google?state=...&code=...
3. HAProxy intercepts and issues a 308 to /api/auth/callback/google/
4. NextAuth receives the modified callback URL and fails with OAuthCallback error

Step three is where everything went wrong. HAProxy was redirecting the OAuth callback with a trailing slash, and the redirect was also dropping the query string. state and code parameters, the ones Google sends back to complete the OAuth handshake, were gone.

The redirect rule responsible was in a backend-config-snippet on an Ingress we had recently updated. But not the Ingress serving the OAuth route. A completely different Ingress, for a completely different domain.

The Behavior Nobody Documents Clearly

HAProxy Ingress Controller merges backend-config-snippet annotations per backend service, not per Ingress resource.

This is the behavior that catches teams off guard. When two or more Ingress resources point to the same Kubernetes Service, every backend-config-snippet from every one of those Ingresses gets injected into the same backend block in the generated haproxy.cfg.

To make it concrete:

# Ingress A - domain: staging.example.com
backend-config-snippet: |
  http-request redirect code 308 location %[path]/ if !{ path_end / }

# Ingress B - domain: www.example.com
backend-config-snippet: |
  http-request redirect code 308 location %[path]/ if !{ path_end / }

# Both point to: my-app-service

What ends up in haproxy.cfg:

backend my-app-service
  ### injected from Ingress A ###
  http-request redirect code 308 location %[path]/ if !{ path_end / }
  ### injected from Ingress B ###
  http-request redirect code 308 location %[path]/ if !{ path_end / }

Both rules are active. Both apply to all traffic hitting that backend, regardless of which domain it came from. Changing one Ingress changes the behavior for traffic from every Ingress that shares the same service.

In our case, a redirect rule on a non-production Ingress was quietly being evaluated against OAuth callbacks on our main domain.

Four Things We Got Wrong

1. Redirect rules do not belong in backend-config-snippet

http-request redirect rules are meant for the frontend. In HAProxy’s architecture, the frontend evaluates the request before routing it to a backend. That is the right place for redirect logic, before you have committed to a backend, and before the rules from unrelated Ingresses can interfere.

Backend is evaluated after the routing decision. Putting redirects there works, but it means the rule runs in a context shared by every Ingress pointing at that service. Use haproxy.org/frontend-config-snippet for redirect rules.

2. Every trailing slash redirect needs an /api exclusion

A redirect rule that catches all paths without exceptions will catch OAuth callbacks, webhook endpoints, and any other API path that cannot tolerate a modified URL.

# This will redirect /api/auth/callback/google to /api/auth/callback/google/
http-request redirect code 308 location %[path]/ if !{ path_end / } !{ path_reg \.(js|css|png|jpg|ico|svg|woff2)$ }

# This will not
http-request redirect code 308 location %[path]/?%[query] if !{ path_end / } !{ path_beg /api } !{ path_reg \.(js|css|png|jpg|ico|svg|woff2)$ }

Any path under /api should be excluded from trailing slash redirects. These paths are consumed by code, not browsers. They do not need normalization, and they cannot tolerate unexpected redirects.

3. Never redirect without %[query]

location %[path]/ drops the query string. location %[path]/?%[query] preserves it.

For OAuth, the query string is the entire payload: state, code, session_state. Losing it means the authorization flow is dead on arrival. This is a one-character difference that breaks authentication completely.

4. The Ingress manifest is not the source of truth

Reading the Ingress YAML tells you what you configured. It does not tell you what HAProxy is actually running. The generated haproxy.cfg is the real source of truth, and it reflects the merged output of every Ingress touching your backend.

When debugging HAProxy behavior, inspect the generated config directly:

kubectl exec -n <haproxy-namespace> <haproxy-pod> -- cat /etc/haproxy/haproxy.cfg

Search for every snippet injected into the backend you care about, not just the ones from the Ingress you edited. That is the only way to see the full picture of what rules are actually evaluating your traffic.

The Blast Radius of Shared Backends

The deeper issue here is not the specific redirect rule. It is the implicit coupling that shared backends create in HAProxy.

In NGINX Ingress, each Ingress resource is relatively self-contained. An annotation on one Ingress does not bleed into another. In HAProxy, backend-config-snippet is pooled. The moment two Ingresses share a backend service, they share a config space. A change on one is a change on both, whether you intended it or not.

This is not a bug; it is a consequence of how HAProxy generates a single unified config file. But it means the operational model is different. You cannot reason about an Ingress in isolation. You have to reason about it in the context of every other Ingress sharing its backend.

Teams that come from NGINX tend to underestimate this. The annotation names look familiar, the concepts seem equivalent, and the differences only show up in production when something unrelated breaks.

HAProxy Controller Is Dynamic, and That Is the Real Risk

Everything described so far assumes you made a deliberate change and then something broke. But HAProxy Ingress Controller introduces a subtler risk that deserves its own attention: the controller is fully dynamic.

Every time any Ingress resource in the cluster changes, HAProxy regenerates its entire haproxy.cfg from scratch and reloads. This includes changes you did not make. A teammate applies an unrelated Ingress update in a different namespace. The controller pod gets restarted due to a node eviction or an OOM kill. A Helm chart upgrade touches a ConfigMap annotation. Any of these events trigger a full config reload, and the resulting config reflects the current state of every Ingress in the cluster at that exact moment.

This means the blast radius of any single change is not limited to the moment it is applied. It persists silently in the config, waiting for the next reload to activate in combination with something else.

The failure pattern is disorienting because there is often no obvious trigger. A service that was working at 2pm stops working at 3pm. No deployment happened to that service. No one touched its Ingress. The on-call engineer checks the usual suspects and finds nothing. What actually happened is that a controller restart re-evaluated the config, and the merged result of some unrelated snippet finally hit the right condition.

This is not hypothetical. It is a real operational pattern that teams running HAProxy Ingress at any meaningful scale will encounter eventually. The controller pod itself becomes a silent failure surface. You cannot reason about what is running in HAProxy without knowing the current merged state of all Ingresses, and that state can change without any intentional action on your part.

The checklist below is not just for when you are about to make a change. It applies any time HAProxy behavior changes unexpectedly. The first question is always: what changed in the generated config, not what changed in the Ingress you are looking at.

Checklist Before Applying Ingress Changes in HAProxy

Before applying any Ingress change that touches backend-config-snippet:

• Find every other Ingress sharing the same backend service: kubectl get ingress -A -o yaml | grep <service-name>
• Inspect the current generated haproxy.cfg for the affected backend before and after your change
• Confirm every redirect rule excludes /api paths
• Confirm every redirect rule preserves the query string with %[query]
• Test OAuth and SSO flows in staging before touching production
• Prefer frontend-config-snippet for redirect rules, because backend snippets should be for backend-specific behavior only

The rule that broke our OAuth was not malicious, not careless, and not written by someone who did not know what they were doing. It was written without knowing that HAProxy merges backend snippets across Ingress boundaries. Once you know that, the rest follows.

This is a post-migration review of what we found, what broke, and what needs to be in place before any team runs this in production.

Executive Summary

The Architectural Divide

NGINX and HAProxy have fundamentally different design philosophies, and understanding this is the prerequisite for a successful migration.

NGINX is modular, flexible, and L7-heavy. Configuration is split across files, and most behavior can be extended through annotations and server blocks. HAProxy is centralized, high-performance, and rigid. There is one config file, one reload path, and one failure mode when something goes wrong.

Three problem categories define most of the friction in this migration:

1. Reload and certificate handling
2. Blast radius from config centralization
3. Feature limitations in regex and annotation support

Findings

TLS Certificate Update Requires Pod Restart

TLS secret updates are not hot-reloaded by the HAProxy ingress controller. When a certificate is rotated in Kubernetes, the change does not propagate until the HAProxy pod is restarted.

This eliminates the possibility of zero-downtime certificate rotation without additional tooling. During a rolling restart, there is a short window where traffic is handed off between pods, which may cause brief disruption depending on how connections are drained.

The practical implication is that certificate rotation must be treated as a planned operation. Schedule it outside peak traffic, implement preStop hooks to drain connections gracefully, and verify readiness probes are configured to prevent traffic from hitting a pod before it is ready to serve.

Single Configuration File

All ingress rules are compiled into a single haproxy.cfg. There is no equivalent to NGINX’s conf.d/ pattern where each domain or service gets its own isolated file.

This matters for failure isolation. In NGINX, a bad config in one file typically prevents that file from loading while leaving the rest untouched. In HAProxy, a single invalid ingress resource causes the entire reload to fail. Every route served by that controller goes down simultaneously.

The blast radius is cluster-wide by default.

Mitigating this requires treating config validation as a mandatory gate. Add haproxy -c -f haproxy.cfg as a required step in the deployment pipeline before any ingress change is applied. Consider an admission webhook to reject invalid ingress resources at submission time, before they can reach the controller. Replicate the full production ingress config in staging to catch errors before they reach production.

TLS Secret Ordering and Lexicographical Priority

When multiple certificates cover the same FQDN, HAProxy selects between them based on the lexicographical sort order of the Kubernetes secret name.

This is not surfaced anywhere in the ingress resource definition. The behavior is implicit, undocumented in most operator documentation, and easy to trigger accidentally. Renaming a secret — even to something semantically equivalent — can silently change which certificate is served to end-users. In the worst case, this means serving an expired certificate in production with no configuration change and no visible error.

The fix is enforcing a one-to-one mapping between certificate secrets and FQDNs. Establish a naming convention for TLS secrets and treat deviations as a deployment error. Set up monitoring to compare served certificate CN/SAN values against expected values so any mismatch is caught before users notice.

Advanced Regex Not Fully Supported

HAProxy does not support complex PCRE patterns. Ingress rules that rely on lookaheads, backreferences, or advanced path rewriting through regex will either fail silently or produce incorrect routing behavior.

NGINX annotations like nginx.ingress.kubernetes.io/use-regex: "true" combined with complex path patterns have no direct equivalent in HAProxy. Path rewrite capabilities are also more limited.

This needs to be assessed before migration starts, not during. Audit every ingress rule that uses regex and categorize it: can it be simplified into a prefix or exact match, or does the logic need to move into the application layer or an API gateway such as Kong or Envoy? Discovering this mid-cutover creates pressure to push incomplete configurations to production.

Annotation Incompatibility

Every nginx.ingress.kubernetes.io/* annotation is ignored by the HAProxy ingress controller. There is no fallback, no warning, and no partial support. Annotations that have been silently handling SSL redirect, rate limiting, CORS, or custom headers for months simply stop working.

The equivalent annotations exist under the haproxy.org/* prefix, but behavior is not always semantically equivalent even when the mapping appears straightforward. Connection timeout handling, rewrite behavior, and header forwarding can differ in ways that are only visible under load or in specific edge cases.

Build a reference mapping table of NGINX-to-HAProxy annotations as part of migration prep. Run dry-run deployments and compare observed behavior against expected outcomes systematically across each service before cutover.

Config Reload Sensitivity

HAProxy enforces strict config validation at reload time. Any syntax or semantic error — including errors introduced by a single ingress resource that generates invalid config — causes a complete reload failure with no partial recovery.

NGINX can tolerate some config errors by isolating the affected block. HAProxy cannot. An operator managing 200 ingress rules in a shared cluster has a high-risk surface area. One misconfigured annotation in one team’s ingress definition can prevent all 200 services from updating their routes.

Managing this risk requires the following at minimum: mandatory haproxy -c validation in the pipeline, canary deployments for high-risk ingress changes, and a known-good config snapshot for fast rollback. Without these, a single bad deploy can take down the entire ingress tier.

Observability and Debugging Complexity

HAProxy logs are lower-level and not structured per ingress resource. Diagnosing a routing issue requires manually inspecting the full generated haproxy.cfg to trace how a request would be matched and forwarded.

Compared to NGINX ingress, where logs are per-request and relatively self-explanatory, debugging HAProxy behavior requires a deeper understanding of the generated config structure and ACL evaluation order. This increases the time to root cause, particularly for engineers who are not familiar with HAProxy internals.

Normalize log output through a parsing layer (Fluentd, Logstash, or similar). Write a troubleshooting runbook that covers the most common failure scenarios: reload failures, certificate mismatches, missing annotation mappings, and ACL evaluation order. Integrate with distributed tracing or APM to close the gap on end-to-end request visibility.

Conclusion

HAProxy is a high-performance proxy with strong operational guarantees. But those guarantees come with a governance model that is stricter than NGINX. Centralized config means higher blast radius. Implicit behaviors around certificate selection and strict reload semantics mean more room for silent failures.

Three areas must be production-ready before cutover:

1. Config validation pipeline — mandatory before every ingress change is applied. No exceptions.
2. TLS rotation procedure — documented, tested, and rehearsed with real traffic patterns, not just in theory.
3. Full ingress audit — every regex pattern, annotation mapping, and certificate assignment reviewed individually. Not at a service level. At a rule level.

The migration is worth doing if the performance characteristics and load balancing capabilities of HAProxy justify the operational cost. But it is not a default drop-in replacement for NGINX. Treat it as an architectural change, not a configuration swap.

The problem was route management. Tyk uses imperative API calls by default, but our infrastructure is fully declarative. Everything lives in Git, deployed with kubectl apply. We needed an operator.

Tyk has one. It’s called Tyk Operator and it’s exactly what we needed: declarative, GitOps-ready.

It’s also enterprise-only.

The open source gateway is free. The tooling that makes it actually usable in a cloud-native setup costs money. Core product free, operational tooling paywalled. So I built my own.

What I Built

tyk-cm-routes-controller is 565 lines of Python using Kopf. The idea is straightforward: watch TykRoute custom resources, pull the API definition from the spec, validate it, write it to a ConfigMap as JSON, optionally trigger a rollout restart, and update the resource status. Done.

It worked perfectly in staging with a dozen routes. We shipped it to production.

Eight months later we’d had three production incidents, merge conflicts became a recurring headache, and we kept running into ConfigMap size limits. The gap between “code that works” and “code that works at scale” hit us hard.

The Merge Conflict Problem

The original design put all routes into a single ConfigMap. Tyk reads route definitions from mounted files at startup, so one ConfigMap with all routes made sense.

It made sense until four teams started deploying independently.

Payments wanted to add a refund endpoint. Products wanted to add an inventory check. Both branched from the same commit. Both modified the same ConfigMap. Git has no idea how to auto-merge 2000 lines of JSON route definitions, so you end up with 57 blobs all marked as conflicting.

An engineer spends 40 minutes resolving it manually. The YAML looks valid, the PR gets approved. Three routes got silently dropped. Six hours later, production starts throwing 404s.

This wasn’t a one-off. The probability of a conflict scales with how often teams deploy. We tried Slack coordination, namespace separation, and not committing the ConfigMap at all. Each workaround introduced a different problem. The root cause was architectural: GitOps assumes independent resources, but a ConfigMap is atomic. The entire object gets versioned as a unit.

Race Conditions

Kopf gives you three handlers: create, update, delete. Simple enough.

The uniqueness check for listen_path wasn’t atomic though. The flow was: read the ConfigMap, check if the path already exists, write the new route if it doesn’t. With multiple operator replicas or two TykRoute resources created at the same time, both checks pass before either write completes. You end up with duplicate listen paths in production.

The rollout restart had its own issue. After updating the ConfigMap, the operator would patch the deployment annotations to trigger a restart, then immediately set the TykRoute status to active. No waiting, no verification. The operator had no idea if the rollout succeeded, if pods were crashlooping, or if Tyk rejected the config.

Fixing this properly would’ve meant integrating with the Tyk admin API, watching rollout status, and adding locking. More complexity, more failure modes. We chose simplicity over robustness.

The 1MB Wall

Kubernetes has a hard 1MB limit per ConfigMap.

A complex route with auth, rate limiting, transforms, and caching is around 10KB. A simple route is about 2KB. Average maybe 5KB. That gives you roughly 200 routes before you hit the ceiling.

Fine with 12 routes in staging. Not fine when you have an entire API surface spread across multiple teams.

The failure mode is silent until you hit it. Kubernetes rejects the update with a cryptic error. New routes stop working, existing routes keep serving traffic. If nobody’s watching closely, it takes a while to figure out what happened.

The workarounds weren’t great either. Multiple ConfigMaps means multiple volume mounts and complexity that spreads into the Tyk deployment config. Increasing the cluster limit is a non-starter with most platform teams. We ended up with manual capacity planning: monitor the size, split across namespaces when we get close.

Automated route deployment, manual storage capacity planning.

Status That Lies

The TykRoute status had fields for state, listenPath, targetConfigMap, and conditions. When you looked at a route and saw state: active, it meant the ConfigMap update succeeded.

It did not mean the route was serving traffic.

A rollout could’ve failed. Pods could be crashlooping. Tyk could’ve rejected the config entirely. The operator only knew about the ConfigMap write. The deployment pipeline saw active and marked success. Monitoring saw active and stayed quiet. Meanwhile, requests were hitting 404s.

Real status would require integrating with the Tyk admin API, watching rollout completion, and running health checks against the routes. That’s a much bigger project, and we chose not to do it. The operator reports what it knows, which turns out not to be what operators actually need to know.

What Tyk Enterprise Probably Does

I can’t say for certain, but the problems we hit are obvious enough that a real product would’ve had to solve them. Probably database writes instead of a single ConfigMap. Actual Tyk API integration so status reflects real availability. Some mechanism for teams to deploy independently without merge conflicts.

The quality gap between a weekend project and production-grade tooling is real and expected.

The Trap

The pattern is predictable in hindsight. Choose open source to avoid the licensing cost. Discover that production use requires the paywalled tooling. By then you’re already committed to the technology. So you build it yourself.

A PoC is a weekend. Production-grade with all the edge cases is a different story. Maintenance over time is harder still.

The comparison spreadsheet made it look obvious: Enterprise at $30k/year versus operator at $0/year. The spreadsheet didn’t have a line for engineering time spent building, debugging, explaining limitations, fielding questions, and responding to incidents. At market rate for a senior engineer, that time adds up past the license cost quickly.

Engineering time doesn’t show up as an incremental cost on budgets the way a vendor invoice does. The CFO sees infrastructure cost down and doesn’t see velocity down. By the time the productivity drag is obvious, you’re maintaining critical infrastructure and the migration cost is high enough that you just keep going.

Tyk’s setup is well-designed for this. The gateway itself is solid and production-ready. The gaps only appear when you start doing K8s-native workflows. By then, traffic is flowing through it, teams depend on it, and migration is a serious project.

We saved the license cost and inherited the development and maintenance cost. Eight months in, I’m not sure we came out ahead.

What I’d Think About Differently

Don’t use a single ConfigMap for data that grows without a bound. Don’t report state as active when you’ve only verified the config write. Coordination logic that works for one team breaks for multiple teams in ways that aren’t obvious until you’re in it.

More importantly: “can we build this” is the wrong question. The right question is whether the operational burden is worth the cost savings. Sometimes the answer is yes, especially when the commercial tool genuinely doesn’t fit your requirements or you have real spare capacity. But when a commercial tool exists and has broad adoption, it usually exists because the problem is harder than it looks. The price reflects engineering effort that’s already been spent solving problems you haven’t hit yet.

For tyk-cm-routes-controller: the routes work and the workflow is declarative. But merge conflicts are a recurring tax, the size limit requires manual management, the status is misleading, and every edge case requires manual verification. Fixing the design problems properly would take weeks of work.

We built it to save money. We might have.

Late Sunday night. I’d finally convinced myself to stop checking Slack every five minutes and actually relax. Big mistake.

By the time I grabbed my laptop, there were 47 unread messages. The monitoring dashboard looked like someone had taken a red marker to it. Every Kafka metric flatlined. Zero brokers available. Zero partitions healthy. Consumer lag climbing into millions. And the beautiful part? Every single health check for services that touched Kafka, which was basically our entire platform, turning red in a cascading wave of failure.

My first thought was “What did we deploy?”

We deployed nothing. It was Sunday night. Nobody deploys on Sunday night. That’s the rule.

My second thought, after SSHing into the cluster was “This doesn’t make any sense.”

The Kafka pods existed. The ZooKeeper ensemble was running. Storage volumes intact. Network connectivity fine. I could curl the broker endpoints. I could see the processes running. Everything was there, just not working. Like walking into your house and finding all your furniture rearranged by ghosts.

The Strimzi operator logs weren’t helpful. Just an endless loop of the same error, over and over.

ERROR PlatformFeaturesAvailability:138 - Detection of Kubernetes version failed.

Kubernetes version detection? What does that have to do with Kafka being down?

I refreshed the GKE console. Cluster status healthy. Recent changes showed one automated upgrade to version 1.33, completed hours earlier.

Oh.

Oh no!!

How We Got Here

Let me rewind a bit. The infrastructure looked textbook-perfect before all this. Strimzi operator managing Kafka on GKE, version 0.43.0, running flawlessly for months. Rock-solid message delivery, zero complaints from application teams, monitoring showing healthy metrics across the board.

Strimzi was the sensible choice. Managing Kafka manually is a nightmare. ZooKeeper coordination, broker configurations, rolling updates, storage management. Why reinvent the wheel when there’s a mature Kubernetes operator that handles all of this? The vendor lock-in was worth it. Or so we thought.

Then GKE decided to auto-upgrade itself to version 1.33.

Nobody noticed at first. Why would we? GKE upgrades happen regularly. Google’s infrastructure is supposed to be reliable. The upgrade completed successfully according to the console. Cluster status green, node health optimal, everything looked normal.

Until the Strimzi operator tried to reconcile its resources.

The failure wasn’t gradual. It was catastrophic. Strimzi operator pods went into CrashLoopBackOff immediately. Every reconciliation loop failed. The operator couldn’t detect the Kubernetes version. Without version detection, it couldn’t manage Kafka resources. Without management, Kafka brokers became orphaned.

Within minutes, the entire Kafka cluster was effectively dead.

Core applications started failing health checks. Message producers couldn’t connect. Consumers stopped processing. Dead letter queues filled up. Circuit breakers tripped across dozens of services. What started as “Kafka is down” cascaded into “half our platform is down.”

The alerts were relentless. PagerDuty, email, Slack, phone calls from on-call engineers, then managers, then directors. Everyone wanted to know what happened and when it would be fixed. The pressure was suffocating.

And I had no answer because I didn’t understand the problem yet.

Digging Through the Wreckage

First instinct was check the logs. Surely the Strimzi operator is telling us something useful?

ERROR PlatformFeaturesAvailability:138 - Detection of Kubernetes version failed.
io.fabric8.kubernetes.client.KubernetesClientException: An error has occurred.

Great. “An error has occurred.” Thanks, Java. Really helpful.

I dug deeper into the stack trace. There’s the real clue, buried twenty lines down.

Caused by: com.fasterxml.jackson.databind.exc.UnrecognizedPropertyException: 
Unrecognized field "emulationMajor" (class io.fabric8.kubernetes.client.VersionInfo), 
not marked as ignorable (9 known properties: "goVersion", "gitTreeState", "platform", 
"minor", "gitVersion", "gitCommit", "buildDate", "compiler", "major"])

GKE 1.33 added a new field called emulationMajor to its version API response. Strimzi’s Kubernetes client library didn’t know how to handle it. Instead of gracefully ignoring unknown fields, it exploded. And when your operator can’t talk to the Kubernetes API properly, your entire Kafka infrastructure becomes unmanageable.

This wasn’t a Kafka problem. This wasn’t even a Strimzi problem. This was a dependency incompatibility introduced by an automatic platform upgrade that nobody had tested against our workloads.

Two Terrible Options

I had two choices, both terrible.

Option A was manual migration. Extract data from the surviving PVCs, spin up a new Kafka cluster, somehow restore topics, partitions, consumer offsets, and ACLs without losing data or breaking every downstream application. In a production environment. While everything is on fire. With no rollback plan.

Option B was fix Strimzi. Somehow make the operator compatible with GKE 1.33, even though we’re running an older version that wasn’t designed for this Kubernetes release.

Option A was high-risk data loss roulette. Kafka’s state is distributed across ZooKeeper and broker storage in complex ways. One wrong move and months of message history could be corrupted or lost. Plus, we’re using Strimzi for a reason. Vendor lock-in means we can’t just export and import like it’s a database dump.

Option B seemed impossible. You don’t just patch compatibility into production operators on the fly. Right?

But Option A would take hours, maybe days, with no guarantee of success. Option B might not work, but if it did, we’d have everything back exactly as it was.

With alerts screaming, applications failing, and stakeholders demanding answers, I chose the impossible option.

The Workarounds That Failed

The first attempt was the obvious one. Maybe I could force the operator to ignore that problematic field. Jackson has a deserialization feature for exactly this, telling it to not explode when encountering unknown properties.

I added an environment variable to the Strimzi operator deployment.

env:
  - name: STRIMZI_JAVA_OPTS
    value: "-Dcom.fasterxml.jackson.databind.DeserializationFeature.FAIL_ON_UNKNOWN_PROPERTIES=false"

This should tell Jackson’s ObjectMapper to gracefully ignore fields it doesn’t recognize instead of throwing exceptions. It’s a nuclear option. You’re basically telling your deserializer “I don’t care what’s in the JSON, just give me what you understand” but desperate times call for desperate measures.

Deployed it. Watched the logs with hope.

Same error. The JVM option wasn’t being picked up correctly, or the failure was happening in a context where this configuration didn’t apply. Either way, the operator still couldn’t detect the Kubernetes version.

Next attempt. Maybe I could bypass version detection entirely. If the operator can’t detect the version automatically, what if I just tell it what version to use?

env:
  - name: STRIMZI_KUBERNETES_VERSION
    value: "1.28"

The idea was to force Strimzi to think it’s running on Kubernetes 1.28, a version it definitely knows how to handle. Skip the problematic version detection API call altogether.

I applied the change. The operator started. Logs looked promising at first. Then everything went sideways again.

WARN PlatformFeaturesAvailability:156 - Forced Kubernetes version 1.28 differs from detected version
ERROR KafkaAssemblyOperator:112 - Reconciliation failed: API version mismatch

Strimzi was still trying to detect the actual version despite being told to use 1.28. The forced version created a conflict rather than a workaround. The operator detected the mismatch and refused to proceed, probably as a safety mechanism to prevent running with incorrect API assumptions.

Two attempts, two failures. The workarounds that should have worked in theory were hitting edge cases in practice. At this point, I’m several hours into the incident. The phone calls have escalated. People are asking if we need to declare a major outage. Application teams are demanding ETAs I can’t give them.

I had to go deeper. No more clever workarounds. Time to understand what was actually broken and fix it at the source.

Reading Source Code at 2 AM

Out of desperation, I did what probably should have been my first move. I cloned the Strimzi repository and started reading source code.

Not the documentation. Not GitHub issues. The actual implementation.

I traced through the PlatformFeaturesAvailability class, following the exact code path that was failing. The logic was straightforward enough. Call Kubernetes API, deserialize version info, detect platform capabilities. The failure happened during deserialization when Jackson encountered a field it didn’t recognize.

Then I checked what the latest Strimzi version was using. Version 0.46.0 had just been released, and it worked fine with newer GKE versions. What were they using?

<fabric8.kubernetes-client.version>7.2.0</fabric8.kubernetes-client.version>

Version 7.2.0. A major version jump from our current 6.13.4.

Here’s where it gets interesting. I couldn’t just upgrade to Strimzi 0.46.0 directly. Our Kafka cluster was still ZooKeeper-based, not yet migrated to KRaft mode. Strimzi’s upgrade path had compatibility requirements. You can’t just jump from 0.43.0 to 0.46.0 with a production Kafka cluster that has months of state.

But what if I backported the dependency fix?

What if I took Strimzi 0.43.0’s source code and simply updated the fabric8 libraries to version 7.2.0?

It was a desperate idea. Dependency upgrades can introduce breaking changes. API incompatibilities. Behavioral differences. This could make things worse. But at this hour with everything on fire, desperate ideas start looking reasonable.

Building a Custom Operator in Production

I modified the pom.xml.

<fabric8.kubernetes-client.version>7.2.0</fabric8.kubernetes-client.version>
<fabric8.openshift-client.version>7.2.0</fabric8.openshift-client.version>
<fabric8.kubernetes-model.version>7.2.0</fabric8.kubernetes-model.version>
<fabric8.zjsonpatch.version>0.3.0</fabric8.zjsonpatch.version>

The progression I’d tried was revealing. Version 6.13.4 was the original that failed with the emulationMajor error. Version 6.14.0 was my first attempt but still failed, the minor version bump wasn’t enough. Version 7.2.0 was from Strimzi 0.46.0, the nuclear option.

I kicked off the Maven build. Waited through compilation, unit tests, integration tests. Every failed test made my stomach drop. Every successful test gave me a sliver of hope.

The build completed successfully.

I pushed the image to our container registry, updated the deployment manifest, applied it to the cluster.

The operator pod started. Logs began flowing. I held my breath, waiting for that familiar error message.

INFO PlatformFeaturesAvailability:124 - Kubernetes version: 1.33.0-gke.3000
INFO KafkaAssemblyOperator:89 - Reconciliation started for Kafka cluster kafka-cluster

It worked.

The operator detected the Kubernetes version. Reconciliation loops began. Within minutes, Kafka brokers were healthy again. Services reconnected. Message backlogs started processing. Health checks turned green.

The alerts stopped.

Fun fact, I didn’t sleep that night. Not because I was still troubleshooting, but because the adrenaline wouldn’t let me. Yeah, this is real DevOps life I guess.

Understanding What Actually Broke

Let me break down the technical reality of what just destroyed our production infrastructure.

GKE 1.33 introduced a new field in their Kubernetes version API response called emulationMajor. This field helps GKE communicate version emulation details for compatibility purposes. Perfectly reasonable addition to their API.

Strimzi 0.43.0 used fabric8 Kubernetes client version 6.13.4. This library included a Java class called VersionInfo that mapped Kubernetes version API responses to strongly-typed objects. That class expected specific fields like major, minor, gitVersion, buildDate, and so on.

When the 6.13.4 client received a response containing emulationMajor, Jackson’s ObjectMapper, the library handling JSON deserialization, threw an UnrecognizedPropertyException because the VersionInfo class didn’t have a field for it.

The exception propagated up through Strimzi’s platform detection logic, causing every reconciliation loop to fail. Without successful reconciliation, the operator couldn’t manage Kafka resources. Kafka brokers became orphaned, unable to receive configuration updates or handle failures.

The fix in fabric8 7.2.0 was likely simple. Either adding @JsonIgnoreProperties(ignoreUnknown = true) to the VersionInfo class or explicitly adding the emulationMajor field. This is a single-line change in the library’s source code, but it made the difference between a working Kafka cluster and complete infrastructure failure.

The Real Problem Wasn’t Technical

Here’s what makes this story terrifying. The technical problem, dependency incompatibility, was completely predictable and preventable. But we never had a chance to prevent it because GKE auto-upgraded itself without warning.

Think about what auto-upgrade actually means in practice.

Your cloud provider decides when your infrastructure changes. Not during planned maintenance windows. Not after testing in staging environments. Not when your team is prepared. Whenever the provider’s release schedule decides it’s time.

The upgrade happens atomically from your perspective. One moment you’re running Kubernetes 1.32, the next you’re on 1.33. No rollback plan, no testing window, no chance to validate workload compatibility.

Breaking changes in Kubernetes APIs can cascade through your entire stack. It’s not just about pods and services. Every operator, every controller, every application that talks to the Kubernetes API is potentially affected.

In our case, GKE 1.33’s emulationMajor field broke Strimzi. But it could have been anything. Istio service mesh failing to configure routing rules due to API changes, breaking service-to-service communication across your entire platform. ArgoCD unable to sync applications, freezing your entire GitOps deployment pipeline with no way to rollback or deploy fixes. Cert-manager failing to renew certificates, leading to cascading TLS failures as certificates expire. Prometheus Operator unable to scrape metrics, leaving you blind during an incident. External DNS controller breaking, making services unreachable as DNS records stop updating.

Any of these failures could trigger the same cascade we experienced with Kafka. One broken operator leading to application failures, alerting storms, and engineers scrambling to fix infrastructure in the middle of the night.

Counting the Real Cost

The Kafka outage rippled through our entire platform. But the real cost wasn’t just downtime metrics.

Revenue impact was brutal. Every service depending on Kafka for async communication was degraded or down. Transaction processing stopped. User notifications failed. Analytics data wasn’t being collected. Calculate those hours against revenue per minute and the number gets uncomfortable fast.

Engineering cost added up quickly. Multiple teams dropped everything to respond. On-call engineers, platform team, application teams trying to understand why their services broke. Call it 15 engineers at an average loaded cost of $150 per hour for 6 hours. That’s $13,500 just in incident response labor.

Opportunity cost was harder to measure but just as real. Those engineers weren’t working on planned features, bug fixes, or improvements. Multiply lost development time across teams and the real cost becomes staggering.

Trust erosion started immediately. Application teams relying on our platform infrastructure lost confidence. Conversations about migrating to managed services started. Engineers questioned whether our operator-based approach was sustainable.

Stress and burnout hit everyone involved. I didn’t sleep that night. Neither did several other engineers. The psychological toll of high-pressure incidents adds up over time, contributing to burnout in ways that don’t show up in incident postmortems.

And all of this could have been prevented by a single configuration setting. Disable auto-upgrade.

This Keeps Happening

This wasn’t an isolated incident. The pattern repeats across infrastructure stacks and workloads.

Different orchestration platforms, different applications, different protocols, but identical fundamental failure mode. Platform auto-upgrade introducing breaking changes that cascade through operator-managed workloads.

Modern infrastructure runs on operators and controllers. Whether you’re running Kubernetes with custom operators, service mesh with control planes, or database clusters with automated management, you’re running distributed systems that make assumptions about platform APIs.

When those APIs change without testing, operators break. When operators break, the applications they manage become unmanageable. When management fails during peak load, cascading failures destroy availability.

The irony is that we’ve built incredibly sophisticated application-layer resilience. Circuit breakers, graceful degradation, chaos engineering. But we’ve left a massive vulnerability at the platform layer by enabling auto-upgrade.

What This Says About How We Build

This pattern reveals something uncomfortable about how we approach infrastructure in the cloud-native era.

We’ve become incredibly comfortable delegating control to cloud providers. “Managed services” sounds great until you realize “managed” means “we decide when and how things change, not you.”

Infrastructure maturity isn’t just about adopting the latest orchestration platform or implementing sophisticated monitoring. It’s about understanding where control matters and being willing to take on operational burden to maintain that control.

For non-critical workloads, auto-upgrade makes sense. Let the provider handle it. Focus your energy elsewhere.

For production systems where downtime costs thousands per minute, auto-upgrade is an unacceptable risk. The convenience isn’t worth the blast radius when something breaks.

Questions You Should Be Asking

Next time you’re architecting infrastructure, think about these things.

Who controls when your infrastructure changes? If the answer is “our cloud provider,” you’ve outsourced not just operations but risk management to a party that doesn’t understand your workloads, your peak traffic patterns, or your business criticality.

What breaks when platform APIs change? Every operator, controller, and integration in your stack makes assumptions about API behavior. You need to list them out. Then ask how you’d test compatibility before upgrading.

Can you rollback a platform upgrade? If your cloud provider upgrades your Kubernetes cluster and something breaks, what’s your rollback plan? Most platforms don’t support rollback. Your only option is forward, fixing compatibility issues under pressure while services are failing.

What’s the blast radius of a compatibility failure? In our case, one operator failure took down Kafka, which cascaded through dozens of services. You need to map your dependencies. Understand what breaks when platform-level components fail.

Are you optimizing for convenience or reliability? Auto-upgrade is convenient. Manual upgrade with proper testing is reliable. You have to choose which matters more for each workload.

How would you debug this failure at 3 AM? When logs just say “version detection failed” and your entire message queue is down, do you have the skills, access, and documentation to trace through operator source code and rebuild custom images? Or are you dead in the water?

Preventing This Nightmare

Here’s what you should be doing instead of enabling auto-upgrade.

First, disable auto-upgrade for production. Your GKE cluster configuration should have release channel set to NONE, not RAPID or REGULAR. Maintenance policy should have auto-upgrade set to false. Manual control over upgrades. Auto-repair for nodes is fine, that’s different, but auto-upgrade needs to be off.

Second, implement staged rollouts. Dev environment gets upgraded first. Then staging with production-like workloads. Then canary production cluster with subset of traffic. Monitor for 48 to 72 hours before full rollout. Only do full production upgrade during scheduled maintenance windows.

Third, maintain a compatibility matrix. Document every operator, controller, and platform integration. Current versions, Kubernetes API versions they depend on, known compatibility issues, upgrade testing checklist. Keep this updated.

Fourth, test before upgrading. Create a test cluster matching production. Same Kubernetes version target, same operators and controllers, representative workload patterns. Run for 24 hours minimum. Monitor for any failures or warnings. If the test cluster breaks, production would have broken too.

Fifth, have rollback plans. Since most cloud platforms don’t support cluster version rollback, you need alternatives. Keep previous cluster as standby. Document migration procedures. Test failover regularly. Understand recovery time objectives.

Sixth, monitor API deprecation warnings. Set up alerts for deprecated API usage in your workloads, operator compatibility announcements, platform upgrade schedules, breaking changes in release notes. Be proactive about this.

Seventh, build operational capability. Ensure your team can read operator source code, build custom operator images if needed, debug platform-level issues, make infrastructure decisions under pressure. These skills matter when everything is on fire.

What Google Knows

Here’s the part that should make every infrastructure engineer reconsider their defaults.

Google builds GKE. They operate it at massive scale. They have world-class SRE teams and decades of distributed systems experience.

And they offer a “No channel” option for release management. Manual control over every upgrade.

If Google, with their expertise and resources, provides manual control as an option, what does that tell us about the risks of auto-upgrade?

They understand the trade-offs. Automatic upgrades reduce toil but increase risk. For workloads where reliability matters more than operational convenience, manual control isn’t optional. It’s essential.

The Uncomfortable Reality

Modern infrastructure has incredible capabilities. We orchestrate thousands of containers, route traffic with millisecond precision, replicate data globally, and handle failures gracefully.

But we’ve convinced ourselves that this complexity can be managed with “set it and forget it” automation. That cloud providers can safely upgrade our infrastructure without coordination. That platform APIs are stable enough that we don’t need to test compatibility.

The Kafka outage proved otherwise. Hours of downtime, thousands in impact, and one sleepless night, all because we enabled a checkbox that said “automatically keep my cluster updated.”

The uncomfortable truth is that infrastructure reliability requires control. Control over when changes happen. Control over testing before production deployment. Control over recovery when things go wrong.

Auto-upgrade is convenient until it’s catastrophic.

Maybe the question isn’t whether cloud platforms should offer auto-upgrade features. Maybe the question is why we keep enabling them in production environments where we can’t afford the risk.

The Technical Debt That Follows

The solution, backporting a dependency update into Strimzi’s source code, worked. But think about what that implies.

I reverse-engineered a production failure, identified the incompatible library version, modified open-source software’s dependency tree, built custom operator images, and deployed them to production. All without vendor support, official documentation, or any guarantee it wouldn’t make things worse.

This is what “fixing” a dependency incompatibility looks like when auto-upgrade takes choice away from you.

We got lucky. The fabric8 library upgrade was backward-compatible enough that Strimzi worked. But it could easily have introduced subtle bugs that wouldn’t surface until later. API behavior changes, memory leaks, concurrency issues.

Now we’re running a forked version of Strimzi that isn’t officially supported. Future upgrades require manually merging our changes. Security patches need custom rebuilds. We’ve traded one vendor lock-in, Strimzi operators, for a worse one, custom-built unsupported operators.

This is the real cost of auto-upgrade failures. Not just the immediate downtime, but the technical debt that follows. The custom workarounds. The unsupported configurations. The increasing fragility as your infrastructure drifts further from standard deployments.

And it all started with a checkbox nobody thought twice about enabling.

The next time you’re setting up a production Kubernetes cluster and you see that “Enable auto-upgrade” option, remember this story. Remember the alerts, the cascading failures, the hours of debugging, the custom builds, the technical debt.

Remember that sometimes the best technology decision is the one that gives you the power to say “not yet.”

Because when your Kafka cluster vanishes in the middle of the night over an upgrade nobody approved, you’ll wish you had that power back.

When distributed systems eat themselves over an auto-upgrade nobody thought to question, it’s worth remembering that “managed” doesn’t mean “maintenance-free.” It means someone else decides when your infrastructure breaks.

Choose wisely who gets that control.