The team was managing Cloudflare WAF rules through the vendor dashboard. ClickOps: no history, no review, no rollback. Terraform was the obvious answer, except its provider destroys and recreates rules on any update, and Cloudflare Log Push references rules by ID. Every rule change orphaned log correlation.
I forked an aging open-source Kubernetes operator, stripped it down to the two CRDs I actually needed, rewrote the vendor API layer against the current Cloudflare REST API, and shipped a GitOps workflow that PATCHes rules in place. Rule IDs stay stable. Log Push stays glued. SIEM correlation survives across deploys.
Start with the actual pain, because this is the part reviewers usually miss.
Cloudflare WAF rules have IDs. Cloudflare Log Push emits security events with those rule IDs in the payload. Any SIEM downstream (Splunk, Elastic, Datadog, whatever) correlates events by rule ID. Update a rule, keep the same ID, correlation is continuous. Change the ID, correlation breaks: your "detection X fired 47 times this week" query returns zero for the days before the change.
Terraform's Cloudflare provider handles rule updates by destroying the old resource and creating a new one. The resource name in the Terraform state stays the same. The rule ID at Cloudflare does not. Every merged PR that touched WAF rules silently reset the SIEM's history for those rules. Nobody noticed until someone tried to build a trend chart across a two-month window and the data was full of holes.
Terraform is the wrong tool for this class of resource. Not because Terraform is bad. Because destroy-and-recreate is the wrong operation for a rule that has downstream consumers keyed on identity.
A Kubernetes operator was the obvious shape. Rules as declarative resources, controller reconciles to the vendor API, GitOps everywhere. But writing a new operator to solve one problem is where senior time evaporates. Six months of work to save a week of dashboard clicking.
There was an existing OSS project, Kubeflare, that already had the CRD scaffolding, controller framework, and reconciliation loop. It was aging, over-scoped for what I needed, and used a Cloudflare API layer that had drifted from current. But the bones were right.
Forked it. Named the reduced version accordingly. Kept what mattered. Cut everything else.
WAFRule CRD shapeRateLimit CRD shapeFocus is a feature. A tool that does two things well is easier to reason about, audit, and hand off than a tool that does ten things at 70 percent.
The upstream Cloudflare API client Kubeflare depended on was old. Some endpoints had moved to v4, some fields were deprecated, error handling was inconsistent. Rather than patch around it, I moved to hitting the current Cloudflare REST API directly from the controller.
Why direct calls over a wrapper library:
Tradeoff: I own the API client code. It's a small surface area. Worth it.
The core behaviour change. When a CR's spec updates, the controller PATCHes the existing Cloudflare rule in place. It does not delete and create. It fetches the current rule by the ID it recorded in status, computes the diff, and sends a PATCH with the changed fields.
Concretely:
1. Read WAFRule CR from the cluster
2. Look up recorded Cloudflare rule ID in CR status
3. GET current rule from Cloudflare, compare with desired spec
4. If drift: PATCH the rule by ID
5. Update CR status: last synced timestamp, ID unchanged
6. If rule missing at Cloudflare (deleted out of band): recreate, record new ID, log the anomalyThe rule ID at Cloudflare is stable for the life of the CR. Log Push continues emitting the same rule ID. SIEM correlation stays intact across every merged PR. This one design choice is the entire reason the tool exists.
Rules live in a repository as YAML manifests. A change follows the normal PR flow: branch, edit, open PR, review, merge. ArgoCD sees the change, syncs it into the cluster, the operator reconciles, the rule updates at the edge. Two to three minutes end to end.
Reviewers see a diff. Reviewers can request changes. Merging is auditable. Reverting is git revert followed by ArgoCD sync. The rule PATCHes back to the previous state. Rule ID still stable. Log Push still intact.
The security team writes rules, reviews each other's work, and hands the merged manifest off to the platform. Nobody needs a Cloudflare dashboard seat to author a rule.
One war story to make the pattern concrete.
Automated signups were flooding the platform. Thousands of fake accounts. IPs rotating across residential proxy pools. User agents rotating across a plausible browser distribution. Standard bot-detection heuristics were losing.
Every request in the abuse cluster shared the same JA3 fingerprint. JA3 hashes the TLS ClientHello, so it fingerprints the client stack, not the client's declared identity. Chrome on Windows, Firefox on Linux, and a headless automation library all produce distinct JA3 hashes even when the user agent lies. Rotating IPs and user agents does not help. The attacker would have needed to swap the entire TLS stack to change the fingerprint.
One WAFRule CR, matching on the JA3 hash, blocking the class. Merged, synced, live in under three minutes. The abuse cluster collapsed. The rule is still deployed. It has never needed a change, because the fingerprint of the underlying tool has not.
The lesson is not "JA3 is magic." The lesson is that when your rule pipeline is GitOps, you can respond to an incident with the same rigor you use for a normal PR. The panic doesn't push you into ClickOps.
The runbook is one line: git revert <sha>, wait for ArgoCD to sync.
The operator sees the reverted spec, computes the diff, PATCHes the rule back to its previous form. Rule ID unchanged. No log continuity break. If the change was catastrophic, you're back to the last known good state in about the same amount of time it takes to make coffee.
Sync latency. GitOps propagation is 30-60 seconds plus the ArgoCD reconcile interval. A dashboard click is instant. In a genuine emergency where seconds matter, the dashboard still wins. That is an intentional escape hatch, not a failure of the design.
Fork maintenance. Owning a fork means keeping up with vendor API changes yourself. WAF endpoints are stable enough that this has been low cost. If they ever change materially, it is a real project. Worth pricing in.
CRD schema evolution. Changing a CRD schema in a running cluster is a real operation. Additive changes are easy. Removing or renaming fields requires care. Nothing exotic, just discipline.
Rule count in production: north of twenty WAF rules and fifteen rate limits at peak, all versioned, all reviewable, all revertable.
Terraform was breaking log continuity. Kubeflare existed and had the right shape but was aging and over-scoped. Building a new operator from scratch was too much work for one problem.
Fork, prune, rewrite the API layer, ship the smallest thing that solves the actual pain. That's the pattern. Applies anywhere an existing tool is 60 percent right and the missing 40 percent is well-defined.