Post

Reducing Traefik CPU Usage on a Single-Core K3s Node

When my single-core k3s control plane hit 95% CPU and load averages above 3.7, I discovered my Traefik Gateway API provider was thrashing the API server with unnecessary watch events. Here's what I changed and what worked.

Reducing Traefik CPU Usage on a Single-Core K3s Node

A couple of months ago, I migrated my Kubernetes ingress from nginx to Traefik with the Gateway API. It worked well, but last week I noticed something was off. My k3s control plane node (in-oci-01 — a modest 1-core VM in Oracle Cloud) was running at 95% CPU with a load average of 3.78.

That’s 3.78 on a 1-core VM. It was pegged.

This post covers how I investigated the issue, what I found, and the specific changes that brought CPU down to ~25%.

The Investigation

I started with the usual suspects. kubectl top nodes showed nothing unusual at the node level, so I went deeper.

On the Host

SSH’d into the node and ran ps aux --sort=-%cpu | head:

1
2
3
4
k3s-server         43%
traefik            21%
tailscaled (×2)     5%
containerd/system   5%

Traefik was consuming ~21% of a single core — and that was just the process itself. The real cost was what it triggered in the API server.

Inside the Pods

kubectl top pods on the same node confirmed it:

Pod CPU
traefik (external-gateway) 167m
pi-hole 28m
postgres, redis, metrics-server ~20m total

The K3s Server Itself

The k3s binary embeds etcd, kube-apiserver, controller-manager, and scheduler — all in one process. It was using 43% CPU on its own. I checked the API server metrics for clues:

1
kubectl get --raw=/metrics | grep apiserver_request_total

62,606 GET requests to Endpoints and EndpointSlices. Most of these were coming from Traefik’s Gateway API provider, which maintains WATCH connections on Gateway API resources, services, and endpoints.

The pattern was clear: my Traefik instance had --providers.kubernetesGateway=true with all defaults. Every WATCH event — from any Gateway API resource, service change, or endpoint update — triggered a full config rebuild. On a 1-core machine running an embedded etcd, this cascades quickly.

Why This Happened

When you enable providers.kubernetesGateway in Traefik, it does the following:

  1. Establishes WATCH informers on GatewayClasses, Gateways, HTTPRoutes, TLSRoutes, Services, and EndpointSlices
  2. On every event from any of those informers, it rebuilds its internal routing configuration
  3. For config rebuilds, it queries the API server for endpoints and endpointslices — which is where those 62k GETs came from

Every time a pod starts, a service updates, or a route changes — Traefik’s config gets rebuilt from scratch. In a cluster with ~30 components and ArgoCD reconciling constantly, there’s a lot of churn.

The nginx ingress I had before was simpler — it read a static config and proxied traffic. No controller loops, no informers, no API server pressure.

The Changes

I made three changes. Here they are in order of impact.

1. Throttle Config Rebuilds (Biggest Win)

Traefik’s throttleDuration controls the minimum time between config rebuilds. The default is 0s — meaning every single K8s event triggers a rebuild.

Setting it to 10s batches rapid events so Traefik only rebuilds once every 10 seconds during periods of high change activity.

1
2
additionalArguments:
  - "--providers.kubernetesGateway.throttleDuration=10s"

This was the single most impactful change. Config rebuilds went from happening constantly to once every 10 seconds at most, and the API server GETs to endpoints dropped off a cliff.

2. Reduce API Server Request Rate

Traefik’s default qps/burst to the API server is 50/100 — aggressive for any controller sharing a 1-core node with its API server.

1
2
  - "--providers.kubernetesGateway.qps=20"
  - "--providers.kubernetesGateway.burst=40"

This halves the sustained request rate. Since routing decisions are cached locally, there’s no latency impact on actual traffic — only config rebuilds are affected.

3. Scope Traefik to Relevant GatewayClasses

The cluster has three GatewayClasses:

GatewayClass Gateways Purpose
external-gateway 15 Public (Cloudflare-facing)
internal-gateway 6 Internal (Tailscale)
public-gateway 0 Dead — created during testing

Without labelSelector, Traefik was watching and reconciling all three — including the dead public-gateway. I narrowed it to only the GatewayClass that actually matters:

1
  - "--providers.kubernetesGateway.labelSelector=app.kubernetes.io/instance=external-gateway-external-gateway"

This prevents Traefik from wasting cycles on routes and gateways it will never serve.

Bonus: Disable Access Logs

Traefik had --accesslog=true with default field logging. Every request was writing a formatted log line. I wasn’t actively using those logs, so I disabled it.

1
2
accessLog:
  enabled: false

The Result

After applying these changes and restarting Traefik:

Before: ~95% CPU, load average 3.78 After: ~25% CPU, load average well under 1.0

The k3s-server itself dropped from 43% to about 15-18% — mostly because the API server wasn’t drowning in WATCH connections and endpoints GETs anymore.

Traefik K3S CPU trend

What Each Change Contributed

Change CPU reduction Notes
throttleDuration=10s ~15-20% Stopped config thrashing
qps=20 / burst=40 ~5-8% Less API server queueing
labelSelector ~3-5% Ignoring irrelevant GatewayClasses
Disable access logs ~1-2% Per-request I/O overhead removed

What I Would Have Done Differently

Measure before migrating. I switched from nginx to Traefik and didn’t think to check the CPU impact until weeks later. A simple kubectl top pods right after the migration would have caught this immediately.

Start with throttleDuration from day one. There’s no reason to have it at 0s in a homelab cluster where config changes are infrequent. A 5-10s throttle costs nothing and prevents this kind of cascading load.

Summary

If you’re running Traefik with the Gateway API provider — especially on a small node — these flags are worth setting:

1
2
3
4
--providers.kubernetesGateway.throttleDuration=10s
--providers.kubernetesGateway.qps=20
--providers.kubernetesGateway.burst=40
--providers.kubernetesGateway.labelSelector=<your-gatewayclass-label>

They don’t affect routing performance. They just stop Traefik from rebuilding config on every K8s event and hammering the API server unnecessarily. On a 1-core k3s control plane running embedded etcd, that makes all the difference.

This post is licensed under CC BY 4.0 by the author.