How to: Install, Use and Debug the Kubernetes Cluster Autoscaler

What is the Kubernetes Cluster Autoscaler?

The Kubernetes Cluster Autoscaler extends the regular capabilities of autoscaling. By default, Kubernetes only supports the scaling of workloads, which translates to adding or removing pods. This allows your current cluster resources to be redistributed, but it does not allow you to change the amount of resources in a given cluster. If you would like to increase or decrease resources based on current usage, you would need to do this manually by adding or removing nodes.

This is where the Cluster Autoscaler comes in: if you have unschedulable pods due to resource constraints, it can add another node, and if the average node utilisation is too low and your workload allows it, it can also remove nodes. This can improve your financial efficiency by only adding compute during peak hours.

How to install & configure the Xelon Kubernetes Cluster Autoscaler?

Before you can install the Xelon Kubernetes Cluster Autoscaler, you have to upgrade your cluster to at least Kubernetes version 1.35.

How to get the pool IDs

You need the ID of the node pool you would like to scale. To get it, you can create a temporary debug pod like this:

kubectl -n kube-system run xelon-probe --rm -it --restart=Never \
  --image=alpine:latest \
  --overrides='{
    "spec": {
      "containers": [{
        "name": "xelon-probe",
        "image": "alpine:latest",
        "stdin": true, "tty": true,
        "env": [
          {"name": "XELON_API_URL",    "value": "https://hq.xelon.ch/api/v2"},
          {"name": "XELON_CLIENT_ID", "valueFrom": {"secretKeyRef": {"name": "xelon-api-credentials", "key": "clientId"}}},
          {"name": "XELON_CLUSTER_ID","valueFrom": {"secretKeyRef": {"name": "xelon-api-credentials", "key": "kubernetesClusterId"}}},
          {"name": "XELON_TOKEN",     "valueFrom": {"secretKeyRef": {"name": "xelon-api-credentials", "key": "token"}}}
        ]
      }]
    }
  }' -- ash

And then get the list of worker pools in your cluster:

apk add curl jq
curl -fsS "${XELON_API_URL}/kubernetes/${XELON_CLUSTER_ID}/pools" \
  -H "Authorization: Bearer ${XELON_TOKEN}" \
  -H "X-User-Id: ${XELON_CLIENT_ID}" |
jq -r '.[] | [.identifier, .name, (.nodes | length)] | @tsv'

How to install with Kustomize

We recommend installing with Kustomize, so you can easily upgrade in the future by simply switching the tag. Be sure to use the autoscaler release closest to your Kubernetes version.

kustomization.yaml:

apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization

resources:
  - https://raw.githubusercontent.com/Xelon-AG/autoscaler/v1.35.2-xelon.2/cluster-autoscaler/cloudprovider/xelon/examples/cluster-autoscaler.yaml

patches:
  - target:
      group: apps
      version: v1
      kind: Deployment
      name: xelon-cluster-autoscaler
      namespace: kube-system
    patch: |-
      - op: replace
        path: /spec/template/spec/containers/0/args/1
        value: --nodes=3:5:pt3s8gen7g
      - op: replace
        path: /spec/template/spec/containers/0/env/0
        value:
          name: XELON_BASE_URL
          value: https://hq.xelon.ch/api/v2/

Be sure to adapt the first patch to your needs: --nodes=<min nodes>:<max nodes>:<pool ID> The example scales the node pool "pt3s8gen7g" between 3 and 5 nodes.

What are scale conditions?

The scale conditions can be tweaked to your needs. The following two sections explain the current defaults and behaviour.

Scale up

A new node gets added if a pod is unschedulable — because the kube-scheduler has set PodScheduled=False with Reason=Unschedulable — and the pod does not fall into any of the filters listed below:

The "already schedulable" filter is why a pending pod sometimes produces no scale-up: CA has counted capacity on a node that is still being provisioned.

Scale down

A node is removed only after passing six sequential gates. Failing any one of them resets the clock in gate 5.

Gate 1: Loop enabled and not in cooldown

Gate 2: Candidate pre-filter

• Node belongs to an autoscaled node group.
• The group is above its MinSize(). With --nodes=1:4, the last node is never a candidate.

Gate 3: Eligibility

• Not already being deleted (no ToBeDeletedByClusterAutoscaler taint).
• No cluster-autoscaler.kubernetes.io/scale-down-disabled: "true" annotation.
• If unready: --scale-down-unready-enabled (default true) must be on.
• Utilization at or below --scale-down-utilization-threshold (default 0.5).

Node prod-w-1-1 unremovable: memory requested (93.2809% of allocatable) is above the scale-down utilization threshold 

DaemonSet and mirror pods can be excluded from the calculation via --ignore-daemonsets-utilization and --ignore-mirror-pods-utilization.

Gate 4: Simulation

Every pod on the node must:

1. Be drainable (no blocking PodDisruptionBudget, not a kube-system pod without a PDB, not a pod without a controller, no local storage unless annotated safe-to-evict) — otherwise Node X cannot be removed: …
2. Have somewhere else to go — otherwise Node X is not suitable for removal (reason NoPlaceToMovePods)

Simulation is capped by --scale-down-simulation-timeout (30s).

Gate 5: Unneeded long enough

The clock resets whenever a node fails any gate in a loop. A node that flickers above the utilization threshold every few minutes will never accumulate 10 continuous minutes. Track it with:

kubectl logs -n kube-system deploy/xelon-cluster-autoscaler | grep "was unneeded for" 

Gate 6: Min size and resource floor

• Group size minus deletions already in flight must stay above MinSize().
• Removing the node must not drop the cluster below the minimums in --cores-total / --memory-total.

How to debug the autoscaler?

Debugging the autoscaler is simple: just follow the logs and check what happens between the "Starting main loop" messages.

kubectl logs -f -n kube-system deploy/xelon-cluster-autoscaler 

cluster-autoscaler.kubernetes.io/safe-to-evict: "true"