요약
Hello! My name is Ssup, and I work as an engineer on the Cluster team within Karrot’s SRE organization. Our team handles a wide range of responsibilities — from operating AWS EKS clusters and configuring Istio-based service mesh networks, to managing monitoring components like Prometheus and Loki.
Most of Karrot’s workloads run on AWS EKS clusters. While workloads can be classified in many ways, one of the most common distinctions is between Server workloads and Job workloads. Unlike Server workloads, Job workloads have a clear start and end — they run once, complete their task, and stop.
One of the key characteristics of Job workloads is that they are difficult to interrupt once started. Most Job workloads need to restart from the beginning if they are interrupted, which means any disruption directly translates to wasted time and compute. This is especially true for long-running Jobs that take an hour or more to complete — interrupting them is simply not an option.
This “hard to interrupt” nature is one of the main reasons why autoscaling EKS Node Groups for Job workloads is so challenging. In this post, I will walk you through how we at Karrot worked around these constraints to successfully enable autoscaling for the EKS Node Groups running our Job workloads.
Previous Approach to Managing Job Workloads
As mentioned earlier, Job workloads are difficult to interrupt, which means any Node running a Job workload cannot be removed during the Scale-in process of autoscaling. This means that the more evenly Job workloads are distributed across Nodes, the more they interfere with autoscaling. For this reason, Karrot operates a dedicated Node Group exclusively for Job workloads, separate from other workloads.
The diagram above illustrates an example of separating the Node Group for Server workloads from the Node Group for Job workloads. Before the separation, Job workloads were running across all Nodes, making it impossible to perform Node Scale-in until all Job workloads had completed. After the separation, however, Node Scale-in for Server workloads can be performed without being affected by Job workloads.
Despite this improvement, the Job workload Node Group was still operating with a fixed number of Nodes, which introduced a new set of problems. When Job workloads were light, the fixed Node count led to unnecessary costs. On the other hand, when more Job workloads than expected came in at once, processing would fall behind and workloads could not start on time.
To guard against these delays, we had no choice but to provision Nodes conservatively with extra headroom — which inevitably drove up costs. In the end, finding the right balance between cost and stability, and continuously maintaining that balance, became a significant operational burden. This led us to explore whether autoscaling could be introduced for the Job workload Node Group as well.
Applying Autoscaling to the Job Workload Node Group
To effectively apply autoscaling to the Job workload Node Group, we needed to address two key challenges. The first was applying Bin-packing to enable smooth Scale-in, and the second was preventing running Job workloads from being forcefully interrupted during the Scale-in process.
Applying Bin-packing to Job Workloads
Since Job workloads are difficult to interrupt once started, they need to be concentrated on as few Nodes as possible rather than spread evenly across many Nodes — this is what makes smooth Scale-in possible later on. In other words, unlike typical Server workloads, Job workloads need to be scheduled using a Bin-packing strategy.
The diagram above illustrates examples of Pod scheduling with and without Bin-packing applied. Without Bin-packing, Job workload Pods are scheduled as evenly as possible across multiple Nodes. With Bin-packing applied, Pods are scheduled to fill up as few Nodes as possible. This means that Bin-packing increases the likelihood of empty Nodes being available, making Node Scale-in more effective.
The default Kubernetes scheduler distributes Pods across as many Nodes as possible, which means it does not perform Bin-packing by default. To address this, we explored several approaches — mainly running a dedicated scheduler for Job workloads, or leveraging PodAffinity.
For the dedicated scheduler approach, we looked into the Bin-packing features offered by well-known schedulers such as Volcano and Yunikorn, as well as the MostAllocated strategy provided by the default Kubernetes scheduler. However, any of these options would require us to run and maintain a separate scheduler, which we wanted to avoid. Instead, we decided to go with PodAffinity.
PodAffinity is a feature that encourages new Pods to be placed on the same Node as certain existing Pods. For example, if two services are sensitive to network latency, PodAffinity can be used to always place their Pods on the same Node, reducing communication overhead. By applying this idea a little differently, we can encourage Job workload Pods to be placed on the same Node as each other, naturally achieving a Bin-packing effect.
Looking at the PodAffinity source code comments, it becomes clear that it does more than simply co-locate Pods on the same Node. The more Pods matching the Affinity condition are already running on a Node, the higher that Node’s weight sum becomes — and the scheduler prioritizes Nodes with the highest weight sum. In other words, the more Job workload Pods are already running on a Node, the more likely new Pods are to be scheduled there as well.
apiVersion: batch/v1
kind: CronJob
metadata:
name: binpacking-test-${index}
spec:
jobTemplate:
spec:
ttlSecondsAfterFinished: 0
template:
metadata:
labels:
group: job
spec:
affinity:
podAffinity:
preferredDuringSchedulingIgnoredDuringExecution:
- weight: 100
podAffinityTerm:
labelSelector:
matchLabels:
group: job
topologyKey: kubernetes.io/hostname
To verify that PodAffinity works as expected, we created multiple CronJobs and ran a test. The YAML file above is the CronJob configuration used for the test. You can see that the Job workload Pods are labeled with group: job, and the PodAffinity is configured to target Pods with the same group: job label. In other words, the more Pods with the same label are already running on a Node, the more likely new Pods are to be scheduled on that Node as well.
The diagram above shows a visualization of the Pod distribution before PodAffinity was applied. Using the kind tool, we set up a Kubernetes cluster consisting of one Master Node (kind-control-plane) and five Worker Nodes (kind-worker[1-5]), then created a number of CronJobs. The cluster state was then visualized using kube-ops-view.
The green boxes inside each Node represent Pods. The green boxes at the bottom are Pods belonging to the kube-system Namespace — the core components required to operate the Kubernetes cluster. The boxes at the top represent general workload Pods that run actual services or tasks.
Since the kind cluster has no workloads other than CronJobs, all Pods inside the red rectangles are CronJob Pods. As Bin-packing has not been applied, you can see that the Job workload Pods are evenly distributed across all five Worker Nodes.
The diagram above shows a visualization of the Pod distribution after PodAffinity was applied. You can see that Job workload Pods are now concentrated on specific Nodes. Initially, the Pods were concentrated on kind-worker2 and kind-worker3, and over time shifted to kind-worker3 and kind-worker5. This confirms that Bin-packing is working exactly as intended.
apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
name: add-job-affinity
spec:
rules:
- name: add-job-label-and-affinity
match:
any:
- resources:
kinds:
- Pod
preconditions:
all:
- key: "{{ request.object.metadata.ownerReferences[0].kind }}"
operator: Equals
value: Workflow
mutate:
patchStrategicMerge:
metadata:
labels:
group: job
spec:
affinity:
podAffinity:
preferredDuringSchedulingIgnoredDuringExecution:
- weight: 100
podAffinityTerm:
labelSelector:
matchLabels:
group: job
topologyKey: kubernetes.io/hostname
PodAffinity settings for Job workload Pods are currently applied in bulk using Kyverno. The YAML file above is an example of a Kyverno rule that applies PodAffinity to Argo Workflow Pods. It checks the Pod’s ownerReference to verify whether the Pod was created by Argo Workflow, and only adds the group: job label and PodAffinity configuration if that condition is met.
Preventing Forced Termination of Job Workload Pods
If a Node is removed during a Scale-in event while a Job workload is still running, the Job workload can be forcefully interrupted.
- Cluster Autoscaler : cluster-autoscaler.kubernetes.io/safe-to-evict: "false"
- Karpenter : karpenter.sh/do-not-disrupt: "true"
To prevent this, both Cluster Autoscaler and Karpenter — the two main Node autoscalers — provide a feature that excludes Nodes running Pods with a specific Annotation from being selected as Scale-in targets.
apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
name: add-job-do-not-disrupt
spec:
rules:
- name: add-do-not-disrupt-annotation
match:
any:
- resources:
kinds:
- Pod
preconditions:
all:
- key: "{{ request.object.metadata.ownerReferences[0].kind }}"
operator: Equals
value: Workflow
mutate:
patchStrategicMerge:
metadata:
annotations:
karpenter.sh/do-not-disrupt: "true"
Job workload Pods are configured with this Annotation to prevent forced termination. Just like the PodAffinity settings, the Annotation is applied in bulk using Kyverno. The YAML file above is an example of a Kyverno rule that applies Karpenter’s forced termination prevention Annotation, karpenter.sh/do-not-disrupt: "true", to Argo Workflow Pods.
If you are using Cluster Autoscaler together with a Multi-AZ ASG (Auto Scaling Group), there is one more thing to keep in mind — you will need to disable the AZ Rebalancing feature of the ASG. This is because the ASG has no awareness of the state of Job workload Pods and simply acts to maintain an equal number of Nodes across Availability Zones.
The diagram above illustrates how AZ Rebalancing occurs. During the Scale-in process, the balance of Nodes across Availability Zones can become uneven, which triggers AZ Rebalancing. When AZ Rebalancing occurs, Job workload Pods can be forcefully interrupted.
Issues Encountered After Applying Autoscaling
After applying Bin-packing and forced termination prevention to Job workload Pods, we confirmed that autoscaling worked correctly in our Alpha environment. However, when we rolled it out to Production, a number of unexpected issues arose. The root cause was that unlike the Alpha environment, Production sees a large number of Job workload Pods being created all at once at specific times, causing a temporary but significant surge in load on certain Nodes.
The graph above shows the number of running Job workload Pods over time. You can see that the Pod count spikes every 10 minutes, with an even larger surge at the top of each hour. Before Bin-packing was applied, Job workload Pods were distributed across many Nodes. After Bin-packing was applied, however, Pods became concentrated on a small number of specific Nodes, causing those Nodes to experience significantly higher load.
Breaking down the issues further, we encountered four distinct problems: kubelet overload, Image Pull failures, EBS Volume Throttling, and CNI Plugin IP assignment delays. Each of these required a different approach to resolve.
kubelet Overload
kubelet is installed on each Node in a Kubernetes cluster and is responsible for managing the Pods running on that Node. As a result, it is one of the first components to experience load as the number of Pods on a Node increases. At Karrot, we measure kubelet load using the kubelet_pleg_relist_duration_seconds_bucket metric, which represents the time it takes for kubelet to relist Pods.
After applying autoscaling, we observed that the kubelet_pleg_relist_duration_seconds_bucket metric value spiked significantly at the top of each hour, as Bin-packing caused a large number of Job workload Pods to be created on a small number of Nodes all at once. This was made worse by the fact that most Job workload Pods had their CPU Resource Request set lower than their actual CPU usage, which allowed far more Pods than expected to be scheduled onto a single Node, further increasing the load.
To reduce kubelet load, we looked for a way to maintain Bin-packing while preventing too many Job workload Pods from concentrating on a single Node. The most ideal solution would have been Pod Right-sizing — ensuring that each Job workload Pod accurately declares its actual CPU usage in the Resource Request. However, we determined that Right-sizing every single Job workload Pod was not practical.
While exploring alternative approaches, the first thing we tried was increasing the kube-reserved Resource for kubelet. kube-reserved is a value that sets aside resources on a Node to ensure kubelet has enough to operate properly. By increasing kube-reserved, the amount of resource available for Pods (Pod Allocatable Resource) decreases, which in turn reduces the number of Pods that can be scheduled onto a single Node. The diagram above briefly illustrates the relationship between kube-reserved and Pod Allocatable Resource.
However, increasing kube-reserved ultimately proved ineffective. Without Pod Right-sizing in place, simply raising kube-reserved was not enough to prevent Pod concentration. We considered increasing it further, but if kube-reserved becomes too large, the allocatable resources available to Pods shrink as well, making it difficult to raise aggressively.
This led us to another approach — adjusting the maxPods value in kubelet. maxPods defines the maximum number of Pods that can be scheduled onto a single Node. By lowering maxPods, we can cap the number of Pods per Node, which prevents excessive Pod concentration while still maintaining Bin-packing.
status:
allocatable:
cpu: 15890m
ephemeral-storage: "95491281146"
hugepages-1Gi: "0"
hugepages-2Mi: "0"
memory: 29310740Ki
pods: "60"
capacity:
cpu: "16"
ephemeral-storage: 104779756Ki
hugepages-1Gi: "0"
hugepages-2Mi: "0"
memory: 32310036Ki
pods: "60"
The configured maxPods value can be verified by checking the pods field under allocatable or capacity in the Node status. The YAML file above shows an example of a Node status. For instance types of 4xlarge or larger in AWS EKS, the default value is 234. At Karrot, we have set this to 60 for Job workload Nodes.
The way to apply the maxPods setting depends on the autoscaler being used. If you are using Cluster Autoscaler, it can be applied through the NodeConfig in EC2 UserData. If you are using Karpenter, it can be set via kubelet.maxPods in the EC2NodeClass.
Image Pull Failures
The next issue we encountered was Image Pull failures. As a large number of Job workload Pods were created all at once at the top of each hour, ImagePullBackOff errors began occurring frequently. ImagePullBackOff is an error that occurs when kubelet fails to pull a Container Image, leaving the Pod stuck in a waiting state and unable to start normally.
After investigating the root cause, we found that the registryPullQPS and registryBurst settings in kubelet were the problem. registryPullQPS defines the maximum number of Image Pull requests per second that kubelet can make to a Container Registry, while registryBurst defines the maximum number of requests allowed in a burst. Both values were set too low by default, causing Image Pull requests to bottleneck when a large number of Pods were created simultaneously.
To resolve this, we increased registryPullQPS from its default value of 5 to 40, and registryBurst from 10 to 60. After making these changes, ImagePullBackOff errors no longer occurred. Note that both values were derived based on the maxPods value of 60 that we had set earlier.
Both registryPullQPS and registryBurst can be configured through the NodeConfig in EC2 UserData, regardless of whether you are using Cluster Autoscaler or Karpenter.
EBS Volume Throttling
E1218 15:20:53.795027 5233 log.go:32] "CreateContainer in sandbox from runtime service failed" err="rpc error: code = Unknown desc = failed to reserve container name \\"init_sync-sa-resources-to-db-1766070900-hook-207168736_krp_f181b699-cbb4-4633-b6d4-b54c531be57e_1\\": name \\"init_sync-sa-resources-to-db-1766070900-hook-207168736_krp_f181b699-cbb4-4633-b6d4-b54c531be57e_1\\" is reserved for \\"1ddbf865c643250c8a88ab890a841409cb21feef6b27e7666a460d226b34db2d\\"" podSandboxID="f2deaecc93076c82e1f7dc686eae07091d12ac3657168ecd65edb3b7836bf69c"
After increasing registryPullQPS and registryBurst to allow kubelet to pull more Images concurrently, we started seeing error logs like the one above appearing intermittently. The error indicated that the Container Runtime had failed to create a Container with a reserved name.
Upon further investigation, we found that the concurrent Image Pulls were causing Throttling on the EBS Volume attached to the EC2 instance. Both IOPS and Bandwidth were being throttled, which prevented the Container Runtime from operating properly. This was the root cause of the errors.
To resolve this, we increased the EBS Volume IOPS from 3,000 to 8,000 and the Throughput from 150MB/s to 800MB/s. After making these changes, the errors no longer occurred.
CNI Plugin IP Assignment Delays
When a large number of Pods are created on a Node all at once, the CNI Plugin responsible for assigning IP addresses to each Pod also comes under heavy load. If you are using AWS VPC CNI, it is generally recommended to consider increasing the number of WARM IPs or adopting Prefix Mode.
WARM IP is an approach where IP addresses are pre-allocated before Pods are created. Rather than requesting a new IP each time a Pod is created, IPs are immediately drawn from a pre-provisioned IP Pool, which reduces IP assignment delays at Pod creation time. Prefix Mode, on the other hand, allocates IP addresses in CIDR block units (/28) rather than one at a time. Since multiple IPs are obtained in a single request, IP addresses can be assigned more quickly even when a large number of Pods are created simultaneously.
At Karrot, however, we took a different approach instead of increasing WARM IPs or adopting Prefix Mode — we applied Host Network Mode. Since Host Network Mode allows Pods to use the Node’s IP address directly rather than having their own dedicated IP, the IP assignment process is bypassed entirely at Pod creation time, which means the CNI Plugin load can be avoided altogether.
How the Metrics Changed After Applying Autoscaling
The graph above shows the number of Nodes in the Job workload Node Groups over time, before and after applying autoscaling. At Karrot, we operate separate Node Groups by CPU Architecture — job is the Node Group for amd64 Architecture, and job-arm is the Node Group for arm64 Architecture.
The graph shows that after applying autoscaling, the Node count fluctuates dynamically. In terms of average Node count, the job Node Group decreased from 5 to 4.198 Nodes — a reduction of approximately 16% — while the job-arm Node Group increased from 2 to 4.18 Nodes, an increase of approximately 109%.
There are two reasons behind the significant increase in the job-arm Node Group. First, the number of Job workloads running on the job-arm Node Group itself increased after autoscaling was introduced. Second, the maxPods setting limited the number of Pods that could be assigned per Node, which caused Job workloads to be distributed across a larger number of Nodes.
The graph above shows how Image Pull times changed before and after applying autoscaling. Image Pull time increased from 3 seconds to 7.972 seconds for the job Node Group, and from 16 seconds to 29.76 seconds for the job-arm Node Group. This is because autoscaling causes Nodes to be repeatedly created and deleted, which reduces the number of Container Images cached on each Node.
The graph above shows the Pending time of Job and Workflow Pods before and after applying autoscaling. Here, Job refers to Pods created by Kubernetes Jobs, and Workflow refers to Pods created by Argo Workflow.
Before autoscaling was applied, the Pending time for workflow Pods spiked up to 14 seconds, and once Pending occurred, it tended to persist for a long time. After autoscaling was applied, the maximum Pending time dropped to 10 seconds, and the pattern shifted to one where Pending occurs briefly and resolves quickly. The average Pending time also decreased from 1.54 seconds to 0.78 seconds — a reduction of approximately 49%.
It is worth noting that Pending time includes Image Pull time. As we saw earlier, Image Pull times actually increased after autoscaling was applied. The fact that Pending time still decreased despite this means that autoscaling is responding quickly enough to place Pods onto Nodes in a timely manner.
The last graph shows the Running time of Job and Workflow Pods before and after applying autoscaling. The Running time for workflow Pods decreased from 138 seconds to 72.6 seconds — a reduction of approximately 47% — while job Pods increased from 290.76 seconds to 321.48 seconds, an increase of approximately 10.56%. Since most workflow Pods run on the job-arm Node Group, and the number of Nodes in that group roughly doubled, it makes sense that the Running time was cut by about half as well.
The table above summarizes the key metric changes introduced throughout this post. After applying autoscaling, the Node count adjusted naturally to the appropriate level, resulting in overall performance improvements including reduced Pod Pending times and shorter Running times.
Closing Thoughts
At Karrot, a wide variety of Job workloads run on AWS EKS clusters — including CronJobs provided through Kontrol, our Internal Developer Platform (IDP), Airflow and Dagster for data orchestration, and GitHub Actions Runners for CI/CD.
We have applied the autoscaling approach introduced in this post to all of these Job workloads, enabling us to utilize Nodes in a cost-efficient and flexible manner. Beyond the performance gains such as reduced Pending times and shorter Running times, the biggest benefit has been freeing ourselves from the operational burden of manually monitoring and adjusting Node counts.
We hope this post has been helpful to those managing Job workloads. We will continue exploring new ways to improve the stability and efficiency of our Kubernetes clusters. Thank you for reading!
Our Journey to Autoscaling EKS Node Groups for Job Workloads was originally published in 당근 테크 블로그 on Medium, where people are continuing the conversation by highlighting and responding to this story.