← 목록으로

Closing the GitHub Actions Monitoring Gap: Observing ARC Runners with an Exporter

Closing the GitHub Actions Monitoring Gap: Observing ARC Runners with an Exporter

요약

Hi, I’m Uno from the Delivery part of Karrot’s SRE team. Our team’s mission is to run Kontrol, our internal deployment platform on Kubernetes, and manage the company-wide CI system that mostly relies on GitHub Actions. In this post I’ll introduce the exporter we built to see which Jobs are waiting and which Runner is running each Job. I’ll also share how we improved it in production.

A Workflow defines what happens when an event such as a push or pull request occurs. Each Workflow is split into Jobs such as test and build that run on execution environments called Runners. If no Runner is free, a Job waits until one becomes available.

name: Simple                    # 1. Workflow: defines the automation
on:
workflow_dispatch:
jobs:
run_shell_command: # 2. Job: the unit that gets a Runner and runs
name: Actions are amazing
runs-on: arc-test-runner
steps: # 3. Step: the commands the Job executes
- name: Say hello
run: |
echo "Hello!"

Runners can be GitHub-hosted or self-hosted. At Karrot we mainly use self-hosted Runners on our own Kubernetes clusters. These are managed by a Kubernetes operator called ARC (Actions Runner Controller).

ARC organizes Runners into groups called Runner Scale Sets and scales each set according to Job demand. Each set defines the Runner image and resources, along with minimum and maximum Runner counts. A Workflow can specify a Runner Scale Set in runs-on so its Jobs are assigned to Runners in that set.

Source: https://docs.github.com/en/actions/concepts/runners/actions-runner-controller

Even with autoscaling, Jobs don’t always start immediately. A GitHub outage or a spike in Job requests can leave them stuck in the Queued state. While operating ARC, we often heard developers say, “My Action is stuck in Queued and won’t run.”

Answering that question wasn’t easy with the data we had. We often had to wait for an announcement on GitHub’s status page or infer the cause from ARC’s built-in metrics.

ARC’s built-in metrics show aggregate counts of Runners and Jobs, but they don’t tell us how many Jobs are waiting for a Runner, how long they’ve been waiting, or which Runner Pod is running a given Job. With that information, we could give developers a better understanding of what was happening with their Jobs.

While looking for a solution, we found an issue in the ARC repository proposing to add Job information to Runner Pods as labels. Adopting that approach would require controller changes and additional permissions, as well as a separate setup to collect the labels as metrics, so we decided to build our own solution.

We built action-runner-job-exporter to connect Runner Pods to their Assigned Jobs and track Jobs still waiting for a Runner. There are three key components for the exporter, which are separate binaries. Collector associates Job names with Runner Pods, Webhook tracks waiting Jobs, and StateStore persists Webhook's state.

action-runner-job-exporter architecture

How ARC Runners work

Before getting into the exporter, let’s look at how ARC creates a Runner Pod and how a Job reaches it.

  1. Each Runner Scale Set has a Listener Pod that keeps a session open with GitHub and waits for new Job assignments.
  2. A Workflow starts in a repository, and GitHub assigns each Job to the Runner Scale Set that matches its runs-on.
  3. The Listener works out how many Runners are needed from the assigned Jobs and the min/max Runner settings, then updates the replicas of the EphemeralRunnerSet.
  4. ARC creates that many EphemeralRunner resources, fetches a one-time registration config (JIT config) from GitHub for each, and starts the Runner Pods.
  5. Once a Runner Pod registers with GitHub, GitHub assigns a Job to it.
  6. After the Job starts, the Listener receives the assignment and writes the repository and Job details into the status field of that EphemeralRunner.
  7. The Runner process exits after the Job finishes and ARC cleans up the Runner’s resources.
Runner creation and Job assignment

ARC creates the Runners and GitHub decides which Job goes to each one. Because Runner Pods aren’t created for a specific Job, the Pod name alone doesn’t tell us which Job it’s running. That becomes visible only after the Listener updates the EphemeralRunner status.

Collector: Mapping Jobs to Runner Pods

We wanted SREs to detect GitHub Actions delays before developers had to ask about them. We started by finding a way to see which Job each Runner Pod was running. Neither Pod labels nor existing metrics carried that information, but an EphemeralRunner and its Runner Pod share the same name. We could use that name to match the Job information in the status to the right Pod.

Our first binary, Collector, exposes that relationship as a metric. It uses a Kubernetes informer to watch EphemeralRunner creations, updates, and deletions, keeping the latest state in an in-memory cache. From that cache, it serves the following metric.

arc_runner_job_info{namespace, pod, runner_name, repository, job_name, workflow_ref, workflow_run_id}

Collector emits this metric only after a Runner’s Job has started and status.jobRepositoryName is populated. The workflow_run_id label identifies the Workflow run, letting us link directly to its page at https://github.com/<org>/<repo>/actions/runs/<workflow_run_id>.

Collector architecture

The pod and namespace labels on this metric collide with those of the exporter Pod itself during a Prometheus scrape. As a result, the Runner Pod name appears in exported_pod and its namespace in exported_namespace. We join those labels with pod and namespace on container resource metrics (container_.*) to attach repository and Job names to CPU, memory, and network usage. The resulting dashboard links each Job to its Runner Pod and shows how many resources it used.

Resource usage panel by Job

Job information remains in the status after the Job ends until the EphemeralRunner is cleaned up. The metric therefore shows the Job assigned to each Runner even if the Job has already finished.

Webhook: Tracking queued Jobs

Collector filled one gap, but the status field is populated only after a Runner receives a Job and starts running it. Jobs still waiting in GitHub’s queue remained invisible, so we couldn’t tell how many were piling up.

To make those Jobs visible, we added a second binary called Webhook. It receives workflow_job events from our GitHub Organization whenever a Job's state changes. Because the events come directly from GitHub rather than through the ARC Listener, we can see waiting Jobs even when the Listener has problems, as long as GitHub delivers the webhooks.

A Job typically progresses through queued → in_progress → completed. The main Job states in workflow_job events are described below.

  • waiting: Waiting for a protection rule, such as a deployment environment approval, to be satisfied. This is different from waiting for a Runner.
  • queued: Waiting to run, usually for a Runner.
  • in_progress: Assigned to a Runner and running.
  • completed: Finished. Includes success, failure, and cancellation.

Webhook keeps Job state in memory using job_id as the key and exposes the following metrics.

  • github_workflow_jobs_queued: number of Jobs waiting to run
  • github_workflow_jobs_in_progress: number of running Jobs
  • github_workflow_job_queued_seconds: when the Job was created
  • github_workflow_job_queue_duration_seconds: histogram of how long Jobs waited before getting a Runner

Rather than storing a wait duration, github_workflow_job_queued_seconds holds the Job's creation time as a Unix timestamp. We can calculate the current wait time with time() - github_workflow_job_queued_seconds without the exporter having to keep updating the value. Once a Job is assigned to a Runner and starts running, it is removed from the metric.

Webhook architecture

Because GitHub doesn’t guarantee that webhook events arrive in order, we treat Job state as moving only queued → in_progress → completed. Late queued events are ignored for Jobs already in_progress or completed. To make that possible even after a Job finishes, we keep completed Jobs in the state map for a while as markers called tombstones. Removing them immediately could cause a late event to be counted as a new waiting Job.

With Webhook showing us how many Jobs were waiting, we could share more accurate status updates when delays affected many Jobs.

Job wait time panel

We also set up alerts for when too many Jobs have been waiting longer than a certain time and when a Runner Scale Set reaches 90% of its maxRunners. These alerts help us detect delays and investigate potential bottlenecks in our ARC configuration.

When the Runner count reaches 90% of maxRunners, When the number of long-waiting Jobs exceeds the threshold

StateStore: Preserving Webhook state

The first version of Webhook kept Job state only in memory, so it lost track of waiting Jobs whenever it was deployed or restarted. Saving and restoring the state through S3 was one option, but Webhook is exposed to the internet to receive GitHub events, and we didn’t want to grant it S3 permissions.

Instead, we built a third binary called StateStore. It runs on the internal network and handles snapshots as the only component with S3 permissions. This separation makes it harder for an attacker to use those permissions directly if Webhook is compromised.

Webhook, StateStore architecture

During a rollout, old and new Pods briefly run together. We use S3 conditional writes (ETag / If-Match) to keep them from overwriting each other's snapshots.

Fixing Job state errors

#1 Stale snapshots

One day an alert reported that 72 Jobs had been waiting a long time for a Runner, even though all of them had completed hours earlier. We traced the miscount to a Webhook restart that had restored an old snapshot.

At the time, --webhook-ttl (default 6h) controlled several things at once: retention of in-progress Job state, tombstones for finished Jobs, and Delivery IDs (X-GitHub-Delivery) used for deduplication. It also set the maximum age of a snapshot that could be restored. As a result, six hours' worth of records accumulated in the state map. We had set an 8MiB snapshot size limit, assuming it would only need to store in-progress Jobs, but eventually the snapshot outgrew it.

  1. Snapshots stopped saving: Delivery IDs and tombstones piled up until the snapshot hit 8MiB, and saving stopped.
  2. Webhook restarted: The Webhook Pod held the whole state map in memory. It went over its memory limit and got OOMKilled.
  3. Stale state restored: The new Pod restored 16,097 Job states from a snapshot saved 2 hours and 44 minutes earlier. The snapshot was old, but the 6-hour restore window still allowed it.
  4. Finished Jobs counted as waiting: completed events received after snapshots stopped saving only existed in the dead Pod's memory. The new Pod never saw them, so it counted those finished Jobs as queued.
Snapshot incident

No further events arrived to correct those restored Jobs, so their apparent wait times kept increasing. The alert cleared only after the 6-hour retention period passed and their state was deleted all at once.

Separate retention for each kind of data

We split retention by data type and set a separate limit for snapshot restore. In-progress Jobs are retained long enough to be tracked while tombstones and Delivery IDs have shorter retention periods. We also kept the restore window short because older snapshots are more likely to diverge from the actual state.

  • --webhook-ttl 6h: In-progress Jobs
  • --tombstone-ttl 1h(now 3h): Tombstones for completed and waiting Jobs
  • --dedup-ttl 10m: Delivery ID (X-GitHub-Delivery)
  • --snapshot-max-age 10m: Max age of a snapshot that can be restored

Only 175 of the 16,097 entries in the state map were in-progress Jobs at the time of the alert; most of the rest were tombstones. Reducing tombstone retention to one hour shrank the snapshot from 8MiB to about 0.66MiB.

We also found that event counters alone made it hard to distinguish stopped delivery from low traffic. To detect stalls, we added metrics for the last snapshot save and the last webhook received. At startup we now record the number of Jobs restored and the snapshot’s age as well.

#2 Failed webhook deliveries

Another source of incorrect state is a missed event. GitHub doesn’t automatically retry failed webhooks, so if Webhook is down or a temporary network issue causes it to miss an in_progress or completed event, the Job can remain in a waiting or running state for the entire 6-hour retention period.

To recover from failed deliveries, StateStore uses a GitHub App to check the Organization’s webhook delivery log every 5 minutes. It requests redelivery of in_progress and completed workflow_job events that failed in the last 30 minutes. We exclude queued and waiting events because they could cause incorrect counts if they arrive late. We also check Delivery IDs to avoid processing the same event twice.

#3 Reconciling Job states

Redelivery couldn’t solve every case. At one point, dozens of Jobs that had already succeeded still appeared as queued. GitHub Status reported a webhook-related incident around the same time.

Their queued events had arrived more than an hour after the Jobs were created, but with the tombstone retention set to one hour, the tombstones have already been expired by the time the events arrived. This lead to misunderstanding the events as new waiting jobs.

We still needed a way to correct the miscount, so we decided to reconcile Jobs whose state hadn’t changed for a certain period with their actual state on GitHub.

  • Every minute Webhook checks queued and in_progress Jobs with no event for over 10 minutes against the Actions API through StateStore. It updates their state from GitHub to completed, in_progress, while keeping transitions forward-only along queued → in_progress → completed. Jobs still queued keep their alerts. Events received during a lookup update the state as usual.
  • The lookups use StateStore’s existing GitHub App with the Actions: read permission added. Each cycle checks at most 50 Jobs starting with the least recently checked. That caps lookups at 3,000 an hour and keeps us within rate limits.
Keeping tombstones longer

Within 5 minutes of deploying the reconciliation logic, 244 completed Jobs were cleaned up and the number of long-waiting Jobs dropped from 30 to 4. Later that afternoon, though, queued events kept arriving 70 to 112 minutes late, causing about 30 Jobs to trigger the alert for roughly 10 minutes before they became eligible for reconcile. We raised the --tombstone-ttl default from 1 hour to 3 hours, leaving reconcile to clean up miscounts caused by events arriving more than 3 hours late.

Runner sizing and bottlenecks

#1 Right-sizing Runners

Collector linked Job names to Runner Pod resource usage so we could see how much CPU and memory each Job actually used. Comparing that usage with the Runner’s configured resource requests helps us determine whether the Runner is right-sized for the Job.

An unnecessarily large Runner wastes resources and can delay Pod scheduling while the cluster secures them. To help teams choose an appropriate sized runner, we use this data in a dashboard that recommends a Runner for each Job.

Recommended Runner panel

#2 ARC controller bottleneck

In another incident, the wait-time alert fired as Jobs waiting for Runners accumulated, and we could see the queue growing on the Webhook Queue dashboard.

We first checked for a GitHub outage, but nothing had been announced, and the call that fetches the Runner registration config (JIT config) had a normal median latency of 0.57 seconds. The problem turned up in ARC instead. The work queue (workqueue_depth) of the EphemeralRunner controller, which creates and cleans up Runners, had risen far above its usual level.

The EphemeralRunner controller work queue growing as Runner creation and cleanup fall behind

The number of workers processing that queue (controller_runtime_active_workers) had been stuck at 2 for 6 hours, the default value of runnerMaxConcurrentReconciles as of 0.12.x. Meanwhile, p99 processing time had risen from 0.9 seconds to 8–10 seconds. The controller couldn't process its work fast enough with only two workers and slower GitHub calls.

Work to create new Runners sat in the controller queue and slowed Pod creation. In one Runner Scale Set, 258 Runners were waiting to be created while only about 150 Pods existed. Cleanup was delayed too. As a result, 95 already-terminated Pods and 55 Runners being deleted continued to count against maxRunners.

We raised runnerMaxConcurrentReconciles from 2 to 5 to give the controller more throughput. Within about 10 minutes, the controller queue fell from 669 to 233, waiting Jobs went from 380 to 168, and Runners waiting in the Runner Scale Set dropped from 224 to 61. Although p99 processing time was still high at 11 seconds, increasing concurrency reduced the backlog. Comparing Webhook's Job wait-time metrics with ARC's controller metrics had helped us identify controller throughput as the bottleneck.

Results of increasing controller concurrency

Conclusion

Even though we operate ARC ourselves, running GitHub Actions inevitably depends on GitHub. We originally collected these metrics to keep developers informed during outages, but once Job states were visible, we found other uses for them too. They now help us size Runners appropriately and track down the sources of delays and bottlenecks.

I hope our experience helps you better understand what’s happening with your Jobs and Runners when using GitHub Actions with the ARC. Thanks for reading.

Closing the GitHub Actions Monitoring Gap: Observing ARC Runners with an Exporter was originally published in 당근 기술 블로그 on Medium, where people are continuing the conversation by highlighting and responding to this story.

← 목록으로