CI: Self-Hosting GitHub Actions Runners on K8s

Summary

The muscle behind CI is the GitHub Actions Runner — the runtime that actually executes the workflow steps (a VM for GitHub-hosted runners, a pod for ARC). Most public repositories use a GitHub-hosted runner: runs-on: ubuntu-latest (or windows, macos) pulls a pre-baked VM from Microsoft, already loaded with the common toolchain. That’s free and unlimited for public repos. For private repos it caps at 2000 free minutes per month (official docs).

For an org with several active dev projects, that cap gets eaten quickly. A self-hosted runner removes the cap, but raw self-hosted (a long-lived VM you patch yourself) means stateful runners — leftover caches, lingering processes, ambient secrets — which is a security and reproducibility problem.

ARC (Actions Runner Controller) is the K8s-native answer: each job runs in a fresh, ephemeral pod. The pod is created when the job is claimed and torn down when the job finishes — clean state every time, autoscaled by the controller, configured via ArgoCD alongside the rest of the cluster’s manifests.

• ARC configurations are managed through ArgoCD (GitOps: the argo-ops Git repo is the single source of truth).
• ARC is what GitHub talks to when a workflow asks for a self-hosted runner that matches its labels (e.g. runs-on: [self-hosted, arc-dev]).

Multiple ARC runner sets can coexist with distinct labels — arc-dev, arc-stage, arc-prod — and each listener claims only the jobs whose runs-on: labels match its own set. That’s how a single cluster routes CI jobs from multiple projects or environments to the right runner pool.

ARC ↔ ArgoCD (Configuration)

Standby

ArgoCD watches the argo-ops Git repo for any change to ARC’s config

ArgoCD watches argo-ops — the GitOps single source of truth — and reconciles cluster state on any change to ARC’s manifests.

On Update

From change-detected to arc-systems / arc-runners reconciled on the on-prem cluster

1. Poll detects the change — ArgoCD’s periodic Git poll (or a webhook from the Git server, when configured) sees the new commit on argo-ops and pulls it.
2. Update lands on the CRs — dx-arc-controller and dx-arc-runner (the project’s ARC Custom Resources) in the ArgoCD cluster reconcile to the new desired state.
3. Propagate to the on-prem cluster — the corresponding arc-systems and arc-runners resources in the on-prem K8s cluster pick up the change, and ARC adjusts the runner pool accordingly.

ARC ↔ GitHub Actions (CI)

The K8s objects ARC creates: an AutoscalingRunnerSet (the user-defined runner pool spec), one AutoscalingListener pod per set (long-polling GitHub), and an EphemeralRunner CR per claimed job (each backed by one pod that’s torn down after the job). The “listener pod” and “ephemeral CR” terms below map onto these K8s types.

Standby

arc-systems on the on-prem cluster listens to the GitHub project repository for job requests matching its runner labels

arc-systems long-polls the corresponding project repository on GitHub for any job whose runs-on: labels match the labels this ARC runner set advertises.

On Update

From job-detected to a runner pod assigned and running the workflow

Direction matters. GitHub Actions doesn’t push jobs to ARC. The listener pod in arc-systems holds an outbound long-polling HTTPS connection to GitHub and pulls job requests as they’re queued. From GitHub’s side, ARC just looks like a self-hosted runner pool that happens to answer when called.

1. Listener detects a job — the AutoscalingListener pod in arc-systems sees a new workflow job whose runs-on: labels match this ARC runner set.
2. EphemeralRunner CR is created — an EphemeralRunner Custom Resource lands in the K8s API server representing the pending runner instance.
3. Controller reconciles — the EphemeralRunner is picked up by the ARC controller in arc-systems, which selects (or creates) a runner pod for it.
4. Pod runs the job — a new pod is spun up and assigned the job (or a warm pod from the pool is reused if one is configured and available), runs the workflow steps, and is torn down on completion.

Each pod is ephemeral: created for one job, destroyed after. That gives clean state per run (no leftover caches, no leaked secrets) but also means the first job after a scale-up pays the cold-start cost. A small warm pool sized to typical concurrency hides this for most teams.

In Action

Here’s ARC sitting in a full CI/CD pipeline — GitHub Actions claims an ARC runner, the runner builds an image, pushes it to Harbor, and ArgoCD picks the new tag up from there. The CI image-build/push and CD halves are covered in their own posts; this clip is the end-to-end view with ARC as the compute layer.

Live demo: the full CI/CD pipeline with ARC as the compute layer