[2026-08-06] Incident Thread: Actions Runner Controller

[github-actions[bot]]

❗ An incident has been declared:

Incident with Actions

Subscribe to this Discussion for updates on this incident. Please upvote or emoji react instead of commenting +1 on the Discussion to avoid overwhelming the thread. Any account guidance specific to this incident will be shared in thread and on the Incident Status Page.

__

🆕 Community Manager Update: August 7, 2026

We appreciate you all reaching out to report this issue.

Yesterday, August 6th, some Actions Runner Controller (ARC) runner pods became stuck in an idle state. Affected users can delete those pods using kubectl or redeploy their Actions Runner Controller application. ARC will automatically create replacement runners.

The next releases of Actions Runner and Actions Runner Controller will include an automatic recovery mechanism, preventing the need for these manual steps in the future.

This incident has been resolved and detailed root cause analysis will be shared as soon as it is available on our incident page.

If you continue to experience issues or have additional details you’d like to share, please comment below.

Thank you for your patience and understanding as we addressed this issue.

[github-actions[bot]]

Update

We are continuing to work on the issue affecting GitHub Actions.

Some workflow runs are still delayed or failing to complete, and some requests to the Actions API are returning errors.

Customers running migrations with GitHub Enterprise Importer may also see failures.

Engineers are actively working towards full recovery.

[rky-ke]

Any progress?

[tannerhelton]

Any updates?

[THC0220]

Welp..Looks like its a no work, relax night....

[desarrollo-yire]

Still waiting for a queued action for 4 hours now.

[vidasrenovadasassembleia-debug]

Alguém já está conseguindo usar?

[github-actions[bot]]

Update

We are continuing to work on an issue affecting multiple GitHub services.

Workflow runs are failing or delayed in starting, and some queued jobs may time out.

Copilot code review, Copilot coding agent, hosted runners, and migrations using GitHub Enterprise Importer might also affected.

Webhook deliveries may be delayed.

Engineers have applied a number of mitigations and are rolling out a further fix across all affected systems now.

[github-actions[bot]]

Update

We are continuing to work on an issue affecting multiple GitHub services.

Workflow runs are still failing or delayed in starting, and some queued jobs may time out.

Customers using self-hosted runners may see errors or rate limiting when runners register.

Copilot code review, Copilot coding agent, hosted runners, and migrations using GitHub Enterprise Importer may also be affected.

Webhook deliveries may be delayed.

Engineers have applied further mitigations and are continuing to work towards full recovery.

[QTI65536]

Still can't deploy on pages, getting following errors during build:

Internal server error. Correlation ID: 42b73796-513e-44d3-a675-add748f7d23f

The job was not acquired by Runner of type hosted even after multiple attempts

[github-actions[bot]]

Update

We are continuing to work on an issue affecting GitHub Actions. Webhook triggers remain throttled to aid recovery, so many push and pull request events are not triggering new workflow runs.

We identified runners being assigned jobs that are no longer valid and are deploying a change to address this issue. Both GitHub-hosted and self-hosted runners are affected.

Copilot code review, Copilot coding agent, and GitHub Pages may experience failures or delays. Migrations using GitHub Enterprise Importer have been paused to support mitigation efforts.

[xdebbie]

Wow it's been over 6hrs... please share a postmortem once it's all recovered, this must be interesting!

[github-actions[bot]]

Update

We have deployed fixes that address runners being assigned invalid jobs and are taking additional steps to clear the backlog of affected jobs. Job completion rates for running workflows have improved significantly, with success rates now at 99%. Global queues for hosted runner assignment are nearly burned down and concurrency queues for customers are being processed. Another change was deployed to accelerate processing the backlog of job requests.

We are gradually restoring throughput for webhook-triggered Actions workflows and monitoring system stability. We have deployed a fix for self-hosted runners that were not picking up jobs and are enabling it incrementally.

GitHub Pages, Copilot code review, and Copilot coding agent may still experience intermittent failures or delays. Migrations using GitHub Enterprise Importer remain paused.

We continue to monitor recovery across all affected services and will provide another update as conditions improve.

[github-actions[bot]]

Update

System-wide queues have been drained, and new jobs are being processed as expected. The fix for self-hosted runners not picking up jobs has been fully rolled out.

Webhook-triggered Actions workflows have been restored to full throughput. GitHub Pages, Copilot code review, and Copilot coding agent are showing recovery. Migrations using GitHub Enterprise Importer remain paused as a precaution.

We are monitoring all affected services for sustained recovery and will provide another update shortly.

[knipknap]

Hmmm, my actions are still stuck in "Queued", and any attempt to cancel them yields "Cancel failed"...

[codemeltDev]

Same behaviour when trying to cancel the stuck queued action as well.

[github-actions[bot]]

Update

We’re investigating reports that some Actions Runner Controller runners are taking longer than expected to recover. We’ll provide an update as our investigation progresses.

[desarrollo-yire]

Still waiting for a queued action for 4 hours now.

[codemeltDev]

Same here, have a queued action for 5 hours as well

[github-actions[bot]]

Incident Resolved

This incident has been resolved.

[knipknap]

Unfortunately not. Workflows still stuck in "Queued", and still "Failed to cancel workflow.".

[github-actions[bot]]

Update

During the incident, some Actions Runner Controller (ARC) runner pods became stuck in an idle state. Affected users can delete those pods using kubectl or redeploy their Actions Runner Controller application. ARC will automatically create replacement runners.

The next releases of Actions Runner and Actions Runner Controller will include an automatic recovery mechanism, preventing the need for these manual steps in the future.

Some workflow-triggering events, including push and pull request events, were not processed during the incident and cannot be replayed automatically. Customers may need to repeat the triggering action by pushing a new commit, updating the pull request, or manually re-running the workflow where applicable.

[codemeltDev]

And it is still stuck in queued

[ventsislav-georgiev]

The real solution was to patch the 'Outdated' phase:

List every wedged set:

CTX=<your-kube-context>;

kubectl --context "$CTX" get autoscalingrunnerset -n arc-runners \
  -o custom-columns=NAME:.metadata.name,PHASE:.status.phase,CUR:.status.currentRunners \
  | grep Outdated

The Outdated phase is sticky. Once set, every reconcile re-enters cleanUpResources (delete listener → delete ERS → delete the scale set on GitHub), returns "done", and never reaches the updateStatus(..., PhaseRunning) that would clear the phase. It loops there forever.

Restarting the controller does not help — we tried it first, no effect. The bad state lives on the object, not in the controller process. Same for deleting pods: there are no pods to delete.

The fix is to clear the phase:

for n in $(kubectl --context "$CTX" get autoscalingrunnerset -n arc-runners -o jsonpath='{range .items[?(@.status.phase=="Outdated")]}{.metadata.name}{"\n"}{end}'); do echo "patching $n"; kubectl --context "$CTX" patch autoscalingrunnerset -n arc-runners "$n" --subresource=status --type=merge -p '{"status":{"phase":"","currentRunners":0,"runningEphemeralRunners":0,"pendingEphemeralRunners":0}}'; done

[knipknap]

Any solution for us mere mortals who are just users on github.com without our own infrastructure?

[barradam]

The team is still looking at this behavior in ARC and whether there's anything more we can do to aid recovery from the server side.

[thboop]

Hey folks, we published an update on the Actions Runner Controller issue here.

[fedemendex]

Also hitting this. Repo fedemendex/patchatlas, workflow ci, run 31123101069 (commit 3debbaa8).

• Created 2026-08-06T17:25:55Z, run_attempt: 2
• Reports status: queued, conclusion: null, zero jobs (jobs: [])
• updated_at has been frozen at 2026-08-06T18:20:53Z since it got stuck — no change since, including after this incident was marked resolved

Every mutation endpoint gives a different, mutually contradictory reason for refusing to touch it:

• POST .../cancel → Cannot cancel a workflow run that is completed
• POST .../force-cancel → 409: Cannot cancel a workflow re-run that has not yet queued.
• POST .../rerun → run 31123101069 cannot be rerun; This workflow is already running
• DELETE .../runs/31123101069 → 403: Could not delete the workflow run

Same exact error sequence as #197015 from the May incident. Not blocking other workflows in the repo so far, just wanted it on record in case it needs a manual backend sweep like that one did. Happy to provide anything else that'd help trace it.

[reemasardana]

Update: The fix is taking longer than expected to ship. We are continuing to work on it and currently expect it to be available tomorrow. We’ll provide another update once it is deployed and ready to use.

[reemasardana]

You can now cancel stuck workflow runs using the GitHub CLI

For a run like this:

https://github.com/github/github/actions/runs/31123656087

Copy the run ID from the end of the URL, then run:

gh run cancel --repo {owner}/{repo} {run_id}

For example:

gh run cancel --repo github/github 31123656087

Please let us know if you encounter any other issues. We are continuing to work on allowing you to cancel these runs directly from the UI. I’ll share another update when that is ready.

[knipknap]

Sweet, it works! All is well again, Thank you for the support!

[barradam]

Thanks for the confirmation @knipknap !

[reemasardana]

You can now cancel stuck runs through the Workflow Runs UI

[AdithyaWijewickrama]

Any update??

[reemasardana]

You can cancel stuck runs using the GitHub CLI. I shared the guidance here.