Your Runner Is Online, Idle, and Silently Ignoring Your Jobs
~4 min read
DevOpsCI/CDgithub-actionsself-hosted-runnersci-cdtroubleshootingworkflow-dispatchdevopssre
Back openDesk Edu for a sovereign, open-source education — every vote counts.
Vote nowSave products you love by clicking the heart icon.
Traditional CI/CD pipelines are evolving into AI/CD systems where agentic workflows handle testing, deployment, rollback, and optimization autonomously. This article explores the architecture, benefits, and implementation patterns for bringing intelligence to your delivery pipelines.
The most confusing state a self-hosted runner can be in is this: it reports
online, it is busy=false, its labels match your workflow exactly, and
your queued job still sits there for two days. I spent an afternoon on
exactly this. The workflow YAML was correct. The runner was healthy. The
job — it turned out — had never really existed.
This is the diagnostic path that finally unblocked the deploy, and the one-line structural fix that means it can't silently happen again.
When I looked at the stuck run through the CLI, the contradiction was the first real clue:
$ gh run view <run> --json status,jobs
queued | # run reads "queued", but...
jobs was an empty list. GitHub had recorded the workflow run, but never
materialized a job for it. There was nothing for the runner to pick up —
no test step, no dispatch, no work item at all. The run read as queued,
yet gh run cancel insisted it was already complete, and gh run rerun
insisted it was already running. It was stuck in a limbo that neither
cancel nor re-run could touch.
That's the signature of a ghost queue entry: the queue metadata broke, not your machine. Restarting the runner will not help, because the runner was never the problem.
Before blaming GitHub, rule out the runner — quickly, because it's the most common cause and it's often the actual one:
Confirm identity and labels. The runner must be registered for the
repository and carry every label your runs-on requires.
gh api repos/$ORG/$REPO/actions/runners \
--jq '.runners[] | "\(.name) \(.status) busy=\(.busy) labels=\([.labels[].name]|join(","))"'
Read the diagnostic log, not the UI. A runner that appears "online"
in the GitHub UI can still have a wedged job-poll loop. The decision
belongs to _diag/Runner_*.log:
Listening for Jobs
Successfully created session with migrated settings
JobDispatcher] Set runner/worker IPC timeout to 30 seconds
If it says Listening for Jobs and stops there with no errors, the runner is genuinely waiting for work.
Restart the systemd unit anyway. A stuck poll loop is the single most common cause of "online but idle," and the restart is cheap and safe:
sudo systemctl restart actions.runner.<org>-<repo>.<name>.service
On the day this incident happened, the other trouble we had was exactly this wedge — and the restart cleared a zombie job in seconds. But for the ghost queue run, the runner stayed idle, because no job existed to hand it.
The restarted runner active, healthy, busy=false, listening — and the
run still queued with zero jobs. That's when the diagnosis is done and the
fault is on GitHub's side.
The structural problem is that the workflow was only triggerable by a push:
on:
push:
branches: [main]
A ghost queue entry can't be re-run (gh run rerun → "already running"),
can't be cancelled (gh run cancel → "already complete"), and can't be
dispatched. The only way to get a fresh materialized job was a new push to
main — which meant some synthetic commit just to nudge the pipeline. That is
an unacceptable way to run a deploy pipeline.
The structural fix is one block and it removes the entire class of failure — if a run ever lands in limbo, you can force a fresh run directly:
on:
push:
branches: [main]
workflow_dispatch: {} # manual re-trigger when a run gets stuck
That's it. Adding workflow_dispatch to your deploy/build workflow turns a
one-way door into a recoverable one. When the next run gets wedged, instead
of pushing a meaningless commit you run:
gh workflow run build-and-push.yml --ref main
In our case, the push of that one legitimate workflow_dispatch line also
created the fresh, healthy run — which the (already restarted) runner picked
up immediately and took straight through to a green deploy, including the
feature that had been bricked behind the ghost queue.
_diag log says otherwise. A
healthy, listening runner with matching labels shifts the burden of proof
to the GitHub queue.jobs is empty but status reads queued, no restart will help.workflow_dispatch. The cost is a few bytes of YAML; the payoff is that
a stuck deploy is always one command away from being re-triggered,
instead of one forced commit away.Self-hosted runners get most of the blame for pipeline stalls. Most of the time it's fair. But when the runner is genuinely healthy and the job never existed, the fix isn't more machine care — it's giving yourself a manual reset button.