Skip to content

ci(release): make downstream release triggers idempotent and race-free - #1005

Merged
ericcurtin merged 1 commit into
mainfrom
harden-release-idempotency
Jul 9, 2026
Merged

ci(release): make downstream release triggers idempotent and race-free#1005
ericcurtin merged 1 commit into
mainfrom
harden-release-idempotency

Conversation

@ilopezluna

Copy link
Copy Markdown
Contributor

Context

While releasing v1.2.6 the pipeline failed repeatedly across three different root causes, and each full-pipeline retry re-triggered steps that had already completed for the tag, producing new failures instead of resuming cleanly:

  1. Yesterdayverify-docker-ce failed because the prepare resolver's v* glob also matched the off-track vdmr-v0.1.1 tag. Already fixed by ci(release): keep image release on its own v* track #1004 (strict vX.Y.Z filtering).
  2. Packaging — a transient Ubuntu mirror flake (ubuntu2510 linux/arm/v7, mk-build-deps exit 29). Retry succeeded.
  3. Desktop403 denied re-pushing docker-model-cli-desktop-module:v1.2.6, because the tag was already published on an earlier attempt.

The recurring, structural problem: release.yml orchestrates non-idempotent cross-repo triggers, and both trigger jobs identify the downstream run by querying "the latest run" right after dispatch — which races with concurrent/stale runs (a job can watch, and pass on, a run it didn't trigger).

Changes

  • release-cli-desktop: log in to Docker Hub and skip the trigger when the desktop module tag already exists — a re-run for an already-published tag no longer 403s.
  • Both trigger jobs (release-cli-desktop, release-cli-docker-ce-trigger): capture the newest run ID before dispatch and wait for the first run with a strictly greater ID (run IDs are monotonic), instead of trusting gh run list --limit 1.
  • github-release: skip creation when the release already exists.

Fixes 2 and 3 originate in external repos (docker/packaging, docker/inference-engine-llama.cpp); this PR makes the orchestrator tolerant of retries so those transient/duplicate conditions no longer break the whole release.

🤖 Generated with Claude Code

The release pipeline chains non-idempotent cross-repo triggers, so a re-run
for an already-published tag failed in new ways instead of resuming cleanly:

- release-cli-desktop re-pushed docker-model-cli-desktop-module:<tag> and got
  "403 denied" because the tag was already published on a previous attempt.
- Both trigger jobs identified the downstream run by querying "the latest run"
  right after dispatch, which races with concurrent/stale runs — a job could
  watch (and pass on) a run that wasn't the one it triggered.

Changes:
- release-cli-desktop: log in to Docker Hub and skip the trigger when the
  desktop module tag already exists (idempotent re-run, no more 403).
- Both trigger jobs: capture the newest run ID before dispatch and wait for the
  first run with a strictly greater ID (run IDs are monotonic) instead of
  trusting "the latest run".
- github-release: skip creation when the release already exists.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@gemini-code-assist

Copy link
Copy Markdown
Contributor

Note

Gemini is unable to generate a review for this pull request due to the file types involved not being currently supported.

@sourcery-ai sourcery-ai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hey - I've found 1 issue, and left some high level feedback:

  • The trigger-and-wait logic for downstream workflows is now duplicated in two places; consider extracting this into a shared script or composite action so the monotonic ID handling and retry behavior stay consistent and easier to maintain.
  • In the new run ID polling loops, you might want to surface more context in the failure case (e.g., last observed BEFORE_ID/CANDIDATE or total wait duration) to make diagnosing races or misconfigurations easier when the run ID cannot be determined.
Prompt for AI Agents
Please address the comments from this code review:

## Overall Comments
- The trigger-and-wait logic for downstream workflows is now duplicated in two places; consider extracting this into a shared script or composite action so the monotonic ID handling and retry behavior stay consistent and easier to maintain.
- In the new run ID polling loops, you might want to surface more context in the failure case (e.g., last observed BEFORE_ID/CANDIDATE or total wait duration) to make diagnosing races or misconfigurations easier when the run ID cannot be determined.

## Individual Comments

### Comment 1
<location path=".github/workflows/release.yml" line_range="538-541" />
<code_context>
-            echo "⚠️ Could not extract run URL from gh output, querying latest run..."
-            sleep 5
-            RUN_ID=$(gh run list \
+          RUN_ID=""
+          for _ in $(seq 1 30); do
+            sleep 3
+            CANDIDATE=$(gh run list \
               --repo docker/inference-engine-llama.cpp \
               --workflow release-cli-dd.yml \
</code_context>
<issue_to_address>
**issue (bug_risk):** Guard against empty or non-numeric CANDIDATE values before numeric comparison

If `gh run list` fails or returns an unexpected payload, `CANDIDATE` may be empty or non-numeric. In that case `[ "$CANDIDATE" -gt "$BEFORE_ID" ]` will error (`integer expression expected`) and fail the job. Please add a guard that checks for non-empty, numeric values before the `-gt` comparison (e.g., skip the iteration when `CANDIDATE` is empty or not `[0-9]+`).
</issue_to_address>

Sourcery is free for open source - if you like our reviews please consider sharing them ✨
Help me be more useful! Please click 👍 or 👎 on each comment and I'll use the feedback to improve your reviews.

Comment on lines +538 to +541
RUN_ID=""
for _ in $(seq 1 30); do
sleep 3
CANDIDATE=$(gh run list \

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

issue (bug_risk): Guard against empty or non-numeric CANDIDATE values before numeric comparison

If gh run list fails or returns an unexpected payload, CANDIDATE may be empty or non-numeric. In that case [ "$CANDIDATE" -gt "$BEFORE_ID" ] will error (integer expression expected) and fail the job. Please add a guard that checks for non-empty, numeric values before the -gt comparison (e.g., skip the iteration when CANDIDATE is empty or not [0-9]+).

@ericcurtin
ericcurtin merged commit 2221f17 into main Jul 9, 2026
14 checks passed
@ericcurtin
ericcurtin deleted the harden-release-idempotency branch July 9, 2026 12:05
pull Bot pushed a commit to TheTechOddBug/model-runner that referenced this pull request Jul 10, 2026
Follow-up to docker#1005 (Sourcery review). The monotonic run-ID polling loops
compared `$CANDIDATE` numerically without validating it. If `gh run list`
fails transiently or returns an unexpected payload, CANDIDATE can be empty or
non-numeric and `[ "$CANDIDATE" -gt "$BEFORE_ID" ]` errors with "integer
expression expected" — the exact kind of flake this logic was meant to absorb.

- Tolerate transient `gh` failures mid-poll (`2>/dev/null || echo ""`) and skip
  any iteration where CANDIDATE is empty/non-numeric instead of erroring.
- Validate BEFORE_ID up front and fail with a clear message (before dispatch)
  if it can't be read.
- Include before/last-seen IDs and wait duration in the failure message to make
  race/misconfig diagnosis easier.

Applied to both the desktop (release-cli-dd.yml) and packaging
(release-model.yml) trigger jobs.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants