[ci] Linux > Build fails on undersized (29 GB disk) NetCore-Public agents (out of disk in toolchain-cache step)

[simonrozsival]

Summary

Linux > Build jobs on the public dotnet-android pipeline (dnceng-public, def 333) are failing intermittently — and the failure is 100% determined by which agent VM the job lands on, not by anything in our build.

The NetCore-Public pool is handing out two very different VM SKUs for the build.Ubuntu.2204.amd64.open image:

Agent OS disk (/)	Build result
248 GB (33 GB used → 13%)	✅ pass
29 GB (28 GB used → 94%+)	❌ fail

Our Linux build produces ~20 GB of output (make jenkins + make create-nupkgs), taking the agent from ~8 GB used to ~28 GB used. That fits comfortably on a 248 GB agent but overflows a 29 GB OS disk. The job then dies in the Post-job: cache Android toolchain archives step when tar can't finish writing the cache archive:

##[warning]Free disk space on / is lower than 5%; Currently used: 96.79%
tar: <hash>_archive.tar: Wrote only 6144 of 10240 bytes
tar: Error is not recoverable: exiting now
##[error]Process returned non-zero exit code: 2

Evidence

Correlation is perfect across recent builds (all failures on 29 GB agents, all successes on 248 GB agents):

Build	Result	Agent	/ disk
1489447	failed	NetCore-Public 284	29 GB (94% before cache step)
1489234	failed	NetCore-Public 41	29 GB
1489023	failed	NetCore-Public 326	29 GB
1489103	failed	NetCore-Public 165	29 GB
1488597	failed	NetCore-Public 577	29 GB
1488717	failed	NetCore-Public 14	29 GB
1488072	succeeded	NetCore-Public 794	248 GB (13%)
1488146	succeeded	NetCore-Public 240	248 GB
1487951	succeeded	NetCore-Public 45	248 GB
1488035	succeeded	NetCore-Public 375	248 GB

Example failing build: https://dev.azure.com/dnceng-public/public/_build/results?buildId=1489447&view=logs&jobId=a1070d6b-7539-53cb-46be-b3395553b4a7

This is NOT an artifact-size regression

The identical build uses only 33 GB (13%) on the normal 248 GB agents — nothing on our side grew. What changed is the pool now (sometimes) provisions 29 GB-disk VMs for build.Ubuntu.2204.amd64.open.

Pipeline config requesting the image:

• build-tools/automation/azure-pipelines-public.yaml:56-57 (LinuxPoolImageNetCorePublic: build.Ubuntu.2204.amd64.open)
• consumed by the Linux Build pool demand ImageOverride -equals $(LinuxPoolImageNetCorePublic) (e.g. line 133)

Why we can't fix this purely on our side

Azure DevOps agent demands only match named capabilities; there is no built-in "free disk ≥ N GB" demand, and on hosted dnceng pools we don't control agent capabilities. OS-disk size is a fixed property of the VM SKU behind the image. So a real fix needs dnceng infra: either stop backing build.Ubuntu.2204.amd64.open with 29 GB-disk SKUs, or provide an image variant with a larger OS disk we can target via ImageOverride.

Options to discuss

1. [Infra, preferred] File dnceng First Responders to fix/replace the 29 GB-disk SKUs (or point us at a larger-disk image variant).
2. [Repo stopgap] Make the Post-job: cache Android toolchain archives (Cache@2 save) non-fatal (continueOnError: true) so a full-disk cache write doesn't fail the whole build. Doesn't fix builds that overflow before the cache step.
3. [Repo stopgap] Free disk mid-build (e.g. delete bin/Build* intermediates / other large trees) before the cache step to claw back a few GB.
4. [Repo] Reduce the ~20 GB build footprint if feasible.

[jonathanpeppers]

Closing because the undersized-agent failure has not recurred in the recent pipeline window and the underlying condition is owned by dnceng pool infrastructure rather than dotnet/android product code.

A fresh Azure DevOps scan found no out-of-disk, low-disk, or toolchain-cache disk failure signature in the latest 100 completed dotnet-android builds (August 3–12, 2026), and none in the latest 22 completed main builds (July 16–August 12, 2026). The last evidence recorded here was from late June.

If build.Ubuntu.2204.amd64.open begins assigning 29 GB OS disks again, please reopen with current build IDs and agent/disk-size evidence so it can be escalated to dnceng infrastructure.