Builds timing out on CI #38964
Description
We're seeing multiple builds canceled due to "timeouts" when they're not really timeouts. This seems like a bug in AzDo that it is considering the "queue" time and time spent waiting for an agent as part of the timeout time.
##[error]The job running on agent NetCorePublic-Pool 172 ran longer than the maximum time of 60 minutes. For more information, see https://go.microsoft.com/fwlink/?linkid=2077134
Pool: NetCorePublic-Pool
Agent: NetCorePublic-Pool 172
Started: Today at 1:54 PM
Duration: 15m 2s
As seen, the duration was really 15m 2s which is not over 60 mins.
Runtime build
| Build | Kind | Timeline Record |
|---|---|---|
| 721340 | PR #38938 | CoreCLR Product Build Windows_NT x64 release |
| 721340 | PR #38938 | CoreCLR Product Build Windows_NT arm64 checked |
| 721340 | PR #38938 | CoreCLR Product Build Windows_NT x86 release |
| 721425 | PR #38952 | CoreCLR Product Build Windows_NT x64 release |
| 721425 | PR #38952 | CoreCLR Product Build Windows_NT arm release |
| 721425 | PR #38952 | CoreCLR Product Build Windows_NT arm64 release |
| 721425 | PR #38952 | CoreCLR Product Build Windows_NT x86 release |
| 721535 | PR #38765 | CoreCLR Product Build OSX x64 checked |
| 721294 | PR #38948 | Mono Product Build Linux x64 debug |
| 721294 | PR #38948 | CoreCLR Product Build Linux_musl x64 release |
| 721294 | PR #38948 | CoreCLR Product Build Linux_musl x64 checked |
| 721294 | PR #38948 | CoreCLR Product Build Linux x64 release |
| 721294 | PR #38948 | CoreCLR Product Build Linux arm release |
| 721294 | PR #38948 | Formatting Linux x64 |
| 721294 | PR #38948 | CoreCLR Product Build FreeBSD x64 release |
| 721294 | PR #38948 | CoreCLR Product Build Linux arm64 release |
| 721294 | PR #38948 | CoreCLR Product Build Linux_musl arm64 release |
| 721294 | PR #38948 | Build Linux arm Debug AllSubsets_Mono |
| 721294 | PR #38948 | Build Linux x64 Debug AllSubsets_Mono_LLVMJIT |
| 721294 | PR #38948 | Mono Product Build Linux arm64 debug |
| 721294 | PR #38948 | Mono Product Build Linux x64 release |
| 721294 | PR #38948 | Build Linux_musl x64 Release AllSubsets_Mono |
| 721294 | PR #38948 | Mono Product Build Linux arm64 release |
| 721294 | PR #38948 | CoreCLR Product Build Linux arm64 checked |
| 721294 | PR #38948 | CoreCLR Product Build Linux_musl arm64 checked |
| 721294 | PR #38948 | CoreCLR Product Build Linux x64 checked |
| 721294 | PR #38948 | CoreCLR Product Build Linux arm checked |
| 721294 | PR #38948 | Build Linux x64 Release AllSubsets_Mono_LLVMAOT |
| 721099 | PR #38707 | Mono Product Build Linux x64 debug |
| 721099 | PR #38707 | CoreCLR Product Build Linux_musl x64 release |
| 721099 | PR #38707 | CoreCLR Product Build Linux x64 release |
| 721099 | PR #38707 | CoreCLR Product Build Linux arm release |
| 721099 | PR #38707 | CoreCLR Product Build FreeBSD x64 release |
| 721099 | PR #38707 | CoreCLR Product Build Linux arm64 release |
| 721099 | PR #38707 | CoreCLR Product Build Linux_musl arm64 release |
| 721099 | PR #38707 | Build Linux arm Debug AllSubsets_Mono |
| 721099 | PR #38707 | Build Linux x64 Release AllSubsets_Mono_LLVMAOT |
| 721099 | PR #38707 | Build Linux x64 Debug AllSubsets_Mono_LLVMJIT |
| 721099 | PR #38707 | Mono Product Build Linux arm64 debug |
| 721099 | PR #38707 | Mono Product Build Linux x64 release |
| 721099 | PR #38707 | Build Linux_musl x64 Release AllSubsets_Mono |
| 721099 | PR #38707 | Mono Product Build Linux arm64 release |
| 721062 | PR #38935 | CoreCLR Product Build Linux_musl x64 release |
| 721062 | PR #38935 | CoreCLR Product Build Linux x64 release |
| 721062 | PR #38935 | Build Linux x64 Release AllSubsets_Mono_LLVMAOT |
| 721160 | PR #37971 | CoreCLR Product Build Windows_NT x86 checked |
| 721161 | PR #38884 | CoreCLR Product Build Windows_NT x64 release |
| 721161 | PR #38884 | CoreCLR Product Build Windows_NT arm release |
| 721161 | PR #38884 | CoreCLR Product Build Windows_NT x86 release |
| 721161 | PR #38884 | CoreCLR Product Build Windows_NT arm64 release |
| 721009 | PR #38815 | Build Linux arm Debug AllSubsets_Mono |
| 721042 | PR #38933 | CoreCLR Product Build Linux arm release |
| 720484 | PR #38893 | CoreCLR Product Build OSX x64 release |
| 719881 | PR #38728 | CoreCLR Product Build Windows_NT x86 checked |
Evaluated 100 builds
Impacted 12 bulids
Impacted 54 jobs
This is impacting around 12% of builds of runtime and 12% of runtime-live-build.
Runtime-live-build
| Build | Kind | Timeline Record |
|---|---|---|
| 721591 | PR #38547 | Build OSX x64 release Runtime_Debug |
| 721451 | PR #38894 | Build OSX x64 release Runtime_Debug |
| 721456 | PR #38616 | Build Windows_NT x64 debug Mono_Libraries |
| 721456 | PR #38616 | Build Windows_NT x86 release Runtime_Debug |
| 721295 | PR #38948 | Build Linux x64 debug RuntimeFlavor_Mono |
| 721295 | PR #38948 | Build Linux x64 debug Runtime_Release |
| 721295 | PR #38948 | Build Linux x64 release SourceBuild |
| 721264 | PR #38547 | Build Linux x64 debug Runtime_Release |
| 721264 | PR #38547 | Build Linux x64 release SourceBuild |
| 721270 | PR #38946 | Build Linux x64 debug RuntimeFlavor_Mono |
| 721270 | PR #38946 | Build Linux x64 debug Runtime_Release |
| 721241 | PR #32520 | Build Windows_NT x64 debug Mono_Libraries |
| 721063 | PR #38935 | Build Windows_NT x64 debug Mono_Libraries |
| 721063 | PR #38935 | Build Linux x64 debug RuntimeFlavor_Mono |
| 721063 | PR #38935 | Build Linux x64 debug Runtime_Release |
| 721063 | PR #38935 | Build Linux x64 release SourceBuild |
| 721100 | PR #38707 | Build Linux x64 debug RuntimeFlavor_Mono |
| 721100 | PR #38707 | Build Linux x64 debug Runtime_Release |
| 721043 | PR #38933 | Build Linux x64 release SourceBuild |
| 721039 | PR #37159 | Build Linux x64 debug Runtime_Release |
| 720680 | PR #38917 | Build OSX x64 release Runtime_Debug |
Evaluated 100 builds
Impacted 12 bulids
Impacted 21 jobs
@MattGal is looking into this and opening a ticket on AzDo
cc: @dotnet/runtime-infrastructure
