[ci] Make Android SDK test setup downloads more resilient#12001
[ci] Make Android SDK test setup downloads more resilient#12001simonrozsival wants to merge 1 commit into
Conversation
Add a conditional hook to the Android archive cache template and use it in test-environment setup for macOS, where the OpenJDK/Android SDK license step has been exhausting retries on large Google SDK archives. Also retry the OpenJDK/Android SDK license setup task so a fully exhausted per-file download attempt can be recovered by a fresh step attempt while preserving already-downloaded archives. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
There was a problem hiding this comment.
Pull request overview
This PR hardens the Azure Pipelines Android test-environment setup against transient download failures (notably macOS ResponseEnded when fetching large Android SDK/OpenJDK archives) by restoring an archive cache earlier and adding an extra end-to-end retry around the SDK setup step.
Changes:
- Invoke the Android toolchain archive cache during macOS test-environment setup (before OpenJDK/SDK downloads) to reduce reliance on live downloads.
- Make
cache-android-archives.yamlconditionally runnable via a newconditiontemplate parameter. - Retry the “install OpenJDK and accept Android SDK licenses” step (and workload pack extraction) to allow recovery after exhausting per-file download retries.
Show a summary per file
| File | Description |
|---|---|
| build-tools/automation/yaml-templates/setup-test-environment-steps.yaml | Restores Android archive cache on macOS before SDK setup; adds task retries for resilient setup. |
| build-tools/automation/yaml-templates/cache-android-archives.yaml | Adds a condition parameter and applies it to the variable-setup steps and Cache@2 step. |
Copilot's findings
- Files reviewed: 2/2 changed files
- Comments generated: 0
|
/review |
|
✅ Android PR Reviewer completed successfully! |
![github-actions[bot]](https://avatars.githubusercontent.com/in/15368?s=60&v=4){kind=small}
There was a problem hiding this comment.
🤖 ✅ LGTM — Well-reasoned, conservative CI hardening. I formed an independent assessment of the mechanics before reading the description, and they hold up:
- The cache actually takes effect on this path. The template restores/saves at
$(ANDROID_ARCHIVES_DIR)=$HOME/android-archives, which matches the MSBuild defaultAndroidToolchainCacheDirectory($(HOME)\android-archivesinConfiguration.props). The followingandroidsdk.csprojrun downloads into that same directory with no explicit override — identical to howbuild-macos-steps.yaml/build-linux-steps.yamlalready consume this template. ✔ - Template expansion is correct. With the caller passing
condition: and(succeeded(), eq(agent.os, 'Darwin')), the Windowspwshvariable step resolves toand(...Darwin..., eq(Agent.OS,'Windows_NT'))= false (correctly skipped on macOS), while the non-Windows variable step and theCache@2task correctly run. ✔ - Scope is correctly limited to macOS. On Linux/Windows test setup the condition is false, so the cache is a no-op there — consistent with the stated deferral of Linux caching (
#11837). ✔ retryCountOnTaskFailure: 2is forwarded correctly byrun-dotnet-preview.yaml(param exists, default0), and the retried work is idempotent (license acceptance + hash-verified downloads; already-fetched archives skip via cache/stamps). It matches the existingextract workload packsretry two steps below. ✔- CI is green. All 42
dotnet-androidchecks (including the macOS test jobs that now exercise this path) pluslicense/clapassed.mergeable_state: blockedis due to the pending required review fromjonathanpeppers, not CI.
Two minor 💡 suggestions left inline (document the new condition parameter; a benign succeeded() behavior change for existing build callers). Neither blocks merge.
Issues: ❌ 0 ·
Generated by Android PR Reviewer for #12001 · 181.6 AIC · ⌖ 20.2 AIC · ⊞ 6.8K
Comment /review to run again
|
|
||
| parameters: | ||
| xaSourcePath: $(System.DefaultWorkingDirectory)/android | ||
| condition: succeeded() |
There was a problem hiding this comment.
🤖 💡 Documentation — The new condition parameter gates every step in this shared template (both prepare ... cache variables steps and the Cache@2 task), but the otherwise-thorough header comment doesn't mention it. A one-line note that callers can pass a condition to scope the whole cache to specific agents — as setup-test-environment-steps.yaml now does for Darwin — would help future maintainers understand why the parameter exists.
Rule: Document shared-template parameters
| - script: echo "##vso[task.setvariable variable=ANDROID_ARCHIVES_DIR]$HOME/android-archives" | ||
| displayName: prepare android-archives cache variables | ||
| condition: ne(variables['Agent.OS'], 'Windows_NT') | ||
| condition: and(${{ parameters.condition }}, ne(variables['Agent.OS'], 'Windows_NT')) |
There was a problem hiding this comment.
🤖 💡 Patterns — For existing callers that don't pass condition (the three build-*-steps.yaml templates), this step's guard changes from ne(Agent.OS, 'Windows_NT') to and(succeeded(), ne(Agent.OS, 'Windows_NT')) — it now also requires prior steps to have succeeded (same for the Windows pwsh step below). This is almost certainly harmless — setting the cache-dir variable after an earlier failure has no value, and it aligns these steps with the Cache@2 task's implicit succeeded() default — but it's a behavior change for build jobs, so worth a quick confirm that it's intended.
Rule: Watch shared-template edits for side effects on existing callers
simonrozsival
added
the
ready-to-review
Summary
This is another incremental hardening PR for the recurring
install OpenJDK and accept Android SDK licensesCI failure where large Android SDK archives fromdl.google.comfail with:The specific recent failure already had the newer retry/backoff behavior: the log shows retries waiting 30s, 60s, 90s, and 120s before ultimately exhausting
x86_64-29_r08-darwin.zip. So this PR does not replace the existing download retry work; it builds on it and covers the next failure mode.Prior work this builds on
src/androidsdk/androidsdk.targets, which is what this setup step builds.android-archivespipeline cache for build jobs and bumped the built-inDownloadFileretries.DownloadFileWithRetrywrapper becauseResponseEndedis a top-levelHttpIOExceptionthat MSBuild's built-inDownloadFileretry logic does not catch.DownloadOneFileWithRetryskip cleanly on no-op builds using per-file stamps, which makes whole-step retries less wasteful once some archives already succeeded.Remaining gap
Those PRs made individual downloads much more resilient, but the failing path is the test environment setup template, not the main build template:
build-{linux,macos,windows}-steps.yamlalready usescache-android-archives.yaml.setup-test-environment-steps.yamldid not use that cache before runningsrc/androidsdk/androidsdk.csproj.install OpenJDK and accept Android SDK licensesrun-dotnet-preview.yamlinvocation also hadretryCountOnTaskFailure: 0through the template default.That means a macOS test setup job could still cold-download several large archives concurrently and fail the entire job if one archive exhausted the per-file retry/backoff loop.
Change
This PR hardens that remaining test-setup path by:
conditionparameter tocache-android-archives.yaml, preserving the existing default behavior for build jobssetup-test-environment-steps.yamlfor macOS test setup before SDK downloads runretryCountOnTaskFailure: 2on theinstall OpenJDK and accept Android SDK licensesstepThe cache is intentionally limited to macOS test setup for now. Linux archive caching already has a separate disk-pressure concern tracked by #11837, so this PR avoids expanding Linux cache usage while still targeting the observed macOS
ResponseEndedfailure.Why step retry helps
The per-file retry wrapper is still the first line of defense. The task-level retry is a second line of defense for the case where a specific archive exhausts all per-file attempts.
On a second task attempt, successfully downloaded archives should be reused/skipped by the existing cache/stamp logic, so the retry mostly focuses on the archive that failed rather than restarting all work from scratch.
Validation
git diff --check