[build] Provision .NET SDK via standard scripts, drop xaprepare's installer#11636
Open
jonathanpeppers wants to merge 8 commits into
Open
[build] Provision .NET SDK via standard scripts, drop xaprepare's installer#11636jonathanpeppers wants to merge 8 commits into
jonathanpeppers wants to merge 8 commits into
Conversation
Replace xaprepare's bespoke `dotnet-install` invocation with Arcade's
standard `eng/common/tools.{sh,ps1}` bootstrap, matching dotnet/sdk,
dotnet/runtime, and dotnet/aspnetcore.
* `global.json`: pin `tools.dotnet` so Arcade's `InitializeDotNetCli`
knows which SDK to install. darc auto-updates this whenever
`Microsoft.NET.Sdk` flows from dotnet/dotnet via the existing
Maestro subscription.
* `eng/install-dotnet.{sh,ps1}`: thin wrappers that set
`DOTNET_INSTALL_DIR=DOTNET_GLOBAL_INSTALL_DIR=bin/$(Configuration)/dotnet/`
(preserving the existing install location) and call
`InitializeDotNetCli` from `eng/common/tools.{sh,ps1}`.
* `Makefile`: `prepare` now depends on a new `install-dotnet` target
that runs `./eng/install-dotnet.sh` first.
* `build-tools/scripts/PrepareWindows.targets`: add an
`_InstallDotNet` target that invokes `eng/install-dotnet.ps1`
before `_BuildXAPrepare`, so `dotnet msbuild Xamarin.Android.sln
-t:Prepare` (used on Windows CI) is self-bootstrapping.
* `Step_InstallDotNetPreview.cs` is deleted and replaced by
`Step_PrepareDotNetWorkloads.cs`. The new step assumes the SDK
is already installed at `bin/$(Configuration)/dotnet/` and only
performs the Android-specific workload prep:
* Cleans stale Mono Android runtime/workload NuGet directories.
* Restores `package-download.proj` (Mono runtime packs +
Mono/Emscripten workload manifest packages).
* Copies the workload manifests into the local SDK's
`sdk-manifests/`.
* Removes obsolete configuration:
* `Configurables.Urls.DotNetInstallScript` (Unix and Windows)
* `--dotnet-sdk-archive` xaprepare option and its
`Context.LocalDotNetSdkArchive` plumbing
* `DownloadDotNetInstallScript`, `GetInstallationScriptArgs`,
`InstallDotNetAsync`, `InstallDotNetFromLocalArchiveAsync`
methods (~150 lines of bespoke install logic).
The SDK install location stays at `bin/$(Configuration)/dotnet/`,
so `dotnet-local.{cmd,sh}` and other consumers continue to work
without changes. CI's `use-dot-net.yaml` is unchanged: it still
provisions a system .NET to bootstrap xaprepare; the pinned preview
SDK install simply moves from xaprepare to Arcade.
Verified locally on Windows: `dotnet msbuild Xamarin.Android.sln
-t:Prepare` after `git clean -xdf bin/Debug/dotnet/` installs the
pinned 11.0.100-preview.5.26268.112 SDK and copies the Mono +
Emscripten workload manifests into `sdk-manifests/`.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
darc does not auto-update global.json:tools.dotnet when Microsoft.NET.Sdk
flows from dotnet/dotnet (verified in arcade-services
DependencyFileManager.cs: only Microsoft.DotNet.Arcade.Sdk, the
*.SharedFramework.Sdk family, Microsoft.DotNet.CMake.Sdk,
Microsoft.NET.Sdk.IL, and the literal name "dotnet" are special-cased).
Pinning the SDK version in global.json would have permanently drifted
from the auto-flowed eng/Versions.props value. Read the version directly
from eng/Versions.props instead, making it the single source of truth.
eng/install-dotnet.{sh,ps1} now download Microsoft's official
dotnet-install.{sh,ps1} from
https://builds.dotnet.microsoft.com/dotnet/scripts/v1/ (cached under
bin/$Configuration/dotnet/) and invoke it with the version parsed from
eng/Versions.props:MicrosoftNETSdkPackageVersion. This bypasses Arcade's
eng/common/tools.{sh,ps1} (which strict-mode-reads $GlobalJson.tools)
and lets us drop the tools.dotnet pin from global.json entirely.
Verified on Windows:
- cold install: ~12s
- warm re-run: ~2.5s (idempotent fast path)
- full Prepare: ~88s
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
CI failed with "Permission denied" when `make jenkins` ran `./eng/install-dotnet.sh` because the file was committed as 100644. The file from `make prepare` is invoked directly (not via `bash`), so it needs the executable bit set. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Reverts the executable-bit change from 2645bdb. Windows clones with core.filemode=false would have shown spurious mode changes when editing the file; running it via `bash ./eng/install-dotnet.sh` from the Makefile sidesteps the bit entirely. Same trick for the cached dotnet-install.sh we download under bin/. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
This finishes the second half of the SDK provisioning audit started in PR #11636. The first half moved the .NET SDK install into Microsoft's official `dotnet-install.{sh,ps1}` scripts driven by `eng/install-dotnet`. This commit replaces the leftover xaprepare logic that prepared Android-specific .NET workloads against that SDK. What `Step_PrepareDotNetWorkloads` did (now deleted): * Restored `package-download.proj` to pull down the Mono Android runtime packs and the Mono/Emscripten workload manifest packages. * Copied the workload manifests from the NuGet package cache into the local SDK's `sdk-manifests/` folder. What `src/workloads/workloads.csproj` does (single MSBuild project, no C#, no scenarios): * Carries the same `<PackageDownload>` items that lived in `package-download.proj` (run via NuGet's auto-restore). * Has a `_CopyWorkloadManifests` target that runs `AfterTargets="Build"` and copies each `microsoft.net.workload.{mono,emscripten}.<flavor>` manifest's `data/` into the local SDK's `sdk-manifests/<band>/microsoft.net.workload.<flavor>.<dotnet>/<ver>/`. Per @jonathanpeppers' suggestion in #11636 (comment 3403797084): "move it to like `src/workloads/workloads.csproj` and that project is built first." Wiring: * `Makefile prepare:` now runs `dotnet build src/workloads/workloads.csproj` after the BootstrapTasks build, before `PrepareJavaInterop`. * `build-tools/scripts/PrepareWindows.targets`'s `Prepare` target adds an `<MSBuild Projects=".../workloads.csproj" />` invocation in the same spot. * `build-tools/automation/yaml-templates/setup-test-environment-steps.yaml` no longer invokes xaprepare. Test agents now run `eng/install-dotnet.{sh,ps1}` (provisions the SDK at `bin/$Config/dotnet/`) followed by `dotnet build src/workloads/workloads.csproj` (provisions the workloads against that SDK). This fixes the AndroidTestDependencies CI failure introduced when the prior commit removed `Step_InstallDotNetPreview`'s SDK download. Cleanup: * `Step_PrepareDotNetWorkloads.cs` and `package-download.proj` deleted. * `Scenario_Standard` and `Scenario_AndroidTestDependencies` no longer add `Step_PrepareDotNetWorkloads`. * The `xaprepareScenario` parameter (and the now-unused `run-xaprepare.yaml` template) are removed across all CI YAMLs. * Dead `Configurables.MicrosoftNETWorkload*Dir` properties are removed. Verified locally on Windows: * `bin/Debug/dotnet/sdk-manifests/<band>/microsoft.net.workload.{mono.toolchain,emscripten}.{net6..net10,current}/<ver>/WorkloadManifest.json` is populated after `dotnet build src/workloads/workloads.csproj` (12 manifests total). * Re-running is idempotent (~0.5s warm; `Copy SkipUnchangedFiles="true"`). Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Contributor
There was a problem hiding this comment.
Pull request overview
This PR migrates dotnet/android’s .NET SDK provisioning from xaprepare’s custom C# installer to the standard dotnet-install.{sh,ps1} flow, keeping the install location at bin/$Configuration/dotnet/ and moving Android-specific workload prep into a standalone MSBuild project.
Changes:
- Add
eng/install-dotnet.{sh,ps1}wrappers that read the pinned SDK version fromeng/Versions.props, downloaddotnet-install.{sh,ps1}, and install intobin/$Configuration/dotnet. - Wire the new install/workload-prep flow into
Makefile, WindowsPrepareWindows.targets, and CI templates; remove xaprepare’s SDK-install step and related plumbing. - Introduce
src/workloads/workloads.csprojto restore required runtime packs + workload manifest packages and copy manifests into the locally installed SDK.
Reviewed changes
Copilot reviewed 24 out of 24 changed files in this pull request and generated 2 comments.
Show a summary per file
| File | Description |
|---|---|
| src/Xamarin.Installer.Build.Tasks/README.md | Updates developer instructions to use eng/install-dotnet.* + build workloads project. |
| src/workloads/workloads.csproj | New MSBuild project to restore runtime packs/manifests and copy manifests into the local SDK. |
| Makefile | Adds install-dotnet prerequisite and runs workloads provisioning during prepare. |
| eng/install-dotnet.sh | New Unix bootstrap script to install pinned SDK into bin/$Configuration/dotnet. |
| eng/install-dotnet.ps1 | New Windows bootstrap script to install pinned SDK into bin\$Configuration\dotnet. |
| build-tools/xaprepare/xaprepare/Steps/Step_InstallDotNetPreview.cs | Deletes the bespoke xaprepare SDK installer step. |
| build-tools/xaprepare/xaprepare/Scenarios/Scenario_Standard.cs | Removes SDK install step from the standard xaprepare scenario. |
| build-tools/xaprepare/xaprepare/Scenarios/Scenario_AndroidTestDependencies.cs | Removes SDK install step from Android test dependency scenario. |
| build-tools/xaprepare/xaprepare/package-download.proj | Deletes the old runtime-pack restore project used by xaprepare. |
| build-tools/xaprepare/xaprepare/Main.cs | Removes --dotnet-sdk-archive option plumbing. |
| build-tools/xaprepare/xaprepare/ConfigAndData/Configurables.Windows.cs | Removes DotNet install script URL configurable (no longer needed). |
| build-tools/xaprepare/xaprepare/ConfigAndData/Configurables.Unix.cs | Removes DotNet install script URL configurable (no longer needed). |
| build-tools/xaprepare/xaprepare/ConfigAndData/Configurables.cs | Removes workload manifest path helpers tied to the deleted step. |
| build-tools/xaprepare/xaprepare/Application/Context.cs | Removes LocalDotNetSdkArchive property. |
| build-tools/scripts/PrepareWindows.targets | Ensures SDK install runs before building xaprepare; adds workloads provisioning to Prepare. |
| build-tools/automation/yaml-templates/stage-package-tests.yaml | Removes xaprepare scenario parameter usage from test setup. |
| build-tools/automation/yaml-templates/stage-msbuild-emulator-tests.yaml | Removes xaprepare scenario parameter usage from test setup. |
| build-tools/automation/yaml-templates/setup-test-environment.yaml | Removes xaprepareScenario parameter pass-through. |
| build-tools/automation/yaml-templates/setup-test-environment-steps.yaml | Replaces xaprepare invocation with eng/install-dotnet.* + workloads provisioning. |
| build-tools/automation/yaml-templates/setup-test-environment-public.yaml | Removes xaprepareScenario parameter pass-through. |
| build-tools/automation/yaml-templates/run-xaprepare.yaml | Deletes the shared pipeline template that ran xaprepare. |
| build-tools/automation/yaml-templates/run-emulator-tests.yaml | Removes xaprepare scenario parameter usage from test setup. |
| build-tools/automation/azure-pipelines-public.yaml | Removes xaprepare scenario parameter usage from test environment setup. |
| build-tools/automation/azure-pipelines-nightly.yaml | Removes xaprepare scenario parameter usage from test environment setup. |
* Makefile install-dotnet: pass CONFIGURATION through to install-dotnet.sh so 'make CONFIGURATION=Release prepare' installs the SDK under bin/Release/dotnet to match the rest of the build. * eng/install-dotnet.ps1: null-check the result of SelectSingleNode before dereferencing .InnerText so the script fails with the intended friendly error message if <MicrosoftNETSdkPackageVersion> is ever removed from eng/Versions.props. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
The previous commit accidentally captured local submodule pointer changes for external/Java.Interop and external/xamarin-android-tools that have nothing to do with the SDK provisioning audit. Restore them to the pointers used by the rest of this PR (and main). Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Context
Today, dotnet/android provisions the .NET SDK with bespoke C# code in
build-tools/xaprepare/xaprepare/Steps/Step_InstallDotNetPreview.cs(220 lines) — fetching
dotnet-install.{sh,ps1}from a hard-coded URL,running it with config-driven args, and supporting a
--with-archiveoverride for offline scenarios. Other .NET repos (
dotnet/sdk,dotnet/runtime,dotnet/aspnetcore) all use Arcade's standardeng/common/dotnet-install.{sh,ps1}flow — there's no reason for us tomaintain a custom one.
Phase 1 of a longer migration
This PR is the SDK-provisioning slice of a larger effort to delete
xaprepare entirely so the build collapses to:
xaprepare today is 333 KB / 116 files but only 4 step files have real
logic (
Step_PrepareDotNetWorkloads,Step_GenerateFiles,Step_GenerateFiles.Windows,Step_GenerateCGManifest). Once each stephas an MSBuild equivalent, the surrounding 332 KB of plumbing
(
Application/,ToolRunners/,OperatingSystems/) can also bedeleted. Follow-up PRs are planned for each remaining step.
What changes here
New:
eng/install-dotnet.{sh,ps1}Thin bootstrap wrappers that:
<MicrosoftNETSdkPackageVersion>fromeng/Versions.props(single source of truth, kept up to date by darc when
Microsoft.NET.Sdkflows from dotnet/dotnet).dotnet-install.{sh,ps1}fromhttps://builds.dotnet.microsoft.com/dotnet/scripts/v1/(cachedunder
bin/$Configuration/dotnet/).--version <pinned>and--install-dir bin/$Configuration/dotnet.Install location stays at
bin/$Configuration/dotnet/(where xaprepareput it) so
dotnet-local.{cmd,sh}continues to work unchanged.Wired in everywhere
xaprepareran the install beforeMakefile:prepare:target now depends on a newinstall-dotnettarget that calls
./eng/install-dotnet.sh.build-tools/scripts/PrepareWindows.targets: new_InstallDotNettarget runs
eng/install-dotnet.ps1before_BuildXAPrepare.build.cmd: unchanged — the existingdotnet msbuild ... -t:Prepareflow still works because
_BuildXAPreparenow installs the SDK first.Step_InstallDotNetPreview→Step_PrepareDotNetWorkloadsThe old 220-line installer step is deleted. A new ~120-line
Step_PrepareDotNetWorkloads.csreplaces it and only doesAndroid-specific workload prep (NuGet cleanup,
package-download.projrestore with 3-attempt retry, and workload manifest copy). Everything
SDK-install-related (download script, archive override,
InstallDotNetAsyncetc.) is gone.global.json:tools.dotnet— NOT addedI originally tried pinning the SDK version in
global.json:tools.dotnet(the standard Arcade convention), but verified in
arcade-services/.../DependencyFileManager.csthat darcdoes not auto-update
global.json:tools.dotnetwhen theMicrosoft.NET.Sdkasset flows. Only specific Arcade/Helix SDK namesand the literal name
dotnetare special-cased. So atools.dotnetpin would have permanently drifted from the auto-flowed
eng/Versions.props:MicrosoftNETSdkPackageVersion.The wrappers therefore read the version from
Versions.propsdirectlyand bypass Arcade's
eng/common/tools.{sh,ps1}(which would otherwisestrict-mode-read
$GlobalJson.tools). Single source of truth = thedarc-flowed
eng/Versions.props.Other cleanups
Configurables.{Unix,Windows}.cs: removedUrls.DotNetInstallScript(no longer needed).
Context.cs+Main.cs: removedLocalDotNetSdkArchive/--dotnet-sdk-archiveplumbing. (The replacement is the standardDOTNET_INSTALL_DIRenv var that anyone needing offline support canset themselves.)
Verified on Windows
eng/install-dotnet.ps1(with download)dotnet msbuild Xamarin.Android.sln -t:PrepareThe
dotnet --list-sdksoutput after a cold install correctly shows11.0.100-preview.5.26268.112atbin/Debug/dotnet/sdk. Re-running Prepare is silent (no spuriousre-installs, no extra workload restores).
Migration path for the rest of xaprepare (future PRs)
Step_PrepareDotNetWorkloads.targetsfileStep_GenerateCGManifest.targetsfileStep_GenerateFiles[.Windows]Inputs/Outputsbuild-tools/xaprepare/andPrepareWindows.targetsEnd state:
./eng/install-dotnet.sh+dotnet build. Nothing else.