Re-enable Microsoft.Extensions.Hosting shutdown functional coverage in CI and harden Helix/NET481 compatibility

[Copilot]

Description

This PR re-enables Microsoft.Extensions.Hosting shutdown functional coverage in CI and hardens the tests for Helix/minimal Linux environments where source layout or common external tools may be unavailable, while also ensuring the updated shutdown signal path compiles for net481 test legs.

Changes Made

• Reworked ShutdownTests deployment to run Microsoft.Extensions.Hosting.TestApp from deployed test output (AppContext.BaseDirectory) instead of source-project lookup.
• Re-enabled Hosting functional tests in CI by removing the CI skip gating (IgnoreForCI) and the assembly-level ActiveIssue.
• Hardened Unix process cleanup to tolerate missing pgrep and improve robustness of shutdown/termination paths in minimal environments.
• Removed the direct dependency on spawning kill for SIGINT delivery in ShutdownTests by using libc kill(pid, SIGINT).
• Fixed net481 compatibility in ShutdownTests by using Marshal.GetLastWin32Error() (instead of Marshal.GetLastPInvokeError()) for the P/Invoke error path, resolving Windows x86 Libraries_NET481 build failures.
• Fixed a structural hazard where ExistingOutputApplicationPublisher returned a PublishedApplication rooted at AppContext.BaseDirectory, whose Dispose() would recursively delete the test output directory:
  • Made PublishedApplication.Dispose() virtual in the integration testing harness.
  • Added a BorrowedPublishedApplication nested class inside ExistingOutputApplicationPublisher that overrides Dispose() as a no-op, since the path is borrowed from test output and must not be deleted.
  • Removed PreservePublishedApplicationForDebugging = true from DeploymentParameters; the structural fix makes this flag unnecessary.
• Added shutdown cleanup hardening to ensure process-termination flows wait for exit and avoid race conditions when asserting post-kill process state:
  • WaitForExitOrKill now waits up to 5 seconds after calling Kill() before accessing ExitCode, preventing InvalidOperationException race conditions.
  • KillProcessUnix uses kill -TERM for graceful termination (SIGTERM) instead of Process.Kill() (SIGKILL), allowing processes to perform cleanup before exiting.
• Increased test timeouts from 60 seconds to 180 seconds (3 minutes) for startup and shutdown waits to handle stress/JIT-stress pipelines and slow Helix environments.

Validation

• Baseline build completed successfully:
  • ./build.sh clr+libs -rc release
• Targeted Hosting functional tests passed:
  • dotnet build /t:test ./tests/FunctionalTests/Microsoft.Extensions.Hosting.Functional.Tests.csproj
• Targeted net481 build validation passed:
  • dotnet build ./tests/FunctionalTests/Microsoft.Extensions.Hosting.Functional.Tests.csproj -p:TargetFramework=net481 -c Release
• Security validation run:
  • codeql_checker executed for this change.

[rosebyte]

@copilot , the shutdown tests are failing with this error on helix machines:

Comments
{ "HelixJobId": "bf5b8a6f-0bdd-40b3-9fbc-f0eec7148f40", "HelixWorkItemName": "Microsoft.Extensions.Hosting.Functional.Tests" }

Error message
System.IO.DirectoryNotFoundException : Could not find Microsoft.Extensions.Hosting.TestApp.csproj from base directory '/root/helix/work/workitem/e/'.

Stack trace
at Microsoft.AspNetCore.Hosting.FunctionalTests.ShutdownTests.GetApplicationPath() in //src/libraries/Microsoft.Extensions.Hosting/tests/FunctionalTests/ShutdownTests.cs:line 173
at Microsoft.AspNetCore.Hosting.FunctionalTests.ShutdownTests.ExecuteShutdownTest(String testName, String shutdownMechanic) in //src/libraries/Microsoft.Extensions.Hosting/tests/FunctionalTests/ShutdownTests.cs:line 53
at Microsoft.AspNetCore.Hosting.FunctionalTests.ShutdownTests.ShutdownTestRun() in /_/src/libraries/Microsoft.Extensions.Hosting/tests/FunctionalTests/ShutdownTests.cs:line 35
--- End of stack trace from previous location ---

[Copilot]

@copilot , the shutdown tests are failing with this error on helix machines:

...

Fixed in b9cd702. I removed the source-tree lookup and switched the shutdown tests to run TestApp from the deployed test output (AppContext.BaseDirectory) using a custom ApplicationPublisher, so Helix no longer requires Microsoft.Extensions.Hosting.TestApp.csproj to exist in the workitem.

[rosebyte]

@copilot , this time it failed with this error:

Test Failures (700 tests failed)
🔹 [All failing tests from runtime]

⚠️ 698 failing tests not displayed for this pipeline due to space limitations

❌ Microsoft.AspNetCore.Hosting.FunctionalTests.ShutdownTests.ShutdownTestRun [Console] [Details] [Artifacts] [0.70% failure rate]
[ 🚧 Create issue against infrastructure team] [ 📄 Create issue in this repo]
Failing Configuration
AzureLinux.3.0.ArmArch.Open
Exception Message
System.ComponentModel.Win32Exception : An error occurred trying to start process 'pgrep' with working directory '/root/helix/work/workitem/e'. No such file or directory
CallStack
at Microsoft.Win32.SafeHandles.SafeProcessHandle.ForkAndExecProcess(ProcessStartInfo startInfo, String resolvedFilename, String[] argv, IDictionary2 env, String cwd, Boolean setCredentials, UInt32 userId, UInt32 groupId, UInt32[] groups, SafeFileHandle stdinHandle, SafeFileHandle stdoutHandle, SafeFileHandle stderrHandle, Boolean usesTerminal, SafeHandle[] inheritedHandles, Holder& waitStateHolder, Boolean throwOnNoExec) in /_/src/libraries/System.Diagnostics.Process/src/Microsoft/Win32/SafeHandles/SafeProcessHandle.Unix.cs:line 369 at Microsoft.Win32.SafeHandles.SafeProcessHandle.StartCore(ProcessStartInfo startInfo, SafeFileHandle stdinHandle, SafeFileHandle stdoutHandle, SafeFileHandle stderrHandle, SafeHandle[] inheritedHandles, Holder& waitStateHolder) in /_/src/libraries/System.Diagnostics.Process/src/Microsoft/Win32/SafeHandles/SafeProcessHandle.Unix.cs:line 213 at System.Diagnostics.Process.StartCore(ProcessStartInfo startInfo, SafeFileHandle stdinHandle, SafeFileHandle stdoutHandle, SafeFileHandle stderrHandle, SafeHandle[] inheritedHandles) in /_/src/libraries/System.Diagnostics.Process/src/System/Diagnostics/Process.Unix.cs:line 363 at System.Diagnostics.Process.Start() in /_/src/libraries/System.Diagnostics.Process/src/System/Diagnostics/Process.cs:line 1272 at System.Diagnostics.Process.Start(ProcessStartInfo startInfo) in /_/src/libraries/System.Diagnostics.Process/src/System/Diagnostics/Process.cs:line 1383 at Microsoft.Extensions.Internal.ProcessExtensions.RunProcessAndWaitForExit(String fileName, String arguments, TimeSpan timeout, String& stdout) in /_/src/libraries/Microsoft.Extensions.Hosting/tests/FunctionalTests/IntegrationTesting/src/ProcessExtensions.cs:line 93 at Microsoft.Extensions.Internal.ProcessExtensions.GetAllChildIdsUnix(Int32 parentId, ISet1 children, TimeSpan timeout) in //src/libraries/Microsoft.Extensions.Hosting/tests/FunctionalTests/IntegrationTesting/src/ProcessExtensions.cs:line 44
at Microsoft.Extensions.Internal.ProcessExtensions.KillTree(Process process, TimeSpan timeout) in //src/libraries/Microsoft.Extensions.Hosting/tests/FunctionalTests/IntegrationTesting/src/ProcessExtensions.cs:line 33
at Microsoft.Extensions.Internal.ProcessExtensions.KillTree(Process process) in //src/libraries/Microsoft.Extensions.Hosting/tests/FunctionalTests/IntegrationTesting/src/ProcessExtensions.cs:line 17
at Microsoft.Extensions.Hosting.IntegrationTesting.ApplicationDeployer.ShutDownIfAnyHostProcess(Process hostProcess) in //src/libraries/Microsoft.Extensions.Hosting/tests/FunctionalTests/IntegrationTesting/src/Deployers/ApplicationDeployer.cs:line 126
at Microsoft.Extensions.Hosting.IntegrationTesting.SelfHostDeployer.Dispose() in //src/libraries/Microsoft.Extensions.Hosting/tests/FunctionalTests/IntegrationTesting/src/Deployers/SelfHostDeployer.cs:line 179
at Microsoft.AspNetCore.Hosting.FunctionalTests.ShutdownTests.ExecuteShutdownTest(String testName, String shutdownMechanic) in //src/libraries/Microsoft.Extensions.Hosting/tests/FunctionalTests/ShutdownTests.cs:line 107
at Microsoft.AspNetCore.Hosting.FunctionalTests.ShutdownTests.ShutdownTestRun() in //src/libraries/Microsoft.Extensions.Hosting/tests/FunctionalTests/ShutdownTests.cs:line 34
--- End of stack trace from previous location ---
❌ Microsoft.AspNetCore.Hosting.FunctionalTests.ShutdownTests.ShutdownTestWaitForShutdown [Console] [Details] [Artifacts] [0.69% failure rate]
[ 🚧 Create issue against infrastructure team] [ 📄 Create issue in this repo]
Failing Configuration
AzureLinux.3.0.ArmArch.Open
Exception Message
System.ComponentModel.Win32Exception : An error occurred trying to start process 'pgrep' with working directory '/root/helix/work/workitem/e'. No such file or directory
CallStack
at Microsoft.Win32.SafeHandles.SafeProcessHandle.ForkAndExecProcess(ProcessStartInfo startInfo, String resolvedFilename, String[] argv, IDictionary2 env, String cwd, Boolean setCredentials, UInt32 userId, UInt32 groupId, UInt32[] groups, SafeFileHandle stdinHandle, SafeFileHandle stdoutHandle, SafeFileHandle stderrHandle, Boolean usesTerminal, SafeHandle[] inheritedHandles, Holder& waitStateHolder, Boolean throwOnNoExec) in /_/src/libraries/System.Diagnostics.Process/src/Microsoft/Win32/SafeHandles/SafeProcessHandle.Unix.cs:line 369 at Microsoft.Win32.SafeHandles.SafeProcessHandle.StartCore(ProcessStartInfo startInfo, SafeFileHandle stdinHandle, SafeFileHandle stdoutHandle, SafeFileHandle stderrHandle, SafeHandle[] inheritedHandles, Holder& waitStateHolder) in /_/src/libraries/System.Diagnostics.Process/src/Microsoft/Win32/SafeHandles/SafeProcessHandle.Unix.cs:line 213 at System.Diagnostics.Process.StartCore(ProcessStartInfo startInfo, SafeFileHandle stdinHandle, SafeFileHandle stdoutHandle, SafeFileHandle stderrHandle, SafeHandle[] inheritedHandles) in /_/src/libraries/System.Diagnostics.Process/src/System/Diagnostics/Process.Unix.cs:line 363 at System.Diagnostics.Process.Start() in /_/src/libraries/System.Diagnostics.Process/src/System/Diagnostics/Process.cs:line 1272 at System.Diagnostics.Process.Start(ProcessStartInfo startInfo) in /_/src/libraries/System.Diagnostics.Process/src/System/Diagnostics/Process.cs:line 1383 at Microsoft.Extensions.Internal.ProcessExtensions.RunProcessAndWaitForExit(String fileName, String arguments, TimeSpan timeout, String& stdout) in /_/src/libraries/Microsoft.Extensions.Hosting/tests/FunctionalTests/IntegrationTesting/src/ProcessExtensions.cs:line 93 at Microsoft.Extensions.Internal.ProcessExtensions.GetAllChildIdsUnix(Int32 parentId, ISet1 children, TimeSpan timeout) in //src/libraries/Microsoft.Extensions.Hosting/tests/FunctionalTests/IntegrationTesting/src/ProcessExtensions.cs:line 44
at Microsoft.Extensions.Internal.ProcessExtensions.KillTree(Process process, TimeSpan timeout) in //src/libraries/Microsoft.Extensions.Hosting/tests/FunctionalTests/IntegrationTesting/src/ProcessExtensions.cs:line 33
at Microsoft.Extensions.Internal.ProcessExtensions.KillTree(Process process) in //src/libraries/Microsoft.Extensions.Hosting/tests/FunctionalTests/IntegrationTesting/src/ProcessExtensions.cs:line 17
at Microsoft.Extensions.Hosting.IntegrationTesting.ApplicationDeployer.ShutDownIfAnyHostProcess(Process hostProcess) in //src/libraries/Microsoft.Extensions.Hosting/tests/FunctionalTests/IntegrationTesting/src/Deployers/ApplicationDeployer.cs:line 126
at Microsoft.Extensions.Hosting.IntegrationTesting.SelfHostDeployer.Dispose() in //src/libraries/Microsoft.Extensions.Hosting/tests/FunctionalTests/IntegrationTesting/src/Deployers/SelfHostDeployer.cs:line 179
at Microsoft.AspNetCore.Hosting.FunctionalTests.ShutdownTests.ExecuteShutdownTest(String testName, String shutdownMechanic) in //src/libraries/Microsoft.Extensions.Hosting/tests/FunctionalTests/ShutdownTests.cs:line 107
at Microsoft.AspNetCore.Hosting.FunctionalTests.ShutdownTests.ShutdownTestWaitForShutdown() in //src/libraries/Microsoft.Extensions.Hosting/tests/FunctionalTests/ShutdownTests.cs:line 41
--- End of stack trace from previous location ---

[Copilot]

Pull request overview

Copilot reviewed 5 out of 5 changed files in this pull request and generated 1 comment.

[Copilot]

Pull request overview

Copilot reviewed 5 out of 5 changed files in this pull request and generated 1 comment.

[Copilot]

Pull request overview

Copilot reviewed 5 out of 5 changed files in this pull request and generated 1 comment.

[rosebyte]

@copilot , you only addressed one of my comments, resolve all of them.

[Copilot]

@copilot , you only addressed one of my comments, resolve all of them.

Fixed in f9172b4. I've addressed both remaining issues:

1. KillProcessUnix now uses kill -TERM for graceful termination (SIGTERM) instead of Process.Kill() (SIGKILL)
2. Test timeouts increased from 60 seconds to 180 seconds (3 minutes) for startup and shutdown waits to handle stress/JIT-stress Helix environments