-
Notifications
You must be signed in to change notification settings - Fork 5.4k
[NO-REVIEW] Batch WASM CoreCLR library test suites on Helix #126157
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Draft
radekdoulik
wants to merge
4
commits into
main
Choose a base branch
from
batch-wasm-coreclr-library-tests
base: main
Could not load branches
Branch not found: {{ refName }}
Loading
Could not load tags
Nothing to show
Loading
Are you sure you want to change the base?
Some commits from the old base branch may be removed from the timeline,
and old review comments may become outdated.
+308
−1
Draft
Changes from all commits
Commits
Show all changes
4 commits
Select commit
Hold shift + click to select a range
b4a1df1
[NO-REVIEW] Batch WASM CoreCLR library test suites on Helix
radekdoulik a751466
Address PR review feedback
radekdoulik face27d
Fix batch timeout: 30m min, 20m/suite for WASM overhead
radekdoulik 54346c1
Address code review feedback
radekdoulik File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,101 @@ | ||
| #!/usr/bin/env bash | ||
|
|
||
| if [[ -z "$HELIX_WORKITEM_UPLOAD_ROOT" ]]; then | ||
| ORIGINAL_UPLOAD_ROOT="$PWD/test-results" | ||
| else | ||
| ORIGINAL_UPLOAD_ROOT="$HELIX_WORKITEM_UPLOAD_ROOT" | ||
| fi | ||
|
|
||
| BATCH_DIR="$PWD" | ||
| SUITE_COUNT=0 | ||
| FAIL_COUNT=0 | ||
| SUITE_NAMES=() | ||
| SUITE_EXIT_CODES=() | ||
| SUITE_DURATIONS=() | ||
|
|
||
| echo "=== WasmBatchRunner ===" | ||
| echo "BATCH_DIR=$BATCH_DIR" | ||
| echo "ORIGINAL_UPLOAD_ROOT=$ORIGINAL_UPLOAD_ROOT" | ||
|
|
||
| for zipFile in "$BATCH_DIR"/*.zip; do | ||
| if [[ ! -f "$zipFile" ]]; then | ||
| echo "No .zip files found in $BATCH_DIR" | ||
| exit 1 | ||
| fi | ||
|
|
||
| suiteName=$(basename "$zipFile" .zip) | ||
| suiteDir="$BATCH_DIR/$suiteName" | ||
|
|
||
| echo "" | ||
| echo "========================= BEGIN $suiteName =============================" | ||
|
|
||
| mkdir -p "$suiteDir" | ||
| if ! unzip -q -o "$zipFile" -d "$suiteDir"; then | ||
| echo "ERROR: Failed to extract $zipFile" | ||
| FAIL_COUNT=$((FAIL_COUNT + 1)) | ||
| SUITE_NAMES+=("$suiteName") | ||
| SUITE_EXIT_CODES+=("1") | ||
| SUITE_DURATIONS+=("0") | ||
| SUITE_COUNT=$((SUITE_COUNT + 1)) | ||
| rm -rf "$suiteDir" | ||
| continue | ||
| fi | ||
|
|
||
| export HELIX_WORKITEM_UPLOAD_ROOT="$ORIGINAL_UPLOAD_ROOT/$suiteName" | ||
| mkdir -p "$HELIX_WORKITEM_UPLOAD_ROOT" | ||
|
|
||
| pushd "$suiteDir" >/dev/null | ||
|
|
||
| chmod +x RunTests.sh | ||
|
|
||
| startTime=$(date +%s) | ||
| ./RunTests.sh "$@" | ||
| suiteExitCode=$? | ||
| endTime=$(date +%s) | ||
|
|
||
| popd >/dev/null | ||
|
|
||
| rm -rf "$suiteDir" | ||
|
|
||
| duration=$((endTime - startTime)) | ||
|
|
||
| SUITE_NAMES+=("$suiteName") | ||
| SUITE_EXIT_CODES+=("$suiteExitCode") | ||
| SUITE_DURATIONS+=("$duration") | ||
| SUITE_COUNT=$((SUITE_COUNT + 1)) | ||
|
|
||
| if [[ $suiteExitCode -ne 0 ]]; then | ||
| FAIL_COUNT=$((FAIL_COUNT + 1)) | ||
| echo "----- FAIL $suiteName — exit code $suiteExitCode — ${duration}s -----" | ||
| else | ||
| echo "----- PASS $suiteName — ${duration}s -----" | ||
| fi | ||
|
|
||
| echo "========================= END $suiteName ===============================" | ||
| done | ||
|
|
||
| # Restore so Helix post-commands write artifacts to the expected root | ||
| export HELIX_WORKITEM_UPLOAD_ROOT="$ORIGINAL_UPLOAD_ROOT" | ||
|
|
||
| echo "" | ||
| echo "=== Batch Summary ===" | ||
| printf "%-60s %-6s %s\n" "Suite" "Status" "Duration" | ||
| printf "%-60s %-6s %s\n" "-----" "------" "--------" | ||
|
|
||
| for i in "${!SUITE_NAMES[@]}"; do | ||
| if [[ ${SUITE_EXIT_CODES[$i]} -eq 0 ]]; then | ||
| status="PASS" | ||
| else | ||
| status="FAIL" | ||
| fi | ||
| printf "%-60s %-6s %ss\n" "${SUITE_NAMES[$i]}" "$status" "${SUITE_DURATIONS[$i]}" | ||
| done | ||
|
|
||
| echo "" | ||
| echo "Total: $SUITE_COUNT | Passed: $((SUITE_COUNT - FAIL_COUNT)) | Failed: $FAIL_COUNT" | ||
|
|
||
| if [[ $FAIL_COUNT -ne 0 ]]; then | ||
| exit 1 | ||
| fi | ||
|
|
||
| exit 0 | ||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -41,10 +41,15 @@ | |
| '$(Scenario)' == 'WasmTestOnChrome' or | ||
| '$(Scenario)' == 'WasmTestOnFirefox'">true</IsRunningLibraryTests> | ||
|
|
||
| <WasmBatchLibraryTests Condition="'$(WasmBatchLibraryTests)' == '' and '$(RuntimeFlavor)' == 'CoreCLR' and '$(Scenario)' == 'WasmTestOnChrome'">true</WasmBatchLibraryTests> | ||
| <WasmBatchLibraryTests Condition="'$(WasmBatchLibraryTests)' == ''">false</WasmBatchLibraryTests> | ||
| <_WasmBatchLargeThreshold Condition="'$(_WasmBatchLargeThreshold)' == ''">52428800</_WasmBatchLargeThreshold> | ||
|
|
||
| <HelixExtensionTargets /> | ||
| <PrepareForBuildHelixWorkItems_WasmDependsOn> | ||
| PrepareHelixCorrelationPayload_Wasm; | ||
| _AddWorkItemsForLibraryTests; | ||
| _AddBatchedWorkItemsForLibraryTests; | ||
| _AddWorkItemsForBuildWasmApps | ||
| </PrepareForBuildHelixWorkItems_WasmDependsOn> | ||
|
|
||
|
|
@@ -172,6 +177,135 @@ | |
|
|
||
| <Import Project="$(RepositoryEngineeringDir)testing\wasm-provisioning.targets" /> | ||
|
|
||
| <UsingTask TaskName="_GroupWorkItems" TaskFactory="RoslynCodeTaskFactory" | ||
| AssemblyFile="$(MSBuildToolsPath)\Microsoft.Build.Tasks.Core.dll"> | ||
| <ParameterGroup> | ||
| <Items ParameterType="Microsoft.Build.Framework.ITaskItem[]" Required="true" /> | ||
| <BatchSize ParameterType="System.Int32" Required="false" /> | ||
| <LargeThreshold ParameterType="System.Int64" Required="false" /> | ||
| <GroupedItems ParameterType="Microsoft.Build.Framework.ITaskItem[]" Output="true" /> | ||
| </ParameterGroup> | ||
| <Task> | ||
| <Using Namespace="System" /> | ||
| <Using Namespace="System.Collections.Generic" /> | ||
| <Using Namespace="System.IO" /> | ||
| <Code Type="Fragment" Language="cs"> | ||
| <![CDATA[ | ||
| if (BatchSize <= 0) BatchSize = 10; | ||
| if (LargeThreshold <= 0) LargeThreshold = 52428800L; // 50MB | ||
|
|
||
| var itemsWithSize = new List<(ITaskItem item, long size)>(); | ||
| foreach (var item in Items) | ||
| { | ||
| long size = 0; | ||
| if (File.Exists(item.ItemSpec)) | ||
| { | ||
| size = new FileInfo(item.ItemSpec).Length; | ||
| } | ||
| itemsWithSize.Add((item, size)); | ||
| } | ||
|
|
||
| // Sort largest first for greedy bin-packing | ||
| itemsWithSize.Sort((a, b) => b.size.CompareTo(a.size)); | ||
|
|
||
| var result = new List<ITaskItem>(); | ||
| int negativeBatchId = -1; | ||
|
|
||
| // Separate large items (each gets its own batch) | ||
| var smallItems = new List<(ITaskItem item, long size)>(); | ||
| foreach (var entry in itemsWithSize) | ||
| { | ||
| if (entry.size > LargeThreshold) | ||
| { | ||
| var newItem = new TaskItem(entry.item); | ||
| newItem.SetMetadata("BatchId", negativeBatchId.ToString()); | ||
| negativeBatchId--; | ||
| result.Add(newItem); | ||
| } | ||
| else | ||
| { | ||
| smallItems.Add(entry); | ||
| } | ||
| } | ||
|
|
||
| // Greedy bin-packing for small items | ||
| if (smallItems.Count > 0) | ||
| { | ||
| int numBatches = Math.Min(BatchSize, smallItems.Count); | ||
| var batchSizes = new long[numBatches]; | ||
| var batchAssignments = new List<ITaskItem>[numBatches]; | ||
| for (int i = 0; i < numBatches; i++) | ||
| batchAssignments[i] = new List<ITaskItem>(); | ||
|
|
||
| foreach (var entry in smallItems) | ||
| { | ||
| // Find batch with smallest total size | ||
| int minIdx = 0; | ||
| for (int i = 1; i < numBatches; i++) | ||
| { | ||
| if (batchSizes[i] < batchSizes[minIdx]) | ||
| minIdx = i; | ||
| } | ||
| batchSizes[minIdx] += entry.size; | ||
| var newItem = new TaskItem(entry.item); | ||
| newItem.SetMetadata("BatchId", minIdx.ToString()); | ||
| batchAssignments[minIdx].Add(newItem); | ||
| } | ||
|
|
||
| for (int i = 0; i < numBatches; i++) | ||
| result.AddRange(batchAssignments[i]); | ||
| } | ||
|
|
||
| GroupedItems = result.ToArray(); | ||
| ]]> | ||
| </Code> | ||
| </Task> | ||
| </UsingTask> | ||
|
|
||
| <UsingTask TaskName="_ComputeBatchTimeout" TaskFactory="RoslynCodeTaskFactory" | ||
| AssemblyFile="$(MSBuildToolsPath)\Microsoft.Build.Tasks.Core.dll"> | ||
| <ParameterGroup> | ||
| <GroupedItems ParameterType="Microsoft.Build.Framework.ITaskItem[]" Required="true" /> | ||
| <BatchIds ParameterType="Microsoft.Build.Framework.ITaskItem[]" Required="true" /> | ||
| <ItemPrefix ParameterType="System.String" Required="true" /> | ||
| <BatchOutputDir ParameterType="System.String" Required="true" /> | ||
| <TimedItems ParameterType="Microsoft.Build.Framework.ITaskItem[]" Output="true" /> | ||
| </ParameterGroup> | ||
| <Task> | ||
| <Using Namespace="System" /> | ||
| <Using Namespace="System.Collections.Generic" /> | ||
| <Code Type="Fragment" Language="cs"> | ||
| <![CDATA[ | ||
| var counts = new Dictionary<string, int>(); | ||
| foreach (var item in GroupedItems) | ||
| { | ||
| string bid = item.GetMetadata("BatchId"); | ||
| if (!counts.ContainsKey(bid)) counts[bid] = 0; | ||
| counts[bid]++; | ||
| } | ||
|
|
||
| var result = new List<ITaskItem>(); | ||
| foreach (var batchId in BatchIds) | ||
| { | ||
| string bid = batchId.ItemSpec; | ||
| int count = counts.ContainsKey(bid) ? counts[bid] : 1; | ||
| // 20 minutes per suite to account for WASM startup overhead + test execution; | ||
| // minimum 30 minutes to handle the heaviest individual suites (e.g. Cryptography ~17m) | ||
| int totalMinutes = Math.Max(30, count * 20); | ||
| var ts = TimeSpan.FromMinutes(totalMinutes); | ||
|
|
||
| var helixItem = new TaskItem(ItemPrefix + "Batch-" + bid); | ||
| helixItem.SetMetadata("BatchDir", BatchOutputDir + "batch-" + bid + "/"); | ||
| helixItem.SetMetadata("Timeout", ts.ToString(@"hh\:mm\:ss")); | ||
| result.Add(helixItem); | ||
|
Comment on lines
+292
to
+300
|
||
| } | ||
|
|
||
| TimedItems = result.ToArray(); | ||
| ]]> | ||
| </Code> | ||
| </Task> | ||
| </UsingTask> | ||
|
|
||
| <Target Name="PrepareHelixCorrelationPayload_Wasm"> | ||
| <Error Condition="'$(Scenario)' != 'WasmTestOnV8' and | ||
| '$(Scenario)' != 'WasmTestOnChrome' and | ||
|
|
@@ -228,7 +362,7 @@ | |
| <Target Name="PrepareForBuildHelixWorkItems_Wasm" | ||
| DependsOnTargets="$(PrepareForBuildHelixWorkItems_WasmDependsOn);$(HelixExtensionTargets)" /> | ||
|
|
||
| <Target Name="_AddWorkItemsForLibraryTests" Condition="'$(IsRunningLibraryTests)' == 'true'"> | ||
| <Target Name="_AddWorkItemsForLibraryTests" Condition="'$(IsRunningLibraryTests)' == 'true' and '$(WasmBatchLibraryTests)' != 'true'"> | ||
| <ItemGroup Label="Add samples"> | ||
| <_WasmWorkItem Include="$(TestArchiveRoot)browseronly/**/*.zip" Condition="'$(Scenario)' == 'WasmTestOnChrome' or '$(Scenario)' == 'WasmTestOnFirefox'" /> | ||
| <_WasmWorkItem Include="$(TestArchiveRoot)chromeonly/**/*.zip" Condition="'$(Scenario)' == 'WasmTestOnChrome'" /> | ||
|
|
@@ -273,4 +407,76 @@ | |
|
|
||
| </ItemGroup> | ||
| </Target> | ||
|
|
||
| <Target Name="_AddBatchedWorkItemsForLibraryTests" | ||
| Condition="'$(IsRunningLibraryTests)' == 'true' and '$(WasmBatchLibraryTests)' == 'true'"> | ||
|
|
||
| <!-- Collect all test ZIPs (batching only runs for Chrome, see WasmBatchLibraryTests condition) --> | ||
| <ItemGroup> | ||
| <_WasmBatchWorkItem Include="$(TestArchiveRoot)browseronly/**/*.zip" /> | ||
| <_WasmBatchWorkItem Include="$(TestArchiveRoot)chromeonly/**/*.zip" /> | ||
| </ItemGroup> | ||
|
|
||
| <!-- Sample apps have different runners (no RunTests.sh), keep them as individual work items --> | ||
| <ItemGroup> | ||
| <_WasmBatchSampleZip Include="$(TestArchiveRoot)runonly/**/*.Browser.*.Sample.zip" /> | ||
|
|
||
| <HelixWorkItem Include="@(_WasmBatchSampleZip -> '$(WorkItemPrefix)%(FileName)')"> | ||
| <PayloadArchive>%(Identity)</PayloadArchive> | ||
| <Command>$(HelixCommand)</Command> | ||
| <Timeout>$(_workItemTimeout)</Timeout> | ||
| </HelixWorkItem> | ||
| </ItemGroup> | ||
|
|
||
| <ItemGroup> | ||
| <_WasmBatchDefaultItems Include="$(WorkItemArchiveWildCard)" Exclude="$(HelixCorrelationPayload)" /> | ||
| </ItemGroup> | ||
|
|
||
| <!-- Batch only test suites (not sample apps) --> | ||
| <ItemGroup> | ||
| <_WasmBatchAllItems Include="@(_WasmBatchWorkItem)" /> | ||
| <_WasmBatchAllItems Include="@(_WasmBatchDefaultItems)" /> | ||
| </ItemGroup> | ||
|
|
||
| <!-- Assign batch IDs via greedy bin-packing --> | ||
| <_GroupWorkItems Items="@(_WasmBatchAllItems)" BatchSize="20" LargeThreshold="$(_WasmBatchLargeThreshold)"> | ||
| <Output TaskParameter="GroupedItems" ItemName="_WasmGroupedItem" /> | ||
| </_GroupWorkItems> | ||
|
|
||
| <!-- Determine unique batch IDs --> | ||
| <ItemGroup> | ||
| <_WasmBatchId Include="@(_WasmGroupedItem -> '%(BatchId)')" /> | ||
| <_WasmUniqueBatchId Include="@(_WasmBatchId->Distinct())" /> | ||
| </ItemGroup> | ||
|
|
||
| <!-- Clean stale batch staging from previous runs, then create fresh directories --> | ||
| <RemoveDir Directories="$(IntermediateOutputPath)helix-batches/" /> | ||
| <MakeDir Directories="$(IntermediateOutputPath)helix-batches/batch-%(_WasmUniqueBatchId.Identity)/" /> | ||
| <Copy SourceFiles="@(_WasmGroupedItem)" DestinationFolder="$(IntermediateOutputPath)helix-batches/batch-%(BatchId)/" /> | ||
| <Copy SourceFiles="$(RepositoryEngineeringDir)testing/WasmBatchRunner.sh" | ||
| DestinationFolder="$(IntermediateOutputPath)helix-batches/batch-%(_WasmUniqueBatchId.Identity)/" /> | ||
|
|
||
| <!-- Compute per-batch timeout: 20 minutes per suite, minimum 30 minutes --> | ||
| <_ComputeBatchTimeout GroupedItems="@(_WasmGroupedItem)" BatchIds="@(_WasmUniqueBatchId)" | ||
| ItemPrefix="$(WorkItemPrefix)" BatchOutputDir="$(IntermediateOutputPath)helix-batches/"> | ||
| <Output TaskParameter="TimedItems" ItemName="_WasmTimedBatchItem" /> | ||
| </_ComputeBatchTimeout> | ||
|
|
||
| <!-- Create ZIP archives from batch directories (sendtohelixhelp.proj requires PayloadArchive) --> | ||
| <ZipDirectory SourceDirectory="%(_WasmTimedBatchItem.BatchDir)" | ||
| DestinationFile="$(IntermediateOutputPath)helix-batches/%(_WasmTimedBatchItem.Identity).zip" | ||
| Overwrite="true" /> | ||
|
|
||
| <!-- Build a batch-specific HelixCommand that runs WasmBatchRunner.sh instead of RunTests.sh --> | ||
| <PropertyGroup> | ||
| <_WasmBatchHelixCommand>$(HelixCommand.Replace('./RunTests.sh', 'chmod +x WasmBatchRunner.sh && ./WasmBatchRunner.sh'))</_WasmBatchHelixCommand> | ||
| </PropertyGroup> | ||
|
|
||
| <ItemGroup> | ||
| <HelixWorkItem Include="@(_WasmTimedBatchItem)"> | ||
| <PayloadArchive>$(IntermediateOutputPath)helix-batches/%(Identity).zip</PayloadArchive> | ||
| <Command>$(_WasmBatchHelixCommand)</Command> | ||
| </HelixWorkItem> | ||
| </ItemGroup> | ||
| </Target> | ||
| </Project> | ||
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This script leaves
HELIX_WORKITEM_UPLOAD_ROOTset to the last suite’s subdirectory when it exits. Helix post-commands (e.g., the CoreCLR dump-doc generation insendtohelixhelp.proj) may run after the main command and useHELIX_WORKITEM_UPLOAD_ROOTto decide where to write artifacts; consider restoringHELIX_WORKITEM_UPLOAD_ROOTback toORIGINAL_UPLOAD_ROOTbefore printing the final summary / exiting so post-commands still write to the expected root.