⚡ Bolt: Parallelize repository downloads in CI

[mentalblank]

User description

⚡ Bolt: Parallelize repository downloads in CI

💡 What:

• Converted the sequential loop for downloading and extracting repositories into a parallel loop using Bash background jobs (&) and wait.
• Refactored download_file to use if ! curl ... pattern, fixing a bug where set -e made the error handling block unreachable.
• Added local declaration for temp_dir in extract_archive to prevent variable collision between parallel jobs.
• Added -sS to curl to silence progress bars but keep errors visible.

🎯 Why:

• The build process involves downloading and processing ~40 repositories. Doing this sequentially is slow. Parallel execution utilizes the runner's bandwidth and CPU more effectively.
• The original error handling was flawed because set -e (default in GitHub Actions) would cause the script to exit immediately if curl failed, skipping the "mandatory file" check and output logic.

📊 Impact:

• Build time for the "Download and Combine Releases" step should decrease significantly, bounded by the longest single repo download/extract time rather than the sum of all of them.
• Robustness against network flakes is improved (proper error reporting).

🔬 Measurement:

• Verify that the workflow completes successfully.
• Compare build times of the new workflow against previous runs (expected ~50-80% reduction in the download step duration).
• Verify that failure of a mandatory download correctly stops the release process.

---

PR created automatically by Jules for task 9657267172498000547 started by @mentalblank

---

PR Type

Enhancement

---

Description

• Parallelize repository downloads using Bash background jobs and wait

• Fix error handling in download_file to work with set -e

• Add -sS flags to curl for cleaner parallel execution logs

• Declare local temp_dir in extract_archive for thread safety

---

Diagram Walkthrough

flowchart LR
  A["Sequential Downloads"] -->|Refactor| B["Parallel Downloads with &"]
  C["Unsafe Error Handling"] -->|Fix| D["if ! curl Pattern"]
  E["Shared Variables"] -->|Declare| F["Local temp_dir"]
  B --> G["Faster Build Times"]
  D --> G
  F --> G

Loading

File Walkthrough

	Relevant files
Enhancement	build.yml Parallelize downloads and fix error handling                          .github/workflows/build.yml Refactored download_file to use if ! curl pattern instead of curl; if [ $? -ne 0 ] for proper error handling with set -e Added -sS flags to curl to silence progress bars while keeping errors visible Added local temp_dir declaration in extract_archive to prevent variable collision between parallel jobs Converted sequential handle_downloads loop to parallel execution using background jobs (&) and wait +7/-3
Documentation	bolt.md Document parallel Bash workflows and error handling            .jules/bolt.md Added learning notes on parallelizing sequential Bash workflows using & and wait Documented best practices for error handling in scripts with set -e enabled Captured action items for identifying and parallelizing independent IO tasks +7/-0

---

[google-labs-jules]

👋 Jules, reporting for duty! I'm here to lend a hand with this pull request.

When you start a review, I'll add a 👀 emoji to each comment to let you know I've read it. I'll focus on feedback directed at me and will do my best to stay out of conversations between you and other bots or reviewers to keep the noise down.

I'll push a commit with your requested changes shortly after. Please note there might be a delay between these steps, but rest assured I'm on the job!

For more direct control, you can switch me to Reactive Mode. When this mode is on, I will only act on comments where you specifically mention me with @jules. You can find this option in the Pull Request section of your global Jules UI settings. You can always switch back!

New to Jules? Learn more at jules.google/docs.

---

For security, I will only act on instructions from the user who triggered this task.

[qodo-code-review]

PR Compliance Guide 🔍

Below is a summary of compliance checks for this PR:

Security Compliance
⚪	Token exfiltration risk Description: The download_file function sends Authorization: token $GITHUB_TOKEN on a curl request to whatever url is passed in, so if any part of the workflow can cause url to point to a non-GitHub/attacker-controlled host, the GitHub token could be exfiltrated via the outbound HTTP request. build.yml [41-45] Referred Code download_file() { local url=$1 output=$2 mandatory=$3 echo "Downloading file: $output from $url" if ! curl -sS -L -H "Authorization: token $GITHUB_TOKEN" -o "$output" "$url"; then echo "Failed to download $output from $url"
Ticket Compliance
⚪	🎫 No ticket provided Create ticket/issue
Codebase Duplication Compliance
⚪	Codebase context is not defined Follow the guide to enable codebase context checks.
Custom Compliance
🟢	Generic: Comprehensive Audit Trails Objective: To create a detailed and reliable record of critical system actions for security analysis and compliance. Status: Passed Learn more about managing compliance generic rules or creating your own custom rules
	Generic: Meaningful Naming and Self-Documenting Code Objective: Ensure all identifiers clearly express their purpose and intent, making code self-documenting Status: Passed Learn more about managing compliance generic rules or creating your own custom rules
	Generic: Secure Error Handling Objective: To prevent the leakage of sensitive system information through error messages while providing sufficient detail for internal debugging. Status: Passed Learn more about managing compliance generic rules or creating your own custom rules
	Generic: Secure Logging Practices Objective: To ensure logs are useful for debugging and auditing without exposing sensitive information like PII, PHI, or cardholder data. Status: Passed Learn more about managing compliance generic rules or creating your own custom rules
🔴	Generic: Robust Error Handling and Edge Case Management Objective: Ensure comprehensive error handling that provides meaningful context and graceful degradation Status: Missed job failures: Parallel downloads are launched in the background but a bare wait does not reliably detect/report failures from any job (it returns only one exit status), risking silent partial failures. Referred Code for repo in $(echo "$RELEASE_VERSIONS" | jq -r '.repos | keys[]'); do handle_downloads "$repo" & done wait echo "Final contents of release/:" Learn more about managing compliance generic rules or creating your own custom rules
	Generic: Security-First Input Validation and Data Handling Objective: Ensure all data inputs are validated, sanitized, and handled securely to prevent vulnerabilities Status: Unsafe word splitting: The for repo in $(...) loop performs shell word-splitting on external JSON-derived data, which can break on whitespace/special characters and should be read safely (e.g., while IFS= read -r). Referred Code for repo in $(echo "$RELEASE_VERSIONS" | jq -r '.repos | keys[]'); do handle_downloads "$repo" & done Learn more about managing compliance generic rules or creating your own custom rules

Compliance status legend

🟢 - Fully Compliant
🟡 - Partial Compliant
🔴 - Not Compliant
⚪ - Requires Further Human Verification
🏷️ - Compliance label

[qodo-code-review]

PR Code Suggestions ✨

Explore these optional code suggestions:

Category	Suggestion	Impact
Possible issue	fail on mandatory download error Change exit 0 to exit 1 for mandatory download failures to ensure the CI step correctly reports an error instead of silently succeeding. .github/workflows/build.yml [46-50] if [ "$mandatory" = "true" ]; then echo "should_continue=false" >> $GITHUB_OUTPUT echo "Mandatory download failure: $output" - exit 0 + exit 1 fi Apply / Chat Suggestion importance[1-10]: 9 __ Why: The suggestion correctly points out that exit 0 masks a critical failure, causing the step to succeed incorrectly. Changing it to exit 1 ensures that a mandatory download failure properly fails the CI job, which is a critical correctness fix.	High
	Add error handling for temp directory creation Add error handling after creating a temporary directory with mktemp -d to prevent the script from continuing with an empty directory variable on failure. .github/workflows/build.yml [54-60] extract_archive() { local file=$1 target=$2 archive_pattern=$3 additional_files_json=$4 local temp_dir - mkdir -p "$target" && temp_dir=$(mktemp -d) + temp_dir=$(mktemp -d) + if [[ -z "$temp_dir" ]]; then + echo "Failed to create temporary directory." >&2 + exit 1 + fi + mkdir -p "$target" case "$file" in *.zip|*.tar.gz|*.7z) Apply / Chat Suggestion importance[1-10]: 8 __ Why: The suggestion correctly identifies that set -e will not catch a failure of mktemp in a command substitution, which could lead to dangerous behavior. Adding an explicit check for the directory creation failure significantly improves the script's robustness.	Medium
High-level	Limit concurrency to prevent instability To prevent overwhelming the CI runner, limit the number of concurrent download jobs. This can be done using a Bash semaphore pattern or a tool like xargs -P to ensure stability. Examples: .github/workflows/build.yml [238-241] for repo in $(echo "$RELEASE_VERSIONS" | jq -r '.repos | keys[]'); do handle_downloads "$repo" & done wait Solution Walkthrough: Before: # .github/workflows/build.yml # ... echo "Downloading files for updated repositories..." for repo in $(get_all_repos); do # This launches a background job for every single repo handle_downloads "$repo" & done # This waits for ALL jobs to finish wait # ... After: # .github/workflows/build.yml # ... MAX_JOBS=10 echo "Downloading files for updated repositories..." for repo in $(get_all_repos); do handle_downloads "$repo" & # Wait if the number of active jobs reaches the limit if [[ $(jobs -r -p | wc -l) -ge $MAX_JOBS ]]; then wait -n fi done wait # ... Suggestion importance[1-10]: 8 __ Why: The suggestion correctly identifies a significant stability risk in running an unlimited number of parallel jobs, which could exhaust CI runner resources and lead to failures.	Medium
General	cleanup temporary directories Add a trap command to automatically clean up the temporary directory created by mktemp when the function exits, preventing resource leaks. .github/workflows/build.yml [56-57] local temp_dir -mkdir -p "$target" && temp_dir=$(mktemp -d) +mkdir -p "$target" +temp_dir=$(mktemp -d) +trap 'rm -rf "$temp_dir"' EXIT Apply / Chat Suggestion importance[1-10]: 7 __ Why: The suggestion correctly proposes using trap for robust temporary directory cleanup, which is a good practice. This prevents leftover files if the script exits unexpectedly, improving resource management in the CI runner.	Medium
More