Advanced Usage Guide

This guide covers advanced features and usage patterns for CE Library Wizard.

Command-Line Automation

Batch Processing

Process multiple libraries using shell scripting:

#!/bin/bash
# add-multiple-libs.sh

libraries=(
    "rust:serde:1.0.195"
    "rust:tokio:1.35.1"
    "c++:https://github.com/fmtlib/fmt:10.2.1"
)

for lib in "${libraries[@]}"; do
    IFS=':' read -r lang name ver <<< "$lib"
    echo "Adding $lang library $name version $ver..."
    ./run.sh -y --build-test=no --lang="$lang" --lib="$name" --ver="$ver"
done

Note: Use -y to skip confirmations and --build-test=no to speed up batch operations.

CI/CD Integration

Use CE Library Wizard in automated workflows:

# .github/workflows/add-library.yml
name: Add Library to CE
on:
  workflow_dispatch:
    inputs:
      language:
        required: true
        type: choice
        options: [rust, c++, c, fortran]
      library:
        required: true
      version:
        required: true
      build_test:
        required: false
        type: choice
        options: [auto, yes, no]
        default: auto

jobs:
  add-library:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v3
      - uses: actions/setup-python@v4
        with:
          python-version: '3.12'
      - run: |
          ./run.sh \
            -y \
            --lang=${{ inputs.language }} \
            --lib=${{ inputs.library }} \
            --ver=${{ inputs.version }} \
            --build-test=${{ inputs.build_test }} \
            --github-token=${{ secrets.GITHUB_TOKEN }}

Note the -y flag to skip confirmation prompts in CI environments.

Advanced Authentication

Using GitHub App Tokens

For organizations, you can use GitHub App tokens:

# Generate app token (example using gh cli extension)
export GITHUB_TOKEN=$(gh app token create --app-id YOUR_APP_ID)
./run.sh --lang=rust --lib=serde --ver=1.0.195

Token Scoping

Minimum required token permissions:

repo - Full repository access (for forking and pushing)
workflow - Update GitHub Actions (if PR includes workflow files)

Installation Testing Deep Dive

Understanding --install-test

The --install-test flag for C/C++ libraries:

What it does:
- Downloads and installs the library locally
- Verifies installation paths match properties
- Ensures library can be used by Compiler Explorer

Requirements:

# Create and set permissions for CE directory
sudo mkdir -p /opt/compiler-explorer
sudo chown -R $USER: /opt/compiler-explorer

What's checked:
- Library downloads successfully
- Files are placed in expected locations
- Paths in properties files are correct

Manual Installation Testing

Test installations manually:

cd /tmp/ce-lib-wizard-*/infra
poetry run ce_install install "library_id version"

Build Testing Deep Dive

Understanding --build-test

The --build-test option tests that libraries requiring compilation actually build correctly:

Modes:

auto (default): Run if compiler available AND library needs building (skips header-only)
yes: Force run build test, fail if no compiler available
no: Skip build testing entirely

# Auto mode (default) - runs if compiler available
./run.sh --lang=c++ --lib=https://github.com/madler/zlib --ver=1.3.1

# Force build test
./run.sh --build-test=yes --lang=c++ --lib=https://github.com/madler/zlib --ver=1.3.1

# Skip build test
./run.sh --build-test=no --lang=c++ --lib=https://github.com/madler/zlib --ver=1.3.1

What it does:
- Detects installed compilers (gcc, clang, rust, gfortran) via ce_install
- Runs a dry-run build using the latest compiler
- Captures all produced artifacts (libraries, headers, pkg-config files)
- Verifies that expected link libraries match what was built
Requirements:
- A compiler must be installed via ce_install (e.g., gcc 14.2.0)
- The infra repository must be cloned
What's checked:
- Build completes successfully
- Expected .a files exist for staticliblink entries
- Expected .so files exist for sharedliblink entries
Auto-skip conditions:
- Header-only libraries (no compilation needed)
- No compiler available (in auto mode only)

Example Output

🔨 Running build test... (Build testing available with gcc 15.2.0 (g152))
✓ Build test passed
  Artifacts produced:
  Libraries: libz.a, libz.so, libz.so.1, libz.so.1.3.1
  Headers: zconf.h, zlib.h
  Other: zlib.pc, zlib.3
  Link library verification:
  ✓ -lz (static)

Manual Build Testing

Test builds manually using ce_install:

cd /opt/compiler-explorer/infra

# List installed compilers
bin/ce_install --filter-match-all list --installed-only --show-compiler-ids --json gcc '!cross'

# Run a build test (dry-run mode, keeps staging directory)
bin/ce_install --debug --dry-run --keep-staging build --temp-install --buildfor g152 'libraries/c++/zlib 1.3.1'

# Check the staging directory for artifacts
ls -la /tmp/ce-cefs-temp/staging/*/install/

Verifying Link Libraries

The build test reads staticliblink and sharedliblink from libraries.yaml and verifies the corresponding library files were produced:

# In libraries.yaml
zlib:
  lib_type: static
  staticliblink:
    - z  # Expects libz.a to be built

If a library is missing, you'll see:

Link library verification:
✗ MISSING -lfoo (static)
⚠️  Missing: libfoo.a

Rust Build Testing

Understanding Rust Build Tests

The --build-test flag also works for Rust crates:

What it does:
- Detects installed Rust compilers via ce_install
- Downloads the crate from crates.io
- Builds with cargo in a staging directory
- Captures .rlib and .rmeta artifacts
Requirements:
- A Rust compiler must be installed via ce_install
What's produced:
- .rlib files (Rust library archives)
- .rmeta files (Rust metadata)

Example Rust Build Output

🔨 Running Rust build test... (Rust build testing available with rust 1.91.0 (r1910))
✓ Build test passed
  Artifacts produced:
  Libraries: libserde-e092d98ee6dac194.rlib, libserde.rlib
  Rust metadata: libserde-e092d98ee6dac194.rmeta, metadata.rmeta

Manual Rust Build Testing

Test Rust builds manually:

cd /opt/compiler-explorer/infra

# List installed Rust compilers
bin/ce_install --filter-match-all list --installed-only --show-compiler-ids --json rust '!nightly'

# Run a Rust build test
bin/ce_install --debug --dry-run --keep-staging build --temp-install --buildfor r1910 'libraries/rust/serde 1.0.219'

# Check the staging directory for artifacts
find /tmp/ce-cefs-temp/staging/*/r1910_*/build/debug -name "*.rlib"

Fortran Build Testing

Understanding Fortran Build Tests

The --build-test option also works for Fortran libraries:

What it does:
- Detects installed Fortran compilers (prefers gfortran from gcc)
- Downloads and builds the FPM package
- Compiles source into static libraries and module files
- Verifies .a and .mod files are produced
Requirements:
- gfortran (included with gcc) or another Fortran compiler via ce_install
What's produced:
- Static library archives (.a files, e.g., libjson-fortran.a)
- Fortran module files (.mod files for use statements)
Compiler preference:
- gfortran (from gcc) - most compatible, preferred
- LFortran - modern Fortran compiler
- Intel Fortran - fallback option

Example Fortran Build Output

🔨 Running Fortran build test... (Fortran build testing available with gfortran (gcc 15.2.0) 15.2.0 (g152))
✓ Build test passed
  Artifacts produced:
  Libraries: libjson-fortran.a
  Fortran modules: json_file_module.mod, json_kinds.mod, json_module.mod, ...

Manual Fortran Build Testing

Test Fortran builds manually:

cd /opt/compiler-explorer/infra

# gfortran comes with gcc - use a gcc compiler ID
bin/ce_install --filter-match-all list --installed-only --show-compiler-ids --json gcc '!cross'

# Run a Fortran build test using gcc compiler ID (includes gfortran)
bin/ce_install --debug --keep-staging build --temp-install --buildfor g152 'libraries/fortran/json_fortran 8.3.0'

# Check the staging directory for Fortran sources
find /tmp/ce-cefs-temp/staging/ -name "*.f90" -o -name "*.F90"

Working with Properties Files

Understanding Property File Structure

C++ Properties (c++.amazon.properties):

libs.fmt.versions=10.2.1:10.2.0:10.1.1
libs.fmt.url.10.2.1=https://github.com/fmtlib/fmt/archive/refs/tags/10.2.1.tar.gz
libs.fmt.path.10.2.1=/opt/compiler-explorer/libs/fmt/10.2.1

Rust Properties (rust.amazon.properties):

libs.serde.versions=1.0.195:1.0.194
libs.serde.crate=serde
libs.serde.dependencies=serde_derive

Fortran Properties (fortran.amazon.properties):

libs.json-fortran.url=https://github.com/jacobwilliams/json-fortran
libs.json-fortran.versions=8.5.0:8.4.0

Note: CE auto-discovers libraries from libs.<id>.* property keys, so no libs= header line is needed.

Debugging Techniques

Verbose Git Operations

# See all Git commands
export GIT_TRACE=1
export GIT_CURL_VERBOSE=1
./run.sh --debug ...

Pinning Repositories to a Specific Commit

When debugging, you may want to test against a specific version of the infra or compiler-explorer repo rather than the latest. Use the CE_DEBUG_PIN_<REPO>_COMMIT environment variable to pin a repo to a specific commit after sync:

# Pin infra to a specific commit
CE_DEBUG_PIN_INFRA_COMMIT=abc123 ./run.sh --lang=c++ --lib=https://github.com/google/re2 --ver=2025-07-17

# Pin compiler-explorer to a specific commit
CE_DEBUG_PIN_COMPILER_EXPLORER_COMMIT=def456 ./run.sh --lang=rust --lib=serde --ver=1.0.219

# Pin both repos
CE_DEBUG_PIN_INFRA_COMMIT=abc123 CE_DEBUG_PIN_COMPILER_EXPLORER_COMMIT=def456 ./run.sh ...

The repo name in the variable is uppercased with hyphens replaced by underscores (e.g. compiler-explorer becomes COMPILER_EXPLORER).

Inspecting Temporary Files

# Keep temp directory after exit for debugging
./run.sh --keep-temp --lang=rust --lib=serde --ver=1.0.195

# Find temp directory
ls -la /tmp/ce-lib-wizard-*

# Explore the cloned repositories
cd /tmp/ce-lib-wizard-*/compiler-explorer
ls -la

cd /tmp/ce-lib-wizard-*/infra
ls -la

Manual ce_install Operations

# Set up environment manually
cd /tmp/ce-lib-wizard-*/infra
make ce

# Run ce_install commands
poetry run ce_install list
poetry run ce_install add-crate serde 1.0.195

Automation Options

Non-Interactive Mode

Skip all confirmation prompts with -y or --yes:

# Automatically proceed without confirmation
./run.sh -y --lang=rust --lib=serde --ver=1.0.219

# Combine with other options for full automation
./run.sh -y --build-test=no --lang=c++ --lib=https://github.com/fmtlib/fmt --ver=10.2.1

Dry Run Mode

Preview changes without committing or creating PRs:

# See what would be changed
./run.sh --dry-run --lang=rust --lib=serde --ver=1.0.219

# Combine with build test to validate everything
./run.sh --dry-run --build-test=yes --lang=c++ --lib=https://github.com/madler/zlib --ver=1.3.1

Dry run mode:

Clones repositories and makes changes locally
Shows diff of all changes that would be made
Does NOT commit changes
Does NOT create pull requests
Useful for validating configuration before actual run

Customization Options

Environment Variables

# Enable verbose Poetry output
export DEBUG=1
# or
export CE_DEBUG=1

# Use personal access token
export GITHUB_TOKEN=your_token_here

# Custom OAuth app credentials (optional)
export CE_GITHUB_CLIENT_ID=your_client_id
export CE_GITHUB_CLIENT_SECRET=your_client_secret

# Pin a repo to a specific commit for debugging (see Debugging section)
export CE_DEBUG_PIN_INFRA_COMMIT=abc123
export CE_DEBUG_PIN_COMPILER_EXPLORER_COMMIT=def456

Custom Fork Names

If you have non-standard fork names:

# The tool will detect existing forks automatically
# But you can pre-create with custom names:
gh repo fork compiler-explorer/compiler-explorer --clone=false --fork-name=my-ce-fork

Working with Branches

Advanced Library Configuration

C++ Library Types

Understanding library type detection:

packaged-headers: Has CMakeLists.txt
- Built and installed during CE setup
- Headers + compiled libraries
header-only: No CMakeLists.txt
- Headers copied directly
- No compilation needed

Forcing Library Type

Override automatic detection:

# Force header-only even if CMakeLists.txt exists
./run.sh --lang=c++ --lib=URL --ver=VERSION --type=header-only

Complex Version Patterns

CE supports various version formats:

Semantic: 1.2.3, v1.2.3
Date-based: 2024.01.15
Git refs: main, develop
Commits: abc123def456

Performance Optimization

Parallel Repository Operations

The tool clones repositories in parallel:

# This happens automatically, but you can verify:
with GitManager() as git_mgr:
    # Both repos clone simultaneously
    main_repo, infra_repo = git_mgr.clone_repositories()

Shallow Clones

The tool uses shallow clones for performance:

git clone --depth 1 --single-branch

Troubleshooting Complex Scenarios

Recovering from Partial Runs

If the tool fails partway:

Check temp directory for state
Manually complete missing steps
Or clean up and retry

Testing Against Local Repositories

For development, test against local clones:

# Create test repos
mkdir ~/ce-test
cd ~/ce-test
git clone https://github.com/compiler-explorer/compiler-explorer
git clone https://github.com/compiler-explorer/infra

# Modify GitManager to use local paths (development only)

Integration with CE Infrastructure

Understanding the Build Pipeline

PR Created → CI runs tests
Maintainer reviews → Approves
Merge → Deployment pipeline
Library available on compiler-explorer.com

Testing Your Library

After PRs are merged:

Visit https://compiler-explorer.com
Select your language
Click libraries dropdown
Find your library

Monitoring Deployment

Watch the deployment:

Check GitHub Actions on the main repo
Monitor pull request status and CI results

Best Practices

Version Selection

Use stable releases, not development branches
Prefer tagged versions over commit hashes
Test the version locally first

PR Descriptions

Good PR descriptions include:

Link to library homepage
Brief description of what it does
Why it's useful for CE users
Any special requirements

Library Naming

Use lowercase with underscores
Match the common name of the library
Be consistent with existing patterns

Remember: The tool handles the complex mechanics, but understanding these advanced features helps you troubleshoot issues and customize behavior for special cases.

FilesExpand file tree

advanced-usage.md

Latest commit

History