feat: tarball build pipeline for process injection by deanq · Pull Request #75 · runpod-workers/flash

deanq · 2026-03-06T20:17:16Z

Summary

Add self-contained runtime tarball build pipeline (build-tarball.sh, bootstrap.sh)
Add Makefile tarball/tarball-test/tarball-test-local targets
Add CI tarball job on every PR push (artifact + PR comment with link)
Add release-tarball.yml workflow for release uploads
Unify all base images on Python 3.11 (matching PyTorch runtime)
Fix VERSION regex and add --platform linux/amd64 for Apple Silicon testing
Update dependency_installer for tarball-aware install paths

Test plan

make tarball-test passes locally (Python 3.11, 3.12, 3.13 verified)
make tarball-test-local passes
CI tarball job produces artifact and posts PR comment
FLASH_WORKER_TARBALL_URL=<artifact-url> flash deploy works end-to-end

Add self-contained runtime tarball that can be injected into any base image via dockerArgs, replacing the need for pre-built Docker images. - Add build-tarball.sh using uv for Python version management - Add bootstrap.sh entry point for injected runtime - Add Makefile tarball/tarball-test/tarball-test-local targets - Add release-tarball.yml CI workflow triggered on release - Unify all base images on Python 3.11 (matching PyTorch runtime) - Fix VERSION regex to handle spaces around = in version.py - Add --platform linux/amd64 to test targets for Apple Silicon - Update dependency_installer for tarball-aware install paths - Add tarball constants (TARBALL_URL_TEMPLATE, TARBALL_INSTALL_DIR)

Copilot

Pull request overview

Adds a self-contained “tarball mode” runtime for process-injection deployments, plus CI/release automation to build, test, and publish the tarball. This also aligns Docker base images to Python 3.11 and updates dependency installation logic to support the tarball layout.

Changes:

Add tarball build + bootstrap scripts and Makefile targets to build/test a self-contained flash-worker runtime.
Add CI job to build/test/upload tarball on PRs and a reusable workflow to upload tarballs on releases.
Update DependencyInstaller to install user deps into the tarball venv when FLASH_WORKER_INSTALL_DIR is set; bump locked dependencies accordingly.

Reviewed changes

Copilot reviewed 10 out of 11 changed files in this pull request and generated 10 comments.

Show a summary per file

File	Description
`uv.lock`	Updates locked dependency versions (incl. FastAPI and other runtime deps) for the new build pipeline/runtime alignment.
`tests/unit/test_dependency_installer.py`	Adds unit tests for tarball-mode dependency installation behavior.
`src/dependency_installer.py`	Adds tarball-mode install path selection via `FLASH_WORKER_INSTALL_DIR`.
`src/constants.py`	Introduces `FLASH_WORKER_INSTALL_DIR_ENV` constant for tarball-mode signaling.
`scripts/build-tarball.sh`	Implements the self-contained tarball build (Python + uv + venv + deps + sources).
`scripts/bootstrap.sh`	Adds runtime bootstrap entrypoint and `--test` self-check for the tarball.
`Makefile`	Adds `tarball`, `tarball-test`, and `tarball-test-local` targets to build/test tarballs in containers.
`Dockerfile-cpu`	Unifies CPU image base to Python 3.11.
`Dockerfile-lb-cpu`	Unifies LB CPU image base to Python 3.11.
`.github/workflows/release-tarball.yml`	Adds workflow to build/test and upload tarball to GitHub Releases (or as artifact for dry runs).
`.github/workflows/ci.yml`	Adds PR tarball job: build/test/upload artifact + PR comment with link; adds tarball release workflow invocation.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

You can also share your feedback on Copilot code review. Take the survey.

Copilot · 2026-03-06T21:37:12Z

src/dependency_installer.py

+        if self._fw_install_dir:
+            # Tarball mode: install user deps into the flash-worker venv using bundled uv
+            uv_bin = os.path.join(self._fw_install_dir, "uv")
+            venv_python = os.path.join(self._fw_install_dir, "venv", "bin", "python")


Tarball mode assumes the directory pointed to by FLASH_WORKER_INSTALL_DIR exists and contains uv and the venv Python, but there’s no validation. If the env var is set incorrectly, installs will fail with a generic subprocess error; consider checking that uv_bin and venv_python exist and returning a clearer FunctionResponse (or ignoring tarball mode if invalid).

Suggested change

venv_python = os.path.join(self._fw_install_dir, "venv", "bin", "python")

venv_python = os.path.join(self._fw_install_dir, "venv", "bin", "python")

# Validate tarball layout before attempting installation to avoid opaque subprocess errors

missing_paths = []

if not os.path.isdir(self._fw_install_dir):

missing_paths.append(f"install directory '{self._fw_install_dir}'")

if not os.path.isfile(uv_bin):

missing_paths.append(f"uv binary '{uv_bin}'")

if not os.path.isfile(venv_python):

missing_paths.append(f"venv python '{venv_python}'")

if missing_paths:

error_msg = (

"Tarball mode is enabled via environment variable "

f"{FLASH_WORKER_INSTALL_DIR_ENV}={self._fw_install_dir}, "

"but the following required paths are missing or invalid: "

+ ", ".join(missing_paths)

)

self.logger.error(error_msg)

return FunctionResponse(success=False, error=error_msg)

Copilot · 2026-03-06T21:37:12Z

scripts/build-tarball.sh

+BUILD_DIR="/tmp/flash-worker-build"
+TARBALL_ROOT="$BUILD_DIR/flash-worker"
+OUTPUT_DIR="$REPO_ROOT/dist"
+TARBALL_NAME="flash-worker-v${VERSION}-py${PYTHON_VERSION}-linux-x86_64.tar.gz"
+
+# Clean previous build
+rm -rf "$BUILD_DIR"


The build uses a fixed BUILD_DIR=/tmp/flash-worker-build and unconditionally rm -rf’s it. This can break concurrent builds on the same machine and can delete unrelated data if the variable is ever changed unexpectedly; consider using mktemp -d (and a trap cleanup) to make the build directory unique and safer.

Suggested change

BUILD_DIR="/tmp/flash-worker-build"

TARBALL_ROOT="$BUILD_DIR/flash-worker"

OUTPUT_DIR="$REPO_ROOT/dist"

TARBALL_NAME="flash-worker-v${VERSION}-py${PYTHON_VERSION}-linux-x86_64.tar.gz"

# Clean previous build

rm -rf "$BUILD_DIR"

BUILD_DIR="$(mktemp -d -t flash-worker-build.XXXXXX)"

TARBALL_ROOT="$BUILD_DIR/flash-worker"

OUTPUT_DIR="$REPO_ROOT/dist"

TARBALL_NAME="flash-worker-v${VERSION}-py${PYTHON_VERSION}-linux-x86_64.tar.gz"

cleanup() {

if [ -n "${BUILD_DIR:-}" ] && [ -d "$BUILD_DIR" ]; then

rm -rf "$BUILD_DIR"

fi

}

trap cleanup EXIT

Copilot · 2026-03-06T21:37:13Z

scripts/build-tarball.sh

+    tar xzf "$OUTPUT_DIR/.cache/uv-${UV_VERSION}.tar.gz" -C "$TARBALL_ROOT" --no-same-owner --strip-components=1 "uv-x86_64-unknown-linux-gnu/uv" 2>/dev/null || true
+else
+    curl -fsSL "$UV_URL" -o "$OUTPUT_DIR/.cache/uv-${UV_VERSION}.tar.gz"
+    tar xzf "$OUTPUT_DIR/.cache/uv-${UV_VERSION}.tar.gz" -C "$TARBALL_ROOT" --no-same-owner --strip-components=1 "uv-x86_64-unknown-linux-gnu/uv" 2>/dev/null || true
+fi


tar ... 2>/dev/null || true suppresses extraction failures for the bundled uv, which can make subsequent failures (e.g., at chmod +x) harder to diagnose. Consider removing || true/stderr suppression and explicitly erroring if the expected uv binary isn’t extracted.

Suggested change

tar xzf "$OUTPUT_DIR/.cache/uv-${UV_VERSION}.tar.gz" -C "$TARBALL_ROOT" --no-same-owner --strip-components=1 "uv-x86_64-unknown-linux-gnu/uv" 2>/dev/null || true

else

curl -fsSL "$UV_URL" -o "$OUTPUT_DIR/.cache/uv-${UV_VERSION}.tar.gz"

tar xzf "$OUTPUT_DIR/.cache/uv-${UV_VERSION}.tar.gz" -C "$TARBALL_ROOT" --no-same-owner --strip-components=1 "uv-x86_64-unknown-linux-gnu/uv" 2>/dev/null || true

fi

tar xzf "$OUTPUT_DIR/.cache/uv-${UV_VERSION}.tar.gz" -C "$TARBALL_ROOT" --no-same-owner --strip-components=1 "uv-x86_64-unknown-linux-gnu/uv"

else

curl -fsSL "$UV_URL" -o "$OUTPUT_DIR/.cache/uv-${UV_VERSION}.tar.gz"

tar xzf "$OUTPUT_DIR/.cache/uv-${UV_VERSION}.tar.gz" -C "$TARBALL_ROOT" --no-same-owner --strip-components=1 "uv-x86_64-unknown-linux-gnu/uv"

fi

if [ ! -f "$TARBALL_ROOT/uv" ]; then

echo "ERROR: Failed to extract uv binary to $TARBALL_ROOT/uv"

exit 1

fi

Copilot · 2026-03-06T21:37:13Z

scripts/build-tarball.sh

+echo "Copying source files..."
+cp -r "$REPO_ROOT/src/"*.py "$TARBALL_ROOT/src/" 2>/dev/null || true
+mkdir -p "$TARBALL_ROOT/src"
+for f in "$REPO_ROOT/src/"*.py; do
+    [ -f "$f" ] && cp "$f" "$TARBALL_ROOT/src/"
+done


cp -r "$REPO_ROOT/src/"*.py "$TARBALL_ROOT/src/" runs before $TARBALL_ROOT/src is created and its failure is ignored, and then the script copies the same files again in a loop. Creating the destination directory first and using a single copy mechanism would avoid silent failures and reduce duplication.

Copilot · 2026-03-06T21:37:13Z

Makefile

+# Tarball targets (process-injectable runtime)
+tarball: # Build self-contained runtime tarball (runs in Docker, linux/amd64)
+	docker run --rm --platform linux/amd64 \
+		-e PYTHON_VERSION=$(TARBALL_PYTHON_VERSION) \
+		-e UV_CACHE_DIR=/workspace/dist/.uv-cache \
+		-v $(PWD):/workspace -w /workspace \
+		python:3.11-slim \
+		bash -c 'apt-get update -qq && apt-get install -y -qq curl > /dev/null 2>&1 && pip install uv -q && bash scripts/build-tarball.sh'
+


make tarball installs uv inside the Docker container, but this Makefile also has a global parse-time check that errors if uv isn’t installed on the host. That makes the tarball targets unusable on machines without host uv; consider moving the uv presence check into only the targets that require host uv, or gating it behind a variable for Docker-only targets.

Copilot · 2026-03-06T21:37:13Z

.github/workflows/ci.yml

+          TARBALL=$(basename dist/flash-worker-v*-py3.11-linux-x86_64.tar.gz)
+          RUN_URL="${{ github.server_url }}/${{ github.repository }}/actions/runs/${{ github.run_id }}"
+          BODY="**Tarball artifact:** [\`${TARBALL}\`](${RUN_URL}#artifacts)
+
+          To test: \`FLASH_WORKER_TARBALL_URL=<download-url> flash deploy\`"


TARBALL=$(basename dist/flash-worker-v*-py3.11-linux-x86_64.tar.gz) can expand to multiple paths; basename will then output multiple lines and the PR comment link text becomes ambiguous. Consider selecting a single artifact deterministically (or failing if more than one match) before composing the comment body.

Copilot · 2026-03-06T21:37:14Z

tests/unit/test_dependency_installer.py

+    def test_tarball_mode_uses_bundled_uv(self):
+        """Test tarball mode uses the bundled uv and venv python."""


This test/docstring says it verifies the bundled uv and venv python are used, but it only asserts _fw_install_dir. Either update the docstring/name to match what’s asserted, or extend the test to validate the actual install command (similar to test_tarball_mode_install_command).

Suggested change

def test_tarball_mode_uses_bundled_uv(self):

"""Test tarball mode uses the bundled uv and venv python."""

def test_tarball_mode_sets_fw_install_dir(self):

"""Test tarball mode sets _fw_install_dir from FLASH_WORKER_INSTALL_DIR."""

Copilot · 2026-03-06T21:37:14Z

.github/workflows/release-tarball.yml

+          TARBALL=$(ls dist/flash-worker-v*-py3.11-linux-x86_64.tar.gz)
+          docker run --rm -v "$(pwd)/dist:/dist" ubuntu:22.04 \
+            bash -c "tar xzf /dist/$(basename $TARBALL) -C /opt && /opt/flash-worker/bootstrap.sh --test"
+
+      - name: Upload tarball to GitHub Release
+        if: inputs.dry_run != 'true' && inputs.tag_name != ''
+        env:
+          GH_TOKEN: ${{ github.token }}
+        run: |
+          TARBALL=$(ls dist/flash-worker-v*-py3.11-linux-x86_64.tar.gz)


Similar to CI, TARBALL=$(ls dist/flash-worker-v*-py3.11-linux-x86_64.tar.gz) + $(basename $TARBALL) assumes exactly one match. If multiple artifacts exist (or no match), this step can behave unpredictably; consider enforcing a single match and quoting variables in the docker command.

Suggested change

TARBALL=$(ls dist/flash-worker-v*-py3.11-linux-x86_64.tar.gz)

docker run --rm -v "$(pwd)/dist:/dist" ubuntu:22.04 \

bash -c "tar xzf /dist/$(basename $TARBALL) -C /opt && /opt/flash-worker/bootstrap.sh --test"

- name: Upload tarball to GitHub Release

if: inputs.dry_run != 'true' && inputs.tag_name != ''

env:

GH_TOKEN: ${{ github.token }}

run: |

TARBALL=$(ls dist/flash-worker-v*-py3.11-linux-x86_64.tar.gz)

shopt -s nullglob

tarballs=(dist/flash-worker-v*-py3.11-linux-x86_64.tar.gz)

if [ "${#tarballs[@]}" -ne 1 ]; then

echo "Expected exactly one tarball in dist/, found ${#tarballs[@]} matches."

exit 1

fi

TARBALL="${tarballs[0]}"

docker run --rm -v "$(pwd)/dist:/dist" ubuntu:22.04 \

bash -c "tar xzf /dist/$(basename \"$TARBALL\") -C /opt && /opt/flash-worker/bootstrap.sh --test"

- name: Upload tarball to GitHub Release

if: inputs.dry_run != 'true' && inputs.tag_name != ''

env:

GH_TOKEN: ${{ github.token }}

run: |

shopt -s nullglob

tarballs=(dist/flash-worker-v*-py3.11-linux-x86_64.tar.gz)

if [ "${#tarballs[@]}" -ne 1 ]; then

echo "Expected exactly one tarball in dist/, found ${#tarballs[@]} matches."

exit 1

fi

TARBALL="${tarballs[0]}"

Copilot · 2026-03-06T21:37:14Z

scripts/build-tarball.sh

+# 1. Install Python via uv (handles version resolution and caching)
+echo "Installing Python ${PYTHON_VERSION} via uv..."
+uv python install "$PYTHON_VERSION"
+PYTHON_BIN=$(uv python find --python-preference only-managed "$PYTHON_VERSION")
+PYTHON_INSTALL_DIR=$(cd "$(dirname "$PYTHON_BIN")/.." && pwd -P)
+cp -r "$PYTHON_INSTALL_DIR" "$TARBALL_ROOT/python"


The script pins UV_VERSION for the tarball, but uses whatever uv is on PATH for uv python install and uv export. This can make builds non-reproducible across environments (host uv version != tarball uv). Consider extracting/using the pinned uv binary for all uv operations (or asserting the host uv --version matches UV_VERSION).

Copilot · 2026-03-06T21:37:14Z

.github/workflows/ci.yml

+      - name: Test tarball in bare ubuntu container
+        run: |
+          TARBALL=$(ls dist/flash-worker-v*-py3.11-linux-x86_64.tar.gz)
+          docker run --rm -v "$(pwd)/dist:/dist" ubuntu:22.04 \
+            bash -c "tar xzf /dist/$(basename $TARBALL) -C /opt && /opt/flash-worker/bootstrap.sh --test"
+


In the tarball self-test step, TARBALL=$(ls dist/flash-worker-v*-py3.11-linux-x86_64.tar.gz) and then $(basename $TARBALL) can behave unexpectedly if the glob matches 0 or >1 files (or if filenames contain spaces). Consider using a safer selection pattern (e.g., ensure exactly one match) and quoting variables when passing them into basename/the docker command.

uv pip install stamps absolute build-time paths in console_scripts shebangs (e.g. /tmp/flash-worker-build/.../python), making binaries like uvicorn unfindable at runtime in the extracted tarball. Rewrite any shebang referencing the build dir to #!/usr/bin/env python3 after dependency installation.

github-actions · 2026-03-07T02:19:18Z

Tarball artifact: flash-worker-v1.1.1-py3.11-linux-x86_64.tar.gz

To test: FLASH_WORKER_TARBALL_URL=<download-url> flash deploy

deanq mentioned this pull request Mar 6, 2026

feat: process injection via LiveServerlessMixin runpod/flash#260

Open

4 tasks

deanq requested a review from Copilot March 6, 2026 21:32

Copilot started reviewing on behalf of deanq March 6, 2026 21:33 View session

Copilot AI reviewed Mar 6, 2026

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat: tarball build pipeline for process injection#75

feat: tarball build pipeline for process injection#75
deanq wants to merge 2 commits intomainfrom
deanq/ae-2390-process-injectable-worker

deanq commented Mar 6, 2026

Uh oh!

Copilot AI left a comment

Uh oh!

Copilot AI Mar 6, 2026

Uh oh!

Copilot AI Mar 6, 2026

Uh oh!

Copilot AI Mar 6, 2026

Uh oh!

Copilot AI Mar 6, 2026

Uh oh!

Copilot AI Mar 6, 2026

Uh oh!

Copilot AI Mar 6, 2026

Uh oh!

Copilot AI Mar 6, 2026

Uh oh!

Copilot AI Mar 6, 2026

Uh oh!

Copilot AI Mar 6, 2026

Uh oh!

Copilot AI Mar 6, 2026

Uh oh!

github-actions bot commented Mar 7, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

-            venv_python = os.path.join(self._fw_install_dir, "venv", "bin", "python")
+            venv_python = os.path.join(self._fw_install_dir, "venv", "bin", "python")
+            # Validate tarball layout before attempting installation to avoid opaque subprocess errors
+            missing_paths = []
+            if not os.path.isdir(self._fw_install_dir):
+                missing_paths.append(f"install directory '{self._fw_install_dir}'")
+            if not os.path.isfile(uv_bin):
+                missing_paths.append(f"uv binary '{uv_bin}'")
+            if not os.path.isfile(venv_python):
+                missing_paths.append(f"venv python '{venv_python}'")
+            if missing_paths:
+                error_msg = (
+                    "Tarball mode is enabled via environment variable "
+                    f"{FLASH_WORKER_INSTALL_DIR_ENV}={self._fw_install_dir}, "
+                    "but the following required paths are missing or invalid: "
+                    + ", ".join(missing_paths)
+                )
+                self.logger.error(error_msg)
+                return FunctionResponse(success=False, error=error_msg)

		def test_tarball_mode_uses_bundled_uv(self):
		"""Test tarball mode uses the bundled uv and venv python."""

Conversation

deanq commented Mar 6, 2026

Summary

Test plan

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Copilot AI Mar 6, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Mar 6, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Mar 6, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Mar 6, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Mar 6, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Mar 6, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Mar 6, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Mar 6, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Mar 6, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Mar 6, 2026

Choose a reason for hiding this comment

Uh oh!

github-actions bot commented Mar 7, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants