Skip to content

Update Deployment Workflows #14

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
cabutlermit merged 1 commit into from
Sep 29, 2025
Merged

Update Deployment Workflows #14

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
48 changes: 42 additions & 6 deletions .github/workflows/dev-build.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,7 +1,10 @@
### This is the Terraform-generated dev-build.yml workflow for the marimo-launcher-dev app repository ###
### If this is a Lambda repo, uncomment the FUNCTION line at the end of the document ###
### If the container requires any additional pre-build commands, uncomment and edit ###
### the PREBUILD line at the end of the document. ###
### This is the Terraform-generated dev-build.yml workflow for the ###
### marimo-launcher-dev app repository. ###
### If this is a Lambda repo, uncomment the FUNCTION line at the end of ###
### the document. If the container requires any additional pre-build ###
### commands, uncomment and edit the PREBUILD line at the end of the ###
### document. ###

name: Dev Container Build and Deploy
on:
workflow_dispatch:
Expand All @@ -11,14 +14,47 @@ on:
paths-ignore:
- '.github/**'

permissions:
id-token: write
contents: read

jobs:
prep:
name: Prep for Build
runs-on: ubuntu-latest
outputs:
cpuarch: ${{ steps.setarch.outputs.cpuarch }}
steps:
- name: Checkout
uses: actions/checkout@v5

- name: Set CPU Architecture
id: setarch
run: |
echo "### :abacus: Architecture Selection" >> $GITHUB_STEP_SUMMARY
if [[ -f .aws-architecture ]]; then
ARCH=$(cat .aws-architecture)
echo "\`$ARCH\` was read from \`.aws-architecture\` and passed to the deploy job." >> $GITHUB_STEP_SUMMARY
else
ARCH="linux/amd64"
echo "No \`.aws-architecture\` file, so default \`$ARCH\` was passed to the deploy job." >> $GITHUB_STEP_SUMMARY
fi
if [[ "$ARCH" != "linux/arm64" && "$ARCH" != "linux/amd64" ]]; then
echo "$ARCH is INVALID architecture!"
echo "$ARCH is INVALID architecture!" >> $GITHUB_STEP_SUMMARY
exit 1
fi
echo "cpuarch=$ARCH" >> $GITHUB_OUTPUT

deploy:
name: Dev Container Deploy
uses: mitlibraries/.github/.github/workflows/ecr-shared-deploy-dev.yml@main
needs: prep
name: Dev Deploy
uses: mitlibraries/.github/.github/workflows/ecr-multi-arch-deploy-dev.yml@main
secrets: inherit
with:
AWS_REGION: "us-east-1"
GHA_ROLE: "marimo-launcher-gha-dev"
ECR: "marimo-launcher-dev"
CPU_ARCH: ${{ needs.prep.outputs.cpuarch }}
# FUNCTION: ""
# PREBUILD:
44 changes: 40 additions & 4 deletions .github/workflows/prod-promote.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,21 +1,57 @@
### This is the Terraform-generated prod-promote.yml workflow for the marimo-launcher-prod repository. ###
### If this is a Lambda repo, uncomment the FUNCTION line at the end of the document. ###
### This is the Terraform-generated prod-promote.yml workflow for the ###
### marimo-launcher-prod repository. ###
### If this is a Lambda repo, uncomment the FUNCTION line at the end of ###
### the document. ###

name: Prod Container Promote
on:
workflow_dispatch:
release:
types: [published]

permissions:
id-token: write
contents: read

jobs:
prep:
name: Prep for Promote
runs-on: ubuntu-latest
outputs:
cpuarch: ${{ steps.setarch.outputs.cpuarch }}
steps:
- name: Checkout
uses: actions/checkout@v5

- name: Set CPU Architecture
id: setarch
run: |
echo "### :abacus: Architecture Selection" >> $GITHUB_STEP_SUMMARY
if [[ -f .aws-architecture ]]; then
ARCH=$(cat .aws-architecture)
echo "\`$ARCH\` was read from \`.aws-architecture\` and passed to the deploy job." >> $GITHUB_STEP_SUMMARY
else
ARCH="linux/amd64"
echo "No \`.aws-architecture\` file, so default \`$ARCH\` was passed to the deploy job." >> $GITHUB_STEP_SUMMARY
fi
if [[ "$ARCH" != "linux/arm64" && "$ARCH" != "linux/amd64" ]]; then
echo "$ARCH is INVALID architecture!"
echo "$ARCH is INVALID architecture!" >> $GITHUB_STEP_SUMMARY
exit 1
fi
echo "cpuarch=$ARCH" >> $GITHUB_OUTPUT

deploy:
name: Prod Container Promote
uses: mitlibraries/.github/.github/workflows/ecr-shared-promote-prod.yml@main
needs: prep
name: Deploy
uses: mitlibraries/.github/.github/workflows/ecr-multi-arch-promote-prod.yml@main
secrets: inherit
with:
AWS_REGION: "us-east-1"
GHA_ROLE_STAGE: marimo-launcher-gha-stage
GHA_ROLE_PROD: marimo-launcher-gha-prod
ECR_STAGE: "marimo-launcher-stage"
ECR_PROD: "marimo-launcher-prod"
CPU_ARCH: ${{ needs.prep.outputs.cpuarch }}
# FUNCTION: ""

48 changes: 42 additions & 6 deletions .github/workflows/stage-build.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,7 +1,10 @@
### This is the Terraform-generated dev-build.yml workflow for the marimo-launcher-stage app repository ###
### If this is a Lambda repo, uncomment the FUNCTION line at the end of the document ###
### If the container requires any additional pre-build commands, uncomment and edit ###
### the PREBUILD line at the end of the document. ###
### This is the Terraform-generated stage-build.yml workflow for the ###
### marimo-launcher-stage app repository. ###
### If this is a Lambda repo, uncomment the FUNCTION line at the end of ###
### the document. If the container requires any additional pre-build ###
### commands, uncomment and edit the PREBUILD line at the end of the ###
### document. ###

name: Stage Container Build and Deploy
on:
workflow_dispatch:
Expand All @@ -11,14 +14,47 @@ on:
paths-ignore:
- '.github/**'

permissions:
id-token: write
contents: read

jobs:
prep:
name: Prep for Build
runs-on: ubuntu-latest
outputs:
cpuarch: ${{ steps.setarch.outputs.cpuarch }}
steps:
- name: Checkout
uses: actions/checkout@v5

- name: Set CPU Architecture
id: setarch
run: |
echo "### :abacus: Architecture Selection" >> $GITHUB_STEP_SUMMARY
if [[ -f .aws-architecture ]]; then
ARCH=$(cat .aws-architecture)
echo "\`$ARCH\` was read from \`.aws-architecture\` and passed to the deploy job." >> $GITHUB_STEP_SUMMARY
else
ARCH="linux/amd64"
echo "No \`.aws-architecture\` file, so default \`$ARCH\` was passed to the deploy job." >> $GITHUB_STEP_SUMMARY
fi
if [[ "$ARCH" != "linux/arm64" && "$ARCH" != "linux/amd64" ]]; then
echo "$ARCH is INVALID architecture!"
echo "$ARCH is INVALID architecture!" >> $GITHUB_STEP_SUMMARY
exit 1
fi
echo "cpuarch=$ARCH" >> $GITHUB_OUTPUT

deploy:
name: Stage Container Deploy
uses: mitlibraries/.github/.github/workflows/ecr-shared-deploy-stage.yml@main
needs: prep
name: Stage Deploy
uses: mitlibraries/.github/.github/workflows/ecr-multi-arch-deploy-stage.yml@main
secrets: inherit
with:
AWS_REGION: "us-east-1"
GHA_ROLE: "marimo-launcher-gha-stage"
ECR: "marimo-launcher-stage"
CPU_ARCH: ${{ needs.prep.outputs.cpuarch }}
# FUNCTION: ""
# PREBUILD:
80 changes: 53 additions & 27 deletions Makefile
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,10 +1,12 @@
SHELL=/bin/bash
DATETIME:=$(shell date -u +%Y%m%dT%H%M%SZ)
### This is the Terraform-generated header for marimo-launcher-dev. ###
ECR_NAME_DEV:=marimo-launcher-dev
ECR_URL_DEV:=222053980223.dkr.ecr.us-east-1.amazonaws.com/marimo-launcher-dev
ECR_NAME_DEV := marimo-launcher-dev
ECR_URL_DEV := 222053980223.dkr.ecr.us-east-1.amazonaws.com/marimo-launcher-dev
CPU_ARCH ?= $(shell cat .aws-architecture 2>/dev/null || echo "linux/amd64")
### End of Terraform-generated header ###


help: # Preview Makefile commands
@awk 'BEGIN { FS = ":.*#"; print "Usage: make <target>\n\nTargets:" } \
/^[-_[:alpha:]]+:.?*#/ { printf " %-15s%s\n", $$1, $$2 }' $(MAKEFILE_LIST)
Expand Down Expand Up @@ -111,30 +113,54 @@ docker-test-run: # Test local docker container with test fixture notebook
# Terraform
####################################

### Terraform-generated Developer Deploy Commands for Dev environment ###
dist-dev: ## Build docker container (intended for developer-based manual build)
docker build --platform linux/amd64 \
-t $(ECR_URL_DEV):latest \
-t $(ECR_URL_DEV):`git describe --always` \
-t $(ECR_NAME_DEV):latest .
### Terraform-generated Developer Deploy Commands for Dev environment ###
check-arch:
Copy link
Copy Markdown
Contributor

@ghukill ghukill Sep 29, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I like this check-arch command. After some cycles with all this, might be neat to weave this into either pre-commit, or make lint, or something.

@ARCH_FILE=".aws-architecture"; \
if [[ "$(CPU_ARCH)" != "linux/amd64" && "$(CPU_ARCH)" != "linux/arm64" ]]; then \
echo "Invalid CPU_ARCH: $(CPU_ARCH)"; exit 1; \
fi; \
if [[ -f $$ARCH_FILE ]]; then \
echo "latest-$(shell echo $(CPU_ARCH) | cut -d'/' -f2)" > .arch_tag; \
else \
echo "latest" > .arch_tag; \
fi

dist-dev: check-arch ## Build docker container (intended for developer-based manual build)
@ARCH_TAG=$$(cat .arch_tag); \
docker buildx inspect $(ECR_NAME_DEV) >/dev/null 2>&1 || docker buildx create --name $(ECR_NAME_DEV) --use; \
Copy link
Copy Markdown
Contributor

@ghukill ghukill Sep 29, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@cabutlermit - so you did end up going with buildx? Am I correct in remembering you were kind of waffling between approaches for a bit? Sounds like maybe this approach won out?

Copy link
Copy Markdown
Contributor Author

@cabutlermit cabutlermit Sep 29, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

yeah, I still ended up withdocker buildx in the end for the Makefile commands (the shared workflows only use docker build). I kept buildx in the Makefile, mostly because I wanted to ensure that the make commands would build either AMD64 or ARM64 regardless of the CPU architecture of the developer's machine.

docker buildx use $(ECR_NAME_DEV); \
docker buildx build --platform $(CPU_ARCH) \
--load \
--tag $(ECR_URL_DEV):$$ARCH_TAG \
--tag $(ECR_URL_DEV):make-$$ARCH_TAG \
--tag $(ECR_URL_DEV):make-$(shell git describe --always) \
--tag $(ECR_NAME_DEV):$$ARCH_TAG \
.

publish-dev: dist-dev ## Build, tag and push (intended for developer-based manual publish)
docker login -u AWS -p $$(aws ecr get-login-password --region us-east-1) $(ECR_URL_DEV)
docker push $(ECR_URL_DEV):latest
docker push $(ECR_URL_DEV):`git describe --always`

### Terraform-generated manual shortcuts for deploying to Stage. This requires ###
### that ECR_NAME_STAGE, ECR_URL_STAGE, and FUNCTION_STAGE environment ###
### variables are set locally by the developer and that the developer has ###
### authenticated to the correct AWS Account. The values for the environment ###
### variables can be found in the stage_build.yml caller workflow. ###
dist-stage: ## Only use in an emergency
docker build --platform linux/amd64 \
-t $(ECR_URL_STAGE):latest \
-t $(ECR_URL_STAGE):`git describe --always` \
-t $(ECR_NAME_STAGE):latest .

publish-stage: ## Only use in an emergency
docker login -u AWS -p $$(aws ecr get-login-password --region us-east-1) $(ECR_URL_STAGE)
docker push $(ECR_URL_STAGE):latest
docker push $(ECR_URL_STAGE):`git describe --always`
@ARCH_TAG=$$(cat .arch_tag); \
aws ecr get-login-password --region us-east-1 | docker login --username AWS --password-stdin $(ECR_URL_DEV); \
docker push $(ECR_URL_DEV):$$ARCH_TAG; \
docker push $(ECR_URL_DEV):make-$$ARCH_TAG; \
docker push $(ECR_URL_DEV):make-$(shell git describe --always); \
echo "Cleaning up dangling Docker images..."; \
docker image prune -f --filter "dangling=true"


### If this is a Lambda repo, uncomment the two lines below ###
# update-lambda-dev: ## Updates the lambda with whatever is the most recent image in the ecr (intended for developer-based manual update)
# @ARCH_TAG=$$(cat .arch_tag); \
# aws lambda update-function-code \
# --region us-east-1 \
# --function-name $(FUNCTION_DEV) \
# --image-uri $(ECR_URL_DEV):make-$$ARCH_TAG

docker-clean: ## Clean up Docker detritus
@ARCH_TAG=$$(cat .arch_tag); \
echo "Cleaning up Docker leftovers (containers, images, builders)"; \
docker rmi -f $(ECR_URL_DEV):$$ARCH_TAG; \
docker rmi -f $(ECR_URL_DEV):make-$$ARCH_TAG; \
docker rmi -f $(ECR_URL_DEV):make-$(shell git describe --always) || true; \
docker rmi -f $(ECR_NAME_DEV):$$ARCH_TAG || true; \
docker buildx rm $(ECR_NAME_DEV) || true
@rm -rf .arch_tag
13 changes: 7 additions & 6 deletions README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -34,6 +34,7 @@ More information about structuring notebooks and dependencies below in "Preparin
## Preparing Notebooks

### Notebook Location

This CLI expects two primary things to discover the notebook to launch:

1. The root directory of the notebook project (either mounted or a cloned Github repository)
Expand Down Expand Up @@ -84,14 +85,13 @@ There are many ways to create this file, [`uv export` is worth consideration](ht

The `Makefile` command `cli-test-reqs-txt-run` will demonstrate this.


## Environment Variables

### Required

```shell
SENTRY_DSN=### If set to a valid Sentry DSN, enables Sentry exception monitoring. This is not needed for local development.
WORKSPACE=### Set to `dev` for local development, this will be set to `stage` and `prod` in those environments by Terraform.
SENTRY_DSN= ### If set to a valid Sentry DSN, enables Sentry exception monitoring. This is not needed for local development.
WORKSPACE= ### Set to `dev` for local development, this will be set to `stage` and `prod` in those environments by Terraform.
```

### Optional
Expand All @@ -102,15 +102,14 @@ Set these if you want to override defaults or pass values via env instead of fla
NOTEBOOK_REPOSITORY= ### repository to clone that contains a notebook and any required assets
NOTEBOOK_REPOSITORY_BRANCH= ### optional branch to checkout on clone
NOTEBOOK_MOUNT= ### either local of Docker context, an accessible root directory that contains notebook(s)
NOTEBOOK_PATH=### Relative path of actual notebook .py file based on cloned repository or mounted directory; defaults to "notebook.py"
NOTEBOOK_PATH= ### Relative path of actual notebook .py file based on cloned repository or mounted directory; defaults to "notebook.py"
NOTEBOOK_REQUIREMENTS= ### filepath to install dependencies from, relative to notebook root; if unset assuming dependencies are inline in notebook

NOTEBOOK_MODE= ### how to launch marimo: "run" to execute, "edit" to open the editor; default "run"
NOTEBOOK_HOST= ### host to bind running notebook to
NOTEBOOK_PORT= ### port to serve running notebook on
```


## CLI Commands

### `launcher`
Expand Down Expand Up @@ -159,6 +158,8 @@ Options:
--help Show this message and exit.
```

## Building for AWS

This application is designed to also run as a Fargate container in AWS. To build and deploy the container to AWS ECR in our Development Account, use the `make publish-dev` command (you must be authenticated to AWS on the CLI first). The infrastructure to run the container (e.g., an ECS Task Definition, Cluster, and Service) are already in place in AWS, so you can easily launch the service via the AWS Console or via the Lambda function trigger.


The infrastructure can support a container built on either `amd64` or `arm64` CPU architecture. There should be a file named `.aws-architecture` at the root of this repository that contains the default architecture for any builds for AWS, either `linux/amd64` or `linux/arm64`. IF the file does not exist, the builds from either the `Makefile` command or the automated deployment with GitHub Actions will default to `amd64`.