Local banzai setup for running at site

[timbeccue]

Changes in service of #422

This PR introduces the local setup we'll use for site deployments of banzai, sans calibration caching which will be added in a subsequent PR.

Local Banzai Notes

To run:

docker compose -f docker-compose-site.yml --env-file .site-banzai-env up -d --build

This requires an env file called .site-banzai-env that should look like this:

# .site-banzai-env

# Database Configuration
DB_ADDRESS=sqlite:////data/banzai.db    # Path for the docker container, not the host.
CAL_DB_ADDRESS=""                       # This should be the address to the AWS banzai database where we get calibrations
SITE_ID=

# API Configuration
API_ROOT=https://archive-api.lco.global/
AUTH_TOKEN=

# Data Paths
HOST_DATA_DIR=./site_banzai # this maps to /data in the container, and should contain unprocessed data in a subdirectory `raw`
HOST_PROCESSED_DIR=./site_banzai/output # path where processed data will be saved on the host

# Container Networking
FITS_BROKER=rabbitmq
FITS_BROKER_URL=amqp://rabbitmq:5672
FITS_EXCHANGE=fits_files
TASK_HOST=redis://redis:6379/0

# Celery Configuration
CELERY_TASK_QUEUE_NAME=e2e_task_queue
CELERY_LARGE_TASK_QUEUE_NAME=e2e_large_task_queue

# Worker Configuration
BANZAI_WORKER_LOGLEVEL=debug
OMP_NUM_THREADS=2
OPENTSDB_PYTHON_METRICS_TEST_MODE=1

In order to send images to be processed, run:

python queue_images.py <host_data_dir>/raw

The data to be processed should be in the directory ${HOST_DATA_DIR}/raw. The output will be saved in ./${HOST_PROCESSED_DIR}.

[timbeccue]

Is the prior docker-compose.yml file important to preserve? If not, I'll replace it with my version, currently docker-compose.local.yml

[cmccully]

This feels like it should be in the main.py parse args code rather than the context. The context stuff doesn't really care what we put into the object itself.

[cmccully]

This is where the cal-db-address should be set. default the arg to None and then check for None in this function.

[timbeccue]

I tried your suggestion of removing the cal_db_address fallback from context.py but that caused issues in the e2e tests because some parts of the code (e2e tests and celery workers) don’t use parse_args to set up the context, and were therefore failing to set the cal_db_address fallback to db_address.

Seems like the solution is either

1. add the init logic back to context.py
2. use access patterns like getattr(runtime_context, ‘cal_db_address’, runtime_context.db_address) everywhere cal_db_address is used (messy)
3. require setting cal_db_address explicitly (might break existing banzai setups)
4. get rid of the separate cal_db_address entirely

Any of these fixes is relatively easy to implement but option 4 seems best from an overall complexity standpoint if we are ok with the docker-compose-local.yml setup requiring users to set up their own db from scratch.

[timbeccue]

I've gone ahead with option 1 (adding init logic back to context.py) just to get tests working again

[timbeccue]

Moved the fallback logic from context.py to settings.py

[cmccully]

And the tests build a mock context object right?

[cmccully]

What is the error trying to check? This feels too specific and like you are trying to solve something else here.

[timbeccue]

I think I encountered some files that were missing the s3 url. Something from when I was working on the old cache setup a while back. Unfortunately I'm fuzzy on the details, but here's an example log with a file that prompted this:

2025-10-22 17:36:36.624 | 2025-10-22 21:36:36.620    ERROR:            sync: Unexpected error downloading lsc0m412-sq35-20240315-bpm-central30x30.fits.fz (frameid: 69366609, type: BPM): 'url' | {"processName": "MainProcess"}
...
2025-10-22 17:36:36.624 |     bytes = buffer.write(requests.get(response.json()['url'], stream=True, timeout=60).content)
2025-10-22 17:36:36.624 | KeyError: 'url'

I don't think the code is hitting this file anymore, and I tried running without this block and there were no issues. So maybe best to remove it?

[cmccully]

I think so. If we don't know why it's there, I'm inclined to remove it.

[cmccully]

This feels redundant with the file utils function. What new things does this add?

[timbeccue]

This uses the file path rather than the frameid with the intended use case of wanting to process a file that exists on the local disk. Maybe it would be cleaner to modify the file utils function to accept a frameid or path?

[timbeccue]

I am leaning towards keeping post_to_processing_queue separate from post_to_archive_queue in file utils because different intended workflows (local disk vs s3), and because this is a small function that's pretty limited in scope and maybe not worth setting up the abstraction elsewhere. Let me know if you disagree.

[cmccully]

I think if we keep them both we should rename them to cover what they actually do. I think the post to archive queue originally used paths as well. I'm just not a fan of the code redunancy.

[cmccully]

Some recommendations for cleanup, but overall looks fine.

[cmccully]

You also need to add a change log message and update the pyproject version number.

[timbeccue]

Added the requested changes. Two of your comments I did not change but replied with details; let me know if there's anything else needed for those to be resolved.

[Copilot]

Pull request overview

Adds a self-contained “BANZAI-at-site” local deployment flow (docker-compose + env defaults + helper script) and introduces configuration for using a separate calibration database, along with a new CLI to create/populate a local DB from a remote calibration DB.

Changes:

• Added docker-compose-site.yml, site-banzai-env.default, and queue_images.py to support local/site deployments.
• Introduced CAL_DB_ADDRESS / --calibration-db-address plumbing and a new banzai_create_local_db entrypoint.
• Updated archive download retry behavior (don’t retry missing frames) and updated docs/versioning/deps accordingly.

Reviewed changes

Copilot reviewed 19 out of 21 changed files in this pull request and generated 11 comments.

Show a summary per file

File	Description
site-banzai-env.default	Adds default env vars for site deployments (DB/broker/paths).
queue_images.py	Helper script to enqueue local FITS files via RabbitMQ.
pyproject.toml	Version bump + torch source/marker changes + new console script entrypoint.
poetry.lock	Dependency lock updates (notably torch/CUDA-related packages).
docs/example_reduction.ipynb	Updates example instrument lookup call.
docker-compose.yml	Removes legacy compose configuration.
docker-compose-site.yml	New compose stack for redis/rabbitmq/listener/workers and host volume mappings.
banzai/utils/fits_utils.py	Adds FrameNotAvailableError + retry filtering + request timeouts.
banzai/utils/file_utils.py	Updates RabbitMQ publish helper to support either frameid or path.
banzai/tests/utils.py	Extends FakeContext with cal_db_address.
banzai/tests/test_end_to_end.py	Updates queue publishing call signature + adds cal_db_address into runtime context.
banzai/main.py	Adds env-driven defaults + new --calibration-db-address + new create_local_db CLI.
banzai/frames.py	Routes calibration DB writes via cal_db_address.
banzai/exceptions.py	Adds FrameNotAvailableError exception.
banzai/dbs.py	Adds create_local_db() and helpers to replicate site/instruments from remote DB.
banzai/context.py	Minor formatting change.
banzai/calibrations.py	Uses cal_db_address when fetching master cal record and when stacking input query.
README.rst	Updates docs for new local/site compose workflow + new local DB creation command.
CHANGES.md	Adds 1.28.0 release notes.
.gitignore	Adds ignores for local/site runtime directories and venvs.
.dockerignore	Excludes local/site runtime directories and venvs from docker build context.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[Copilot]

RabbitMQ is started with default credentials, but FITS_BROKER_URL in the env example is amqp://rabbitmq:5672 (i.e., guest/guest). The default guest account is restricted to localhost in RabbitMQ, so other containers (listener/workers) typically cannot authenticate. Configure an explicit user/pass (e.g., via RABBITMQ_DEFAULT_USER / RABBITMQ_DEFAULT_PASS and optionally a vhost) and require FITS_BROKER_URL to include those credentials.

[Copilot]

The torch dependency selection via extra markers looks inconsistent with the resulting poetry.lock (which contains overlapping CPU/CUDA torch entries for the same platforms). One contributing factor is that torch is also listed unconditionally under [project].dependencies, so Poetry may resolve torch outside the conditional [tool.poetry.dependencies] matrix. Consider removing the unconditional torch from [project].dependencies and gating CPU vs CUDA wheels solely via mutually-exclusive markers/extras, then re-generating the lockfile to ensure only one torch variant can match a given environment.

[jchate6]

I have looked at this, and nothing really jumps out at me as being obviously problematic, but I am the first to admit that it isn't clear to me that this will have no impact on normal operations.

I'm also not clear on exactly why we are running banzai at site, and how this will affect our ability to maintain tools and address banzai issues when they arise. There are tools and documentation for checking logs in the current banzai-workers, or re-queuing frames that failed to process. This happens a lot. Is there any coverage in those tools or documentation for what happens if these things break at a site? I know I'm a little late to this party, and it's possible all of these questions have been addressed elsewhere.

[jchate6]

Should this be logged somewhere rather than just printed? This might be helpful for debugging if it's captured somewhere.

[markBowman]

I'm also not clear on exactly why we are running banzai at site, and how this will affect our ability to maintain tools and address banzai issues when they arise. There are tools and documentation for checking logs in the current banzai-workers, or re-queuing frames that failed to process. This happens a lot. Is there any coverage in those tools or documentation for what happens if these things break at a site? I know I'm a little late to this party, and it's possible all of these questions have been addressed elsewhere.

Great question. We already have several pieces of 'flash reduction' at site. Smart stacking, occultation and other observing techniques mean we need more image processing capabilities local to the data. It seems to have reached a threshold where using banzai makes more sense than adding more ad-hoc applications at site. That being said, we will definitely have to address SciOps visibility and capability as the project progresses. It is unlikely to be worse than the current myriad of technical-debt-laced Docker containers, understood' by Steve and nobody else. In most cases there will be very little opportunnity to reprocess data later as the underlying frames are considered volatile - the banzai image is the raw data product to upload to the archive.

[jchate6]

In most cases there will be very little opportunnity to reprocess data later as the underlying frames are considered volatile - the banzai image is the raw data product to upload to the archive.

This seems bad. While the banzai-reduced image is good enough for many purposes, there are still many cases where that reduction fails or isn't what the user requires. Raw images are necessary to recover the data. We get bad flats, or poor solves all the time that have to be re-done either by us or by users. It feels like we are breaking a contract with the user to consider the raw images volatile in "most cases".