README.md

SpectrumX Data System | Gateway

Metadata management and web interface for SDS.

• SpectrumX Data System | Gateway
  • System requirements
  • Development setup
  • Quick deploy
  • Just recipes
  • More SDS Gateway docs

Tip

Deploying SDS in production? Start with the network component.

System requirements

The recommended operating system is a Linux distribution.

Make sure you have the following binaries available:

• docker - container runtime
• just - command runner
• uv - python package and project manager

Development setup

Run the following just recipe and install any missing dependencies:

just dev-setup

Quick deploy

After cloning this repo, follow the steps below:

# for a local (dev / evaluation) environment:
./scripts/deploy.sh local

# for ci:
./scripts/deploy.sh ci
# to force a CI environment, set the environment variable:
#   export CI=1
#   just env

# for production:
./scripts/deploy.sh production
# to force a production environment, list the current machine's hostname as production:
#   cp ./scripts/prod-hostnames.env.example ./scripts/prod-hostnames.env
#   hostname >> ./scripts/prod-hostnames.env
#   just env

Important

This deploy script does several tasks, and it's generally safe to run it multiple times. Nonetheless, a production deployment is expected to require manual configuration of items like:

• Storage backend to match your hardware and your use case;
• Network configuration (reverse proxy, TLS certs, subdomains, etc);
• Allowed hosts, email server, and other settings in django.env.

So, for a production deployment, make sure to check the detailed deployment instructions after running the script.

This should leave you with the application running using default configurations. If that doesn't happen, feel free to open an issue or reach out for help.

1. Access the web interface:

   Open the web interface at localhost:8000 (localhost:18000 in production). You can create regular users by signing up there, or:

   You can sign in with the superuser credentials at localhost:8000/admin (or localhost:18000/<admin-path-set-in-django.env> in production) to access the admin interface.

   [!TIP] The superuser credentials are the ones provided in a step above, or during an interactive execution of the deploy.sh script. If the credentials were lost, you can reset the password with:

   just uv run manage.py changepassword <email>

   Or create one:

   just uv run manage.py createsuperuser

2. Run the test suite:

   # run all gateway tests available for the current environment:
   just test

Alternatively, follow the steps that are not automated for a first-time setup.

Just recipes

Next, you might be interested in other available just recipes.

# print the currently selected environment:
just env

# stream logs until interrupted, or without following it:
just logs
just logs-once

# rebuilds and restarts services, showing logs (press Ctrl+C to exit logs):
just redeploy

# stop all gateway services:
just down

Get the full list with:

just --list

Available recipes:
    default                         # show available recipes

    [development]
    dev-setup                       # sets up the development environment
    gact *args                      # runs GitHub Actions locally
    pre-commit                      # runs the pre-commit hooks with dev dependencies [alias: hooks]
    update                          # upgrades pre-commit hooks and gateway dependencies to their latest compatible versions [alias: upgrade]
    watch *args                     # watch file changes when in local env mode

    [monitoring]
    logs *args                      # streams logs until interrupted (tails 10k lines); args are passed to compose logs
    logs-once *args                 # prints all recent logs once; args are passed to compose logs
    snapshot                        # captures a snapshot of the configured environment

    [qa]
    deptry                          # runs deptry to check for missing and unused python dependencies
    gact *args                      # runs GitHub Actions locally
    serve-coverage                  # serves pytest coverage HTML locally
    test *args                      # runs all tests (python and javascript); args are passed to pytest
    test-js *args                   # runs javascript tests inside the app container
    test-py *args                   # validates templates and runs pytest inside the app container

    [service]
    down *args                      # stops and remove compose services; args are passed to compose down
    redeploy services=''            # rebuilds then restarts services and shows logs
    restart *args                   # restarts running compose services
    up *args                        # starts services in detached mode; if env is local, starts process to watch files [alias: run]
    watch *args                     # watch file changes when in local env mode

    [setup]
    build *args                     # pulls and rebuild the compose services with optional args
    build-full *args                # pulls and rebuilds from scratch without cache
    generate-secrets env_type *args # generates environment secrets for local/production/ci environments

    [utilities]
    clean                           # removes ephemeral files, like python caches and test coverage reports
    dc +args                        # runs a generic docker compose command e.g. `just dc ps`
    env                             # prints currently selected environment, for debugging and validation purposes
    uv +args                        # shorthand for 'uv' commands (e.g. `just uv run manage.py migrate`)

More SDS Gateway docs

• Setting up a dev environment
• Production
  • Detailed production deploy instructions
  • Production backups
• Others
  • OpenSearch Query Tips
  • Handling migration conflicts