diff --git a/.docs/docker-setup-with-plugins.md b/.docs/docker-setup-with-plugins.md new file mode 100644 index 00000000..12afafdc --- /dev/null +++ b/.docs/docker-setup-with-plugins.md @@ -0,0 +1,132 @@ +# Deploying with plugins + +To automatically build a Giscube Admin with plugins you must add additional `app.env` variables and create a file `app-plugins.env` in the root directory of the repo (alongside `app.env`). + +If `app-plugins.env` does not exist, it will not install any plugin. + +If you remove `app-plugins.env`, make sure you also remove the `app.env` variables described below. + +_**Warning:** A theme counts as a plugin!_ + +## Environment variables (`app.env`) + +### GISCUBE_PLUGINS + +The list of plugins (excluding theme) to install in Giscube (without the name of the repo). Example: + +``` +GISCUBE_PLUGINS=mapia_ms_identity,mapiasantcugat_search,mapiareus_search,mapiareus_renatur +``` + +These plugins must match those defined in [`app-plugins.env`](#giscube_plugins_paths) (without the repo folder). + +### THEME_PLUGINS + +The name of the theme you want to install, if any. + +``` +THEME_PLUGINS=mapia_theme +``` + +This must match the one defined in [`app-plugins.env`](#giscube_plugins_paths) (without the repo folder). + +## Environment variables (`app-plugins.env`) + +All variables are required. + +### GISCUBE_PLUGINS_REPOS + +Contains a list of the GIT repos where the plugins can be found, comma separated. Example: + +``` +GISCUBE_PLUGINS_REPOS=git@bitbucket.org:infraplan/mapia-server-theme.git,git@bitbucket.org:infraplan/mapiasantcugat-giscube-admin.git,git@bitbucket.org:infraplan/mapiareus-giscube-admin.git +``` + +Or: + +``` +theme_repo=git@bitbucket.org:infraplan/mapia-server-theme.git +repo1=git@bitbucket.org:infraplan/mapiasantcugat-giscube-admin.git +repo2=git@bitbucket.org:infraplan/mapiareus-giscube-admin.git + +GISCUBE_PLUGINS_REPOS=${theme_repo},${repo1},${repo2} +``` + +### GISCUBE_PLUGINS_PATHS + +Contains a list of each plugin to be installed, it must include the folder where they have been cloned (the name of their repo). Example: + +``` +GISCUBE_PLUGINS_PATHS=mapia-server-theme/mapia_theme,mapiasantcugat-giscube-admin/mapia_ms_identity,mapiasantcugat-giscube-admin/mapiasantcugat_search,mapiareus-giscube-admin/mapiareus_search,mapiareus-giscube-admin/mapiareus_renatur +``` + +Or: + +``` +giscube_theme=mapia-server-theme/mapia_theme +plugin1=mapiasantcugat-giscube-admin/mapia_ms_identity +plugin2=mapiasantcugat-giscube-admin/mapiasantcugat_search +plugin3=mapiareus-giscube-admin/mapiareus_search +plugin4=mapiareus-giscube-admin/mapiareus_renatur + +GISCUBE_PLUGINS_PATHS=${giscube_theme},${plugin1},${plugin2},${plugin3},${plugin4} +``` + +You can use a specific branch by adding it with a `hash`. Example: + +``` +GISCUBE_PLUGINS_PATHS=mapia-server-theme/mapia_theme#much-cooler-version +``` + +### HOST_UID & HOST_GID + +These variables are required to grant privileges to the repository directories created inside the container, so they can be seen and edited in the `plugins` directory of the host. + +**Warning: DO NOT CHANGE THESE LINES** + +## Building and running Docker Compose + +_**Important:** Make sure you have an SSH key defined in the computer where you want to install this and the key has access to the repos of the plugins._ + +Once the environment variables are set, you will run docker compose normally. + +`docker compose build` will execute `docker-custom/plugins/plugins.sh` to clone the repos and store them in a volume inside the containers. + +`docker compose up` will execute `docker-custom/plugins/plugins_links.sh` to create the appropriate symbolic links to a Django accessible folder before running Django. + +### Local development + +The plugin repos will be visible in the directory `plugins` of the host. They are accessible as GIT repos, so they can be edited and the changes pushed. Changes in the plugins' code will also trigger Django's hot-reload and be visible immediately. + +### Known caveats + +With the current implementation, there are a few caveats: + +#### Inconsistent success message + +The script will always print **PLUGINS successfully cloned** at the end, even though errors may have been returned during cloning. Look for **ERROR** or **fatal** in the logs to search for errors. + +_**Tip**: Running `docker compose build --progress=plain` will also provide additional information in case of errors._ + +#### Pulling changes from already installed plugin repositories + +If someone pushes changes into one of the plugin remote repositories we already installed, a simple `docker compose build` will not pull those changes into our local repo. + +This is by design, Docker compose caches building steps. If the code did not change (in `docker-compose.yml`, `Dockerfile`, `plugins.sh` or `.env` files), the `plugins.sh` step is skipped. + +**To update the plugins with the latest changes you have to remove the cache with `docker compose build --no-cache`.** + +#### Switching from a branch to master + +If you first installed a plugin using a branch, removing the branch name in the `app-plugins.env` file will not checkout the base branch (`master`|`main`). + +To go back to the base branch, you must specify it in the `app-plugins.env`. Example: + +``` +GISCUBE_PLUGINS_PATHS=mapia-server-theme/mapia_theme#master +``` + +## Development improvements + +- Fix some of the caveats above. +- Some common constant variables are defined inside plugins.sh and plugins_links.sh. Check if the constants could be hardcoded in the docker-compose.yml file and pass them to both scripts. diff --git a/.gitignore b/.gitignore index e3558cd3..5ce8bd44 100644 --- a/.gitignore +++ b/.gitignore @@ -9,6 +9,7 @@ test.db docker-test/requirements.txt tests/media/* app.env +app-plugins.env .env example-data/ __pycache__ diff --git a/README.md b/README.md index 5f296f3a..247e7cd1 100644 --- a/README.md +++ b/README.md @@ -56,6 +56,9 @@ Create the giscube docker network: docker network create giscube ``` +### (optional) Theme & Plugins + +If you want to use Giscube plugins or a Giscube theme, you must define [additional environment variables](.docs/docker-setup-with-plugins.md). ### (optional) Build the images diff --git a/app-plugins.env-example b/app-plugins.env-example new file mode 100644 index 00000000..25f971df --- /dev/null +++ b/app-plugins.env-example @@ -0,0 +1,18 @@ +# Copy it into app-plugins.env only if you want to install a theme or plugins. + +theme_repo=git@bitbucket.org:mycompany/my_theme_repo.git +repo1=git@bitbucket.org:mycompany/cool_plugins.git +repo2=git@bitbucket.org:mycompany/even_cooler_plugins.git + +GISCUBE_PLUGINS_REPOS=${theme_repo},${repo1} + +giscube_theme=my_theme_repo/my_theme +plugin1=cool_plugins/advanced_search +plugin2=even_cooler_plugins/marquee +plugin3=even_cooler_plugins/make_unicorns + +GISCUBE_PLUGINS_PATHS=${giscube_theme},${plugin1},${plugin2},${plugin3} + +# Host user/group for proper file ownership (DO NOT CHANGE) +HOST_UID=$(id -u) +HOST_GID=$(id -g) diff --git a/app.env-example b/app.env-example index 5c5a6e4a..194f1147 100644 --- a/app.env-example +++ b/app.env-example @@ -64,9 +64,8 @@ EMAIL_HOST_USER= EMAIL_HOST_PASSWORD= DEFAULT_FROM_EMAIL= - -#GISCUBE_PLUGINS_PATH= #GISCUBE_PLUGINS= +#THEME_PLUGINS=giscube_theme GISCUBE_SEARCH_DEFAULT_DICTIONARY=catalan diff --git a/docker-compose.yml b/docker-compose.yml index 351d3da5..ed1d9a6f 100644 --- a/docker-compose.yml +++ b/docker-compose.yml @@ -1,21 +1,24 @@ -version: "3.7" services: - requirements: container_name: giscube_giscube-admin_requirements build: context: . - dockerfile: docker/django/ubuntu-20.04/Dockerfile + dockerfile: docker-custom/plugins/Dockerfile + ssh: + - default args: - EXTRA_REQUIREMENTS=devel - PIP_PACKAGES=setuptools==57.5.0 - APT_PACKAGES= - APT_PACKAGES_BUILD= image: giscube_giscube-admin_requirements - entrypoint: /bin/bash -c "chown -v www-data:www-data /app_data" - command: [] + # Initialize plugins: copy from image to host mount and set permissions + entrypoint: /bin/bash /docker/plugins_links.sh --init-requirements + env_file: + - app.env volumes: - app_data:/app_data + - ./plugins:/var/lib/plugins_repos db: image: postgis/postgis:12-3.0 @@ -36,10 +39,11 @@ services: image: giscube_giscube-admin_requirements env_file: - app.env - command: ["/bin/bash", "/app/docker/scripts/wait-for-it.sh", "db:5432", "--", "/bin/bash", "/app/docker-custom/django/runserver.sh"] + command: ["/bin/bash", "/app/docker/scripts/wait-for-it.sh", "db:5432", "--", "/bin/bash", "/docker/plugins_links.sh", "/bin/bash", "/app/docker-custom/django/runserver.sh"] volumes: - .:/app:cached - app_data:/app_data + - ./plugins:/var/lib/plugins_repos # Mounted from host for live editing uwsgi: @@ -49,12 +53,13 @@ services: image: giscube_giscube-admin_requirements env_file: - app.env - command: ["/bin/bash", "/app/docker/scripts/wait-for-it.sh", "db:5432", "--", "/bin/bash", "/app/docker-custom/uwsgi/uwsgi.sh"] + command: ["/bin/bash", "/app/docker/scripts/wait-for-it.sh", "db:5432", "--", "/bin/bash", "/docker/plugins_links.sh", "/bin/bash", "/app/docker-custom/uwsgi/uwsgi.sh"] volumes: - .:/app:cached - app_data:/app_data - ../giscube-admin-ee-plugins:/giscube-admin-ee-plugins - static:/static + - ./plugins:/var/lib/plugins_repos # Mounted from host for live editing environment: - DEBUG=False - STATIC_ROOT=/static @@ -70,11 +75,12 @@ services: - app.env ports: - "5555:5555" - command: ["/bin/bash", "/app/docker/scripts/wait-for-it.sh", "db:5432", "--", "/bin/bash", "/app/docker-custom/django/celery_devel.sh"] + command: ["/bin/bash", "/app/docker/scripts/wait-for-it.sh", "db:5432", "--", "/bin/bash", "/docker/plugins_links.sh", "/bin/bash", "/app/docker-custom/django/celery_devel.sh"] volumes: - .:/app:cached - app_data:/app_data - ../giscube-admin-ee-plugins:/giscube-admin-ee-plugins + - ./plugins:/var/lib/plugins_repos # Mounted from host for live editing redis: @@ -130,5 +136,5 @@ volumes: networks: default: - external: - name: giscube + name: giscube + external: true diff --git a/docker-custom/plugins/Dockerfile b/docker-custom/plugins/Dockerfile new file mode 100644 index 00000000..2d55b9c1 --- /dev/null +++ b/docker-custom/plugins/Dockerfile @@ -0,0 +1,101 @@ +# +# BUILD TEMPORARY BUILD-SYSTEM +# +# Downloads all the pip with the given ssh key. Allows to use private repos +# without the key +# +FROM microdisseny/py3-development:ubuntu-20.04 as build-system + +ADD docker/config/known_hosts /root/.ssh/known_hosts + +ADD ./requirements*.txt /tmp/ + +# Download python requirements +ARG EXTRA_REQUIREMENTS=devel +ARG PIP_PACKAGES +ARG APT_PACKAGES + +RUN \ + set -e; \ + if [ -n "$APT_PACKAGES_BUILD" ]; then \ + apt-get update && \ + apt -y install $APT_PACKAGES_BUILD; \ + fi; \ + if [ -n "$PIP_PACKAGES" ]; then \ + pip3 install $PIP_PACKAGES; \ + fi; + +RUN --mount=type=ssh \ + set -e; \ + pip3 install -U pip; \ + mkdir /pip; \ + if [ -n "$EXTRA_REQUIREMENTS" -a -f "/tmp/requirements-$EXTRA_REQUIREMENTS.txt" ]; then \ + pip3 download -d /pip -r /tmp/requirements.txt -r "/tmp/requirements-$EXTRA_REQUIREMENTS.txt"; \ + else \ + pip3 download -d /pip -r /tmp/requirements.txt; \ + fi; + + +# +# BUILD RUNTIME SYSTEM +# +# The actuall system that will be run +# +FROM microdisseny/py3-development:ubuntu-20.04 + +# Copy files +RUN mkdir -p /app +RUN mkdir -p /docker +ADD docker/django/entrypoint.sh /docker/ +ADD docker/django/bash.sh /docker/ + +# Install python requirements +COPY --from=build-system /pip /pip +ARG PIP_PACKAGES + +RUN \ + set -e; \ + if [ -n "$APT_PACKAGES" ]; then \ + apt-get update && \ + apt -y install $APT_PACKAGES; \ + fi; \ + if [ -n "$PIP_PACKAGES" ]; then \ + pip3 install $PIP_PACKAGES; \ + fi; +RUN pip3 install -U pip +RUN pip3 install /pip/* +RUN rm -rf /pip + +# Configure run + +# Configuración para plugins +ADD docker-custom/plugins/plugins.sh /docker/ +ADD docker-custom/plugins/plugins_links.sh /docker/ +# Copy app-plugins.env if it exists, otherwise create empty file +COPY app-plugins.env* /tmp/ +RUN if [ -f /tmp/app-plugins.env ]; then \ + cp /tmp/app-plugins.env /docker/app-plugins.env; \ + else \ + touch /docker/app-plugins.env; \ + fi && \ + rm -rf /tmp/app-plugins.env* +RUN chmod +x /docker/*.sh + +# Script para clonar repos y configurar plugins +RUN --mount=type=ssh \ + set -e; \ + mkdir -p ~/.ssh && \ + chmod 700 ~/.ssh && \ + ssh-keyscan bitbucket.org >> ~/.ssh/known_hosts && \ + ssh-keyscan github.com >> ~/.ssh/known_hosts && \ + chmod 644 ~/.ssh/known_hosts && \ + echo "Building Giscube plugins..." && \ + bash /docker/plugins.sh && \ + echo "Giscube plugins built." + +# Create plugins_links directory with proper permissions +RUN mkdir -p /var/lib/plugins_links && chmod 777 /var/lib/plugins_links + +WORKDIR /app +ENTRYPOINT ["/docker/entrypoint.sh"] +CMD ["/bin/bash", "/app/docker-custom/plugins/plugins_links.sh"] diff --git a/docker-custom/plugins/plugins.sh b/docker-custom/plugins/plugins.sh new file mode 100644 index 00000000..9eb82a3f --- /dev/null +++ b/docker-custom/plugins/plugins.sh @@ -0,0 +1,69 @@ +#!/usr/bin/env bash + +if [ ! -f "/docker/app-plugins.env" ]; then + echo "PLUGINS SETUP: app-plugins.env file not found. Skipping plugins setup." + exit 0 +fi + +set -a +source /docker/app-plugins.env +set +a + +# Hardcoded internal paths - not user-configurable +GISCUBE_INTERNAL_PLUGINS_PATH="/var/lib/plugins_links" +PLUGINS_CLONES_DIR="/tmp/plugins_clones" + +if [ -z "$GISCUBE_PLUGINS_REPOS" ] && [ -z "$GISCUBE_PLUGINS_PATHS" ]; then + echo "PLUGINS SETUP: No plugins configuration found. Skipping plugins setup." + exit 0 +fi + +if [ -z "$GISCUBE_PLUGINS_REPOS" ] || [ -z "$GISCUBE_PLUGINS_PATHS" ]; then + echo "PLUGINS SETUP: GISCUBE_PLUGINS_REPOS and GISCUBE_PLUGINS_PATHS environment variables must be set." + exit 0 +fi + +mkdir -p "$PLUGINS_CLONES_DIR" +echo "$GISCUBE_PLUGINS_REPOS" + +IFS=',' read -ra GISCUBE_PLUGINS_REPOS_LIST <<< "$GISCUBE_PLUGINS_REPOS" + +for repo in "${GISCUBE_PLUGINS_REPOS_LIST[@]}"; do + IFS='#' read -ra REPO_PARTS <<< "$repo" + REPO_URL="${REPO_PARTS[0]}" + BRANCH="${REPO_PARTS[1]}" + REPO_NAME=$(basename "$REPO_URL" .git) + echo "Doing $REPO_URL" + + if [ -d "$PLUGINS_CLONES_DIR/$REPO_NAME" ]; then + echo "* Updating existing repository $REPO_NAME" + if [ -n "$BRANCH" ]; then + git -C "$PLUGINS_CLONES_DIR/$REPO_NAME" checkout "$BRANCH" + git -C "$PLUGINS_CLONES_DIR/$REPO_NAME" pull + fi + git -C "$PLUGINS_CLONES_DIR/$REPO_NAME" pull + continue + fi + + echo "* Cloning repository $REPO_NAME" + if [ -n "$BRANCH" ]; then + git clone --branch "$BRANCH" "$REPO_URL" "$PLUGINS_CLONES_DIR/$REPO_NAME" + else + git clone "$REPO_URL" "$PLUGINS_CLONES_DIR/$REPO_NAME" + fi +done + +IFS=',' read -ra GISCUBE_PLUGINS_PATHS_LIST <<< "$GISCUBE_PLUGINS_PATHS" + +echo "Installing plugin requirements..." +set -e +for plugin_path in "${GISCUBE_PLUGINS_PATHS_LIST[@]}"; do + REQ_FILE="$PLUGINS_CLONES_DIR/$plugin_path/src/requirements.txt" + if [ -f "$REQ_FILE" ]; then + echo "Installing requirements for $plugin_path" + pip3 install -r "$REQ_FILE" + fi +done + +echo "PLUGINS successfully cloned at $PLUGINS_CLONES_DIR:" +ls -la "$PLUGINS_CLONES_DIR" diff --git a/docker-custom/plugins/plugins_links.sh b/docker-custom/plugins/plugins_links.sh new file mode 100644 index 00000000..600c0152 --- /dev/null +++ b/docker-custom/plugins/plugins_links.sh @@ -0,0 +1,67 @@ +#!/usr/bin/env bash + +# Hardcoded internal paths - not user-configurable +GISCUBE_INTERNAL_PLUGINS_PATH="/var/lib/plugins_links" +PLUGINS_REPOS_PATH="/var/lib/plugins_repos" +PLUGINS_CLONES_DIR="/tmp/plugins_clones" + +# Check if app-plugins.env exists +if [ ! -f "/docker/app-plugins.env" ]; then + echo "PLUGINS SETUP: app-plugins.env file not found. Skipping plugins setup." + # For --init-requirements, still need to set app_data ownership + if [ "$1" = "--init-requirements" ]; then + chown -v www-data:www-data /app_data + echo "Requirements initialization completed (no plugins)." + exit 0 + fi + exec "$@" +fi + +set -a +source /docker/app-plugins.env +set +a + +# REQUIREMENTS CONTAINER: Initialize plugins directory +if [ "$1" = "--init-requirements" ]; then + echo "Initializing plugins for requirements container..." + + # Fix app_data ownership + chown -v www-data:www-data /app_data + + # Copy plugins from image to host mount only if plugins exist + if [ -d "$PLUGINS_CLONES_DIR" ] && [ "$(ls -A $PLUGINS_CLONES_DIR 2>/dev/null)" ]; then + cp -rn $PLUGINS_CLONES_DIR/* $PLUGINS_REPOS_PATH/ 2>/dev/null || true + chown -R ${HOST_UID:-1000}:${HOST_GID:-1000} $PLUGINS_REPOS_PATH + echo "Plugins copied and ownership set." + else + echo "No plugins to copy." + fi + + echo "Requirements initialization completed." + exit 0 +fi + +# DJANGO/UWSGI/CELERY CONTAINERS: Create plugin symlinks +if [ -z "$GISCUBE_PLUGINS_REPOS" ] && [ -z "$GISCUBE_PLUGINS_PATHS" ]; then + echo "ERROR PLUGINS SETUP: No plugins configuration found. Skipping plugins setup." + exec "$@" +fi + +if [ -z "$GISCUBE_PLUGINS_REPOS" ] || [ -z "$GISCUBE_PLUGINS_PATHS" ]; then + echo "ERROR PLUGINS SETUP: GISCUBE_PLUGINS_REPOS and GISCUBE_PLUGINS_PATHS environment variables must be set." + exec "$@" +fi + +IFS=',' read -ra GISCUBE_PLUGINS_PATHS_LIST <<< "$GISCUBE_PLUGINS_PATHS" + +echo "Creating plugin symlinks in: $GISCUBE_INTERNAL_PLUGINS_PATH" +mkdir -p "$GISCUBE_INTERNAL_PLUGINS_PATH" + +for plugin_path in "${GISCUBE_PLUGINS_PATHS_LIST[@]}"; do + PLUGIN_NAME=$(basename "$plugin_path") + ln -sf "$PLUGINS_REPOS_PATH/$plugin_path" "$GISCUBE_INTERNAL_PLUGINS_PATH/$PLUGIN_NAME" +done + +echo "PLUGINS LINKS: Completed successfully." + +exec "$@" diff --git a/giscube/settings.py b/giscube/settings.py index fdaeb6df..094e3706 100644 --- a/giscube/settings.py +++ b/giscube/settings.py @@ -58,7 +58,7 @@ INSTALLED_APPS = [] # Theme plugin -GISCUBE_PLUGINS_PATH = os.environ.get('GISCUBE_PLUGINS_PATH', None) +GISCUBE_PLUGINS_PATH = '/var/lib/plugins_links' # Internal container path (DO NOT CHANGE) THEME_PLUGIN = os.environ.get('THEME_PLUGINS', '') if GISCUBE_PLUGINS_PATH and THEME_PLUGIN: