#02040 servidor intern

.docs/docker-setup-with-plugins.md

@@ -0,0 +1,132 @@
+# Deploying with plugins
+
+To automatically build a Giscube Admin with plugins you must add additional `app.env` variables and create a file `app-plugins.env` in the root directory of the repo (alongside `app.env`).
+
+If `app-plugins.env` does not exist, it will not install any plugin.
+
+If you remove `app-plugins.env`, make sure you also remove the `app.env` variables described below.
+
+_**Warning:** A theme counts as a plugin!_
+
+## Environment variables (`app.env`)
+
+### GISCUBE_PLUGINS
+
+The list of plugins (excluding theme) to install in Giscube (without the name of the repo). Example:
+
+```
+GISCUBE_PLUGINS=mapia_ms_identity,mapiasantcugat_search,mapiareus_search,mapiareus_renatur
+```
+
+These plugins must match those defined in [`app-plugins.env`](#giscube_plugins_paths) (without the repo folder).
+
+### THEME_PLUGINS
+
+The name of the theme you want to install, if any.
+
+```
+THEME_PLUGINS=mapia_theme
+```
+
+This must match the one defined in [`app-plugins.env`](#giscube_plugins_paths) (without the repo folder).
+
+## Environment variables (`app-plugins.env`)
+
+All variables are required.
+
+### GISCUBE_PLUGINS_REPOS
+
+Contains a list of the GIT repos where the plugins can be found, comma separated. Example:
+
+```
+GISCUBE_PLUGINS_REPOS=git@bitbucket.org:infraplan/mapia-server-theme.git,git@bitbucket.org:infraplan/mapiasantcugat-giscube-admin.git,git@bitbucket.org:infraplan/mapiareus-giscube-admin.git
+```
+
+Or:
+
+```
+theme_repo=git@bitbucket.org:infraplan/mapia-server-theme.git
+repo1=git@bitbucket.org:infraplan/mapiasantcugat-giscube-admin.git
+repo2=git@bitbucket.org:infraplan/mapiareus-giscube-admin.git
+
+GISCUBE_PLUGINS_REPOS=${theme_repo},${repo1},${repo2}
+```
+
+### GISCUBE_PLUGINS_PATHS
+
+Contains a list of each plugin to be installed, it must include the folder where they have been cloned (the name of their repo). Example:
+
+```
+GISCUBE_PLUGINS_PATHS=mapia-server-theme/mapia_theme,mapiasantcugat-giscube-admin/mapia_ms_identity,mapiasantcugat-giscube-admin/mapiasantcugat_search,mapiareus-giscube-admin/mapiareus_search,mapiareus-giscube-admin/mapiareus_renatur
+```
+
+Or:
+
+```
+giscube_theme=mapia-server-theme/mapia_theme
+plugin1=mapiasantcugat-giscube-admin/mapia_ms_identity
+plugin2=mapiasantcugat-giscube-admin/mapiasantcugat_search
+plugin3=mapiareus-giscube-admin/mapiareus_search
+plugin4=mapiareus-giscube-admin/mapiareus_renatur
+
+GISCUBE_PLUGINS_PATHS=${giscube_theme},${plugin1},${plugin2},${plugin3},${plugin4}
+```
+
+You can use a specific branch by adding it with a `hash`. Example:
+
+```
+GISCUBE_PLUGINS_PATHS=mapia-server-theme/mapia_theme#much-cooler-version
+```
+
+### HOST_UID & HOST_GID
+
+These variables are required to grant privileges to the repository directories created inside the container, so they can be seen and edited in the `plugins` directory of the host.
+
+**Warning: DO NOT CHANGE THESE LINES**
+
+## Building and running Docker Compose
+
+_**Important:** Make sure you have an SSH key defined in the computer where you want to install this and the key has access to the repos of the plugins._
+
+Once the environment variables are set, you will run docker compose normally.
+
+`docker compose build` will execute `docker-custom/plugins/plugins.sh` to clone the repos and store them in a volume inside the containers.
+
+`docker compose up` will execute `docker-custom/plugins/plugins_links.sh` to create the appropriate symbolic links to a Django accessible folder before running Django.
+
+### Local development
+
+The plugin repos will be visible in the directory `plugins` of the host. They are accessible as GIT repos, so they can be edited and the changes pushed. Changes in the plugins' code will also trigger Django's hot-reload and be visible immediately.
+
+### Known caveats
+
+With the current implementation, there are a few caveats:
+
+#### Inconsistent success message
+
+The script will always print **PLUGINS successfully cloned** at the end, even though errors may have been returned during cloning. Look for **ERROR** or **fatal** in the logs to search for errors.
+
+_**Tip**: Running `docker compose build --progress=plain` will also provide additional information in case of errors._
+
+#### Pulling changes from already installed plugin repositories
+
+If someone pushes changes into one of the plugin remote repositories we already installed, a simple `docker compose build` will not pull those changes into our local repo.
+
+This is by design, Docker compose caches building steps. If the code did not change (in `docker-compose.yml`, `Dockerfile`, `plugins.sh` or `.env` files), the `plugins.sh` step is skipped.
+
+**To update the plugins with the latest changes you have to remove the cache with `docker compose build --no-cache`.**
+
+#### Switching from a branch to master
+
+If you first installed a plugin using a branch, removing the branch name in the `app-plugins.env` file will not checkout the base branch (`master`|`main`).
+
+To go back to the base branch, you must specify it in the `app-plugins.env`. Example:
+
+```
+GISCUBE_PLUGINS_PATHS=mapia-server-theme/mapia_theme#master
+```
+
+## Development improvements
+
+- Fix some of the caveats above.
+- Some common constant variables are defined inside plugins.sh and plugins_links.sh. Check if the constants could be hardcoded in the docker-compose.yml file and pass them to both scripts.

.gitignore

@@ -9,6 +9,7 @@ test.db
 docker-test/requirements.txt
 tests/media/*
 app.env
+app-plugins.env
 .env
 example-data/
 __pycache__

README.md

@@ -56,6 +56,9 @@ Create the giscube docker network:
 docker network create giscube
 ```
 
+### (optional) Theme & Plugins
+
+If you want to use Giscube plugins or a Giscube theme, you must define [additional environment variables](.docs/docker-setup-with-plugins.md).
 
 ### (optional) Build the images
 

app-plugins.env-example

@@ -0,0 +1,18 @@
+# Copy it into app-plugins.env only if you want to install a theme or plugins.
+
+theme_repo=git@bitbucket.org:mycompany/my_theme_repo.git
+repo1=git@bitbucket.org:mycompany/cool_plugins.git
+repo2=git@bitbucket.org:mycompany/even_cooler_plugins.git
+
+GISCUBE_PLUGINS_REPOS=${theme_repo},${repo1}
+
+giscube_theme=my_theme_repo/my_theme
+plugin1=cool_plugins/advanced_search
+plugin2=even_cooler_plugins/marquee
+plugin3=even_cooler_plugins/make_unicorns
+
+GISCUBE_PLUGINS_PATHS=${giscube_theme},${plugin1},${plugin2},${plugin3}
+
+# Host user/group for proper file ownership (DO NOT CHANGE)
+HOST_UID=$(id -u)
+HOST_GID=$(id -g)

app.env-example

@@ -64,9 +64,8 @@ EMAIL_HOST_USER=
 EMAIL_HOST_PASSWORD=
 DEFAULT_FROM_EMAIL=
 
-
-#GISCUBE_PLUGINS_PATH=
 #GISCUBE_PLUGINS=
+#THEME_PLUGINS=giscube_theme
 
 GISCUBE_SEARCH_DEFAULT_DICTIONARY=catalan
 

docker-compose.yml

@@ -1,21 +1,24 @@
-version: "3.7"
 services:
-
   requirements:
     container_name: giscube_giscube-admin_requirements
     build:
       context: .
-      dockerfile: docker/django/ubuntu-20.04/Dockerfile
+      dockerfile: docker-custom/plugins/Dockerfile
+      ssh:
+        - default
       args:
         - EXTRA_REQUIREMENTS=devel
         - PIP_PACKAGES=setuptools==57.5.0
         - APT_PACKAGES=
         - APT_PACKAGES_BUILD=
     image: giscube_giscube-admin_requirements
-    entrypoint: /bin/bash -c "chown -v www-data:www-data /app_data"
-    command: []
+    # Initialize plugins: copy from image to host mount and set permissions
+    entrypoint: /bin/bash /docker/plugins_links.sh --init-requirements
+    env_file:
+     - app.env
     volumes:
       - app_data:/app_data
+      - ./plugins:/var/lib/plugins_repos
 
   db:
     image: postgis/postgis:12-3.0
@@ -36,10 +39,11 @@ services:
     image: giscube_giscube-admin_requirements
     env_file:
      - app.env
-    command: ["/bin/bash", "/app/docker/scripts/wait-for-it.sh", "db:5432", "--", "/bin/bash", "/app/docker-custom/django/runserver.sh"]
+    command: ["/bin/bash", "/app/docker/scripts/wait-for-it.sh", "db:5432", "--", "/bin/bash", "/docker/plugins_links.sh", "/bin/bash", "/app/docker-custom/django/runserver.sh"]
     volumes:
       - .:/app:cached
       - app_data:/app_data
+      - ./plugins:/var/lib/plugins_repos  # Mounted from host for live editing
 
 
   uwsgi:
@@ -49,12 +53,13 @@ services:
     image: giscube_giscube-admin_requirements
     env_file:
      - app.env
-    command: ["/bin/bash", "/app/docker/scripts/wait-for-it.sh", "db:5432", "--", "/bin/bash", "/app/docker-custom/uwsgi/uwsgi.sh"]
+    command: ["/bin/bash", "/app/docker/scripts/wait-for-it.sh", "db:5432", "--", "/bin/bash", "/docker/plugins_links.sh", "/bin/bash", "/app/docker-custom/uwsgi/uwsgi.sh"]
     volumes:
       - .:/app:cached
       - app_data:/app_data
       - ../giscube-admin-ee-plugins:/giscube-admin-ee-plugins
       - static:/static
+      - ./plugins:/var/lib/plugins_repos  # Mounted from host for live editing
     environment:
       - DEBUG=False
       - STATIC_ROOT=/static
@@ -70,11 +75,12 @@ services:
      - app.env
     ports:
       - "5555:5555"
-    command: ["/bin/bash", "/app/docker/scripts/wait-for-it.sh", "db:5432", "--",  "/bin/bash", "/app/docker-custom/django/celery_devel.sh"]
+    command: ["/bin/bash", "/app/docker/scripts/wait-for-it.sh", "db:5432", "--",  "/bin/bash", "/docker/plugins_links.sh", "/bin/bash", "/app/docker-custom/django/celery_devel.sh"]
     volumes:
       - .:/app:cached
       - app_data:/app_data
       - ../giscube-admin-ee-plugins:/giscube-admin-ee-plugins
+      - ./plugins:/var/lib/plugins_repos  # Mounted from host for live editing
 
 
   redis:
@@ -130,5 +136,5 @@ volumes:
 
 networks:
   default:
-    external:
-      name: giscube
+    name: giscube
+    external: true

docker-custom/plugins/Dockerfile

@@ -0,0 +1,101 @@
+#
+#                        BUILD TEMPORARY BUILD-SYSTEM
+#
+# Downloads all the pip with the given ssh key. Allows to use private repos
+#  without the key
+#
+FROM microdisseny/py3-development:ubuntu-20.04 as build-system
+
+ADD docker/config/known_hosts /root/.ssh/known_hosts
+
+ADD ./requirements*.txt /tmp/
+
+# Download python requirements
+ARG EXTRA_REQUIREMENTS=devel
+ARG PIP_PACKAGES
+ARG APT_PACKAGES
+
+RUN \
+  set -e; \
+  if [ -n "$APT_PACKAGES_BUILD" ]; then \
+    apt-get update && \
+    apt -y install $APT_PACKAGES_BUILD; \
+  fi; \
+  if [ -n "$PIP_PACKAGES" ]; then \
+    pip3 install $PIP_PACKAGES; \
+  fi;
+
+RUN --mount=type=ssh \
+  set -e; \
+  pip3 install -U pip; \
+  mkdir /pip; \
+  if [ -n "$EXTRA_REQUIREMENTS" -a -f "/tmp/requirements-$EXTRA_REQUIREMENTS.txt" ]; then \
+    pip3 download -d /pip -r /tmp/requirements.txt -r "/tmp/requirements-$EXTRA_REQUIREMENTS.txt"; \
+  else \
+    pip3 download -d /pip -r /tmp/requirements.txt; \
+  fi;
+
+
+#
+#                            BUILD RUNTIME SYSTEM
+#
+# The actuall system that will be run
+#
+FROM microdisseny/py3-development:ubuntu-20.04
+
+# Copy files
+RUN mkdir -p /app
+RUN mkdir -p /docker
+ADD docker/django/entrypoint.sh /docker/
+ADD docker/django/bash.sh /docker/
+
+# Install python requirements
+COPY --from=build-system /pip /pip
+ARG PIP_PACKAGES
+
+RUN \
+  set -e; \
+  if [ -n "$APT_PACKAGES" ]; then \
+    apt-get update && \
+    apt -y install $APT_PACKAGES; \
+  fi; \
+  if [ -n "$PIP_PACKAGES" ]; then \
+    pip3 install $PIP_PACKAGES; \
+  fi;
+RUN pip3 install -U pip
+RUN pip3 install /pip/*
+RUN rm -rf /pip
+
+# Configure run
+
+# Configuración para plugins
+ADD docker-custom/plugins/plugins.sh /docker/
+ADD docker-custom/plugins/plugins_links.sh /docker/
+# Copy app-plugins.env if it exists, otherwise create empty file
+COPY app-plugins.env* /tmp/
+RUN if [ -f /tmp/app-plugins.env ]; then \
+      cp /tmp/app-plugins.env /docker/app-plugins.env; \
+    else \
+      touch /docker/app-plugins.env; \
+    fi && \
+    rm -rf /tmp/app-plugins.env*
+RUN chmod +x /docker/*.sh
+
+# Script para clonar repos y configurar plugins
+RUN --mount=type=ssh \
+  set -e; \
+  mkdir -p ~/.ssh && \
+  chmod 700 ~/.ssh && \
+  ssh-keyscan bitbucket.org >> ~/.ssh/known_hosts && \
+  ssh-keyscan github.com >> ~/.ssh/known_hosts && \
+  chmod 644 ~/.ssh/known_hosts && \
+  echo "Building Giscube plugins..." && \
+  bash /docker/plugins.sh && \
+  echo "Giscube plugins built."
+
+# Create plugins_links directory with proper permissions
+RUN mkdir -p /var/lib/plugins_links && chmod 777 /var/lib/plugins_links
+
+WORKDIR /app
+ENTRYPOINT ["/docker/entrypoint.sh"]
+CMD ["/bin/bash", "/app/docker-custom/plugins/plugins_links.sh"]