From 5c455e50cd2fe40e531e05916f0e864f5b2cc8c5 Mon Sep 17 00:00:00 2001 From: ionous Date: Tue, 9 Jul 2024 10:45:16 -0700 Subject: [PATCH] support win dev by moving environment config windows can't run bash scripts without linux for windows ( which imposes some extra headaches for shift development; setup of wsl, permission issues, etc. ) but, the bash script sets up the environment needed for docker compose to work ( which is why we can't use "docker compose" directly at the command line ) docker does, however, support a ".env" file next to the compose file just for this purpose. changes: - moved envvars from ./shift to .env - for letsencrypt and eventimages, put the required production paths in shift.overrides.production ( because these two had/need special handling ) - moved local event images from "/backend/eventimages" to "/app/eventimages" . the latter is what "npm run dev" uses and makes fake event creation work inside docker ( granted both should probably be using /bin/eventimages or something ) - added a new placeholder directory "./services/nginx/letsencrypt/". this placeholder is only used for development. previously mac/linux dev would use "/tmp/letsencrypt" just like production does; but windows can't use "/tmp" folders. - removed unused envvars from the 'secrets.example' - for windows users: added npm scripts for up, down, watch, and emails to mimic those commands from ./shift; and tweaked the npm build script to build hugo into the expected docker location. - updated the readme to describe the windows steps - moved the old, probably inaccurate local netlify development steps from the readme to a separate doc file with notes fixed: - fixed: the node container in docker-compose.yml should have been using the EVENTIMAGES envvar just like nginx does ( at the top of the file ) --- .env | 37 +++++++++++++++++ README.md | 45 +++++++++------------ docker-compose.yml | 4 +- docs/netlify.md | 11 +++++ package.json | 21 +++++++--- secrets.example | 10 +++-- services/nginx/letsencrypt/.gitkeep | 7 ++++ shift | 63 +++++++---------------------- shift.overrides.production | 8 ++++ 9 files changed, 121 insertions(+), 85 deletions(-) create mode 100644 .env create mode 100644 docs/netlify.md create mode 100644 services/nginx/letsencrypt/.gitkeep diff --git a/.env b/.env new file mode 100644 index 00000000..5de92ef9 --- /dev/null +++ b/.env @@ -0,0 +1,37 @@ +# this file is read by docker to configure its environment. +# https://docs.docker.com/compose/environment-variables/ +# +# on production, these settings are overridden via the './shift' script. +# ( see also: 'shift.overrides.production' and the 'secrets.example' ) +# +#### DOCKER #### +COMPOSE_PROJECT_NAME="shift" # set the prefix of docker containers + +#### DB #### +MYSQL_ROOT_PASSWORD=da642315cb2d390714590dad93e07c50 +MYSQL_DATABASE=shift # can force node to use a in-memory db, by changing this to "sqlite" +MYSQL_USER=shift +MYSQL_PASSWORD=ok124 + +### NGINX ### +NGINX_HTTPS_PORT=4443 # used by nginx and node configuration +NGINX_HTTP_PORT=4480 # used only by nginx ( for lets encrypt i think ) +LETSENCRYPT_WEBROOT=./services/nginx/letsencrypt + +### NODE #### +SHIFT_DOMAIN="localhost" # for generating sharable links +SHIFT_IMAGE_DIR=/opt/backend/eventimages # images as seen by the node container +EVENTIMAGES=./app/eventimages # source of event images on local machine + +#### EMAIL #### +# with empty values, the default behavior is to log to console. +# overridden in production via the secrets.example file. +SMTP_HOST= +SMTP_USER= +SMTP_PASS= + +# optional configuration for ethereal email. +# ( see readme.md for more info ) +# export SMTP_HOST=smtp.ethereal.email +# export SMTP_USER=h42vgcxjpznr6qhx@ethereal.email +# export SMTP_PASS=E18RgFGSnhwNMzmBdJ diff --git a/README.md b/README.md index eb78ba33..a6074b5d 100644 --- a/README.md +++ b/README.md @@ -27,37 +27,33 @@ The backend uses: - If you want to change something about the site configuration or theme, [pull requests](https://help.github.com/articles/creating-a-pull-request/) are welcome. Once you create a PR, you can immediately check out a link to the [build status, log, and a preview of your changes](https://app.netlify.com/sites/shift-docs/deploys). - If you only want to edit CONTENT rather than any code or site styling, (including creating new pages), [this doc shows how to easily do so without writing code](/docs/UPDATING.md). -# Frontend development with Netlify - -While creating a pull request does automatically deploy a preview of the frontend to Netlify, you can also create previews manually: this could help you do things like theme development in your own repository before submitting your pr. - -1. [fork repo](https://help.github.com/articles/fork-a-repo/) -2. read the comments in the netlify.toml file around changing the build command in the `[context.production]` section and make changes if needed. -2. [deploy on Netlify](https://app.netlify.com/start) by linking your forked repo. Included configuration file `netlify.toml` should mean 0 additional configuration required to get the site running. If you get a build failure around access denied for ssh, you probably need the advice in step 2 just above this! - -If you have trouble with it please [file an issue](https://github.com/shift-org/shift-docs/issues/new) to let us know what you tried and what happened when you did. - # Local development with Docker -The production backend is run in several docker containers; including nginx, mysql, and the node server. - -The docker configuration also supports running your own frontend and backend server locally. The following steps assume a Linux, or MacOs development environment. On Windows, you'll need something like the [Windows Subsystem for Linux](https://learn.microsoft.com/en-us/windows/wsl/install). +Docker runs the production backend, and Netlify runs the frontend. When developing locally, however, you can use Docker to run both. The following steps assume a Linux, or MacOs development environment. 1. Install Docker: [https://docs.docker.com/get-docker/](https://docs.docker.com/get-docker/) +1. Install Node: https://nodejs.org/en/download 2. Download source code: `git clone https://github.com/shift-org/shift-docs.git` -4. Start shift site: `cd shift-docs ; ./shift up` -5. If you're standing up the site for the first time, add database tables with the setup script: `./shift mysql-pipe < services/db/seed/setup.sql`. -6. Visit `https://localhost:4443/` . If this leads to an SSL error in chrome, you may try flipping this flag: chrome://flags/#allow-insecure-localhost +3. Start your local server: `cd shift-docs ; ./shift up` ( On Windows, use `npm run up` ) +4. Optionally, watch for content changes: `./shift watch` ( On Windows, use `npm run watch` ) +5. Visit `https://localhost:4443/` . If this leads to an SSL error in chrome, you may try flipping this flag: chrome://flags/#allow-insecure-localhost + +Note that no changes **inside** the Docker containers should ever be needed. The containers read from your **local** filesystem, so updating the files on your machine will automatically update docker too. ( Sometimes after a `./shift down`, `./shift up` to restart the server. ) -Note that no changes to the filesystems **inside** the container should ever be needed; they read from your **local** filesystem so updating the local FS will show up in the container (perhaps after a restart). Updating, changing branches, etc can be done with git commands **outside** of the container (`git checkout otherbranch` or `git pull`). +So now you can hopefully access the site. But for a real end-to-end test of your setup, you'll need to create an event: -So - now you can hopefully access the site. But a real end-toend test of yoursetup, would be creating an event: +1. Visit https://localhost:4443/addevent/ +2. Fill out all required fields (ones marked with an asterisk) for a date a day or two in the future. +3. Save the event (fix any validation errors around missing fields to ensure it saves.) +4. In production, we send you an email with a link to confirm the ride listing. For local development, we don't actually send the email. Instead, use `./shift emails` ( or `npm run emails` ) to view the email log file. +5. Publish your event by visiting the url printed to the log file. +6. See your event on the https://localhost:4443/calendar page! -1. visit https://localhost:4443/addevent/ -2. fill out all required fields (ones marked with an asterisk), for a date a day or two in the future. -3. save the event (fix any validation errors around missing fields to ensure it saves) -4. In production, we send you an email with a link to confirm the ride listing; we also write a copy of that email to the file `services/node/shift-mail.log`. For local development, we don't actually send the email, so get the confirmation link from that mail log, visit it, and hit publish event -5. hopefully see your event on the https://localhost:4443/calendar page! +You can also create some sample test events using the command: + +1. `npm run docker-create-events` + +It will print the event urls to the console, but they are automatically published to your test site. ## Important Project Files @@ -97,7 +93,6 @@ So - now you can hopefully access the site. But a real end-toend test of yourse * This will show the persistent volumes that docker knows about. The shift project volumes are prefixed with `shift_` * The "shift_" docker namespace comes from the shift file: `export COMPOSE_PROJECT_NAME="shift"` - # Local development with Node.js You can also do local development with node. These steps will setup a local node server which, by default, acts as both frontend and backend. @@ -116,14 +111,12 @@ When you new create events, the link for activating those events will be written If you are writing javascript code in the node backend, you can test everything is working as expected using `npm test`. - ### Local previews using production data As an alternative to `npm run dev`, you can preview a local frontend with the actual production backend by using: `npm run -w tools preview`. **NOTE:** any events you create while previewing this way *will* be seen by the world! - ### Ethereal email Before executing `npm run dev`, you can configure email debugging using `npm run -w tools new-ethereal-cfg`. It will generate a `shift-email.cfg` file in your `bin` directory which will be used when adding new events. Use the username and password listed in that file to check for emails here: https://ethereal.email/login. diff --git a/docker-compose.yml b/docker-compose.yml index 56be316a..e1134f58 100644 --- a/docker-compose.yml +++ b/docker-compose.yml @@ -18,6 +18,8 @@ services: # NGINX CONF: - ./services/nginx/:/opt/nginx/ - ./services/nginx/conf.d:/etc/nginx/conf.d + # maps the hosts letsencrypt onto the ngnix's container + # see also services/certbot.sh and ./shift script - ${LETSENCRYPT_WEBROOT}:/tmp/letsencrypt-auto links: # lets shift.conf reference node ( ex. http://node:3080 ) @@ -97,7 +99,7 @@ services: # the right-hand side has to be SHIFT_IMAGE_DIR # todo: change the host side to a named volume? # fix: can the rhs here use the environment var? - - ./backend/eventimages/:/opt/backend/eventimages/ + - "${EVENTIMAGES}:/opt/backend/eventimages" # DEACTIVATED: access for email log: # email logging disabled has been disabled in emailer.js diff --git a/docs/netlify.md b/docs/netlify.md new file mode 100644 index 00000000..b87aebda --- /dev/null +++ b/docs/netlify.md @@ -0,0 +1,11 @@ +# Frontend development with Netlify + +This page is archived from the main readme. Running locally with Netlify is no longer a core supported configuration. Netlify does however still run the production frontend. So this information might be useful. + +While creating a pull request does automatically deploy a preview of the frontend to Netlify, you can also create previews manually: this could help you do things like theme development in your own repository before submitting your pr. + +1. [fork repo](https://help.github.com/articles/fork-a-repo/) +2. read the comments in the netlify.toml file around changing the build command in the `[context.production]` section and make changes if needed. +2. [deploy on Netlify](https://app.netlify.com/start) by linking your forked repo. Included configuration file `netlify.toml` should mean 0 additional configuration required to get the site running. If you get a build failure around access denied for ssh, you probably need the advice in step 2 just above this! + +If you have trouble with it please [file an issue](https://github.com/shift-org/shift-docs/issues/new) to let us know what you tried and what happened when you did. diff --git a/package.json b/package.json index 150d8727..e280faaa 100644 --- a/package.json +++ b/package.json @@ -13,19 +13,30 @@ "tools" ], "scripts": { + "postinstall": "hugo-installer --version 0.128.0", + "start": "npm -w app start", "prestart": "npm i --omit dev --audit false --fund false --ignore-scripts", + "dev": "concurrently -k \"npm:dev-*\"", "predev": "env-cmd -f tools/dev.env node tools/setupEventImages.js", "dev-app": "env-cmd -f tools/dev.env npm start -w app", "dev-hugo": "exec-bin bin/hugo/hugo -w -s site -d ../bin/dist", - "build": "npm run build-hugo", - "build-hugo": "exec-bin bin/hugo/hugo -s site -d ../bin/dist", - "postinstall": "hugo-installer --version 0.128.0", - "test": "npm run test -w app", + + "watch": "exec-bin bin/hugo/hugo -w -s site -d public", + "build": "exec-bin bin/hugo/hugo -s site -d public", + "prebuild": "npm i", + + "preup": "npm run build", + "up": "docker compose up -d", + "down": "docker compose down", + "emails": "docker compose logs node -f", + "preview": "concurrently -k \"npm:dev-*\"", "preview-hugo": "npm run dev-hugo", - "preview-app": " npm -w tools run preview" + "preview-app": " npm -w tools run preview", + "docker-create-events": "docker compose exec node node ./tools/makeFakeEvents.js", + "test": "npm run test -w app" } } \ No newline at end of file diff --git a/secrets.example b/secrets.example index bf198db7..fc4b2b5b 100644 --- a/secrets.example +++ b/secrets.example @@ -1,8 +1,10 @@ -# production file at api.shift2bikes.org:/opt/shift-docs/secrets +# --------------------------------------- +# production environment variables for smtp configuration +# the real file is on api.shift2bikes.org at `/opt/shift-docs/secrets` +# ( it contains some outdated unused vars; the ones here are the only ones needed. ) +# see also: shift.overrides.production +# --------------------------------------- export SMTP_HOST= export SMTP_USER= export SMTP_PASS= -export SMTP_LOGIN=DIGEST-MD5 -export SMTP_DOMAIN= -export CAL_ADMIN_PASSWORD= diff --git a/services/nginx/letsencrypt/.gitkeep b/services/nginx/letsencrypt/.gitkeep new file mode 100644 index 00000000..ceaab8c0 --- /dev/null +++ b/services/nginx/letsencrypt/.gitkeep @@ -0,0 +1,7 @@ +# tldr: this directory makes nginx happy during development. +# +# the nginx container expects the environment variable LETSENCRYPT_WEBROOT +# to tell it a location on the host to map its /tmp/letscrypt directory +# on production certbot.sh **requires** that the host directory also be called /tmp/letscrypt +# but not all development machines have such a location (ie. windows) +# see also: docs/PRODUCTION_CONFIGURATION.md \ No newline at end of file diff --git a/shift b/shift index 2fec9eda..0670935f 100755 --- a/shift +++ b/shift @@ -1,53 +1,12 @@ #!/usr/bin/env bash set -e -#### DB #### -export MYSQL_ROOT_PASSWORD=da642315cb2d390714590dad93e07c50 -export MYSQL_DATABASE=shift -export MYSQL_USER=shift -export MYSQL_PASSWORD=ok124 -# can force the node version to use a sqlite in memory database instead of mysql: -# export MYSQL_DATABASE=sqlite - -export NGINX_HTTPS_PORT=4443 -export NGINX_HTTP_PORT=4480 -export LETSENCRYPT_WEBROOT="/tmp/letsencrypt-auto" - -export SHIFT_DOMAIN="localhost" - -#### DOCKER #### -export COMPOSE_PROJECT_NAME="shift" - -#### SMTP #### -# with this configuration, emails will log to console, and to a file -# ./shift emails -# - or - -# ./shift attach node -# $ tail -f /var/log/shift-mail.log -# -# it will also try to "send" ethereal emails -# https://ethereal.email/messages -# -# export SMTP_HOST=smtp.ethereal.email -# export SMTP_USER=h42vgcxjpznr6qhx@ethereal.email -# export SMTP_PASS=E18RgFGSnhwNMzmBdJ -# export SMTP_LOGIN=587 -SMTP_HOST= -SMTP_USER= -SMTP_PASS= -SMTP_LOGIN= -SMTP_DOMAIN= - -##### -# === HERE BE DRAGONS === -# These are some convenience pattens and functions that vincent@khougaz.com has found most helpful -# Feel free to copy into your own projects, or mostly don't worry about it! -##### - -# ENV +# ==================== +# setup environment overrides +# ==================== + +# Get location of this script export ROOT="$( cd "$( dirname "${BASH_SOURCE[0]}" )" && pwd )" -export EVENTIMAGES="${ROOT}/backend/eventimages" -export SHIFT_IMAGE_DIR="/opt/backend/eventimages" # Check for an overrides file if [ -f ${ROOT}/shift.overrides ]; then @@ -59,9 +18,8 @@ if [ -f $ROOT/secrets ]; then source $ROOT/secrets fi -SCRIPT_NAME=$(basename $0) - # ==================== +# helper to wait for node startup. # https://www.marksayson.com/blog/wait-until-docker-containers-initialized/ # Max query attempts before consider setup failed @@ -94,6 +52,7 @@ function waitUntilServiceIsReady() { } # ==================== +SCRIPT_NAME=$(basename $0) sub_help() { # - Show this text echo "Usage: ./${SCRIPT_NAME} [options]" @@ -104,7 +63,13 @@ sub_help() { # - Show this text sub_up() { # - Bring all services up if [ ! -d $LETSENCRYPT_WEBROOT ] ; then - mkdir $LETSENCRYPT_WEBROOT + # tbd: maybe we shouldn't mkdir, and instead let docker/nginx fail? + # rationale: certbot.sh creates and fills this directory. + # and if certbot hasn't already already run https will fail, + # but it won't be obvious until you visit the site. + # see docs/PRODUCTION_CONFIGURATION.md + echo "*** if this is production: have you run certbot? ***" + mkdir -p $LETSENCRYPT_WEBROOT fi sub_compose up --force-recreate -d diff --git a/shift.overrides.production b/shift.overrides.production index 23cb28b9..238cf3c8 100755 --- a/shift.overrides.production +++ b/shift.overrides.production @@ -1,3 +1,11 @@ +# --------------------------------------- +# production environment variables for nginx and node +# the './shift' bash script 'sources' "shift.overrides" +# and on the production server that file is soft-linked to this file ( `/opt/shift-docs/shift.overrides.production` ) +# unlike secrets.example, nothing here is privileged info. +# --------------------------------------- export NGINX_HTTPS_PORT=443 export NGINX_HTTP_PORT=80 export SHIFT_DOMAIN=shift2bikes.org +export LETSENCRYPT_WEBROOT="/tmp/letsencrypt-auto" +export EVENTIMAGES="/opt/shift-docs/backend/eventimages/" \ No newline at end of file