Fix mismatches between `.env.example` and `.env`

[david1542]

Overview

We tend to modify our environment variables a lot. We do this by changing our .env locally, propagate the changes to our containers via docker-compose-common.yaml and then update .env.example.

However, when people pull the latest changes, their local .env remains the same since it is not committed go git (can contains secret stuff, like Slack bot token). This creates bugs and mismatches in the environments.

We need to find a better way to sync everyone to the latest environment variables setup.

[KrishEthan]

@david1542 Should we create a Makefile with a command to run a Python script that syncs .env with .env.example before pushing code?

[david1542]

@KrishEthan That's a good idea. Makefiles are a good approach, kinda like in opentelemetry-demo. On Windows people would have to install it somehow, but seems like there are good ways to do that. There's also this library that we can take inspiration from, regarding merging .env.example and .env.

Similar to opentelemetry-demo, we can create a "start" script that syncs the .env files and then runs docker compose up --build -d. I'd tie that logic to the start of the services though, rather then at the commit phase, because that's where people experience most of the bugs related to the .env changes. However, using Python/Node.js here would require people to have them installed first on their machine, which is suboptimal if they only wanna use the Docker setup.

Do you have an idea maybe of solving it in an OS-agnostic way?

[KrishEthan]

@david1542

1. To make it OS-agnostic, we can set up Docker to run a bash script that copies keys from .env to .env.example.
2. We can also use pre-commit hooks to handle this setup automatically.

[david1542]

@KrishEthan That sounds good 🚀 If you wanna give it a shot and create a PR, I'd be happy to review it & assist you if needed on Slack :)

[KrishEthan]

@david1542 sure I will take that, which approach should I go with pre-commit hooks or docker?

[david1542]

@KrishEthan I think the docker approach sounds good. I think all the other containers should rely on it (depends_on) to be successful, so we need to exit with an appropriate status code once it finishes.

[KrishEthan]

@david1542 I made a small test project to check if the scripts worked correctly before using them on the full project.

Bash Script

# Check if .env file exists
if [ ! -f .env ]; then
    echo ".env file does not exist."
    exit 1
fi

# Read .env file and extract keys
keys=$(grep -v '^#' .env | cut -d '=' -f1)

# Write keys to .env.example
> .env.example
for key in $keys; do
    echo "$key=" >> .env.example
done

echo "Keys from .env have been copied to .env.example"

Dockerfile

FROM alpine:3.14

RUN apk add --no-cache bash

WORKDIR /app

COPY copy_env_keys.sh .

RUN chmod +x copy_env_keys.sh

CMD ["./copy_env_keys.sh"]

Video Walkthrough

sync-env.mp4

[david1542]

@KrishEthan Looks good! One small note though - we want to also do the other way around. Namely, detect if there are any new variables in the .env.example.

Basically, the script can take all the new variables from both files (.env and .env.example) and merge them to each other.

[KrishEthan]

@david1542

Bash Script

#!/bin/bash

extract_keys() {
    grep -v '^#' "$1" | cut -d '=' -f1
}

merge_keys() {
    local keys=$1
    local file=$2
    local temp_file=$(mktemp)

    grep -v '^#' "$file" > "$temp_file" 2>/dev/null

    for key in $keys; do
        if ! grep -q "^$key=" "$file" 2>/dev/null; then
            echo "$key=" >> "$temp_file"
        fi
    done

    sort -u "$temp_file" > "$file"
    rm "$temp_file"
}

if [ ! -f .env ]; then
    echo ".env file does not exist."
    exit 1
fi

if [ ! -f .env.example ]; then
    touch .env.example
fi

env_keys=$(extract_keys .env)
example_keys=$(extract_keys .env.example)

merge_keys "$env_keys $example_keys" .env

merge_keys "$env_keys $example_keys" .env.example

echo "Keys from .env and .env.example have been synchronized."

Video Walkthrough

sync-env.mp4

[david1542]

@KrishEthan Super cool!! thanks a lot for your contribution.
Do you want me to embed your code to our setup, or would you like to create a PR and I'll review it?

[KrishEthan]

Hey @david1542 I think it would be better if you embed code, Setup on my end would take a lot of time.

[david1542]

@KrishEthan Sure, I'll do that then. Thx again. I'll update here once it's merged