Skip to content
This repository was archived by the owner on Dec 14, 2023. It is now read-only.

[RFR] use pgmigrate for schema migrations #759

Merged
pypt merged 49 commits into from
Jul 22, 2021
Merged

[RFR] use pgmigrate for schema migrations #759

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
49 commits
Select commit Hold shift + click to select a range
63d5ef9
start on pgmigrate implementation
jtotoole Jan 21, 2021
a17394c
don't initialize db with schema
jtotoole Jan 22, 2021
65eb9bc
fix psycopg2 build
jtotoole Jan 22, 2021
76a3216
pgmigrate working
jtotoole Jan 24, 2021
b25f584
experiment with pg_dump
jtotoole Jan 25, 2021
b74d75f
dump schema file during build
jtotoole Jan 27, 2021
7c8c6bd
port schema comments to COMMENT ON statements
jtotoole Jan 27, 2021
2fa3d48
add documentation + get_schema script
jtotoole Jan 29, 2021
a750cb7
fix path to pgmigrate-callbacks in Dockerfile
jtotoole Jan 29, 2021
d092536
more cleanup
jtotoole Jan 29, 2021
e98b54e
moar cleanup
jtotoole Jan 29, 2021
8a0586b
delete test migration
jtotoole Jan 29, 2021
302b683
add comment in get_schema.sh
jtotoole Feb 1, 2021
56a8832
review feedback
jtotoole Jun 15, 2021
d486adb
dump schema to correct dir
jtotoole Jun 15, 2021
2806c27
fix container paths
jtotoole Jun 16, 2021
44c8ca1
typo fix + add test table
jtotoole Jun 16, 2021
05a5991
fix test migration
jtotoole Jun 16, 2021
50926d1
switch pgmigrate arg order
jtotoole Jun 16, 2021
f6a4734
put pgmigrate args on one line
jtotoole Jun 16, 2021
afd8f06
pgmigrate working on second run
jtotoole Jun 16, 2021
6f7fca2
remove test migration
jtotoole Jun 16, 2021
a8a8664
fix get_schema.sh + mkdirs in Dockerfile
jtotoole Jun 21, 2021
4770aa3
temporarily add step to debug GH actions
jtotoole Jun 21, 2021
9ca6515
change permissions on /schema dir
jtotoole Jun 21, 2021
663be86
move pg_dump statement
jtotoole Jun 21, 2021
7e245b2
try probably superfluous mkdir
jtotoole Jun 21, 2021
62063b2
more mkdir experimenting
jtotoole Jun 21, 2021
cb9b5b1
touch test file to see if dir appears in build
jtotoole Jun 22, 2021
bc71af3
more GH actions debugging
jtotoole Jun 22, 2021
a4469ea
more debugging
jtotoole Jun 22, 2021
6b9ba24
move pg_dump back to initialize_db.sh
jtotoole Jun 22, 2021
038692c
remove foo.txt test
jtotoole Jun 25, 2021
2172459
do not ignore /schema/mediawords.sql
jtotoole Jun 29, 2021
793709a
print test fails
jtotoole Jun 29, 2021
194c34b
Merge branch 'master' of https://github.com/mediacloud/backend into j…
jtotoole Jul 2, 2021
29dbb3f
remove tmate step from GHA
jtotoole Jul 2, 2021
937ee2e
actually gitignore mediawords.sql
jtotoole Jul 2, 2021
fef95ab
do not mount /postgresql-server/schema
jtotoole Jul 2, 2021
60de341
refactor dir structure + mount pgmigrate dir + do not mount schema
jtotoole Jul 6, 2021
cf3b6ab
put pgmigrate callbacks in subdir
jtotoole Jul 6, 2021
db8b9b6
copy pgmigrate dir recursively
jtotoole Jul 6, 2021
9af0a35
run pg_dump in Dockerfile
jtotoole Jul 6, 2021
dde0b1d
remove erroneous chmod
jtotoole Jul 6, 2021
d08fc82
run pg_dump in initialize_db.sh, mv in Dockerfile
jtotoole Jul 6, 2021
60763f0
typo in Dockerfile
jtotoole Jul 6, 2021
a69189c
Merge branch 'master' into jot-pgmigrate
pypt Jul 19, 2021
f70b20c
fix mount point in docker-compose.tests.yml files
jtotoole Jul 20, 2021
5944625
typo in mediawords.sql path
jtotoole Jul 20, 2021
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
The table of contents is too big for display.
Diff view
Diff view
  •  
  •  
  •  
3 changes: 3 additions & 0 deletions .gitignore
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -25,6 +25,9 @@ nytprof*
# Devel::Cover output
cover_db/

# Schema file dumped for reference in development
mediawords.sql

# MediaWords::TM::Dump output
/dump_layout_*

Expand Down
4 changes: 2 additions & 2 deletions apps/cliff-fetch-annotation-and-tag/docker-compose.tests.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -51,8 +51,8 @@ services:
source: ./../postgresql-server/bin/
target: /opt/mediacloud/bin/
- type: bind
source: ./../postgresql-server/schema/
target: /opt/mediacloud/schema/
source: ./../postgresql-server/pgmigrate/
target: /opt/postgresql-server/pgmigrate/
- type: bind
source: ./../postgresql-base/conf/
target: /etc/postgresql/13/main/
4 changes: 2 additions & 2 deletions apps/common/docker-compose.tests.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -94,8 +94,8 @@ services:
source: ./../postgresql-server/bin/
target: /opt/mediacloud/bin/
- type: bind
source: ./../postgresql-server/schema/
target: /opt/mediacloud/schema/
source: ./../postgresql-server/pgmigrate/
target: /opt/postgresql-server/pgmigrate/
- type: bind
source: ./../postgresql-base/conf/
target: /etc/postgresql/13/main/
Expand Down
4 changes: 2 additions & 2 deletions apps/crawler-ap/docker-compose.tests.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -90,8 +90,8 @@ services:
source: ./../postgresql-server/bin/
target: /opt/mediacloud/bin/
- type: bind
source: ./../postgresql-server/schema/
target: /opt/mediacloud/schema/
source: ./../postgresql-server/pgmigrate/
target: /opt/postgresql-server/pgmigrate/
- type: bind
source: ./../postgresql-base/conf/
target: /etc/postgresql/13/main/
Expand Down
4 changes: 2 additions & 2 deletions apps/crawler-fetcher/docker-compose.tests.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -53,8 +53,8 @@ services:
source: ./../postgresql-server/bin/
target: /opt/mediacloud/bin/
- type: bind
source: ./../postgresql-server/schema/
target: /opt/mediacloud/schema/
source: ./../postgresql-server/pgmigrate/
target: /opt/postgresql-server/pgmigrate/
- type: bind
source: ./../postgresql-base/conf/
target: /etc/postgresql/13/main/
Expand Down
4 changes: 2 additions & 2 deletions apps/crawler-provider/docker-compose.tests.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -46,8 +46,8 @@ services:
source: ./../postgresql-server/bin/
target: /opt/mediacloud/bin/
- type: bind
source: ./../postgresql-server/schema/
target: /opt/mediacloud/schema/
source: ./../postgresql-server/pgmigrate/
target: /opt/postgresql-server/pgmigrate/
- type: bind
source: ./../postgresql-base/conf/
target: /etc/postgresql/13/main/
4 changes: 2 additions & 2 deletions apps/create-missing-partitions/docker-compose.tests.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -40,8 +40,8 @@ services:
source: ./../postgresql-server/bin/
target: /opt/mediacloud/bin/
- type: bind
source: ./../postgresql-server/schema/
target: /opt/mediacloud/schema/
source: ./../postgresql-server/pgmigrate/
target: /opt/postgresql-server/pgmigrate/
- type: bind
source: ./../postgresql-base/conf/
target: /etc/postgresql/13/main/
4 changes: 2 additions & 2 deletions apps/cron-generate-daily-rss-dumps/docker-compose.tests.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -43,8 +43,8 @@ services:
source: ./../postgresql-server/bin/
target: /opt/mediacloud/bin/
- type: bind
source: ./../postgresql-server/schema/
target: /opt/mediacloud/schema/
source: ./../postgresql-server/pgmigrate/
target: /opt/postgresql-server/pgmigrate/
- type: bind
source: ./../postgresql-base/conf/
target: /etc/postgresql/13/main/
4 changes: 2 additions & 2 deletions apps/cron-generate-media-health/docker-compose.tests.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -49,8 +49,8 @@ services:
source: ./../postgresql-server/bin/
target: /opt/mediacloud/bin/
- type: bind
source: ./../postgresql-server/schema/
target: /opt/mediacloud/schema/
source: ./../postgresql-server/pgmigrate/
target: /opt/postgresql-server/pgmigrate/
- type: bind
source: ./../postgresql-base/conf/
target: /etc/postgresql/13/main/
4 changes: 2 additions & 2 deletions apps/cron-generate-user-summary/docker-compose.tests.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -43,8 +43,8 @@ services:
source: ./../postgresql-server/bin/
target: /opt/mediacloud/bin/
- type: bind
source: ./../postgresql-server/schema/
target: /opt/mediacloud/schema/
source: ./../postgresql-server/pgmigrate/
target: /opt/postgresql-server/pgmigrate/
- type: bind
source: ./../postgresql-base/conf/
target: /etc/postgresql/13/main/
4 changes: 2 additions & 2 deletions apps/cron-print-long-running-job-states/docker-compose.tests.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -43,8 +43,8 @@ services:
source: ./../postgresql-server/bin/
target: /opt/mediacloud/bin/
- type: bind
source: ./../postgresql-server/schema/
target: /opt/mediacloud/schema/
source: ./../postgresql-server/pgmigrate/
target: /opt/postgresql-server/pgmigrate/
- type: bind
source: ./../postgresql-base/conf/
target: /etc/postgresql/13/main/
4 changes: 2 additions & 2 deletions apps/cron-refresh-stats/docker-compose.tests.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -43,8 +43,8 @@ services:
source: ./../postgresql-server/bin/
target: /opt/mediacloud/bin/
- type: bind
source: ./../postgresql-server/schema/
target: /opt/mediacloud/schema/
source: ./../postgresql-server/pgmigrate/
target: /opt/postgresql-server/pgmigrate/
- type: bind
source: ./../postgresql-base/conf/
target: /etc/postgresql/13/main/
4 changes: 2 additions & 2 deletions apps/cron-rescrape-due-media/docker-compose.tests.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -44,8 +44,8 @@ services:
source: ./../postgresql-server/bin/
target: /opt/mediacloud/bin/
- type: bind
source: ./../postgresql-server/schema/
target: /opt/mediacloud/schema/
source: ./../postgresql-server/pgmigrate/
target: /opt/postgresql-server/pgmigrate/
- type: bind
source: ./../postgresql-base/conf/
target: /etc/postgresql/13/main/
Expand Down
4 changes: 2 additions & 2 deletions apps/cron-rescraping-changes/docker-compose.tests.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -43,8 +43,8 @@ services:
source: ./../postgresql-server/bin/
target: /opt/mediacloud/bin/
- type: bind
source: ./../postgresql-server/schema/
target: /opt/mediacloud/schema/
source: ./../postgresql-server/pgmigrate/
target: /opt/postgresql-server/pgmigrate/
- type: bind
source: ./../postgresql-base/conf/
target: /etc/postgresql/13/main/
4 changes: 2 additions & 2 deletions apps/cron-set-media-primary-language/docker-compose.tests.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -49,8 +49,8 @@ services:
source: ./../postgresql-server/bin/
target: /opt/mediacloud/bin/
- type: bind
source: ./../postgresql-server/schema/
target: /opt/mediacloud/schema/
source: ./../postgresql-server/pgmigrate/
target: /opt/postgresql-server/pgmigrate/
- type: bind
source: ./../postgresql-base/conf/
target: /etc/postgresql/13/main/
4 changes: 2 additions & 2 deletions apps/cron-set-media-subject-country/docker-compose.tests.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -49,8 +49,8 @@ services:
source: ./../postgresql-server/bin/
target: /opt/mediacloud/bin/
- type: bind
source: ./../postgresql-server/schema/
target: /opt/mediacloud/schema/
source: ./../postgresql-server/pgmigrate/
target: /opt/postgresql-server/pgmigrate/
- type: bind
source: ./../postgresql-base/conf/
target: /etc/postgresql/13/main/
4 changes: 2 additions & 2 deletions apps/dump-table/docker-compose.tests.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -40,8 +40,8 @@ services:
source: ./../postgresql-server/bin/
target: /opt/mediacloud/bin/
- type: bind
source: ./../postgresql-server/schema/
target: /opt/mediacloud/schema/
source: ./../postgresql-server/pgmigrate/
target: /opt/postgresql-server/pgmigrate/
- type: bind
source: ./../postgresql-base/conf/
target: /etc/postgresql/13/main/
4 changes: 2 additions & 2 deletions apps/export-tables-to-backup-crawler/docker-compose.tests.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -46,8 +46,8 @@ services:
source: ./../postgresql-server/bin/
target: /opt/mediacloud/bin/
- type: bind
source: ./../postgresql-server/schema/
target: /opt/mediacloud/schema/
source: ./../postgresql-server/pgmigrate/
target: /opt/postgresql-server/pgmigrate/
- type: bind
source: ./../postgresql-base/conf/
target: /etc/postgresql/13/main/
4 changes: 2 additions & 2 deletions apps/extract-and-vector/docker-compose.tests.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -66,8 +66,8 @@ services:
source: ./../postgresql-server/bin/
target: /opt/mediacloud/bin/
- type: bind
source: ./../postgresql-server/schema/
target: /opt/mediacloud/schema/
source: ./../postgresql-server/pgmigrate/
target: /opt/postgresql-server/pgmigrate/
- type: bind
source: ./../postgresql-base/conf/
target: /etc/postgresql/13/main/
Expand Down
4 changes: 2 additions & 2 deletions apps/facebook-fetch-story-stats/docker-compose.tests.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -50,8 +50,8 @@ services:
source: ./../postgresql-server/bin/
target: /opt/mediacloud/bin/
- type: bind
source: ./../postgresql-server/schema/
target: /opt/mediacloud/schema/
source: ./../postgresql-server/pgmigrate/
target: /opt/postgresql-server/pgmigrate/
- type: bind
source: ./../postgresql-base/conf/
target: /etc/postgresql/13/main/
Expand Down
4 changes: 2 additions & 2 deletions apps/import-solr-data-for-testing/docker-compose.tests.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -47,8 +47,8 @@ services:
source: ./../postgresql-server/bin/
target: /opt/mediacloud/bin/
- type: bind
source: ./../postgresql-server/schema/
target: /opt/mediacloud/schema/
source: ./../postgresql-server/pgmigrate/
target: /opt/postgresql-server/pgmigrate/
- type: bind
source: ./../postgresql-base/conf/
target: /etc/postgresql/13/main/
Expand Down
4 changes: 2 additions & 2 deletions apps/import-solr-data/docker-compose.tests.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -72,8 +72,8 @@ services:
source: ./../postgresql-server/bin/
target: /opt/mediacloud/bin/
- type: bind
source: ./../postgresql-server/schema/
target: /opt/mediacloud/schema/
source: ./../postgresql-server/pgmigrate/
target: /opt/postgresql-server/pgmigrate/
- type: bind
source: ./../postgresql-base/conf/
target: /etc/postgresql/13/main/
Expand Down
4 changes: 2 additions & 2 deletions apps/import-stories-feedly/docker-compose.tests.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -49,8 +49,8 @@ services:
source: ./../postgresql-server/bin/
target: /opt/mediacloud/bin/
- type: bind
source: ./../postgresql-server/schema/
target: /opt/mediacloud/schema/
source: ./../postgresql-server/pgmigrate/
target: /opt/postgresql-server/pgmigrate/
- type: bind
source: ./../postgresql-base/conf/
target: /etc/postgresql/13/main/
4 changes: 2 additions & 2 deletions apps/import-stories-scrapehtml/docker-compose.tests.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -49,8 +49,8 @@ services:
source: ./../postgresql-server/bin/
target: /opt/mediacloud/bin/
- type: bind
source: ./../postgresql-server/schema/
target: /opt/mediacloud/schema/
source: ./../postgresql-server/pgmigrate/
target: /opt/postgresql-server/pgmigrate/
- type: bind
source: ./../postgresql-base/conf/
target: /etc/postgresql/13/main/
4 changes: 2 additions & 2 deletions apps/munin-cron/docker-compose.tests.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -60,8 +60,8 @@ services:
source: ./../postgresql-server/bin/
target: /opt/mediacloud/bin/
- type: bind
source: ./../postgresql-server/schema/
target: /opt/mediacloud/schema/
source: ./../postgresql-server/pgmigrate/
target: /opt/postgresql-server/pgmigrate/
- type: bind
source: ./../postgresql-base/conf/
target: /etc/postgresql/13/main/
Expand Down
4 changes: 2 additions & 2 deletions apps/munin-httpd/docker-compose.tests.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -87,8 +87,8 @@ services:
source: ./../postgresql-server/bin/
target: /opt/mediacloud/bin/
- type: bind
source: ./../postgresql-server/schema/
target: /opt/mediacloud/schema/
source: ./../postgresql-server/pgmigrate/
target: /opt/postgresql-server/pgmigrate/
- type: bind
source: ./../postgresql-base/conf/
target: /etc/postgresql/13/main/
Expand Down
4 changes: 2 additions & 2 deletions apps/munin-node/docker-compose.tests.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -40,8 +40,8 @@ services:
source: ./../postgresql-server/bin/
target: /opt/mediacloud/bin/
- type: bind
source: ./../postgresql-server/schema/
target: /opt/mediacloud/schema/
source: ./../postgresql-server/pgmigrate/
target: /opt/postgresql-server/pgmigrate/
- type: bind
source: ./../postgresql-base/conf/
target: /etc/postgresql/13/main/
Expand Down
4 changes: 2 additions & 2 deletions apps/nytlabels-fetch-annotation-and-tag/docker-compose.tests.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -49,8 +49,8 @@ services:
source: ./../postgresql-server/bin/
target: /opt/mediacloud/bin/
- type: bind
source: ./../postgresql-server/schema/
target: /opt/mediacloud/schema/
source: ./../postgresql-server/pgmigrate/
target: /opt/postgresql-server/pgmigrate/
- type: bind
source: ./../postgresql-base/conf/
target: /etc/postgresql/13/main/
4 changes: 2 additions & 2 deletions apps/podcast-transcribe-episode/docker-compose.tests.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -65,8 +65,8 @@ services:
source: ./../postgresql-server/bin/
target: /opt/mediacloud/bin/
- type: bind
source: ./../postgresql-server/schema/
target: /opt/mediacloud/schema/
source: ./../postgresql-server/pgmigrate/
target: /opt/postgresql-server/pgmigrate/
- type: bind
source: ./../postgresql-base/conf/
target: /etc/postgresql/13/main/
Expand Down
45 changes: 36 additions & 9 deletions apps/postgresql-server/Dockerfile
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -5,30 +5,57 @@
FROM gcr.io/mcback/postgresql-base:latest

USER root
RUN \
apt-get -y --no-install-recommends install python3 python3-pip python3-setuptools && \
#
# Upgrade Pip
pip3 install -U pip && \
#
# https://github.com/pypa/pip/issues/5221#issuecomment-382069604
hash -r pip3 && \
apt-get -y --no-install-recommends install build-essential python3-dev libffi-dev libpq-dev && \
#
# Install package to manage schema migrations
pip3 install yandex-pgmigrate==1.0.6 && \
#
# Remove temporary dependencies
apt-get -y remove build-essential python3-dev libffi-dev libpq-dev && \
#
# Cleanup
apt-get -y autoremove && \
Comment thread
jtotoole marked this conversation as resolved.
apt-get -y clean && \
rm -rf /root/.cache/ && \
#
true

# Copy helper scripts, migrations, pgmigrate callbacks/config, create schema dir
RUN \
mkdir -p \
/opt/postgresql-server/bin/ \
/opt/postgresql-server/schema/ \
/opt/postgresql-server/pgmigrate \
&& \
chmod ugo+rw /opt/postgresql-server/pgmigrate && \
true

# Copy helper scripts, schema, migrations
COPY bin/* /opt/postgresql-server/bin/
COPY schema/ /opt/postgresql-server/schema/
COPY pgmigrate/ /opt/postgresql-server/pgmigrate

# Initialize data volume, create users, a database, and initialize it with
# schema
# Initialize data volume, create users + database
# If a new empty volume gets mounted to /var/lib/postgresql/ upon
# container start, Docker will copy the files from the container to the volume

USER postgres
RUN /opt/postgresql-server/bin/initialize_schema.sh
RUN /opt/postgresql-server/bin/initialize_db.sh

# Remove the init script so that someone doesn't accidentally run it in
# production
USER root
RUN rm /opt/postgresql-server/bin/initialize_schema.sh
USER postgres
RUN rm /opt/postgresql-server/bin/initialize_db.sh

# dump schema file for reference in development (run ./dev/get_schema.sh to get local copy)
RUN mv /opt/postgresql-server/pgmigrate/mediawords.sql /opt/mediawords.sql

USER postgres
ENV \
PATH="/opt/postgresql-server/bin:${PATH}" \
#
Expand All @@ -43,4 +70,4 @@ ENV \
VOLUME /var/lib/postgresql/

# Use our own wrapper script which runs schema upgrades first
CMD ["/opt/postgresql-server/bin/postgresql.sh"]
CMD ["/opt/postgresql-server/bin/postgresql.sh"]
Loading