Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
38 commits
Select commit Hold shift + click to select a range
e8589bc
enh: simplify CI workflow with test.ini approach
RaghavaAlajangi Apr 24, 2025
edd0f0f
fix: update MinIO health check command in CI workflow
RaghavaAlajangi Apr 24, 2025
427f6b3
fix: correct health check commands for PostgreSQL and MinIO in CI wor…
RaghavaAlajangi Apr 24, 2025
0449c59
fix: options block to a single line
RaghavaAlajangi Apr 24, 2025
a842913
fix: add missing environment variables for PostgreSQL and MinIO servi…
RaghavaAlajangi Apr 24, 2025
06ba8f8
fix: update CKAN command to use docker-test.ini for database initiali…
RaghavaAlajangi Apr 24, 2025
657ace4
fix: install DCOR extensions in the DCOR-CKAN container
RaghavaAlajangi Apr 24, 2025
adfece6
fix: health check for CKAN before installing requirements
RaghavaAlajangi May 5, 2025
aa557db
fix: update CKAN image to version 2.11 and add hostname display step
RaghavaAlajangi May 6, 2025
8f3f585
fix: update CKAN image to version 2.11.1 in workflow configuration
RaghavaAlajangi May 6, 2025
f0cb710
fix: add hostname option to CKAN container configuration
RaghavaAlajangi May 6, 2025
d1f5696
fix: remove hostname display and health check steps for CKAN in workflow
RaghavaAlajangi May 6, 2025
e055385
fix: update test configuration to use docker-test.ini for pytest
RaghavaAlajangi May 6, 2025
bfb6544
ci: install latest dcor_control
paulmueller May 7, 2025
93cc537
ci: set correct hostname
paulmueller May 7, 2025
a350b3b
ci: install latest dcor_control from gh
paulmueller May 7, 2025
8b24044
enh: set correct docker postgres credentials
paulmueller May 7, 2025
87b2e75
ci: show ckan config
paulmueller May 7, 2025
194d1db
ci: use latest ckan and postgres directly from ckan devs
paulmueller May 7, 2025
67c4659
ci: initialize db
paulmueller May 7, 2025
0c07d15
ci: base everything on test-core.ini
paulmueller May 7, 2025
063f83f
ci: remove datastore config due to error
paulmueller May 7, 2025
09acf12
ci: datastore allowed_functions.txt not found
paulmueller May 7, 2025
2094518
ci: datastore allowed_functions.txt not found
paulmueller May 7, 2025
9913e35
ci: remove bad db init and remove docker-test.ini
paulmueller May 7, 2025
21a9d37
ci: curl not available
paulmueller May 7, 2025
1c1e87f
ci: try to fix extension module not found
paulmueller May 7, 2025
705e3f3
docs
paulmueller May 7, 2025
84f286a
ci: expand matrix
paulmueller May 7, 2025
a4224f1
ci: remove ckan 2.10
paulmueller May 7, 2025
a25fc09
ci: docs
paulmueller May 7, 2025
fcc73ae
ci: use minio official image and configure it in CI for transparency
RaghavaAlajangi May 9, 2025
f1c5f0f
ci: update MinIO image to edge-cicd and remove entrypoint command
RaghavaAlajangi May 9, 2025
3899786
ci: update MinIO client download method
RaghavaAlajangi May 9, 2025
b71582f
ci: update MinIO secret key environment variable name
RaghavaAlajangi May 9, 2025
ffa3db2
ci: configure MinIO to run in distributed mode with console address
RaghavaAlajangi May 9, 2025
ce18883
ci: remove MinIO distributed mode command and comment out policy atta…
RaghavaAlajangi May 9, 2025
34b79ad
ci: update MinIO configuration and remove user management commands
RaghavaAlajangi May 9, 2025
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
127 changes: 92 additions & 35 deletions .github/workflows/check.yml
Original file line number Diff line number Diff line change
@@ -1,47 +1,104 @@
name: Checks

on:
push:
pull_request:

# only allow one job to run
concurrency: general_workflow
on: [push, pull_request]

jobs:
test:
runs-on: ubuntu-latest
strategy:
matrix:
ckan-version: ["2.11"]
dcor_control: ["develop", "pypi"]
fail-fast: false

steps:
- name: Checkout code
uses: actions/checkout@v2

- name: Set up Docker
uses: docker/setup-buildx-action@v2
name: DCOR-CKAN ${{ matrix.ckan-version }} ${{ matrix.dcor_control }}
runs-on: ubuntu-latest
container:
image: ckan/ckan-dev:${{ matrix.ckan-version }}
options: --user root --hostname dcor-dev-docker

- name: Install Docker Compose
run: |
sudo curl -L "https://github.com/docker/compose/releases/download/v2.17.2/docker-compose-$(uname -s)-$(uname -m)" -o /usr/local/bin/docker-compose
sudo chmod +x /usr/local/bin/docker-compose
docker-compose --version # Verify installation
services:
solr:
image: ckan/ckan-solr:${{ matrix.ckan-version }}-solr9
postgres:
image: ckan/ckan-postgres-dev:${{ matrix.ckan-version }}
env:
POSTGRES_USER: postgres
POSTGRES_PASSWORD: postgres
POSTGRES_DB: postgres
options: >-
--health-cmd "pg_isready"
--health-interval 10s
--health-timeout 5s
--health-retries 10
redis:
image: redis:3
minio:
image: raghavaalajangi95/dcor-minio:latest
# Note:
# The official MinIO setup uses standalone mode, which has limitations:
# - Certain 'mc admin' commands, such as user and policy management, are not supported.
# - Using these commands will result in errors like:
# "This 'admin' API is not supported by server in 'mode-server-fs'"
# - `minio/minio:edge-cicd` container from MinIO is a good alternative,
# but it is still not supporting 'mc admin policy' command and the container is not managed properly.
# References:
# - https://github.com/minio/minio/discussions/16099
# - https://github.com/orgs/community/discussions/26688#discussioncomment-4183701
# - https://stackoverflow.com/questions/64031598/creating-a-minios3-container-inside-a-github-actions-yml-file/64188150#64188150
env:
MINIO_ROOT_USER: minioadmin
MINIO_ROOT_PASSWORD: minioadmin
MINIO_NEW_USER: dcor
MINIO_NEW_SECRET_KEY: secretkey12345678
options: >-
--health-cmd "curl -f http://localhost:9000/minio/health/ready || exit 1"
--health-interval 10s
--health-timeout 5s
--health-retries 10
ports:
- 9000:9000
- 9001:9001

- name: Start Docker Compose services
run: docker-compose --env-file docker-ci.env -f docker-compose.ci.yml up -d
env:
CKAN_SQLALCHEMY_URL: postgresql://ckan_default:pass@postgres/ckan_test
CKAN_DATASTORE_WRITE_URL: postgresql://datastore_write:pass@postgres/datastore_test
CKAN_DATASTORE_READ_URL: postgresql://datastore_read:pass@postgres/datastore_test
CKAN_SOLR_URL: http://solr:8983/solr/ckan
CKAN_REDIS_URL: redis://redis:6379/1

- name: Wait for CKAN to be healthy
run: docker exec dcor_ckan bash -c "while ! wget -qO- http://localhost:5000; do sleep 5; done"
steps:
- uses: actions/checkout@v4

- name: List running services (for debugging)
run: docker-compose -f docker-compose.ci.yml ps
- name: Install and setup DCOR
run: |
# pip3 install dcor_control
pip3 install git+https://github.com/DCOR-dev/dcor_control
# use test-core.ini as default
cp /srv/app/src/ckan/test-core.ini /srv/app/ckan.ini
# initialize CKAN database
ckan -c /srv/app/ckan.ini db init
# fix location of allowed datastore commands
# https://github.com/ckan/ckan/issues/8926
mkdir -p /srv/app/ckanext/datastore/tests/
wget -O /srv/app/ckanext/datastore/tests/allowed_functions.txt https://raw.githubusercontent.com/ckan/ckan/refs/heads/master/ckanext/datastore/allowed_functions.txt
# complement configuration
dcor inspect --assume-yes

- name: Run Tests
run: bash ./docker-run-tests.sh
- name: Install latest DCOR extensions from GitHub
if: matrix.dcor_control == 'develop'
run: |
dcor develop --yes
dcor inspect --assume-yes

- name: Upload Coverage to GitHub
uses: actions/upload-artifact@v4
with:
name: coverage-report
path: coverage.xml
- name: Install extension and requirements
run: |
pip install -e .
pip install -r ./ckanext/dc_view/tests/requirements.txt

- name: Bring Down Docker Compose
if: always()
run: docker-compose -f docker-compose.ci.yml down
- name: Run tests
run: |
pytest -v --ckan-ini=/srv/app/ckan.ini \
--cov=ckanext.dc_view \
--cov-report=xml \
--cov-append \
--disable-warnings \
ckanext/dc_view/tests/
83 changes: 0 additions & 83 deletions docker-ci.env

This file was deleted.

91 changes: 0 additions & 91 deletions docker-compose.ci.yml

This file was deleted.

38 changes: 0 additions & 38 deletions docker-run-tests.sh

This file was deleted.