Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
19 commits
Select commit Hold shift + click to select a range
25445ff
First step towards v5 compatibility
bartvanb Sep 22, 2025
bfb2d79
Implement data extraction functions, fix imports, start of preprocess…
bartvanb Sep 22, 2025
3fd50d9
Add preprocessing function, update docs, and rename kwargs to arguments
bartvanb Sep 22, 2025
234accf
Replaced setup.py by pyproject.toml
bartvanb Sep 23, 2025
8d66739
Minor bugfixes and changes, also optimization of spacing
bartvanb Sep 23, 2025
231b842
Remove dataframe decorator for preprocessing functions as it is now i…
bartvanb Sep 23, 2025
4e8eb6c
Seperated test files for the different actions
frankcorneliusmartin Oct 20, 2025
8df35ea
Merge pull request #12 from vantage6/change/update-mockclient-to-mock…
bartvanb Oct 24, 2025
14e945b
Specify v5 in Dockerfile base
bartvanb Nov 13, 2025
2b2d907
Minor updates and bugfixes to work with v5
bartvanb Nov 13, 2025
ae186cf
Move definition method and arguments out of input dict, as is require…
bartvanb Nov 13, 2025
8c79591
Modify headers test data
bartvanb Nov 14, 2025
5f84161
Default test scripts with 3 nodes instead of 1
bartvanb Nov 14, 2025
b3af417
Add test script for preprocessing tasks, and bugfixes in data extract…
bartvanb Nov 14, 2025
cf0d0ae
Use proper databases definition for v5
bartvanb Nov 21, 2025
808085a
Change directory of mock tools
bartvanb Nov 26, 2025
e9bb2a4
Fix typo with prereleases
bartvanb May 12, 2026
b12fac0
Change references to harbor to ghcr.io
bartvanb May 12, 2026
41204f4
Update automated image build to use vantage6 standard github action p…
bartvanb May 19, 2026
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
19 changes: 1 addition & 18 deletions Dockerfile.jinja
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
# basic python3 image as base
FROM harbor2.vantage6.ai/infrastructure/algorithm-base
FROM ghcr.io/vantage6/infrastructure/algorithm-base:5.0

# This is a placeholder that should be overloaded by invoking
# docker build with '--build-arg PKG_NAME=...'
Expand All @@ -9,23 +9,6 @@ ARG PKG_NAME="{{algorithm_name}}"
COPY . /app
RUN pip install /app

{% if use_vpn %}
# Specify the ports that are used for VPN communication, along with a label
# that helps you identify them. As an example, port 8888 is used here. The label
# must be specified as the port number with a 'p' prefix, e.g. 'p8888'.
{% if vpn_expose %}
{%- for port_dict in vpn_expose %}
EXPOSE {{port_dict.port}}
LABEL p{{port_dict.port}} = '{{port_dict.label}}'
{% endfor %}
{% else %}
# TODO provide a sensible label below. Feel free to add more ports if needed by
# adding additional EXPOSE and LABEL commands.
EXPOSE 8888
LABEL p8888='some_label'
{% endif %}
{% endif %}

# Set environment variable to make name of the package available within the
# docker image.
ENV PKG_NAME=${PKG_NAME}
Expand Down
40 changes: 40 additions & 0 deletions Makefile.jinja
Original file line number Diff line number Diff line change
@@ -0,0 +1,40 @@
VANTAGE6_VERSION ?= 5.0.0
TAG ?= latest
IMAGE ?= {% if has_gh_pipeline %}{{ docker_image }}{% else %}ghcr.io/vantage6/algorithm/{{ algorithm_name }}{% endif %}
PLATFORMS ?= linux/amd64

# Major segment of VANTAGE6_VERSION (e.g. 5.0.0 -> 5)
VANTAGE6_MAJOR := $(firstword $(subst ., ,$(VANTAGE6_VERSION)))

# When true, also tag/push ${IMAGE}:${VANTAGE6_MAJOR}. CI sets false on algorithm
# prereleases if that major tag already exists (see vantage6-workflows release).
INCLUDE_V6_MAJOR_TAG ?= true

# Use `make PUSH_REG=true` to push images to registry after building
PUSH_REG ?= false

# We use a conditional (true on any non-empty string) later. To avoid
# accidents, we don't use user-controlled PUSH_REG directly.
# See: https://www.gnu.org/software/make/manual/html_node/Conditional-Functions.html
_condition_push :=
ifeq ($(PUSH_REG), true)
_condition_push := not_empty_so_true
endif

.PHONY: image
image:
@set -e; \
echo "Building ${IMAGE}:${TAG}-v6-${VANTAGE6_VERSION}"; \
echo "Building ${IMAGE}:latest"; \
EXTRA_MAJOR=""; \
if [ "$(INCLUDE_V6_MAJOR_TAG)" = true ]; then \
echo "Building ${IMAGE}:${VANTAGE6_MAJOR}"; \
EXTRA_MAJOR='--tag ${IMAGE}:${VANTAGE6_MAJOR}'; \
fi; \
docker buildx build \
--tag ${IMAGE}:${TAG}-v6-${VANTAGE6_VERSION} \
--tag ${IMAGE}:latest \
$$EXTRA_MAJOR \
--platform ${PLATFORMS} \
-f ./Dockerfile \
$(if ${_condition_push},--push .,.)
54 changes: 36 additions & 18 deletions README.md.jinja
Original file line number Diff line number Diff line change
Expand Up @@ -18,15 +18,16 @@ Please ensure to execute the following steps. The steps are also indicated with
TODO statements in the generated code - so you can also simply search the
code for TODO instead of following the checklist below.

- [ ] Include a URL to your code repository in setup.py.
- [ ] Fill out the fields in the `pyproject.toml` file, such as a URL to your code
repository. Alternatively, remove these fields.
- [ ] Implement your algorithm functions.
- [ ] You are free to add more arguments to the functions. Be sure to add them
*after* the `client` and dataframe arguments.
- [ ] When adding new arguments, if you run the `test/test.py` script, be sure
to include values for these arguments in the `client.task.create()` calls
that are available there.
- [ ] If you are using Python packages that are not in the standard library, add
them to the `requirements.txt` and `setup.py` file.
them to the `pyproject.toml` file. Note that `pandas` is already included by default.
{% if has_docs %}
- [ ] Fill in the documentation template. This will help others to understand your
algorithm, be able to use it safely, and to contribute to it.
Expand All @@ -37,14 +38,19 @@ code for TODO instead of following the checklist below.
{% endif %}
- [ ] If you want to submit your algorithm to a vantage6 algorithm store, be sure
to fill in everything in ``algorithm_store.json`` (and be sure to update
it if you change function names, arguments, etc.).
it if you change function names, arguments, etc.). It is recommended to run
``v6 algorithm generate-store-json`` to automatically generate the file - this
should work especially well if you have added proper docstrings to your functions.
Note that you do need the `vantage6` CLI to be able to use this command, which can be
installed by e.g. running `pip install vantage6` (or `uv pip install vantage6`).
{% if has_gh_pipeline %}
- [ ] Create a ``DOCKER_USER`` and ``DOCKER_PASSWORD`` secret in the GitHub repository
settings. This will be used to push the Docker image to the registry in the github
pipeline.
{% endif %}
{% if use_vpn %}
- [ ] Review the EXPOSE and LABEL commands in the Dockerfile for VPN
- [ ] Ensure the GitHub repository can publish packages (for ``ghcr.io``, grant
``packages: write`` — the reusable workflow does this; you need a ``GITHUB_TOKEN``
with appropriate permissions, which is the default for Actions). Alternatively, you
can create your own custom GitHub Actions workflow file.
- [ ] If you use a registry other than GitHub Container Registry, add a repository
secret named ``registry_password`` for login (the reusable workflow uses
``secrets: inherit``). For GHCR with the default token, no extra secret is required.
{% endif %}
- [ ] Finally, remove this checklist section to keep the README clean.

Expand All @@ -56,15 +62,27 @@ create a Docker image of your algorithm.
{% if has_gh_pipeline %}
#### Automatically

The easiest way to create a Docker image is to use the GitHub Actions pipeline to
automatically build and push the Docker image. All that you need to do is push a
commit to the ``main`` branch.

#### Manually
The repository includes a release workflow that calls
``vantage6/vantage6-workflows`` (``algorithm-release.yml``). Push a
**semantic version tag** (``x.y.z`` or with a prerelease suffix) to trigger a build;
the workflow runs ``make image`` with the correct tags and pushes to your configured
image name.

{% endif %}
A Docker image can be created by executing the following command in the root of your
algorithm directory:
#### Manually via ``make``

You can build (and optionally push) locally:

```bash
make image
# or with push:
make image PUSH_REG=true TAG=1.0.0 VANTAGE6_VERSION=5.0.0
```

#### Manually via ``docker``

Alternatively, a Docker image can be created by executing the following command in the
root of your algorithm directory:

```bash
docker build -t [my_docker_image_name] .
Expand All @@ -79,9 +97,9 @@ docker tag [my_docker_image_name] [another_image_name]
```

This way, you can e.g. do
`docker tag local_average_algorithm harbor2.vantage6.ai/algorithms/average` to
`docker tag local_average_algorithm ghcr.io/vantage6/algorithm/average` to
make the algorithm available on a remote Docker registry (in this case
`harbor2.vantage6.ai`).
`ghcr.io`).

Finally, you need to push the image to the Docker registry. This can be done
by running
Expand Down
27 changes: 11 additions & 16 deletions algorithm_store.json.jinja
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
{
"name": "{{algorithm_name}}",
"image": "{{docker_image}}",
"vantage6_version": "4.6",
"vantage6_version": "5.0",
"code_url": "https://mygitrepo.org",
"documentation_url": "",
"partitioning": "horizontal",
Expand All @@ -10,17 +10,12 @@
{
"name": "{{central_function_name}}",
"description": "{{algorithm_description}}",
"type": "central",
"step_type": "central_compute",
"databases": [
{%- for idx in range(partial_function_number_databases) -%}
{%- for idx in range(federated_function_number_databases) -%}
{
"name": "Partial database {{idx + 1}}"
}{%- if not loop.last or central_function_number_databases > 0 -%},{%- endif -%}
{% endfor %}
{% for idx in range(central_function_number_databases) %}
{
"name": "Central database {{idx + 1}}"
}{%- if not loop.last -%},{%- endif -%}
"name": "Central database {{idx + 1}}"
}{%- if not loop.last or federated_function_number_databases > 0 -%},{%- endif -%}
{% endfor %}
],
"arguments": [
Expand All @@ -33,21 +28,21 @@
{% endfor %}
]
}{%- endif -%}
{%- if has_central_function and has_partial_function -%},{%- endif -%}
{% if has_partial_function -%}
{%- if has_central_function and has_federated_function -%},{%- endif -%}
{% if has_federated_function -%}
{
"name": "{{partial_function_name}}",
"name": "{{federated_function_name}}",
"description": "",
"type": "federated",
"step_type": "federated_compute",
"databases": [
{%- for idx in range(partial_function_number_databases) -%}
{%- for idx in range(federated_function_number_databases) -%}
{
"name": "Database {{idx + 1}}"
}{%- if not loop.last -%},{%- endif -%}
{% endfor %}
],
"arguments": [
{%- for arg in partial_args -%}
{%- for arg in federated_args -%}
{
"name": "{{arg}}",
"type": "",
Expand Down
10 changes: 6 additions & 4 deletions cleanup.py
Original file line number Diff line number Diff line change
Expand Up @@ -24,10 +24,12 @@ def cleanup() -> None:
print("Removing LICENSE file as no license was chosen...")
Path("LICENSE").unlink()

# Remove partial function files if partial function is not defined
if not copier_config.get("has_partial_function"):
print("Removing partial function file as partial function is not defined...")
Path(algorithm_name, "partial.py").unlink()
# Remove federated function files if federated function is not defined
if not copier_config.get("has_federated_function"):
print(
"Removing federated function file as federated function is not defined..."
)
Path(algorithm_name, "federated.py").unlink()

# Remove central function files if central function is not defined
if not copier_config.get("has_central_function"):
Expand Down
Loading