Skip to content

Commit ee9916f

Browse files
dkazancdkazanc
authored
New testing structure with auto-pipelines generation (#542)
* another cpu generated pipeline added, tests reorganisation, a new CI for pipelines * fixing tests1 * offloading yaml pipeline generator scripts to httomo-backends * fixes * fixes2 * changes in the building script * adding pipelines_full folder * docs update, tests modification and helping scripts * fixing ui tests * fixing pipelines ci * first test for big data * removing raw data folder * tests/scripts/ * scripts to generate ground truth extract for comparison in the test * modifying test to exclude image saving * modifying flag to output folder * fixing path to npz file * fixing the bigdata test * further bigtest corrections * corrections to script and test * further tests fixes * temp test debugging * temp test debugging2 * the fix in the name of the file to be opened * further test corrections * working Jenkins test * renaming flags, adding more tests * updates to tests and docs * restart CI * fixing the flag for tests * further test improvments * adding github ci job to run framework tests * github ci correction * github ci correction2 * github ci correction3 * github ci correction4 * github ci correction5 * github ci correction6 * github ci correction7 * github ci correction8 * tests modification * tests modification2 * fixing the bug in parameters modifier * completing the denoising test * test correction * change_value_parameters_method_pipeline correction * tests work * tests work2 * tests work3 * further tests * further tests2 * further tests3 * completing 360 test with paganin * correcting docs * revised corrections * making change_value_parameters_method_pipeline public * remove nbsphinx from the depenedncies in docs build * forcing the older build of sphinx and not 8.2.0 * adding nbsphinx back
1 parent 74b94c9 commit ee9916f

27 files changed

Lines changed: 1276 additions & 1727 deletions

‎.github/workflows/httomo_docs.yml‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -35,9 +35,9 @@ jobs:
3535
run: |
3636
pip install --no-deps httomo-backends
3737
38-
- name: Generate full yaml pipelines using directives
38+
- name: Generate full yaml pipelines using pipeline directives
3939
run: |
40-
python ./docs/source/scripts/yaml_pipelines_generator.py -i ./docs/source/pipelines_full/gpu_pipeline1_directive.yaml -o ./docs/source/pipelines_full/gpu_pipeline1.yaml
40+
python ./docs/source/scripts/execute_pipelines_build.py -o ./docs/source/pipelines_full/
4141
4242
- name: Build docs
4343
run: sphinx-build -a -E -b html ./docs/source/ ./docs/build/

.github/workflows/run_tests_iris.yml renamed to .github/workflows/run_tests_framework_iris.yml

Lines changed: 8 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -1,4 +1,4 @@
1-
name: HTTomo tests
1+
name: HTTomo framework tests
22

33
on:
44
pull_request:
@@ -38,9 +38,14 @@ jobs:
3838
pip install --upgrade --force-reinstall pillow
3939
pip install httomolibgpu tomobar
4040
pip install --no-deps httomo-backends
41-
pip install .
41+
pip install .
4242
micromamba list
4343
44-
- name: Run HTTomo tests
44+
- name: Generate full yaml pipelines using pipeline directives
45+
run: |
46+
pip install "ruamel.yaml>0.18.0"
47+
python ./docs/source/scripts/execute_pipelines_build.py -o ./docs/source/pipelines_full/
48+
49+
- name: Run HTTomo framework tests
4550
run: |
4651
pytest tests/

‎.github/workflows/run_tests_pipelines_iris.yml‎

Lines changed: 51 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,51 @@
1+
name: HTTomo pipelines tests
2+
3+
on:
4+
pull_request:
5+
branches:
6+
- main
7+
push:
8+
branches:
9+
- main
10+
11+
jobs:
12+
iris-gpu:
13+
runs-on: iris-gpu
14+
container:
15+
image: nvidia/cuda:12.6.3-devel-ubi8
16+
env:
17+
NVIDIA_VISIBLE_DEVICES: ${{ env.NVIDIA_VISIBLE_DEVICES }}
18+
19+
defaults:
20+
run:
21+
shell: bash -l {0}
22+
23+
steps:
24+
- name: Checkout repository code
25+
uses: actions/checkout@v4
26+
27+
- name: Create conda environment
28+
uses: mamba-org/setup-micromamba@v1
29+
with:
30+
environment-file: conda/environment.yml
31+
environment-name: httomo
32+
post-cleanup: 'all'
33+
init-shell: bash
34+
35+
- name: Install httomo libraries, httomo-backends and httomo
36+
run: |
37+
micromamba activate httomo
38+
pip install --upgrade --force-reinstall pillow
39+
pip install httomolibgpu tomobar
40+
pip install --no-deps httomo-backends
41+
pip install .
42+
micromamba list
43+
44+
- name: Generate full yaml pipelines using pipeline directives
45+
run: |
46+
pip install "ruamel.yaml>0.18.0"
47+
python ./docs/source/scripts/execute_pipelines_build.py -o ./docs/source/pipelines_full/
48+
49+
- name: Run HTTomo pipelines tests (small data)
50+
run: |
51+
pytest tests/test_pipeline_small.py --small_data

‎docs/source/doc-conda-requirements.yml‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -4,11 +4,11 @@ channels:
44
dependencies:
55
- python>=3.10,<3.13
66
- numpy
7-
- sphinx
7+
- sphinx>=8.0,<8.2.0
88
- sphinx-book-theme
9-
- nbsphinx
109
- pandoc
1110
- jinja2
11+
- nbsphinx
1212
- sphinx-design
1313
- sphinx-copybutton
1414
- pyyaml

‎docs/source/pipelines/yaml.rst‎

Lines changed: 26 additions & 30 deletions
Original file line numberDiff line numberDiff line change
@@ -3,70 +3,66 @@
33
Full YAML pipelines
44
==============================
55

6-
This is a collection of ready to be used pipeline templates aka process lists for HTTomo.
6+
This is a collection of ready to be used full pipelines or process lists for HTTomo.
77
See more on :ref:`explanation_process_list` and how to :ref:`howto_process_list`.
88

9-
.. _tutorials_pl_templates_cpu:
9+
HTTomo mainly targets GPU computations, therefore the use of :ref:`tutorials_pl_templates_gpu` is
10+
preferable. However, when the GPU device is not available or a GPU method is not implemented, the use of
11+
:ref:`tutorials_pl_templates_cpu` is possible.
1012

11-
CPU Pipeline templates
12-
----------------------------
13+
.. note:: The combination of both GPU and CPU methods is possible. If one expects to achieve the faster performance, please use the GPU methods provided, where possible.
1314

14-
CPU-pipelines mostly use TomoPy methods that are executed on the CPU and expected to be slower.
15+
.. _tutorials_pl_templates_gpu:
1516

16-
.. dropdown:: Basic TomoPy's (CPU-only) pipeline for the classical 180-degrees scan
17+
GPU Pipeline templates
18+
-----------------------
1719

18-
.. literalinclude:: ../../../tests/samples/pipeline_template_examples/pipeline_cpu1.yaml
19-
:language: yaml
20+
The GPU-pipelines consist of methods from httomolibgpu (GPU) and httomolib (CPU) backend :ref:`backends_list`. Those libraries are supported directly by the HTTomo development team.
2021

21-
.. dropdown:: TomoPy's pipeline where :ref:`previewing` is demonstrated
22+
.. dropdown:: GPU pipeline with auto-centering and the FBP reconstruction method.
2223

23-
.. literalinclude:: ../../../tests/samples/pipeline_template_examples/pipeline_cpu2.yaml
24+
.. literalinclude:: ../pipelines_full/gpu_pipelineFBP.yaml
2425
:language: yaml
2526

26-
.. dropdown:: This pipeline shows how "calculate_stats" module extracts global statistics in order to rescale data for saving 8-bit images
27+
.. dropdown:: GPU pipeline as above and Total Variation denoising on the result of the FBP reconstruction.
2728

28-
.. literalinclude:: ../../../tests/samples/pipeline_template_examples/pipeline_cpu3.yaml
29+
.. literalinclude:: ../pipelines_full/gpu_pipelineFBP_denoising.yaml
2930
:language: yaml
3031

31-
.. _tutorials_pl_templates_gpu:
32+
.. _tutorials_pl_templates_cpu:
3233

33-
GPU Pipeline templates
34-
----------------------------
34+
CPU Pipeline templates
35+
-----------------------
3536

36-
It is recommended to use GPU-based pipelines and methods from the httomolib and httomolibgpu libraries. Those libraries are supported directly by HTTomo development team.
37+
The CPU-pipelines mostly use TomoPy methods. They are executed solely on the CPU and therefore expected to be slower than the GPU pipelines.
3738

38-
.. dropdown:: Basic GPU pipeline which uses functions from the httomolibgpu library.
39+
.. dropdown:: CPU pipeline using auto-centering and the gridrec reconstruction method on the CPU (TomoPy).
3940

40-
.. literalinclude:: ../pipelines_full/gpu_pipeline1.yaml
41+
.. literalinclude:: ../pipelines_full/cpu_pipeline_gridrec.yaml
4142
:language: yaml
4243

4344

4445
.. _tutorials_pl_templates_dls:
4546

4647
DLS-specific templates
47-
----------------------------
48+
----------------------
4849

49-
Those pipelines will use the methods from the httomolib and httomolibgpu libraries.
50-
51-
.. dropdown:: An example of a typical DIAD (k11) beamline piepeline.
52-
53-
.. literalinclude:: ../../../tests/samples/pipeline_template_examples/DLS/01_diad_pipeline_gpu.yaml
54-
:language: yaml
50+
Those pipelines are specific to Diamond Light Source processing strategies and can vary between different tomographic beamlines.
5551

56-
.. dropdown:: Pipeline for 360-degrees data with automatic CoR finding and stitching to 180-degrees data.
52+
.. dropdown:: An example of DIAD-k11 beamline pipeline with auto-centering and FBP reconstruction on the GPU.
5753

58-
.. literalinclude:: ../../../tests/samples/pipeline_template_examples/pipeline_360deg_gpu2.yaml
54+
.. literalinclude:: ../pipelines_full/gpu_diad_FBP.yaml
5955
:language: yaml
6056

61-
.. dropdown:: Pipeline for 360-degrees data with automatic CoR finding and stitching to 180-degrees data. Iterative reconstruction
57+
.. dropdown:: Pipeline for 360-degrees data with automatic CoR/overlap finding and stitching to 180-degrees data.
6258

63-
.. literalinclude:: ../../../tests/samples/pipeline_template_examples/pipeline_360deg_iterative_gpu3.yaml
59+
.. literalinclude:: ../pipelines_full/gpu_360_paganin_FBP.yaml
6460
:language: yaml
6561

6662
.. _tutorials_pl_templates_sweeps:
6763

6864
Parameter Sweeps templates
69-
----------------------------
65+
--------------------------
7066

7167
These templates demonstrate how to perform a sweep across multiple values of a
7268
single parameter (see :ref:`parameter_sweeping` for more details).

‎docs/source/pipelines_full/__init__.py‎

Whitespace-only changes.

‎docs/source/pipelines_full/gpu_pipeline1_directive.yaml‎

Lines changed: 0 additions & 18 deletions
This file was deleted.

‎docs/source/scripts/execute_pipelines_build.py‎

Lines changed: 61 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,61 @@
1+
#!/usr/bin/env python3
2+
# -*- coding: utf-8 -*-
3+
# ---------------------------------------------------------------------------
4+
# Copyright 2022 Diamond Light Source Ltd.
5+
#
6+
# Licensed under the Apache License, Version 2.0 (the "License");
7+
# you may not use this file except in compliance with the License.
8+
# You may obtain a copy of the License at
9+
#
10+
# http://www.apache.org/licenses/LICENSE-2.0
11+
#
12+
# Unless required by applicable law or agreed to in writing, software
13+
# distributed under the License is distributed on an "AS IS" BASIS,
14+
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
15+
# See the License for the specific language governing permissions and
16+
# limitations under the License.
17+
# ---------------------------------------------------------------------------
18+
# Created By : Tomography Team <scientificsoftware@diamond.ac.uk>
19+
# Created Date: 30/January/2025
20+
# version ='0.1'
21+
# ---------------------------------------------------------------------------
22+
"""Executing full-pipeline generation for HTTomo using YAML templates from httomo-backends
23+
and yaml_pipelines_generator script available also in httomo-backends.
24+
"""
25+
26+
import argparse
27+
import os
28+
import glob
29+
import httomo_backends
30+
from httomo_backends.scripts.yaml_pipelines_generator import yaml_pipelines_generator
31+
32+
33+
def get_args():
34+
parser = argparse.ArgumentParser(
35+
description="Script that generates YAML pipelines for HTTomo "
36+
"using YAML templates from httomo-backends."
37+
)
38+
parser.add_argument(
39+
"-o",
40+
"--output",
41+
type=str,
42+
default="./",
43+
help="Full path to the output pipelines folder.",
44+
)
45+
return parser.parse_args()
46+
47+
48+
if __name__ == "__main__":
49+
path_to_httomobackends = os.path.dirname(httomo_backends.__file__)
50+
args = get_args()
51+
path_to_httomo_pipelines = args.output
52+
pipelines_folder = path_to_httomobackends + "/pipelines_full/"
53+
# loop over all pipeline directive files and running the generator
54+
for filepath in glob.iglob(pipelines_folder + "*.yaml"):
55+
basename = os.path.basename(filepath)
56+
outputfile_name = os.path.normpath(basename.replace(r"_directive", r""))
57+
yaml_pipelines_generator(
58+
filepath, path_to_httomobackends, path_to_httomo_pipelines + outputfile_name
59+
)
60+
message_str = f"{outputfile_name} has been generated."
61+
print(message_str)

0 commit comments

Comments
 (0)