Changement de la parallélisation des traitements avec rioxarray

.github/workflows/cd.yml

@@ -0,0 +1,43 @@
+name: CD Workflow
+
+on:
+  push:
+    branches:
+      - "main"
+
+permissions:
+  contents: read
+  id-token: write
+
+jobs:
+  deploy:
+    runs-on: ubuntu-latest
+    environment:
+      name: pypi
+    steps:
+      - uses: actions/checkout@v4
+
+      - name: Setup Python 3.8.13
+        uses: actions/setup-python@v3
+        with:
+          python-version: "3.8.13"
+
+      - name: Setup Miniconda
+        uses: conda-incubator/setup-miniconda@v2
+        with:
+          python-version: 3.8.13
+          auto-activate-base: false
+
+      - name: Build package
+        shell: bash -l {0}
+        run: |
+          conda create -n deploy_env python=3.8.13 libgdal=3.5.2 build -c conda-forge -y
+          conda activate deploy_env
+          python -m build -C--global-option=bdist_wheel -C--global-option=--build-number=0 --wheel
+
+      - name: Publish package
+        uses: pypa/gh-action-pypi-publish@release/v1
+        with:
+          user: __token__
+          verbose: true
+          password: ${{ secrets.PYPI_PASSWORD }}

.github/workflows/ci.yml

@@ -0,0 +1,53 @@
+name: CI Workflow
+
+on:
+  pull_request:
+    branches:
+      - "main"
+permissions:
+  contents: read
+
+jobs:
+  quality:
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v4
+      - name: Setup Python 3.8.13
+        uses: actions/setup-python@v3
+        with:
+          python-version: "3.8.13"
+      - name: Create test env
+        shell: bash -l {0}
+        run: |
+          pip install pylint mccabe
+      - name: code quality
+        shell: bash -l {0}
+        run: |
+          pylint --disable=all --fail-under=10 --enable=too-many-statements src/eolab/rastertools/
+          pylint --disable=all --fail-under=10 --enable=too-many-nested-blocks src/eolab/rastertools/
+          ./continuous_integration/scripts/check_mccabe_complexity.sh 25 src/eolab/rastertools
+  test:
+    runs-on: ubuntu-latest
+    steps:
+    - uses: actions/checkout@v4
+    - name: Setup Python 3.8.13
+      uses: actions/setup-python@v3
+      with:
+        python-version: "3.8.13"
+    - name: Setup Miniconda
+      uses: conda-incubator/setup-miniconda@v2
+      with:
+        python-version: 3.8.13
+        auto-activate-base: false
+    - name: Create test env
+      shell: bash -l {0}
+      run: |
+        conda create -n test_env python=3.8.13 libgdal=3.5.2 -c conda-forge -c defaults -y
+        conda activate test_env
+        PIP_NO_BINARY=rasterio,rioxarray pip install .
+        pip install pylint mccabe
+    - name: test
+      shell: bash -l {0}
+      run: |
+        conda activate test_env
+        pytest --cov-fail-under=65 --compare --durations=0

.gitignore

@@ -5,7 +5,6 @@
 # Distribution / packaging 
 .Python build/ 
 develop-eggs/ 
-dist/ 
 downloads/ 
 eggs/ 
 .eggs/ 
@@ -15,10 +14,11 @@ parts/
 sdist/ 
 var/ 
 wheels/ 
-*.whl
+dist/
 *.egg-info/ 
 .installed.cfg 
 *.egg 
+*.whl
 *.manifest 
 *.spec 
 

METADATA

@@ -0,0 +1,195 @@
+Metadata-Version: 2.1
+Name: rastertools
+Version: 0.1.0
+Summary: Collection of tools for raster data
+Home-page: https://github.com/CNES/rastertools
+Author: Olivier Queyrut
+Author-email: olivier.queyrut@cnes.fr
+License: apache v2
+Project-URL: Source, https://github.com/cnes/rastertools
+Project-URL: Changelog, https://github.com/cnes/rastertools/blob/master/CHANGELOG.rst
+Project-URL: Issues, https://github.com/cnes/rastertools/issues
+Platform: any
+Classifier: Development Status :: 5 - Production/Stable
+Classifier: Programming Language :: Python
+Requires-Python: ==3.8.13
+Description-Content-Type: text/x-rst; charset=UTF-8
+License-File: LICENSE.txt
+License-File: AUTHORS.rst
+Requires-Dist: click
+Requires-Dist: rasterio==1.3.0
+Requires-Dist: pytest>=3.6
+Requires-Dist: pytest-cov
+Requires-Dist: geopandas==0.13
+Requires-Dist: python-dateutil==2.9.0
+Requires-Dist: kiwisolver==1.4.5
+Requires-Dist: fonttools==4.53.1
+Requires-Dist: matplotlib==3.7.3
+Requires-Dist: packaging==24.1
+Requires-Dist: Shapely==1.8.5.post1
+Requires-Dist: tomli==2.0.2
+Requires-Dist: Rtree==1.3.0
+Requires-Dist: Pillow==9.2.0
+Requires-Dist: pip==24.2
+Requires-Dist: pyproj==3.4.0
+Requires-Dist: matplotlib
+Requires-Dist: scipy==1.8
+Requires-Dist: pyscaffold
+Requires-Dist: gdal==3.5.0
+Requires-Dist: tqdm==4.66
+Provides-Extra: testing
+Requires-Dist: setuptools; extra == "testing"
+Requires-Dist: pytest; extra == "testing"
+Requires-Dist: pytest-cov; extra == "testing"
+
+============
+Raster tools
+============
+
+This project provides a command line named **rastertools** that enables various calculation tools:
+
+
+- the calculation of radiometric indices on satellite images
+- the calculation of the speed of evolution of radiometry from images between two dates
+- the calculation of zonal statistics of the bands of a raster, that is to say statistics such as min, max, average, etc.
+  on subareas (defined by a vector file) of the area of interest.
+
+The **rastertools** project also aims to make the handling of the following image products transparent:
+
+- Sentinel-2 L1C PEPS (https://peps.cnes.fr/rocket/#/search)
+- Sentinel-2 L2A PEPS (https://peps.cnes.fr/rocket/#/search)
+- Sentinel-2 L2A THEIA (https://theia.cnes.fr/atdistrib/rocket/#/search?collection=SENTINEL2)
+- Sentinel-2 L3A THEIA (https://theia.cnes.fr/atdistrib/rocket/#/search?collection=SENTINEL2)
+- SPOT 6/7 Ortho de GEOSUD (http://ids.equipex-geosud.fr/web/guest/catalog)
+
+It is thus possible to input files in the command line in any of the formats above. 
+It is also possible to specify your own product types by providing a JSON file as a parameter of the command line (cf. docs/usage.rst)
+
+Finally, **rastertools** offers an API for calling these different tools in Python and for extending its capabilities, for example by defining new radiometric indices.
+
+Installation
+============
+
+Create a conda environment by typing the following:
+
+.. code-block:: bash
+
+  conda env create -f environment.yml
+  conda env update -f env_update.yml
+
+The following dependencies will be installed in the ``rastertools`` environment:
+
+- pyscaffold
+- geopandas
+- scipy
+- gdal
+- rasterio
+- tqdm
+
+Install ``rastertools`` in the conda environment by typing the following:
+
+.. code-block:: bash
+
+  conda activate rastertools
+  pip install -e .
+
+.. note::
+
+  Note: Installing in a *virtualenv* does not work properly for this project. For unexplained reasons, 
+  the VRTs that are created in memory by rastertools to handle image products are not properly managed 
+  with an installation in a virtualenv.
+
+For more details, including installation as a Docker or Singularity image, please refer to the documentation. : `Installation <docs/install.rst>`_
+
+
+Usage
+=====
+
+rastertools
+^^^^^^^^^^^
+The rastertools command line is the high-level command for activating the various tools.
+
+.. code-block:: console
+
+  $ rastertools --help
+  usage: rastertools [-h] [-t RASTERTYPE] [--version] [-v] [-vv]
+                     {filter,fi,radioindice,ri,speed,sp,svf,hillshade,hs,zonalstats,zs,tiling,ti}
+                     ...
+
+  Collection of tools on raster data
+
+  optional arguments:
+    -h, --help            show this help message and exit
+    -t RASTERTYPE, --rastertype RASTERTYPE
+                          JSON file defining additional raster types of input
+                          files
+    --version             show program's version number and exit
+    -v, --verbose         set loglevel to INFO
+    -vv, --very-verbose   set loglevel to DEBUG
+
+  Commands:
+    {filter,fi,radioindice,ri,speed,sp,svf,hillshade,hs,zonalstats,zs,tiling,ti}
+      filter (fi)         Apply a filter to a set of images
+      radioindice (ri)    Compute radiometric indices
+      speed (sp)          Compute speed of rasters
+      svf                 Compute Sky View Factor of a Digital Height Model
+      hillshade (hs)      Compute hillshades of a Digital Height Model
+      zonalstats (zs)     Compute zonal statistics
+      tiling (ti)         Generate image tiles
+
+Calling rastertools returns the following exit codes:
+
+.. code-block:: console
+
+    0: everything went well
+    1: processing error
+    2: incorrect invocation parameters
+
+Details of the various subcommands are presented in the documentation : `Usage <docs/cli.rst>`_
+
+
+Tests & documentation
+=====================
+
+To run tests and generate documentation, the following dependencies must be installed in the conda environment. :
+
+- py.test et pytest-cov (tests execution)
+- sphinx (documentation generation)
+
+Pour cela, exécuter la commande suivante :
+
+.. code-block:: console
+
+  conda env update -f env_test.yml
+
+
+Tests
+^^^^^
+
+The project comes with a suite of unit and functional tests. To run them, 
+launch the command ``pytest tests``. To run specific tests, execute ``pytest tests -k "<nom_du_test>"``.
+
+The tests may perform comparisons between generated files and reference files. 
+In this case, the tests depend on the numerical precision of the platforms. 
+To enable these comparisons, you need to add the option. "--compare" for instance ``pytest tests --compare``.
+
+The execution of the tests includes a coverage analysis via pycov.
+
+Documentation generation
+^^^^^^^^^^^^^^^^^^^^^^^^
+
+To generate the documentation, run: 
+
+.. code-block:: console
+
+  cd docs
+  sphinx-quickstart
+  make html
+
+The documentation is generated using the theme "readthedocs".
+
+Note
+====
+
+This project has been set up using PyScaffold. For details and usage
+information on PyScaffold see https://pyscaffold.org/.

RECORD

@@ -0,0 +1,71 @@
+eolab/rastertools/__init__.py,sha256=DbSf1azum6xDyKo310y0ykW-nJYZW2HOI4_1soFygGo,1387
+eolab/rastertools/filtering.py,sha256=KJZVcX4eQYQbHk5VfAbX8q-VvAtq5r5Q6LmCY9hV2PU,8240
+eolab/rastertools/hillshade.py,sha256=2QkUkAfYEZyIFfc-hJ_nEmvIdoa5XFB5Grd7YJ-DPu4,7084
+eolab/rastertools/main.py,sha256=JMe_9GB754LqywhjBVCIFqgwJW3vm9T5WH5qT6CQ2Ek,9038
+eolab/rastertools/radioindice.py,sha256=paQXc1yjWFPKzigI5O1mhoEFLTlusShmLiSgHrlVXU0,25856
+eolab/rastertools/rastertools.py,sha256=BB5W_bJX1Nq6tNAJsdtpupK_wBoFFVypaTLnG3Noofo,7375
+eolab/rastertools/speed.py,sha256=82gn8hXhvyd7fGkd8P2mB6MiyYma57m_5IFCWxpdtvk,7041
+eolab/rastertools/svf.py,sha256=LJ2Q_A1tcwU_iXXewpNKaboSiaVC85NN5X_6lYTb1M0,5910
+eolab/rastertools/tiling.py,sha256=krRVt1dyPw3LaCqNZosfLPCTvu1shzuC9ByY9BoZDB0,8415
+eolab/rastertools/timeseries.py,sha256=qN7BGA_jM97H2SRnVB7mq60KrHqbRJV7qwM_M37VxOQ,10203
+eolab/rastertools/utils.py,sha256=ce_ZKhr4lI-BQPEPvEm9gFAs0bi4Z5kt4g5YdDQVHVI,2997
+eolab/rastertools/zonalstats.py,sha256=T-op7SucTlXA_bf3yLe22JmKZSJp6pRVshEjJnuENak,25706
+eolab/rastertools/__pycache__/__init__.cpython-38.pyc,sha256=we04K-dthGOyVFpOGT49h_ta9VxeqYuzX7lL0mMsur0,1168
+eolab/rastertools/__pycache__/filtering.cpython-38.pyc,sha256=qTAHn_YMwao4ZUhWXnxMcf1j2kGEoynF1GXI9akRo3Q,6678
+eolab/rastertools/__pycache__/hillshade.cpython-38.pyc,sha256=P9cfRDL3Upttnz-JzrPZ9lvn2cX_dBpjbbFPbtt2Ti4,5330
+eolab/rastertools/__pycache__/main.cpython-38.pyc,sha256=ZQkljzVKCq99AqC5XfNOtqfGilDr5l4LraXzwZwNYsY,8449
+eolab/rastertools/__pycache__/radioindice.cpython-38.pyc,sha256=S-JcrFkIHkcrdSLgZ7lgRDPqr9gNuvjBpg35z0jQGao,9723
+eolab/rastertools/__pycache__/rastertools.cpython-38.pyc,sha256=xbVzuffc_o-zNLdcnMBfdP42VGVQd77NI0RNAO3PDjk,7904
+eolab/rastertools/__pycache__/speed.cpython-38.pyc,sha256=o4oPXSbMXKUkOvNTRAGjPzm0WhDsqmai-eOpR5O-v_g,5934
+eolab/rastertools/__pycache__/svf.cpython-38.pyc,sha256=IRYM2sc95TElKvnkiAwaQFDmC-b9f53bI9KDPBpFQ9g,5953
+eolab/rastertools/__pycache__/tiling.cpython-38.pyc,sha256=Gw_QGu8DHGtGKwLJYEjdHQwaZsYnnUEGFkxTq1tGcnY,7250
+eolab/rastertools/__pycache__/timeseries.cpython-38.pyc,sha256=05PbNI5h0odVU_mEJBup6t-iZVJTrV0U0LZLDxnFuaw,8371
+eolab/rastertools/__pycache__/utils.cpython-38.pyc,sha256=WNbk152mH27fEvwqh7vspa_dmeV0UmesTIW8FTtL6pc,3645
+eolab/rastertools/__pycache__/zonalstats.cpython-38.pyc,sha256=TnUEpK3N3qJ6rCTCVCU2qUK1V6mpb9lGRBpInS9g7rU,20867
+eolab/rastertools/cli/__init__.py,sha256=_GH5byj-EOmqUTHK_I_AT0piYoHOM9AvWfFkWq5Ezlg,2071
+eolab/rastertools/cli/filtering.py,sha256=fhGY-oOoKrDtrlaTNUWZJLOxYyVTg3nwB3oADq4F0q4,4627
+eolab/rastertools/cli/hillshade.py,sha256=JjinEhyxPs9uzmEDKR7iuggd-97wUtOCqNn9v0Wh6og,2221
+eolab/rastertools/cli/radioindice.py,sha256=LWMDxu3fau8DtEp1Jmuq2UfKTpogo_CrN7w3pZ28AY4,5212
+eolab/rastertools/cli/speed.py,sha256=v2OuHu0QjNaq9ycY8ilpYwlQOVGQLStBsTBFW7nUF9E,1535
+eolab/rastertools/cli/svf.py,sha256=gnyr72ha-TlVfNbWdjZzdFZtZv13gg0N3Nfg8iP7C1k,2313
+eolab/rastertools/cli/tiling.py,sha256=W6dC5fC0FC7JvKSN7WMnhTgeI8bPRWotBClpiQ50BYM,3187
+eolab/rastertools/cli/timeseries.py,sha256=0OpKK0jCcJJnLb8XC8pjUkKVPcCLx9F84LJxv8tPwPI,3033
+eolab/rastertools/cli/utils_cli.py,sha256=29nyMTEtJ0fhiRfgnq01f42t4y5pw_pETwQJX5J-12Q,3974
+eolab/rastertools/cli/zonalstats.py,sha256=2jBAm87Wh3RsHNEIumt-510okKLqSW9qxbRcwCTrVVo,4819
+eolab/rastertools/cli/__pycache__/__init__.cpython-38.pyc,sha256=bZAxTPmh9GkIIuuRsXCQifP4HFtAvvrKSSFLCyIAeoo,1910
+eolab/rastertools/cli/__pycache__/filtering.cpython-38.pyc,sha256=H_l1uP8Vt41dKMRQXSZRofCZT6IJG-UJXe08Rl2XrxM,2828
+eolab/rastertools/cli/__pycache__/hillshade.cpython-38.pyc,sha256=d7btMY8OTuSW6M2Yai_p2WZvjqq8Edl_LQS9gqe8LEM,2909
+eolab/rastertools/cli/__pycache__/radioindice.cpython-38.pyc,sha256=PjPnBaTmne2zaAwRmgjVE_5V-UqRcB24HI8tBYqLipI,4567
+eolab/rastertools/cli/__pycache__/speed.cpython-38.pyc,sha256=ejc98LZyezfe8eGNC1IcYO-Ym4xItbhqGFThvlsuHBw,2110
+eolab/rastertools/cli/__pycache__/svf.cpython-38.pyc,sha256=2nnA5pIVPFvk9c955hcf56VpxQ9NQcB2D-8fweJrFew,2727
+eolab/rastertools/cli/__pycache__/tiling.cpython-38.pyc,sha256=tD6THKbDeF-cFFxg6wGZz5XO8phqi2_V39NcqDpwtM4,3307
+eolab/rastertools/cli/__pycache__/timeseries.cpython-38.pyc,sha256=Z2s2j2Ji0GnsPU-MCgx7PFWl1Ktv78snaVtSJCXrfAA,3264
+eolab/rastertools/cli/__pycache__/zonalstats.cpython-38.pyc,sha256=RmF6AbIdB1_otub76D7BiMcQ9IbxN7KwirGD0CTo7bA,5417
+eolab/rastertools/processing/__init__.py,sha256=ODB6rtMFA17jWtEJgE_ixrrzAAe1rRRrAqxXV8khZ5k,999
+eolab/rastertools/processing/algo.py,sha256=C0Yu_7KS2xHgQRZvY2hOFCevhtvSrJd1I8rDm3jFR_k,32409
+eolab/rastertools/processing/rasterproc.py,sha256=rya9aWsV1ESDGCE_qgfSzeFlCSXrOrvf_zxWX5DMU-g,9547
+eolab/rastertools/processing/sliding.py,sha256=-fDkygwN7s6oyx6-uB9_bMDbbRFsksBxkDbzdVOePuw,11623
+eolab/rastertools/processing/stats.py,sha256=j56dCl-ErdOTL9fYs30oWMJrzNCvWoh7K9Gf8xSwuV4,22996
+eolab/rastertools/processing/vector.py,sha256=phz_JQLlEJSXPivHAUSjxRDChBnCNvhQQ-1JIOX_ZMU,17852
+eolab/rastertools/processing/__pycache__/__init__.cpython-38.pyc,sha256=tPHsY3nLkszJ3K5RK0b_4fLlPSvf20uk-ubl8mi4ddw,824
+eolab/rastertools/processing/__pycache__/algo.cpython-38.pyc,sha256=Z2LuukZoZx4QKQnkLURq7NPV0_im8EAs8zyymW6WSCw,14808
+eolab/rastertools/processing/__pycache__/rasterproc.cpython-38.pyc,sha256=RCucwDicycqLnB6ITNGMRJ8HL4p8mBUttXHqfAZPriY,9427
+eolab/rastertools/processing/__pycache__/sliding.cpython-38.pyc,sha256=w6Mujw6nKeH104uoe-cXW82-pmNjCubf1-zI2QyEZpA,7822
+eolab/rastertools/processing/__pycache__/stats.cpython-38.pyc,sha256=GTXjwU2qJHiGi-ix6uuL5ur47JpcoYKci33MtMlmTjo,14262
+eolab/rastertools/processing/__pycache__/vector.cpython-38.pyc,sha256=LWVaibJUZ_2Z7JUY6IpE6XU6rt1SxY5i_AT3UwC2ZtU,13412
+eolab/rastertools/product/__init__.py,sha256=Vh8R8brVzLY8sxcTZhn7J_-uZOR66uY_8ZAXjBzE8Wk,834
+eolab/rastertools/product/rasterproduct.py,sha256=i_M2o8-Xgex-ZcGs7pTrLUkUvqTECaE3VpTKOCeYHS4,30177
+eolab/rastertools/product/rastertype.py,sha256=8Uza3yuLi0ZCSRDi2Cp8HVC2CcL0Sg8AnvxjlT3K7q0,13948
+eolab/rastertools/product/rastertypes.json,sha256=0gfiOaOW5calf9B5nNYU0uqQoHg69C1HZSSOdWxp-Sc,6524
+eolab/rastertools/product/vrt.py,sha256=xLflRLFiUkY6-9Cdrwj8W7cfVmj53JsVEJYBoKKrZMQ,6131
+eolab/rastertools/product/__pycache__/__init__.cpython-38.pyc,sha256=eKG-UruwbnxXAiv9UdevyfVwWAinIYu-9jleiE5haaY,993
+eolab/rastertools/product/__pycache__/rasterproduct.cpython-38.pyc,sha256=_AaZErO4Ax78CI7nBPkSJvNxNZbIVFW2Ll6DSwiA7L8,19766
+eolab/rastertools/product/__pycache__/rastertype.cpython-38.pyc,sha256=9LHvYsyboWcmrp3ttfU1YC18jfP2XXyglvBLkN0sxQk,15064
+eolab/rastertools/product/__pycache__/vrt.cpython-38.pyc,sha256=tMdO02V0-zIgtwwCVEM8QO4QJYuXEVmi4dP1zvCPvBM,4900
+rastertools-0.1.0.dist-info/AUTHORS.rst,sha256=99aFJCitK64uanDEQmcudASd-9ZkBphip7iLAFPpgbs,206
+rastertools-0.1.0.dist-info/LICENSE.txt,sha256=Pd-b5cKP4n2tFDpdx27qJSIq0d1ok0oEcGTlbtL6QMU,11560
+rastertools-0.1.0.dist-info/METADATA,sha256=Tb2-yzcqeBl2_U1dYG-TTTVq6NVpea-Y2Ev1OsEu7jQ,6491
+rastertools-0.1.0.dist-info/WHEEL,sha256=OpXWERl2xLPRHTvd2ZXo_iluPEQd8uSbYkJ53NAER_Y,109
+rastertools-0.1.0.dist-info/entry_points.txt,sha256=AKcER9TFv3luXEf6dCzgyzwQ4y6HUysdwvs6QlVOtaE,72
+rastertools-0.1.0.dist-info/top_level.txt,sha256=MVosVtKjYB3qvKmsjhMnKUiUtuPjN73wjTROJ92QJoU,6
+rastertools-0.1.0.dist-info/RECORD,,

WHEEL

@@ -0,0 +1,6 @@
+Wheel-Version: 1.0
+Generator: setuptools (75.3.0)
+Root-Is-Purelib: true
+Tag: py2-none-any
+Tag: py3-none-any
+