Migrate to pyproject.toml (#379)

* Migrate to pyproject.toml, and convert the build to use py-build-cmake

* Switch to pkgutil.extend_path (revisit PEP420, this may be unneeded), and update paths for ionc headers in extension

* Remove install.py; Update ion-c to latest (fixes sub-directory builds)

* Update benchmark-cli to use new python code path

* Add new pyproject.toml and CMakeLists

* Adjust IONC_BUILD_TESTS override

* Update main github actions workflow

* Install build module

* Lower pyparsing version to support 3.8

* Further migration to pyproject.toml; move 3rd party configs into pyproject, update release workflow, and address max-int digits issue for test vectors

* Update main workflow to include test dependencies when installing

* Update extension to link against decNumber as a static lib

* Update extension name, changed to _ioncmodule and is now at the top-level

* Fix version, it got lost in the rebase

* Remove requirements files; add py-build-cmake cache to gitignore

* Add ionc_version method to return the version of Ion C used in the extension

* Quick update pass for C Extension documentation

* Remove libm from linked libraries for windows build

* Update regression test to use pyproject for PR source (will need to be updated to do the same for baseline once merged)

* Update package cache to only install dependencies, not our package

* Update benchmark cli's path for PR content to account for updated layout

* Update virtualenv version to match HEAD's dependabot update

* Remove unneeded virtualenv install; expand wording on C ext use; remove commented cmake code

* Address actionlint issues

Author: nirosys

.github/workflows/main.yml

@@ -21,9 +21,8 @@ jobs:
 
       - name: Create a virtual environment
         run: git submodule init && git submodule update && python3 -m venv ./venv && . venv/bin/activate
-      - run: pip install --upgrade setuptools
-      - run: pip install -r requirements.txt
-      - run: pip install -e .
+      - run: python -m pip install build
+      - run: python -m pip install '.[test]'
       # Run our tests, but do not run the benchmark tests since they pull in libraries that do not have pypy support.
       - run: py.test --ignore tests/test_benchmark_cli.py --ignore tests/test_benchmark_spec.py
 
@@ -42,8 +41,8 @@ jobs:
 
       - name: Create a virtual environment
         run: git submodule init && git submodule update && python3 -m venv ./venv && . venv/Scripts/activate
-      - run: pip install --upgrade setuptools
-      - run: pip install -r requirements.txt
-      - run: pip install -e .
+      - run: python -m pip install build
+      - run: python -m build
+      - run: python -m pip install -e '.[test]'
       # Run our tests, but do not run the benchmark tests since they pull in libraries that do not have pypy support.
       - run: py.test --ignore tests/test_benchmark_cli.py --ignore tests/test_benchmark_spec.py

.github/workflows/performance-regression.yml

@@ -15,6 +15,7 @@ env:
   specs: '{command:read,format:ion_text} {command:write,format:ion_text} {command:read,format:ion_binary} {command:write,format:ion_binary}'
   test_data_id: 'generated-test-data'
   run_cli: 'python amazon/ionbenchmark/ion_benchmark_cli.py'
+  run_cli_new: 'python src-python/amazon/ionbenchmark/ion_benchmark_cli.py'
 
 
 jobs:
@@ -38,7 +39,7 @@ jobs:
           mkdir -p testData
           for test in nestedStruct nestedList sexp realWorldDataSchema01 realWorldDataSchema02 realWorldDataSchema03
           do
-            java -jar $jar_file generate -S ${{env.data_size}} --input-ion-schema $schema_dir/${test}.isl testData/${test}.10n
+            java -jar "$jar_file" generate -S "${{env.data_size}}" --input-ion-schema "$schema_dir/${test}.isl" "testData/${test}.10n"
           done
       - name: Upload test Ion Data to artifacts
         uses: actions/upload-artifact@v4
@@ -55,23 +56,20 @@ jobs:
       - uses: actions/checkout@v4
       - uses: actions/setup-python@v5
         with:
-          python-version: '3.10'
+          python-version: '3.13'
           cache: 'pip'
-          cache-dependency-path: |
-            **/requirements.txt
-            **/requirements_benchmark.txt
+          cache-dependency-path: pyproject.toml
       - run: |
-          pip install -r requirements.txt
-          [ -e "requirements_benchmark.txt" ] && pip install -r requirements_benchmark.txt # include benchmark requirements if they exist.
-          # TODO: See if there's a way to cache the ion-c build output if it hasn't changed
+          python -m pip freeze '.[test,benchmarking]' > requirements.txt # capture only dependencies.
+          python -m pip install -r requirements.txt
 
   detect-regression:
     name: Check
     runs-on: ubuntu-latest
     needs: [generate-test-data, prepopulate-pip-cache]
     strategy:
       matrix:
-        python-version: ['3.9', '3.11']
+        python-version: ['3.13']
         test-data: ['nestedStruct', 'nestedList', 'sexp', 'realWorldDataSchema01', 'realWorldDataSchema02', 'realWorldDataSchema03']
       fail-fast: false
     steps:
@@ -95,9 +93,7 @@ jobs:
         with:
           python-version: ${{ matrix.python-version }}
           cache: 'pip'
-          cache-dependency-path: |
-            **/requirements.txt
-            **/requirements_benchmark.txt
+          cache-dependency-path: pyproject.toml
 
       - name: Download Test Data
         id: 'download'
@@ -106,12 +102,13 @@ jobs:
           name: ${{env.test_data_id}}
 
       # Generates performance results for the previous commit
-      - name: Create a virtual environment
+      - name: Create a virtual environment for baseline
         working-directory: ./baseline
         run: |
           pip install -r requirements.txt
           [ -e "requirements_benchmark.txt" ] && pip install -r requirements_benchmark.txt # include benchmark requirements if they exist.
           pip install .
+
       - name: Run baseline performance benchmark
         id: 'baseline'
         working-directory: ./baseline
@@ -125,17 +122,15 @@ jobs:
           echo "report=$PWD/report.ion" >> "$GITHUB_OUTPUT"
 
       # Generates performance results for the current commit
-      - name: Create a virtual environment and setup the package
+      - name: Create a virtual environment for PR changes
         working-directory: ./new
-        run: |
-          pip install -r requirements.txt
-          [ -e "requirements_benchmark.txt" ] && pip install -r requirements_benchmark.txt # include benchmark requirements if they exist.
-          pip install .
+        run: pip install '.[test,benchmarking]'
+
       - name: Run new performance benchmark
         id: 'new'
         working-directory: ./new
         run: |
-          ${{env.run_cli}} spec '${{env.specs}}' -d '${{env.spec_defaults}}' \
+          ${{env.run_cli_new}} spec '${{env.specs}}' -d '${{env.spec_defaults}}' \
               -O '{input_file:"${{steps.download.outputs.download-path}}/${{ matrix.test-data }}.10n"}' \
               -o "$PWD/report.ion" -r '${{env.report_statistics}}'
           echo "::group::Ion Report"
@@ -146,4 +141,4 @@ jobs:
       # Compare results and identify regression
       - name: Detect performance regression
         working-directory: ./new
-        run: ${{env.run_cli}} compare --fail ${{steps.baseline.outputs.report}} ${{steps.new.outputs.report}} -c '${{env.compare_statistics}}'
+        run: ${{env.run_cli_new}} compare --fail ${{steps.baseline.outputs.report}} ${{steps.new.outputs.report}} -c '${{env.compare_statistics}}'

.github/workflows/release.yml

@@ -20,13 +20,10 @@ jobs:
 
       - name: Create a virtual environment
         run: git submodule init && git submodule update && python3 -m venv ./venv && . venv/bin/activate
-      - name: Install dependencies..
-        run: |
-          pip install --upgrade setuptools
-          pip install -r requirements.txt
-          pip install -e .
-      - name: Run Tests
-        run: py.test --ignore tests/test_benchmark_cli.py --ignore tests/test_benchmark_spec.py
+      - run: python -m pip install build
+      - run: python -m build .
+      - run: python -m pip install -e .
+      - run: py.test --ignore tests/test_benchmark_cli.py --ignore tests/test_benchmark_spec.py
 
   source-distribution:
     if: startsWith(github.ref, 'refs/tags/')
@@ -40,13 +37,13 @@ jobs:
       - uses: actions/checkout@v4
 
       - name: Build sdist
-        run: python3 setup.py sdist
+        run: python -m pip install build && python -m build -s
 
       - name: Upload source to Github
         uses: actions/upload-artifact@v4
         with:
           name: source
-          path: ./dist/amazon.ion-*.tar.gz
+          path: ./dist/amazon_ion-*.tar.gz
 
       - uses: aws-actions/configure-aws-credentials@v4
         with:
@@ -60,7 +57,7 @@ jobs:
 
       - name: Upload source to s3
         run: |
-          zip ion-python-source ./dist/amazon.ion-*.tar.gz
+          zip ion-python-source ./dist/amazon_ion-*.tar.gz
           aws s3 mv ion-python-source.zip ${{ secrets.AWS_SOURCE_BUCKET_URL }} --acl bucket-owner-full-control
 
       - name: Install the released source from PYPI

.gitignore

@@ -17,4 +17,4 @@
 /amazon/ion/ion-c-build/
 /temp_*.ion
 /temp_*.10n
-
+/.py-build-cmake_cache

.gitmodules

@@ -3,6 +3,6 @@
 	url = https://github.com/amazon-ion/ion-tests
 	branch = master
 [submodule "ion-c"]
-	path = ion-c
+	path = src/ion-c
 	url = https://github.com/amazon-ion/ion-c
 	branch = master

CMakeLists.txt

@@ -0,0 +1,4 @@
+cmake_minimum_required(VERSION 3.6 FATAL_ERROR)
+project(amazon)
+
+add_subdirectory(src)

C_EXTENSION.md

@@ -76,16 +76,73 @@ b'\xe0\x01\x00\xea\xe9\x81\x83\xd6\x87\xb4\x83abc\xd3\x8a!{'
 
 ## Development
 
-Architecture of Ion Python C extension:
+The Ion Python C extension is built as part of the PEP 517 build process using py-build-cmake, and leaning on
+Ion C's existing cmake build. A revision of Ion C is included as a submodule in this repo under `src/ion-c`.
+If you would like to update the version of Ion C, simply update the submodule to point to the desired revision.
+
+The file `src/CMakeLists.txt` acts as the build script for the C extension itself, which then includes the Ion C
+codebase into the build tree.
+
+With the extension built, it will be exposed to python as `amazon._ioncmodule`. For example:
+```python
+>>> import amazon._ioncmodule as ionc
+>>> ionc.ionc_version()
+'v1.1.3 (rev: d61c09a)'
+>>>
 ```
-                                   ioncmodule.c
-                                        |
-                                        | 
-                                        ↓ 
-Ion C -------> Ion C binaries -----> setup.py ------> C extension -------------------> Ion Python simpleion module
-      compile                                  setup               import ionc module   
+
+The `amazon.ion.simpleion` module then makes use of this extension when it is available to provide more efficient
+Ion reading and writing. Importing `amazon._ioncmodule` directly can determine if it is available, however simpleion
+also provides the field `__IS_C_EXTENSION_SUPPORTED`.
+```python
+>>> import amazon.ion.simpleion as ion
+>>> ion.__IS_C_EXTENSION_SUPPORTED
+True
+>>>
+```
+
+In order to build the extension, along with the package itself, we can use python's build module:
+```python
+ion-python# python -m build .
+* Creating isolated environment: venv+pip...
+* Installing packages in isolated environment:
+  - py-build-cmake~=0.1.8
+* Getting build dependencies for sdist...
+* Building sdist...
+* Building wheel from sdist
+* Creating isolated environment: venv+pip...
+* Installing packages in isolated environment:
+  - py-build-cmake~=0.1.8
+* Getting build dependencies for wheel...
+* Building wheel...
+...
+Successfully built amazon_ion-0.13.0.tar.gz and amazon_ion-0.13.0-cp310-cp310-linux_x86_64.whl
+```
+This will build both the source wheel, and the binary wheel for the current system. Installing the module can
+be done with pip. Depending on what you're doing with the package you may want to install different dependencies.
+Different sets of optional dependencies are provided, such as `test`, and `benchmarking`. More details can be
+found in the `pyproject.toml`.
+
+To install the package and dependencies for unit tests you can run:
+```python
+ion-python# python -m pip install '.[test]'
+Processing /ion-python
+  Installing build dependencies ... done
+  Getting requirements to build wheel ... done
+  Preparing metadata (pyproject.toml) ... done
+Building wheels for collected packages: amazon_ion
+  Building wheel for amazon_ion (pyproject.toml) ... done
+  Created wheel for amazon_ion: filename=amazon_ion-0.13.0-cp310-cp310-linux_x86_64.whl size=573770 sha256=96ef01efea7519a1a38d9fc9e42139ab82125aafa328cfb4a4823458de802fa1
+  Stored in directory: /root/.cache/pip/wheels/78/55/f9/c6d69051d6a93c725251429f62d0d5b7d16d8c982a3772d666
+Successfully built amazon_ion
+Installing collected packages: amazon_ion
+  Attempting uninstall: amazon_ion
+    Found existing installation: amazon_ion 0.13.0
+    Uninstalling amazon_ion-0.13.0:
+      Successfully uninstalled amazon_ion-0.13.0
+Successfully installed amazon_ion-0.13.0
 ```
-After setup, C extension will be built and imported to simpleion module. If there are changes in `ioncmodule.c`, build the latest C extension by running `python setup.py build_ext --inplace`.
+Installing with `-e` will also allow you to update the python side of the package without having to re-install.
 
 
 ## Technical Details