Package and distribute on PyPI and Conda-Forge

[CAM-Gerlach]

Particularly with the tweaks in #57 , it should be pretty straightforward to build source and binary wheel distributions and upload to PyPI and Conda-Forge, if there is any interest—it certainly should make the project more accessible to newcomers, particularly for students and those newer the field, simplify package and dependency management for users, and increase visibility to reach a wider audience.

Task list of things the need to be done:

• Modernize packaging (again)
  • Fix any immediate packaging/Python compat problems
  • Move static config to pyproject.toml
  • Update manifest & pyproject.toml
  • Update and simplify remaining setup.py
  • Update Readme/etc. with up to date guidance
  • Update/drop any other legacy stuff/Python support
• Add support for publishing wheels to PyPI
  • Drop in, test and tweak cibuildwheel GHA workflow
  • Add key to repo secrets & drop in PyPI GHA workflow
  • Tag a new 1.4.0 release
• Publish to Conda-Forge
  • Run greyskull on PyPI package
  • Tweak/manually add anything necessary
  • Push to staged-recipies & go through review process

Mostly out of date now that cibuildwheel and greyskull exist

Doing so should be as simple as running python setup.py sdist on any machine, then python setup.py bdist_wheel on any platforms and Python versions you'd like to build binary wheels for, to make installation easier, faster and less cumbersome and error-prone (of course, source installation will still work on any platform that has the compilers). and finally twine upload dist/* to upload to PyPI. The only somewhat complex factor is build platforms; of course, building binary distributions is optional—if any of this is too much work you can only distribute select builds or source-only, which is minimal effort but brings all the benefits of pip/PyPI distribution aside from avoiding the need to build locally

Alternatively (and probably preferably), you could go source-only on PyPI (or only a few select wheels, e.g. Windows), and use conda-forge to automate most of the build process, like PyART does. I should be able to help get that set up, if there's interest, and it shouldn't be too involved. If we go with that, then most of the remaining text is unnecessary since the whole build process is almost entirely automated; it just requires some initial configuration and conda-build/conda-smithy handles the rest, and all the builds are automatically done in the CIs. Of course, the advantage there is that the package is natively installable with conda without resorting to pip, and binaries are automatically compiled for all major platforms and Python versions making installation and maintenance a breeze (rebuilds or updates are mostly automated, and only require changing at most a line or two).~~

Let me know if I can be of further help!

Even more out of date build matrix details now that cibuildwheel exists

The most critical OS is (of course) Windows due to the difficulty and overhead of building it themselves; you can build all the required package versions automatically with the free Appveyor CI service, or in a few minutes on any local machine with the Anaconda/Miniconda (to switch Python versions quickly) and the compilers installed; I'd be happy to help there if needed. Similarly, macOS is less critical but can similarly be done locally, or with Travis now that they offer macOS builds. Linux is a bit more complicated due to the variety of distributions and glibc versions so building locally can be rather fragile and not always work on other distros, but thankfully there's the manylinux project, officially endorsed by the PSF/PyPA, that offers pre-built docker containers that can easily be run locally or automatically via Travis to generate the required builds.

As for your build matrix, given 64-bit systems have been widely available for well over 10 years (and exclusively available for close to that), you probably can stick with just x64 builds; Python 2.7, 3.5. 3.6 and 3.7 (if the package actually runs properly on it; it very likely does but I haven't formally tested) should cover the overwhelming majority of users (in our statistics on the Spyder repo, with somewhere around 3000+ issue reports per year and growing, IIRC its been years since we've seen a Python 3.x user with <3.5), and avoids the complexity of building on 3.4 due to the problem VS versions and hacky workarounds necessary for x64 builds).

[tjlang]

Yes, this has been a long-needed change to the package, but I have had neither the time nor expertise to address it. I see you are UAH, feel free to stop by my office sometime to discuss this. I will be back the week of 12/3.

[CAM-Gerlach]

@tjlang Okay, thanks! Since you already had a full setup.py in place, the minor changes I made should ensure going from source repo to published on PyPI is as easy as two simple commands.

All you would need to do after creating a PyPI account (basically just setting a username and password) and updating setuptools and twine to the latest versions is run python setup.py sdist to create the package, and then twine upload dist/* to upload it. That's it! That gives all the benefits of having a pip-installable project, just without the pre-built binary wheels.

Theoretically, I could do it right now in a few minutes time (I already tested it locally). The latter is slightly less trivial can be more easily accomplished with conda-forge; we can discuss that more when you return. I should be available 2018-12-04 or after when you get back (from RELAMPAGO, I'm guessing?) from 10:30-3:30 PM any day, if you want to pencil in a time.

I'll also drop that other cleanup PR probably sometime tomorrow. Writing some unit tests would be great and not too terribly difficult, not to mention cleaning up a bunch of Cython type warnings (that turn into fatal compile-time errors under language_level=3, soon to be the default) in your PYX code. However, as I'm sure you can relate to, with everything on my plate right now between research, courses, Spyder, web development, and getting it standards-compliant, packaged and distributed is the most I can fairly promise.

[mgrover1]

@CAM-Gerlach @tjlang is there still interest in this? Happy to help move this along.

[tjlang]

Yes, there is. I think the main issue is I don't have time or (personal) bandwidth to create/test the packages, particularly on different platforms (notably Windows, which I don't use professionally). I can handle squashing bugs and adding new features, but packaging this kind of stuff for conda/PyPI is mostly out of my wheelhouse.

[CAM-Gerlach]

The scientific packaging and distribution ecosystem (and, hopefully, my ability to be more concise :) has come a long way since 2018, which makes this a lot simpler than the wall of text above—at least if we just focus on binary builds for the Cython version rather than the f2py alternative (though that is possible, if more annoying).

On the packaging side; most if not almost all of the complexity currently in the setup.py script can now be either moved over to the static pyproject.toml config file, or eliminated entirely, and user-side the legacy python setup.py install and manual uninstall are a thing of the past; users can now just run pip install . to install a local package (though they do need a compiler to install from source, if you don't ship pre-compiled Cython files).

The PyPI binary distribution story has improved a ton; instead of all the manual stuff I talked about above, there's cibuildwheel which handles basically the entire process I outlined above (building binary packages on a bunch of different architectures in CI) automatically; you just drop in the GitHub Actions workflow file to the repo, tweak any necessary options, and push, and then wheels for all platforms will automatically be built and tested whenever you push code or open a PR. Furthermore, you can have it upload a new release to PyPI whenever you tag a release here, so its completely hands off.

And for Conda-Forge, there's now tools like grayskull to automatically generate the Conda recipe from the PyPI package, so the initial workflow is just run grayskull, tweak things as needed then stage the recipe, then its just a matter of merging an auto-generated update PR whenever a new version hits PyPI.

I've updated the OP with a checklist of the main items; I can start with the packaging upgrade (basically #57 round 2), then we can see about adding cibuildwheel and Conda-Forge (which will need a bit of cooperation on your end). I can't make any immediate promises as I've got a couple things still to finish up for Python 3.11 that we just released yesterday then a big launch coming up, but I'll put it on my list.