Spliting Pymodaq into several Python packages does not necessarily imply to split it into several Git repositories

[quantumm]

Hello,

This suggestion is mainly for the core developers of Pymodaq and follows a discussion that I had with a friend of mine regarding the spliting of Pymodaq core into several Python packages (_utils, _gui and _data) since the last major release 5.

My friend was suggesting that the spliting of Pymodaq core into several Git repositories could imply some additional complexity regarding the development and testing of the all thing. In fact, using different Git repositories implies having different histories and additional version numbers evolving independently, and add extra depency relations that could be difficult to solve: "if I develop a feature in pymodaq_gui, will it be compatible with the current version of pymodaq_data?" In brief, authorizing the free evolution of each repository could compromise the stability of the all thing.

It is actually possible, and quite simple apparently, to have a single Git repository that contains several folders that correspond to the different Python packages. If we consider this single repository scenario, at each new release it would be possible to configure the release process to issue the release of three Python package on PyPI that could be downloaded independently if needed. The version number of each of those packages would be the same. One drawback could be that an additional version incrementation is given to a package whereas no change has been issued for him. But this drawback has to be compared to the advantage of simplifying the development process and increasing the all Pymodaq stability.

I hope what I say makes sense, I do not feel totally competent for this suggestion since I am not much involved in the core development. I just wanted to share this discussion in case you did not consider this option ;)

This is quite related to issue #595 as it also suggests to reduce the number of free evolving version numbers, but I finally preferred to split them as they could be implemented independently.

[seb5g]

It is true that some development flow are a bit more complex due to the splitting but I'm not convinced that it will be often the case so I am a bit reluctant to do what you propose. Other opinions?

[bernhardlang]

I just ran into a problem which--I guess--is linked to that. Working on #374 I have to apply changes in PyMoDAQ and in pymodaq_gui at the same time. Since that is a new feature, I started by forking PyMoDAQ 5.1.x_dev and pymodaq_gui 5.0.x_dev (there is no 5.1...dev branch of the latter). After running pip install -e . in the PyMoDAQ folder and starting the dashboard for testing, the following strange versions of _utils and _gui were detected

After the upgrade the dashboard doesn't start any more (stuck until kill -KILL / task manager end). No hints from the log file what is going on. Reverting by issuing again pip install -e . in the PyMoDAQ folder and further ignoring any update suggestions solves this problem. Forking instead 5.0.x_dev there is no such problem (similar to #598, probably related).

2. when running now pip install -e . in the pymodaq_gui folder (be that with pymo 5.0.x_dev or 5.1.x_dev), pip complains

ERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts.
pymodaq 5.0.6 requires pymodaq-gui>=5.0.17, but you have pymodaq-gui 0.1.dev2450+g57815fe which is incompatible.

No idea where this strange _gui version comes from. I forked from 5.0.x_dev. The dashboard starts, not yet further tested. All that happened in a dedicated virtual environment to keep things isolated. Python 3.10 and 3.11 on Linux Debian, reproduced on two different computers.

What looked like a pretty simple wrap-up of already tested and running things, collecting stuff from applications coded on slightly older versions ("older" on the scale of a few months), turned into messing around with different versions on top of the necessary changes 4.x -> 5.x. The foreseen time was already used up before I could even start working on the real task.

Not sure whether this is really due to incompatibilities between packages and their versions or just me not yet being familiar enough with the corresponding internals. However, after this adventure I'd strongly recommend to group the packages together into one single repository and to rely on automated testing rather than on manually checking compatibility between different repros, which tends to be error prone. This way, pip install should hopefully always be able to pull a workable starting point from the development branch.

As an additional note I would like to add the suggestion to concentrate on stable APIs between the PyMoDAQ core on one and plugins/applications/extensions on the other side. Which would potentially prevent quite some user-supplied code from getting obsolete unless constantly being maintained. Furthermore, when numerous plugins out there don't run because they are outdated, PyMoDAQ's reputation will suffer from "doesn't work => pip uninstall".

cheers

[seb5g]

hi all, well sometimes mistakes happen and that is what happened at the time. I fixed it quite quickly so hope it doesn't make people uninstall pymodaq ;-)

The main issue is to let people know that there are available new versions (the popup at dashboard startup) bit also that these new versions are compatible between each other. Pip is doing it quite nicely but there is no api to use pip features... @malik-irain tried to look for this but its tricky...

@malik-irain could we not try to install all packages selected in the popup using a subprocess and a virtual env, if there is dependency issues, the output from pip should tell us?

[malik-irain]

There's actually several problem at once:
- First I advise to deactivate the update tool in the general configuration if you are in a development use-case, as you was your version numbers may not always be accurate. Also, installing from the tool, will end up in running pip install pymodaq_gui==5.0.20, so you lose the -e install flag and the local install. You generally don't want that when working as a dev.
- About these version numbers, they are tricky to maintain on a fork. They're generated using tags, that are specific to the main repo and not automatically synced. To update them you have to add the main principal repo as another git remote, sync the tags, push them on your fork (so you keep them) and eventually remove the new remote. Here is a command-line version of that:

```bash
	git remote add upstream https://github.com/PyMoDAQ/pymodaq_gui
	git fetch upstream --tags
	git push origin --tags
	git remote rm upstream
```
- There's also an issue when new branches appear, or the main one change, like what happened when PyMoDAQ 5 released. The old main 5.0.x_dev was renamed the 5.0.x branch and was replaced as the new branche by 5.1.x_dev

I was thinking of a way to keep the same repo, and the same package and using optional dependancies to install only a subset like: pip install pymodaq would install everything but pip install pymodaq[utils] wouls install only the utils parts, pip install pymodaq[gui] would install gui, data and utils. However it seems that's not possible in this direction. Stuff into [brackets] are optional and installed in addition to the main package.

When using a branch on a forked repo, one may end up spending some time troubleshooting git issues with many "points of failure" (or at least several spots where it could go wrong), so yeah the current workflow is not optimal.

@seb5g Isn't it what happens right now? Except if you mean create a new environment, try the install and do it for real if succesful? I don't know, it seems tricky. It would mean find which tool is available to create a virtualenv (if any...), find an unused name for the virtualenv (I don't know for all environments but if I create a new one with an already existing one it just erases the old one ...), and also clean afterwards.

I'm not so much convinced of spending so much time & efforts on details on a functionality that not part of the core of PyMoDAQ, and for which the only bug report we have is on the dev side. I mean, it's cool if it works but it's alright if it's not here.

[seb5g]

@aurore-finco also for the pymodaq days...