Having CLMM tests skip if a backend is not available means some bugs might be left unseen

[caioolivv]

For example, until recently we had a hard check on CCL's version in order to prevent us from trying to use an unsupported version of CCL. Still, in the CI we installed the latest version of CCL, which of course meant that every test that required CCL as a backend was skipped because of the hard check. That means that for the last few months we weren't testing anything with CCL and had no way to know this.

[hsinfan1996]

We were only checking the major version of CCL after the release of version 3. I think up until version 1.16.9, all backends were tested. But please correct me if I am wrong. It has been a while since I visited the CLMM repo.

[hsinfan1996]

On my local machine with CLMM version 1.16.9 and the latest pyccl from the master repo (version 3.2.2.dev6), all unit tests related to CCL passed and CLMM uses CCL as the backend with no problem. Unless you are using ccl version 2.7.1.dev10 and below, the check does not affect anything and it is designed to protect users when pyccl 4 is out (I do not think there is a plan for pyccl 4 right now).

[hsinfan1996]

RIght now CLMM 1.16.10 has removed the CCL version check, so it is unreasonable to blame it for not completing the unit tests. Some skipped tests were due to the "testnotabackend" backend, not CCL, nor the ccl version check.

[caioolivv]

Just to clarify, I observed this at 1.16.9 before removing the check. But I think you're right, I hadn't realized that the max version constraint was on the major version, so the skipped tests were definitely not caused by this.

[caioolivv]

Still, I think that at least for the CI/CD these tests should fail if a backend is somehow missing, so that we at least know something is going on.

[hsinfan1996]

@caioolivv Since the failing tests are not caused by the ccl version check, can you remove or revise the misleading comments? And in PR #651, the easier fix would be sys.modules["pyccl"].__version__ = "3.1.2", but is CCL 3.1.1 or the versions before not compatible with the current version of CLMM?