Fix build for Python 3.11

[gcroci2]

I opened an issue in torch repository: pytorch/pytorch#116707.

If it's not picked up soon, we could think about work arounds, such as regenerating the problematic files with pytest fixtures (it may be that saving AND loading files with Python 3.11 solves the issue).

[gcroci2]

In PR #549 I removed Python 3.11 from the CI, until this issue won't be solved.

While fixing this issue, remember to switch the python requirement to python>=3.10,<3.12 in the yml file, so that then the CI can set both 3.10 and 3.11 according to the different build.

[github-actions]

This issue is stale because it has been open for 30 days with no activity.