[Bug]: `np.complex_` removed in `numpy` 2.0 release.

[SylviaWhittle]

Checklist

• Find the offending file in the output. If processing halts, re-run analysis with topostats --core 1 process.
• Describe the bug.
• Include the configuration file.
• Copy of the log-file from running with topostats --log-level debug <command>.
• The exact command that failed. This is what you typed at the command line, including any options.
• TopoStats version, this is reported by topostats --version
• Operating System and Python Version

Describe the bug

• Any file
• Any config
• Issue happens when saving height profiles to JSON

Seems that our dict_to_json function is broken in numpy >= 2.0, upon trying to use the NumpyEncoder class from numpyencoder package.

Copy of the log-file from running with topostats --log-level debug <command>

[Sun, 01 Jun 2025 16:36:30] [INFO    ] [topostats] Saving all height profiles to output_abs_thresh/height_profiles.json
Traceback (most recent call last):
  File "/Users/sylvi/.pyenv/versions/topostats/bin/topostats", line 8, in <module>
    sys.exit(entry_point())
             ^^^^^^^^^^^^^
  File "/Users/sylvi/Documents/TopoStats/topostats/entry_point.py", line 1281, in entry_point
    args.func(args)
  File "/Users/sylvi/Documents/TopoStats/topostats/run_modules.py", line 301, in process
    dict_to_json(data=height_profile_all, output_dir=config["output_dir"], filename="height_profiles.json")
  File "/Users/sylvi/Documents/TopoStats/topostats/io.py", line 1172, in dict_to_json
    json.dump(data, f, indent=indent, cls=NumpyEncoder)
  File "/Users/sylvi/.pyenv/versions/3.11.9/lib/python3.11/json/__init__.py", line 179, in dump
    for chunk in iterable:
  File "/Users/sylvi/.pyenv/versions/3.11.9/lib/python3.11/json/encoder.py", line 432, in _iterencode
    yield from _iterencode_dict(o, _current_indent_level)
  File "/Users/sylvi/.pyenv/versions/3.11.9/lib/python3.11/json/encoder.py", line 406, in _iterencode_dict
    yield from chunks
  File "/Users/sylvi/.pyenv/versions/3.11.9/lib/python3.11/json/encoder.py", line 406, in _iterencode_dict
    yield from chunks
  File "/Users/sylvi/.pyenv/versions/3.11.9/lib/python3.11/json/encoder.py", line 406, in _iterencode_dict
    yield from chunks
  [Previous line repeated 2 more times]
  File "/Users/sylvi/.pyenv/versions/3.11.9/lib/python3.11/json/encoder.py", line 439, in _iterencode
    o = _default(o)
        ^^^^^^^^^^^
  File "/Users/sylvi/.pyenv/versions/3.11.9/envs/topostats/lib/python3.11/site-packages/numpyencoder/numpyencoder.py", line 31, in default
    elif isinstance(obj, (np.complex_, np.complex64, np.complex128)):
                          ^^^^^^^^^^^
  File "/Users/sylvi/.pyenv/versions/3.11.9/envs/topostats/lib/python3.11/site-packages/numpy/__init__.py", line 400, in __getattr__
    raise AttributeError(
AttributeError: `np.complex_` was removed in the NumPy 2.0 release. Use `np.complex128` instead.

Include the configuration file

Any config file

To Reproduce

Run topostats using any config, see that it fails to save the heights to JSON at the end.

TopoStats Version

Git main branch

Python Version

3.11

Operating System

GNU/Linux

Python Packages

probably relevant packages:

numpy                                   2.2.6
numpyencoder                            0.3.1

all packages:

absl-py==2.2.2
AFMReader @ git+https://github.com/AFM-SPM/AFMReader@794e2d100217be949e3d2fe83d159576273a18c6
art==6.5
astroid==3.3.10
asttokens==3.0.0
astunparse==1.6.3
biopython==1.85
black==25.1.0
certifi==2025.4.26
cfgv==3.4.0
charset-normalizer==3.4.2
cheap_repr==0.5.2
click==8.2.1
contourpy==1.3.2
coverage==7.8.2
cycler==0.12.1
decorator==5.2.1
dill==0.4.0
distlib==0.3.9
docstring_parser==0.16
et_xmlfile==2.0.0
execnet==2.1.1
executing==2.2.0
filelock==3.18.0
filetype==1.2.0
flatbuffers==25.2.10
fonttools==4.58.0
gast==0.6.0
google-pasta==0.2.0
grpcio==1.71.0
h5py==3.13.0
icdiff==2.0.7
identify==2.6.12
idna==3.10
igor2==0.5.12
imageio==2.37.0
iniconfig==2.1.0
ipython==8.36.0
isort==6.0.1
jedi==0.19.2
Jinja2==3.1.6
joblib==1.5.0
keras==3.10.0
kiwisolver==1.4.8
lazy_loader==0.4
libclang==18.1.1
llvmlite==0.44.0
loguru==0.7.3
magicgui==0.10.0
Markdown==3.8
markdown-it-py==3.0.0
MarkupSafe==3.0.2
matplotlib==3.9.4
matplotlib-inline==0.1.7
mccabe==0.7.0
mdurl==0.1.2
ml_dtypes==0.5.1
mypy_extensions==1.1.0
namex==0.0.9
networkx==3.4.2
nodeenv==1.9.1
numba==0.61.2
numpy==2.2.6
numpyencoder==0.3.1
openpyxl==3.1.5
opt_einsum==3.4.0
optree==0.15.0
packaging==25.0
pandas==2.2.3
parso==0.8.4
pathspec==0.12.1
pexpect==4.9.0
pillow==11.2.1
platformdirs==4.3.8
pluggy==1.6.0
pprintpp==0.4.0
pre_commit==4.2.0
prompt_toolkit==3.0.51
protobuf==5.29.4
psutil==5.9.8
psygnal==0.13.0
ptyprocess==0.7.0
pure_eval==0.2.3
pyconify==0.2.1
Pygments==2.19.1
pylint==3.3.7
pyparsing==3.2.3
pyspm @ git+https://github.com/AFM-SPM/pySPM@dc832253a158440ab93063a058be55f88cf36904
pytest==8.3.5
pytest-cov==6.1.1
pytest-durations==1.5.2
pytest-github-actions-annotate-failures==0.3.0
pytest-icdiff==0.9
pytest-mpl==0.17.0
pytest-regtest==2.3.1
pytest-testmon==2.1.3
pytest-xdist==3.7.0
python-dateutil==2.9.0.post0
pytz==2025.2
pyupgrade==3.20.0
PyYAML==6.0.2
QtPy==2.4.3
requests==2.32.3
rich==14.0.0
ruamel.yaml==0.18.10
ruamel.yaml.clib==0.2.12
schema==0.7.7
scikit-image==0.25.2
scikit-learn==1.6.1
scipy==1.15.3
seaborn==0.13.2
six==1.17.0
skan==0.12.2
snoop==0.6.0
stack-data==0.6.3
superqt==0.7.3
tensorboard==2.19.0
tensorboard-data-server==0.7.2
tensorflow==2.19.0
tensorflow-io-gcs-filesystem==0.37.1
termcolor==3.1.0
threadpoolctl==3.6.0
tifffile==2025.5.21
tokenize_rt==6.2.0
tomlkit==0.13.2
toolz==1.0.0
topoly==1.1.0
-e git+ssh://git@github.com/AFM-SPM/TopoStats.git@40bacb4ff709d203f5851ab35a49726eebbce50c#egg=topostats
tqdm==4.67.1
traitlets==5.14.3
typing_extensions==4.13.2
tzdata==2025.2
urllib3==2.4.0
virtualenv==20.31.2
wcwidth==0.2.13
Werkzeug==3.1.3
wrapt==1.17.2

[SylviaWhittle]

Circumventable by turning off extract_height_profile in config > grainstats

[ns-rse]

Ah good find, this could be why things are breaking in the tests, will investigate get onto this when back on TopoStats tomorrow, thanks @SylviaWhittle

[ns-rse]

Should be able to sort this by making a PR against numpyencode (numpyencoder.py L31) to remove np.complex_.

[ns-rse]

I already made a PR to have np.complex_ removed because of the Numpy deprecation two weeks ago (see here). Removing it would break backwards compatibility it seems. I'll chase this up.

[ns-rse]

Two(three) options...

1. Wait for upstream to fix.
   2a) Remove extraction of height profiles and leave it to users to take the slices they want.
   2b) Extract the height profiles and add to the GrainCrop dataclass

I recall the height profiles being important for some reason (I can't recall why or find the issue right now) but for 2b) it looks like I've already added the height_profiles to the GrainCrop dataclass in #1118 so I'm kind of inclined to go with 2b) as the solution.

I think we threw in writing height profiles to file as a quick fix but with the shift to using dataclasses which are stored in HDF5 files this kind of becomes redundant and we should remove. #1102 was, and will soon be, being worked on and will align the internal representation of a topostats object with the HDF5 file. This in turn will unblock development of the #517 entry-points.

Would/do we need to improve the documentation to show users how to access data stored in HDF5 files?

[ns-rse]

As per above we may remove this entirely, in the mean time and further to an earlier PR where I suggested removing
np.complex_ I have proposed a work around that is conditional on the Numpy version, see #7 Numpy 2.0
deprecations.

[ns-rse]

I've also just realised in addressing issues with #1169 that I modified pyproject.toml such that it would install a
forked and modified version of numpyencoder that remove np.complex_ from the list of possible types when I raised
the issue with upstream two weeks ago.

This isn't a permanent solution though because as I've learned we can not publish to PyPI with dependencies that rely on
installation from GitHub, but until we have #1102 sorted at least we won't be making another release and we can
hopefully decide if we should remove ditc_to_json() before then.

[ns-rse]

Closed by #1185