.oswpq fails in docker container

[jcharkow]

pyprophet score command with .oswpq fails in docker container because pyarrow is not installed. I am using singularity so the docker container is converted to the .sif format.

Singularity> pyprophet score tests/data/test_data.oswpq
/opt/venv/lib/python3.12/site-packages/pyprophet/glyco/report.py:463: SyntaxWarning: invalid escape sequence '\p'
  '$\pi_0$ = ' + str(np.around(pi0_['pi0'], decimals=3))
/opt/venv/lib/python3.12/site-packages/pyprophet/glyco/report.py:471: SyntaxWarning: invalid escape sequence '\p'
  '$\pi_0$ = ' + str(np.around(pi0_['pi0'], decimals=3))
/opt/venv/lib/python3.12/site-packages/pyprophet/glyco/report.py:499: SyntaxWarning: invalid escape sequence '\p'
  title = part.capitalize() + ' $\pi_0$ smoothing fit plot'
/opt/venv/lib/python3.12/site-packages/pyprophet/glyco/report.py:501: SyntaxWarning: invalid escape sequence '\p'
  ylabel = part.capitalize() + ' $\pi_0$($\lambda$)'
/opt/venv/lib/python3.12/site-packages/pyprophet/glyco/report.py:505: SyntaxWarning: invalid escape sequence '\p'
  title = '$\pi_0$ smoothing fit plot'
/opt/venv/lib/python3.12/site-packages/pyprophet/glyco/report.py:507: SyntaxWarning: invalid escape sequence '\p'
  ylabel = '$\pi_0$($\lambda$)'
/opt/venv/lib/python3.12/site-packages/pyprophet/glyco/report.py:510: SyntaxWarning: invalid escape sequence '\l'
  xlabel = '$\lambda$'
/opt/venv/lib/python3.12/site-packages/pyprophet/glyco/report.py:746: SyntaxWarning: invalid escape sequence '\p'
  'Total $\pi_0$ = ' + str(np.around(
PyProphet v3.0.5
Execution time: 2025-12-19 13:25:08
System: OS: Linux 6.8.0-90-generic | Python: 3.12.3 | CPU: 16 cores | RAM: 31.0 GB
Command: /opt/venv/bin/python /opt/venv/bin/pyprophet score tests/data/test_data.oswpq

Usage: pyprophet score [OPTIONS]
Try 'pyprophet score --help' for help.

Error: Missing option '--in'.
Singularity> pyprophet score --in tests/data/test_data.oswpq
/opt/venv/lib/python3.12/site-packages/pyprophet/glyco/report.py:463: SyntaxWarning: invalid escape sequence '\p'
  '$\pi_0$ = ' + str(np.around(pi0_['pi0'], decimals=3))
/opt/venv/lib/python3.12/site-packages/pyprophet/glyco/report.py:471: SyntaxWarning: invalid escape sequence '\p'
  '$\pi_0$ = ' + str(np.around(pi0_['pi0'], decimals=3))
/opt/venv/lib/python3.12/site-packages/pyprophet/glyco/report.py:499: SyntaxWarning: invalid escape sequence '\p'
  title = part.capitalize() + ' $\pi_0$ smoothing fit plot'
/opt/venv/lib/python3.12/site-packages/pyprophet/glyco/report.py:501: SyntaxWarning: invalid escape sequence '\p'
  ylabel = part.capitalize() + ' $\pi_0$($\lambda$)'
/opt/venv/lib/python3.12/site-packages/pyprophet/glyco/report.py:505: SyntaxWarning: invalid escape sequence '\p'
  title = '$\pi_0$ smoothing fit plot'
/opt/venv/lib/python3.12/site-packages/pyprophet/glyco/report.py:507: SyntaxWarning: invalid escape sequence '\p'
  ylabel = '$\pi_0$($\lambda$)'
/opt/venv/lib/python3.12/site-packages/pyprophet/glyco/report.py:510: SyntaxWarning: invalid escape sequence '\l'
  xlabel = '$\lambda$'
/opt/venv/lib/python3.12/site-packages/pyprophet/glyco/report.py:746: SyntaxWarning: invalid escape sequence '\p'
  'Total $\pi_0$ = ' + str(np.around(
PyProphet v3.0.5
Execution time: 2025-12-19 13:25:17
System: OS: Linux 6.8.0-90-generic | Python: 3.12.3 | CPU: 16 cores | RAM: 31.0 GB
Command: /opt/venv/bin/python /opt/venv/bin/pyprophet score --in tests/data/test_data.oswpq

[ 2025-12-19 at 13:25:17 | ERROR   | util::536                   ] An error has been caught in function 'wrapper', process 'MainProcess' (54131), thread 'MainThread' (128689185468544):
Traceback (most recent call last):
  File "/opt/venv/lib/python3.12/site-packages/pyprophet/io/util.py", line 61, in _ensure_pyarrow
    import pyarrow as pa  # pylint: disable=C0415
    ^^^^^^^^^^^^^^^^^^^^
ModuleNotFoundError: No module named 'pyarrow'

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/opt/venv/lib/python3.12/site-packages/pyprophet/io/util.py", line 343, in is_parquet_file
    pa, ArrowInvalid, ArrowIOError = _ensure_pyarrow()  # pylint: disable=C0103
                                     ^^^^^^^^^^^^^^^^^
  File "/opt/venv/lib/python3.12/site-packages/pyprophet/io/util.py", line 78, in _ensure_pyarrow
    raise click.ClickException(
click.exceptions.ClickException: Parquet support requires 'pyarrow'. Install with 'pip install pyarrow' or 'pip install pyprophet[parquet]'.

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/opt/venv/bin/pyprophet", line 7, in <module>
    sys.exit(cli())
             ^^^^^
  File "/opt/venv/lib/python3.12/site-packages/click/core.py", line 1485, in __call__
    return self.main(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/venv/lib/python3.12/site-packages/click/core.py", line 1406, in main
    rv = self.invoke(ctx)
         ^^^^^^^^^^^^^^^^
  File "/opt/venv/lib/python3.12/site-packages/pyprophet/cli/util.py", line 250, in invoke
    return super().invoke(ctx)
           ^^^^^^^^^^^^^^^^^^^
  File "/opt/venv/lib/python3.12/site-packages/pyprophet/cli/util.py", line 103, in invoke
    return super().invoke(ctx)
           ^^^^^^^^^^^^^^^^^^^
  File "/opt/venv/lib/python3.12/site-packages/click/core.py", line 1873, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
                           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/venv/lib/python3.12/site-packages/click/core.py", line 1269, in invoke
    return ctx.invoke(self.callback, **ctx.params)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/venv/lib/python3.12/site-packages/click/core.py", line 824, in invoke
    return callback(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/venv/lib/python3.12/site-packages/click/decorators.py", line 34, in new_func
    return f(get_current_context(), *args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/venv/lib/python3.12/site-packages/pyprophet/cli/util.py", line 596, in wrapper
    return ctx.invoke(f, *args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/venv/lib/python3.12/site-packages/click/core.py", line 824, in invoke
    return callback(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/venv/lib/python3.12/site-packages/pyprophet/cli/util.py", line 536, in wrapper
    result = func(*args, **kwargs)
             ^^^^^^^^^^^^^^^^^^^^^
  File "/opt/venv/lib/python3.12/site-packages/loguru/_logger.py", line 1297, in catch_wrapper
    return function(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/venv/lib/python3.12/site-packages/pyprophet/cli/score.py", line 327, in score
    config = RunnerIOConfig.from_cli_args(
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/venv/lib/python3.12/site-packages/pyprophet/_config.py", line 412, in from_cli_args
    return cls(
           ^^^^
  File "<string>", line 9, in __init__
  File "/opt/venv/lib/python3.12/site-packages/pyprophet/_config.py", line 273, in __post_init__
    super().__post_init__()
  File "/opt/venv/lib/python3.12/site-packages/pyprophet/_base.py", line 54, in __post_init__
    elif is_valid_single_split_parquet_dir(infile):
         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/venv/lib/python3.12/site-packages/pyprophet/io/util.py", line 353, in is_valid_single_split_parquet_dir
    return os.path.isdir(path) and all(
                                   ^^^^
  File "/opt/venv/lib/python3.12/site-packages/pyprophet/io/util.py", line 354, in <genexpr>
    os.path.isfile(os.path.join(path, f)) and is_parquet_file(os.path.join(path, f))
                                              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/venv/lib/python3.12/site-packages/pyprophet/io/util.py", line 346, in is_parquet_file
    except (ArrowInvalid, ArrowIOError, OSError):
            ^^^^^^^^^^^^
UnboundLocalError: cannot access local variable 'ArrowInvalid' where it is not associated with a value

[singjc]

Is this using the github container registry docker image?

[jcharkow]

This is the dockerhub docker container.

[jcharkow]

Can you please post the link to the GitHub one? I could not find it.

[singjc]

I moved pyarrow as an optional dependency while slimming down the dependencies and size of the project, since parquet is still experimental.

Would need to install with pip install pyprophet[parquet]. I forgot to add the optional dependency in the Docker build. I just updated it now, so it should work once it finishes building.

The github container registry for PyProphet can be found here

[jcharkow]

Thanks! That will solve the issue.

Maybe if that is the case there should be an automatic failure/more informative error that this? What was confusing to me is that the .oswpq conversion worked because that is dependent only on duckdb and only in scoring did things fail.

e.g. an error if .oswpq or .parquet is used that says something like Error parquet support requires the optional pyarrow dependency, please install with pip install pyarrow.

If you think thats a good idea I can do a PR for that.

[singjc]

Looking at your console log, there actually is a more informative error message that's printed prior to the Traceback.

PyProphet v3.0.5
Execution time: 2025-12-19 13:25:17
System: OS: Linux 6.8.0-90-generic | Python: 3.12.3 | CPU: 16 cores | RAM: 31.0 GB
Command: /opt/venv/bin/python /opt/venv/bin/pyprophet score --in tests/data/test_data.oswpq

[ 2025-12-19 at 13:25:17 | ERROR   | util::536                   ] An error has been caught in function 'wrapper', process 'MainProcess' (54131), thread 'MainThread' (128689185468544):
Traceback (most recent call last):
  File "/opt/venv/lib/python3.12/site-packages/pyprophet/io/util.py", line 61, in _ensure_pyarrow
    import pyarrow as pa  # pylint: disable=C0415
    ^^^^^^^^^^^^^^^^^^^^
ModuleNotFoundError: No module named 'pyarrow'

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/opt/venv/lib/python3.12/site-packages/pyprophet/io/util.py", line 343, in is_parquet_file
    pa, ArrowInvalid, ArrowIOError = _ensure_pyarrow()  # pylint: disable=C0103
                                     ^^^^^^^^^^^^^^^^^
  File "/opt/venv/lib/python3.12/site-packages/pyprophet/io/util.py", line 78, in _ensure_pyarrow
    raise click.ClickException(
click.exceptions.ClickException: Parquet support requires 'pyarrow'. Install with 'pip install pyarrow' or 'pip install pyprophet[parquet]'.

During handling of the above exception, another exception occurred:

Traceback (most recent call last):

We have a soft import check for pyarrow, but it would probably be good to signal a hard program failure if the program is taking a route where pyarrow is needed, to avoid the convoluted downstream errors.

[jcharkow]

Yes I did see the import error earlier but I was thinking that it would be good to have a hard failure since the entire message is quite long.

[singjc]

I agree, having a hard failure would make sense.

I am wondering if maybe we should just switch it back to having pyarrow as a required dependency. I was mainly reducing dependencies when working on integrating PyProphet into the OpenMS ecosystem (due to the pyprophet binary being too large).

I am also wondering, from a quick browsing of the code, we only seem to mainly use pyarrow for one thing, which is creating an arrow table from a pandas df to register a table in duckdb. Maybe we can get around this somehow and completely remove pyarrow?

[jcharkow]

I think the best option is to remove the dependency and just do everything with duckdb if possible

I know duckdb has its own built in arrow reader so I think it should be possible.