Skip to content

Attempt to run humann (3) with the official container#53

Open
miraep8 wants to merge 8 commits into
devfrom
add_humann_3.9.1
Open

Attempt to run humann (3) with the official container#53
miraep8 wants to merge 8 commits into
devfrom
add_humann_3.9.1

Conversation

@miraep8
Copy link
Copy Markdown
Contributor

@miraep8 miraep8 commented Apr 29, 2026 •
edited
Loading

Hi team!

Given the recent feedback we got I thought it might be worth taking a stab at running humann3 with the official container. Initially had been planning to upgrate to 3.9.1, but the signularity index for the galaxy projects only goes up to 3.9, so using that for now.

For the main step that required our hacky patch in the home-baked container (the humann_regroup submodule) I now have a hacky python script that tries to modify the config python object (ie doesn't need to actually change anything written in the container) before calling humann regroup. I think it works ok.... lets definitely do a full test before merging, just adding for now to share what I have been working on with this.

@miraep8 miraep8 requested review from nickp60 and vinisalazar and removed request for vinisalazar April 29, 2026 18:11
@nf-core-bot
Copy link
Copy Markdown
Member

nf-core-bot commented Apr 29, 2026

Warning

Newer version of the nf-core template is available.

Your pipeline is using an old version of the nf-core template: 3.5.2.
Please update your pipeline to the latest version.

For more documentation on how to update your pipeline, please see the nf-core documentation and Synchronisation documentation.

@miraep8
Copy link
Copy Markdown
Contributor Author

miraep8 commented Apr 29, 2026

I promise I ran these tests locally first y'all lol. I actually think this might not be an issue with my changes (?!?) but a nextflow version issue? A bit confused because I think they are all supposed to be running 24.10.5 but looking into the logs spotted this: Input version '' resolved to Nextflow v26.04.0 so might need to fix the CI a bit to make sure its actually testing the version it says it is.

nickp60
nickp60 requested changes Apr 29, 2026
View reviewed changes
Copy link
Copy Markdown
Collaborator

@nickp60 nickp60 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

A few things
-- this reverts to the old versions.yml instead of the updated versions topic, so all the versions logic changed here needs to be reverted
-- the meta should all be tagged as single_end at this point, as the tool does not accept paired data. The merging happens upstream and should be changing the single_end tag to true accordingly. Not sure why this is popping up now though...
-- the wrapper script is a nice idea! I hope it works. We might be back into headache territory due to the larger container size, but maybe we cross that bridge later

@github-actions
Copy link
Copy Markdown

github-actions Bot commented Apr 29, 2026 •
edited
Loading

nf-core pipelines lint overall result: Failed ❌

Posted for pipeline commit 271ebb1

+| ✅ 206 tests passed       |+
#| ❔   2 tests were ignored |#
#| ❔   1 tests had warnings |#
!| ❗  58 tests had warnings |!
-| ❌   3 tests failed       |-
Details

❌ Test failures:

  • nextflow_config - Config variable not found: params.input
  • files_unchanged - .github/workflows/linting.yml does not match the template
  • schema_params - Default value for param input invalid: Not in pipeline parameters. Check nextflow.config.

❗ Test warnings:

  • readme - README contains the placeholder zenodo.XXXXXXX. This should be replaced with the zenodo doi (after the first release).
  • pipeline_todos - TODO string in README.md: Include a figure that guides the user through the major workflow steps. Many nf-core
  • pipeline_todos - TODO string in README.md: If applicable, make list of people who have also contributed
  • pipeline_todos - TODO string in README.md: Add citation for pipeline after first release. Uncomment lines below and update Zenodo doi and badge at the top of this file.
  • pipeline_todos - TODO string in README.md: Add bibliography of tools and data used in your pipeline
  • pipeline_todos - TODO string in nextflow.config: Optionally, you can add a pipeline-specific nf-core config at https://github.com/nf-core/configs
  • pipeline_todos - TODO string in nextflow.config: Update the field with the details of the contributors to your pipeline. New with Nextflow version 24.10.0
  • pipeline_todos - TODO string in awsfulltest.yml: You can customise AWS full pipeline tests as required
  • pipeline_todos - TODO string in nextflow.config: Specify any additional parameters here
  • pipeline_todos - TODO string in methods_description_template.yml: #Update the HTML below to your preferred methods description, e.g. add publication citation for this pipeline
  • pipeline_todos - TODO string in meta.yml: #Add a description of the subworkflow and list keywords
  • pipeline_todos - TODO string in meta.yml: #Add a list of the modules and/or subworkflows used in the subworkflow
  • pipeline_todos - TODO string in meta.yml: #List all of the channels used as input with a description and their structure
  • pipeline_todos - TODO string in meta.yml: #List all of the channels used as output with a descriptions and their structure
  • pipeline_todos - TODO string in main.nf.test: Once you have added the required tests, please run the following command to build this file:
  • pipeline_todos - TODO string in main.nf.test: Add tags for all modules used within this subworkflow. Example:
  • pipeline_todos - TODO string in main.nf.test: Change the test name preferably indicating the test-data and file-format used
  • pipeline_todos - TODO string in main.nf.test: //Add all required assertions to verify the test output.
  • pipeline_todos - TODO string in meta.yml: #Add a description of the subworkflow and list keywords
  • pipeline_todos - TODO string in meta.yml: #Add a list of the modules and/or subworkflows used in the subworkflow
  • pipeline_todos - TODO string in meta.yml: #List all of the channels used as input with a description and their structure
  • pipeline_todos - TODO string in meta.yml: #List all of the channels used as output with a descriptions and their structure
  • pipeline_todos - TODO string in meta.yml: #Add a description of the subworkflow and list keywords
  • pipeline_todos - TODO string in meta.yml: #Add a list of the modules and/or subworkflows used in the subworkflow
  • pipeline_todos - TODO string in meta.yml: #List all of the channels used as input with a description and their structure
  • pipeline_todos - TODO string in meta.yml: #List all of the channels used as output with a descriptions and their structure
  • pipeline_todos - TODO string in main.nf.test: Once you have added the required tests, please run the following command to build this file:
  • pipeline_todos - TODO string in main.nf.test: //Add all required assertions to verify the test output.
  • pipeline_todos - TODO string in main.nf.test: //Add all required assertions to verify the test output.
  • pipeline_todos - TODO string in test_full.config: Specify the paths to your full test data ( on nf-core/test-datasets or directly in repositories, e.g. SRA)
  • pipeline_todos - TODO string in test_full.config: Give any required params for the test so that command line flags are not needed
  • pipeline_todos - TODO string in base.config: Check the defaults for all processes
  • pipeline_todos - TODO string in base.config: Customise requirements for specific processes.
  • pipeline_todos - TODO string in main.nf: See section in main README for further information regarding finding and adding container addresses to the section below.
  • pipeline_todos - TODO string in main.nf: Named file extensions MUST be emitted for ALL output channels
  • pipeline_todos - TODO string in main.nf: A stub section should mimic the execution of the original module as best as possible
  • pipeline_todos - TODO string in main.nf: If the module doesn't use arguments ($args), you SHOULD remove:
  • pipeline_todos - TODO string in meta.yml: # Add a description of the module and list keywords
  • pipeline_todos - TODO string in meta.yml: #Add a description and other details for the software below
  • pipeline_todos - TODO string in meta.yml: ##Add a description of all of the variables used as input
  • pipeline_todos - TODO string in meta.yml: ##Add a description of all of the variables used as output
  • pipeline_todos - TODO string in main.nf.test: Once you have added the required tests, please run the following command to build this file:
  • pipeline_todos - TODO string in main.nf.test: Change the test name preferably indicating the test-data and file-format used
  • pipeline_todos - TODO string in main.nf.test: If you are created a test for a chained module
  • pipeline_todos - TODO string in main.nf.test: define inputs of the process here. Example:
  • pipeline_todos - TODO string in main.nf.test: //Add all required assertions to verify the test output.
  • pipeline_todos - TODO string in main.nf.test: Change the test name preferably indicating the test-data and file-format used but keep the " - stub" suffix.
  • pipeline_todos - TODO string in main.nf.test: define inputs of the process here. Example:
  • pipeline_todos - TODO string in main.nf: Named file extensions MUST be emitted for ALL output channels
  • pipeline_todos - TODO string in main.nf: List additional required output channels/values here
  • pipeline_todos - TODO string in meta.yml: # Add a description of the module and list keywords
  • pipeline_todos - TODO string in meta.yml: #Add a description and other details for the software below
  • pipeline_todos - TODO string in meta.yml: ##Add a description of all of the variables used as input
  • pipeline_todos - TODO string in meta.yml: ##Add a description of all of the variables used as output
  • pipeline_todos - TODO string in main.nf.test: define inputs of the process here. Example:
  • pipeline_todos - TODO string in main.nf.test: //Add all required assertions to verify the test output.
  • schema_params - Schema param input not found from nextflow config
  • local_component_structure - concatall.nf in subworkflows/local should be moved to a SUBWORKFLOW_NAME/main.nf structure

❔ Tests ignored:

  • files_unchanged - File ignored due to lint config: .github/PULL_REQUEST_TEMPLATE.md
  • files_unchanged - File ignored due to lint config: assets/nf-core-funcprofiler_logo_light.png

❔ Tests fixed:

✅ Tests passed:

Run details

  • nf-core/tools version 3.5.2
  • Run at 2026-04-29 19:41:29

@miraep8
Copy link
Copy Markdown
Contributor Author

miraep8 commented Apr 29, 2026

Updated the versions and marked everying as single ended. I think the linting failure is coming again from a nextflow version update issue, but attempts to set the nextflow version to a lwoer version also cause the linter to fail (because the linting.yaml file is no longer the same as the template lol). Bit of a catch 22 so maybe we take this as a prodding to comply idk.

miraep8
miraep8 commented Apr 29, 2026
View reviewed changes
Comment thread modules/local/humann/renorm/tests/main.nf.test.snap Outdated
@nickp60 nickp60 self-requested a review April 29, 2026 20:48
@nickp60
Copy link
Copy Markdown
Collaborator

nickp60 commented Apr 29, 2026

Thanks! LGTM. Feel free to bump the template if that makes merging easier.

@vinisalazar
Copy link
Copy Markdown
Collaborator

vinisalazar commented May 12, 2026

@miraep8 @nickp60 could we close this in favour of nf-core/modules#11201?

@nickp60
Copy link
Copy Markdown
Collaborator

nickp60 commented May 12, 2026

@vinisalazar What @miraep8 is proposing here is a way to avoid having to use the patched versions that allow for the config location variable. If this works (and it seems to be!) then we don't have to ask people to use our custom docker images, which should make the review process easier.

@nickp60
Copy link
Copy Markdown
Collaborator

nickp60 commented May 12, 2026

I think this should be incorporated into your module PR for humann @vinisalazar. Then we can use the official conda/docker containers and be off the hook for supporting them in the future.

@miraep8
Copy link
Copy Markdown
Contributor Author

miraep8 commented May 12, 2026

I think that Vini's solution in #11201 also makes use of the official docker images actually! His solution is different than mine ie:

STATIC_CONFIG=`python -c "import humann; print(humann.__file__.replace('__init__.py', 'humann.cfg'))"`
    cat \$STATIC_CONFIG  | sed "s|utility_mapping = .*|utility_mapping = ${utility_db}|g" > humann.cfg
    export HUMANN_CONFIG=humann.cfg

If this approach works for humann regroup (and it seems to be) then I would definitely be in favor of closing this in favor of your PR Vini as I think your solution seems more elegant than mine! I have just wanted to try your version out first and haven't had a chance to yet :) but I suspect it will be fine if its working in your hands.

@nickp60
Copy link
Copy Markdown
Collaborator

nickp60 commented May 12, 2026

The HUMANN_CONFIG is the stuff I added in the patched version of humann -- that will not work in non-patched containers. The tests pass because (A) its only being run in -stub mode and (B) it regrouping to one of the tables included in the package data: its not using the data in the untarred utility database output from the UNTAR rule. you can confirm this by disabling the -stub option and looking at the log:

#cat .nf-test/tests/e58ff39934787b1547058a8617437648/work/b8/26c94ac1d35b4c2f3744ef8492089d/.command.err
Loading table from: input.tsv
Loading mapping file from: /usr/local/lib/python3.10/site-packages/humann/tools/../data/pathways/metacyc_reactions_level4ec_only.uniref.bz2
Original Feature Count: 300; Grouped 1+ times: 0 (0.0%); Grouped 2+ times: 0 (0.0%)

you can also add a line to the script section

humann_regorup_table --help

and look for which group options are available:

  -g {uniref90_rxn,uniref50_rxn}, --groups {uniref90_rxn,uniref50_rxn}
                        Built-in grouping options

which is missing all the ones from the utility database

@miraep8
Copy link
Copy Markdown
Contributor Author

miraep8 commented May 12, 2026

Thanks Nick! I had thought for some reason that Vini was doing something slightly different from what you had before. In that case Vini - maybe it would be worth integrating my changes into your human module pull instead. (At least for the regroup module)...sorry for the confusion on my part! And thank you both for your work on this :)

vinisalazar added a commit to vinisalazar/modules that referenced this pull request May 14, 2026
@vinisalazar @miraep8
humann3/regroup: replace HUMANN_CONFIG with Python in-memory config
90c80e8 
HUMANN_CONFIG env var only works in patched containers. Use Python
to set config.utility_mapping_database in-memory before calling
humann.tools.regroup_table.main() directly.

Also remove HUMANN_CONFIG block from humann/humann (no-op in official
containers).

Refs nf-core/funcprofiler#53

Co-authored-by: Mirae Baichoo <miraep8@gmail.com>
vinisalazar added a commit to vinisalazar/modules that referenced this pull request May 14, 2026
@vinisalazar @miraep8
humann3/regroup/tests: add non-stub test with real genefamilies input
7048843 
Add test_genefamilies.tsv with UniRef90 IDs present in utility_nfDEMO
mapping. Add non-stub test to verify utility_db is actually used.
Fix stub test input (was fastq, now correct TSV).

Refs nf-core/funcprofiler#53

Co-authored-by: Mirae Baichoo <miraep8@gmail.com>
@vinisalazar
Copy link
Copy Markdown
Collaborator

vinisalazar commented May 14, 2026

Just updated nf-core/modules#11201, please have look if possible

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@nickp60 nickp60 nickp60 approved these changes

@vinisalazar vinisalazar Awaiting requested review from vinisalazar

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

4 participants

@miraep8 @nf-core-bot @nickp60 @vinisalazar