`evaluation_pca_chosen`/`evaluation_per_algo_pca_chosen` rules read undeclared pathway files causing fatal errors with HTCondor

[jhiemstrawisc]

The evaluation_pca_chosen and evaluation_per_algo_pca_chosen rules fail with FileNotFoundError when running on HTCondor in the absence of a shared filesystem because they read pathway files that are not declared as rule inputs.

The Snakemake executor for HTCondor only knows to transfer data files that are either explicitly given to it via a rule's inputs, or are manually added to a rule's resources using htcondor_transfer_input_files.

It looks like these rules function on shared filesystems because Evaluation.pca_chosen_pathway() constructs these paths dynamically , which is why things still work in the presence of a shared filesystem.

To fix this, all inputs to these rules should be declared.

[ntalluri]

The pca_chosen_pathway method selects representative pathways based on PCA coordinates at runtime, so the specific pathway files it reads cannot be known until after PCA runs. There is no way to declare the chosen pathways in rule all via final_input.extend because the selection depends on KDE peak distances computed from PCA coordinates, which are only available after the ml_analysis and ml_analysis_aggregate_algo rules complete. We need a way to make Snakemake to know of this dependency without being able to declare the chosen pathways.

I think the fix has two parts:

1. Declare the pathway files as inputs to both evaluation rules so Snakemake can enforce the dependency. Specifically, add pathways = collect_pathways_per_dataset as an input to both evaluation_pca_chosen and evaluation_per_algo_pca_chosen.

2. In make_final_input, the evaluation_pca_chosen and evaluation_per_algo_pca_chosen final outputs need an algorithm wildcard added to the expand calls with either algorithm=algorithms_mult_param_combos or algorithm=algorithms so collect_pathways_per_dataset can work.

[ntalluri]

@jhiemstrawisc I can try to work on this. Would the current HTCondor SPRAS setup work for testing these changes?

[tristan-f-r]

We should also have a way to test this to prevent this from happening - how easy would it be to have a Snakemake executor that symlinks input files and copies over output files to a central directory, that way we can enforce that all input files exist?

[agitter]

no way to declare the chosen pathways in rule all via final_input.extend

Is this part necessary or is the fix isolated to the input statements of these PCA-related rules?

[ntalluri]

Based on what I looked over for the evaluation code, this seems to be only related to the pathways chosen for the pca-related evaluations. Nothing else was picking pathways and dynamically building the paths to them; they were directly using the output pathways or the ensembled pathway.