KNIME dependencies sounds not correctly installed

[tduigou]

Tested on MacOS, @breakthewall also met the issue with linux.

The KNIME error about missing dependencies is back (above).

ERROR	 main Node	 Loading model settings failed: Data cell implementation 'org.knime.chem.types.SmartsCell' not found.

EDIT: I tried a new installation, running for 2 hours now but the process is still active. It looks like downloading extensions from Zenodo take a very long time (on MacOS system monitor, the download rate is around 2 MB/sec). However, direct download from firefox is way more quick, which means this is not a connexion issue, but rather a limitation specific to the download made through retropath2_wrapper...

[tduigou]

Update: after a while, the installation went well this time.

I suspect the download time is so long that there is substancial chances of download timeout / loss of network, which leads to a corrupted installation.

[guillaume-gricourt]

Tested on linux without errors using only python and for galaxy wrapper (brsynth/galaxytools#227) on a local galaxy.

If the problem is downloading dependencies from Zenodo, perhaps we could let users download them and add them via the command line to create an offline installation.

python -m retropath2_wrapper.knime \
    --kinstall "/path/to/knime/root/dir" \
    --extension-knime org.knime.update.analytics-platform_4.6.4.zip \
    --extension-community TrustedCommunityContributions_4.6_202212212136.zip \ 
   --kexec knime_4.6.4.app.macosx.cocoa.x86_64.dmg | knime_4.6.4.linux.gtk.x86_64.tar.gz | knime_4.6.4.win32.win32.x86_64.zip 

[tduigou]

That's a good idea, that's would help in case of failure.
Side question: how long was the installation on your local galaxy?

[guillaume-gricourt]

Around 1h30, 11GB to download

[breakthewall]

Wow, this is huge and not doable in a Galaxy environment because I think KNIME is downloaded in the working directory of each job, so:

• the manual installation is not possible
• 1h30 waiting the download during the job is very long
• 11G per job will explode the storage space, or we have to delete the folder at the end of the Cheetah code

What do you think?

[guillaume-gricourt]

Few comments

Galaxy wrapper:

• the installation is done in $TMPDIR
• no problem of storage capacity
• Knime install can be done manually through the conda env installed by Galaxy in the $TMPDIR

Python package:

• the installation is done once, only if no knime executable is not found in the kinstall directory. So, only, the first execution takes time.
• deletes these 11GB of data after the installation

If the wait is a problem to install Knime, I can think of two solutions:

• Come back to the original way
• Use docker

[breakthewall]

Thanks for your reply Guillaume.

Ok I misinterpreted the var ${TMPDIR:-.}, in the TMPDIR is perfect since it will be downloaded at most once per Galaxy launch. And it can be installed manually once at Galaxy boot.

What do you mean by: deletes these 11GB of data after the installation. ?

[guillaume-gricourt]

Knime requires some plugins to run the Retropath2 workflows. Nine are installed, but themselves have plugins dependencies, and I don't know which plugins are installed at the end.
Lots of plugins are downloaded from these 11GB of data, but in fact few of them are installed in the Knime installation.

[breakthewall]

Yes, I (or we with Thomas, can't remember) I had tested almost each of them in order to find the good combination and I thikn we stored a suitable version on Zenodo.

[breakthewall]

@guillaume-gricourt, the thing is by default a Docker container has no TMPDIR environment variable. So currently ${TMPDIR:-.} gives . in my Galaxy Docker. Therefore, KNIME is installed in . (= job's working directory) at each new submitted job of retropath2_wrapper...

I added ENV TMPDIR=/galaxy/database/tmp to the Dockerfile of Galaxy and now TMPDIR is set to /galaxy/database/tmp in the Galaxy Docker container.

After a first submitted job on a fresh Galaxy instance, it takes about 30mn to crash with the following message:

('Connection broken: IncompleteRead(148811112 bytes read, 1651916855 more expected)', IncompleteRead(148811112 bytes read, 1651916855 more expected))

but I can see a folder named knime_4.6.4 in /galaxy/database/tmp.

I'll try now manual installation, which is possible after the first failed submitted job on a Galaxy Docker because retropath2_wrapper has to be installed first to run the manual CLI command.

[breakthewall]

@guillaume-gricourt would it sound reasonable to you to give "\${TMPDIR:-.}"/knime to --kinstall option in the wrapper of retropath2_wrapper?

It will create a knime folder where several version can coexist and it will be much easier to manage with Galaxy data volumes.

[guillaume-gricourt]

Sure, it's ok