Parallelization support?

[gabriel-reynald]

Hello,

I'm exploring SketchMap for a very larger datasets from molecular dynamics. Is there any support for parallelization in SketchMap?

Thank you for your time and consideration.

[andrejberg]

Hi Gabriel,

I wasn’t part of the development team, so I can’t say for sure. When I used it, I relied on fairly trivial parallelization approaches.

[kevinsawade]

Hello Gabriel,

Only the calls to the dimproj program can be parallelized. Here's a snippet from an old script we've used to parallelize it:

# get n cpus
CPU=$(nproc)

# sigmoid parameters
DIMNUM
S = 
A = 
B =
s = 
a = 
b = 

# get an array of highd files. Here, these files are suffixed with .highd
shopt -s nullglob
FOLDER = /path/to/folder/with/highd_files/in/it
FILES=(${FOLDER}/*.highd)
NFILES=${#FILES[@]}
shopt -u nullglob

# run dimproj in parallel
var=0
for i in `seq 1 $(($NFILES/$CPU+1))`; do
  for j in `seq 0 $(($CPU-1))`; do
    if [ $var -le $NFILES] ; then
       file=${FILES[var]}
       filen=$( echo "$col" | sed "s/.*\///" | sed "s/\..*//")
       echo "Projecting ${file}."
       dimproj -D $DIMNUM -d 2 -P /path/to/highd-landmarks -p /path/to/lowd-landmarks -grid 100.0,21,201 -fun-hd ${S},${A},${B| -fun-ld ${s},${a},${b} -cgmin 10 < ${file} > ${filen}.lowd &
      var=$((var + 1)) ;
    fi
  done
  wait
done

[ceriottm]

Hello! Glad to see that this is still being of use. I fear that the chance we will be actively doing any development here is low, but we are discussing about re-implementing sketch-map here https://scikit-matter.readthedocs.io/en/latest/index.html, and landmark selection is implemented there https://scikit-matter.readthedocs.io/en/latest/getting-started.html#getting-started-selection so maybe that can help partially to parse through your large dataset to select a small number of references.

You may also want to try a general-purpose dimensionality reduction model https://chemiscope.org/docs/examples/6-explore.html