Yank processes not cleaning exiting

[karalets]

Hi,

I am trying to get some gpus in interactive mode and I am really having a hard time getting them. Normally this would be fine and I'd wait until the cluster clears up, but... the gpus are simply not really in use, so I am not competing against anybody to get them.

As such, the cluster seems to prefer to let the gpus gather dust over giving them to me. Shocker!

Is there any explanation for this? Or, to be actionable: can I do something to change that?

Best and thanks,

Theo

[tatarsky]

Why do you feel the GPUs are not in use?

[karalets]

Because I checked for the running gpu jobs and found very few.

[tatarsky]

How are you checking? As I do not see that. Removing username.

7205572.hal-sched1.loc  USER       gpu      i_hsa_dansylamid  13454     3     12    --   72:00:00 R  49:58:48   gpu-1-16/15,20,24,34+gpu-1-17/2-3,13,24+gpu-2-7/32-35
7205573.hal-sched1.loc  USER       gpu      i_hsa_dansylglyc  20518     3     12    --   72:00:00 R  49:57:32   gpu-1-7/32-35+gpu-2-6/13,28,31-32+gpu-2-9/11-12,20,27
7205574.hal-sched1.loc  USER       gpu      i_hsa_indomethac  13848     3     12    --   72:00:00 R  49:55:40   gpu-1-13/27,29,32,34+gpu-1-15/15-17,28+gpu-1-11/32-35
7205575.hal-sched1.loc  USER       gpu      i_hsa_lapatinib    5463     3     12    --   72:00:00 R  49:54:25   gpu-1-12/25,33-35+gpu-1-10/32-35+gpu-1-6/28-30,35
7212730.hal-sched1.loc  USER       gpu      i_hsa_naproxen    23815     3     12    --   72:00:00 R  26:58:28   gpu-2-12/15,32-34+gpu-1-4/19,32-34+gpu-3-9/32-35
7212731.hal-sched1.loc  USER       gpu      i_hsa_phenylbuta    993     3     12    --   72:00:00 R  26:09:30   gpu-2-11/32-35+gpu-1-14/32-35+gpu-2-5/19-20,25-26
7212733.hal-sched1.loc  USER       gpu      i_hsa_ponatinib   17700     3     12    --   72:00:00 R  26:08:09   gpu-2-17/23,33-35+gpu-3-8/7,33-35+gpu-1-5/2,32-34
7212769.hal-sched1.loc  USER       gpu      hsa_dansylamide    1332     2      8    --   48:00:00 R  25:12:37   gpu-2-16/24,32-34+gpu-1-8/29,31,33,35

Double checking but that sure seems like a fair chunk.

[tatarsky]

The program is called "yank" and I show it on lots of GPUs.

[jchodera]

yank is a program from our lab, and does use lots of GPUs. I'm looking into trying to figure out exactly which GPUs are free at the moment.

Oddly enough, there are some GPUs that are running jobs from two different people, which shouldn't be happening:

gpu-2-5
Tue May 17 17:11:59 2016       
+------------------------------------------------------+                       
| NVIDIA-SMI 352.39     Driver Version: 352.39         |                       
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|===============================+======================+======================|
|   0  GeForce GTX TIT...  Off  | 0000:03:00.0     Off |                  N/A |
| 22%   53C    P2    83W / 250W |    521MiB / 12287MiB |      0%      Default |
+-------------------------------+----------------------+----------------------+
|   1  GeForce GTX TIT...  Off  | 0000:04:00.0     Off |                  N/A |
| 22%   51C    P2    82W / 250W |    522MiB / 12287MiB |      0%      Default |
+-------------------------------+----------------------+----------------------+
|   2  GeForce GTX TIT...  Off  | 0000:83:00.0     Off |                  N/A |
| 22%   57C    P2    89W / 250W |    522MiB / 12287MiB |      0%      Default |
+-------------------------------+----------------------+----------------------+
|   3  GeForce GTX TIT...  Off  | 0000:84:00.0     Off |                  N/A |
| 22%   55C    P2    81W / 250W |    521MiB / 12287MiB |      0%      Default |
+-------------------------------+----------------------+----------------------+

+-----------------------------------------------------------------------------+
| Processes:                                                       GPU Memory |
|  GPU       PID  Type  Process name                               Usage      |
|=============================================================================|
|    0      6897    C   /cbio/jclab/home/misik/anaconda/bin/python     201MiB |
|    0     13653    C   /cbio/jclab/home/luirink/anaconda/bin/python   170MiB |
|    0     31953    C   /cbio/jclab/home/misik/anaconda/bin/python     122MiB |
|    1      6896    C   /cbio/jclab/home/misik/anaconda/bin/python     201MiB |
|    1     13652    C   /cbio/jclab/home/luirink/anaconda/bin/python   171MiB |
|    1     31952    C   /cbio/jclab/home/misik/anaconda/bin/python     122MiB |
|    2      6895    C   /cbio/jclab/home/misik/anaconda/bin/python     201MiB |
|    2     13651    C   /cbio/jclab/home/luirink/anaconda/bin/python   171MiB |
|    2     31951    C   /cbio/jclab/home/misik/anaconda/bin/python     122MiB |
|    3      6894    C   /cbio/jclab/home/misik/anaconda/bin/python     201MiB |
|    3     13650    C   /cbio/jclab/home/luirink/anaconda/bin/python   170MiB |
|    3     31950    C   /cbio/jclab/home/misik/anaconda/bin/python     122MiB |
+-----------------------------------------------------------------------------+

gpu-2-11
Tue May 17 17:12:04 2016       
+------------------------------------------------------+                       
| NVIDIA-SMI 352.39     Driver Version: 352.39         |                       
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|===============================+======================+======================|
|   0  GeForce GTX TITAN   Off  | 0000:03:00.0     Off |                  N/A |
| 36%   54C    P0    70W / 250W |    228MiB /  6143MiB |      0%      Default |
+-------------------------------+----------------------+----------------------+
|   1  GeForce GTX TITAN   Off  | 0000:04:00.0     Off |                  N/A |
| 36%   56C    P0    72W / 250W |    228MiB /  6143MiB |      0%      Default |
+-------------------------------+----------------------+----------------------+
|   2  GeForce GTX TITAN   Off  | 0000:83:00.0     Off |                  N/A |
| 38%   60C    P0    72W / 250W |    228MiB /  6143MiB |      0%      Default |
+-------------------------------+----------------------+----------------------+
|   3  GeForce GTX TITAN   Off  | 0000:84:00.0     Off |                  N/A |
| 42%   68C    P0   157W / 250W |    227MiB /  6143MiB |     99%      Default |
+-------------------------------+----------------------+----------------------+

+-----------------------------------------------------------------------------+
| Processes:                                                       GPU Memory |
|  GPU       PID  Type  Process name                               Usage      |
|=============================================================================|
|    0      1049    C   /cbio/jclab/home/misik/anaconda/bin/python      80MiB |
|    0     10711    C   /cbio/jclab/home/luirink/anaconda/bin/python   129MiB |
|    1      1048    C   /cbio/jclab/home/misik/anaconda/bin/python      80MiB |
|    1     10710    C   /cbio/jclab/home/luirink/anaconda/bin/python   129MiB |
|    2      1047    C   /cbio/jclab/home/misik/anaconda/bin/python      80MiB |
|    2     10709    C   /cbio/jclab/home/luirink/anaconda/bin/python   130MiB |
|    3      1046    C   /cbio/jclab/home/misik/anaconda/bin/python      80MiB |
|    3     10708    C   /cbio/jclab/home/luirink/anaconda/bin/python   128MiB |
+-----------------------------------------------------------------------------+

[karalets]

alright, I misinterpreted that and thought these are single jobs/gpus.

Thank you, Paul. I will probably need to deal with this diplomatically ;)

[jchodera]

I show luirink as having no running jobs, so her jobs should not be still running on GPUs:

[chodera@mskcc-ln1 ~/scripts]$ qstat -u luirink
[chodera@mskcc-ln1 ~/scripts]$ 

It looks like this is related to #409 (comment)

[karalets]

John, do you think it would be possible to keep 5-10 gpus useable or unoccupied by yank until friday (NIPS deadline)?

I really will not be needing any more, but just having 3 as is the case now is a bit tight.

[tatarsky]

I show processes by luirink on gpu-2-5. If the code is not exiting cleanly it will need to be fixed.

luirink  13650 93.9  0.2 308770540 739620 ?    Rl   May11 8806:31 /cbio/jclab/home/luirink/anaconda/bin/python /cbio/jclab/home/luirink/anaconda/bin/yank run --store=output --verbose --mpi -i 5000 --platform=OpenCL --phase=complex-explicit
luirink  13651 94.4  0.2 308770796 743364 ?    Rl   May11 8848:40 /cbio/jclab/home/luirink/anaconda/bin/python /cbio/jclab/home/luirink/anaconda/bin/yank run --store=output --verbose --mpi -i 5000 --platform=OpenCL --phase=complex-explicit
luirink  13652 92.8  0.2 308770800 729816 ?    Rl   May11 8704:07 /cbio/jclab/home/luirink/anaconda/bin/python /cbio/jclab/home/luirink/anaconda/bin/yank run --store=output --verbose --mpi -i 5000 --platform=OpenCL --phase=complex-explicit
luirink  13653 94.2  0.2 308770536 739504 ?    Rl   May11 8834:58 /cbio/jclab/home/luirink/anaconda/bin/python /cbio/jclab/home/luirink/anaconda/bin/yank run --store=output --verbose --mpi -i 5000 --platform=OpenCL --phase=complex-explicit

[tatarsky]

But basically @karalets the statement in this Git that "the cluster seems to prefer to let the gpus gather dust over giving them to me" is incorrect and I'm still waiting for how you determined that so we can resolve the actually requested support. I show yank consuming all the GPUs.

[tatarsky]

I see your note @karalets. Ignore. If you want my methods of parsing the qstat I can document.

[jchodera]

@MehtapIsik: Can you halt one or more of your yank jobs right now so @karalets can use some GPUs?

[karalets]

Paul @tatarsky , I must have made a mistake as said earlier, I was just counting the number of jobs, not the number of gpus each one of these jobs was using. So please put my comments down to my ignorance of the cluster-diagnostics.

[tatarsky]

Noted your Git response already. Just an ordering. I am happy to share my method of determining.

[karalets]

I was doing 'qstat | grep gpu' which obviously was suboptimal.