runOSCommand throws error with R 4.0 (missing shQuote)

[aschersleben]

Hi,
I'm working on a debian server and try to use makeClusterFunctionsSSH(...) to distribute jobs to other debian servers. It throws the following error:
syntax error near unexpected token `('
The error is caused by runOSCommand(...).
In 75a5b70 you removed the shQuote for R 4.0 and higher. But as far as I was able to reproduce the error, this shQuote is still needed with R 4.0.2.

MWE directly built from the code of Worker and runOSCommand:
(you have to set the nodename variable)

library(batchtools)
### error:

nodename = "XXXXX"
sys.args = c("-e", shQuote("message(\"[bt] --BOF--\\n\", \"[bt] \", system.file(\"bin/linux-helper\", package = \"batchtools\"), \"\\n[bt] --EOF--\\n\")"))
sys.cmd = "Rscript"
checkmate::assertCharacter(sys.cmd, any.missing = FALSE, len = 1L)
checkmate::assertCharacter(sys.args, any.missing = FALSE)
checkmate::assertString(nodename, min.chars = 1L)
command = sprintf("%s %s", sys.cmd, stringi::stri_flatten(sys.args, " "))
command = stringi::stri_replace_all_fixed(command, "\\$", "$")
sys.args = c("-q", nodename, command)
sys.cmd = "ssh"
system2(command = sys.cmd, args = sys.args, stdin = "", stdout = TRUE, stderr = TRUE, wait = TRUE)
#> Warning in system2(command = sys.cmd, args = sys.args, stdin = "", stdout =
#> TRUE, : running command ''ssh' -q XXXXX
#> Rscript -e 'message("[bt] --BOF--\n", "[bt] ", system.file("bin/linux-helper",
#> package = "batchtools"), "\n[bt] --EOF--\n")' 2>&1' had status 1
#> [1] "bash: -c: line 0: syntax error near unexpected token `('"                                                                                                   
#> [2] "bash: -c: line 0: `Rscript -e message(\"[bt] --BOF--\\n\", \"[bt] \", system.file(\"bin/linux-helper\", package = \"batchtools\"), \"\\n[bt] --EOF--\\n\")'"
#> attr(,"status")
#> [1] 1


#### works:

command = shQuote(command)
sys.args = c("-q", nodename, command)
sys.cmd = "ssh"
system2(command = sys.cmd, args = sys.args, stdin = "", stdout = TRUE, stderr = TRUE, wait = TRUE)
#> [1] "[bt] --BOF--"                                                               
#> [2] "[bt] /home/pa/R/x86_64-pc-linux-gnu-library/4.0/batchtools/bin/linux-helper"
#> [3] "[bt] --EOF--"                                                               
#> [4] ""

^{Created on 2020-07-17 by the reprex package (v0.3.0)}

Can you reproduce this?

[franzbischoff]

I was just opening another issue, but I see it is already found.

My lazy repro is:

doesn't work

args <- c("-e", shQuote("message(\"[bt] --BOF--\\n\", \"[bt] \", system.file(\"bin/linux-helper\", 
  package = \"batchtools\"), \"\\n[bt] --EOF--\\n\")"))
command <- sprintf("%s %s", "Rscript", stringi::stri_flatten(args, " "))
# command = shQuote(command)
command <- stringi::stri_replace_all_fixed(command, "\\$", "$")
sys.args <- c("-q", "myremotehost", command)
sys.cmd <- "ssh"
res <- suppressWarnings(system2(command = sys.cmd, args = sys.args, stdin = "", 
   stdout = TRUE, stderr = TRUE, wait = TRUE))
output <- as.character(res)
output

[1] "bash: -c: line 0: syntax error near the symbol \"(\" unexpected"

works

args <- c("-e", shQuote("message(\"[bt] --BOF--\\n\", \"[bt] \", system.file(\"bin/linux-helper\", 
  package = \"batchtools\"), \"\\n[bt] --EOF--\\n\")"))
command <- sprintf("%s %s", "Rscript", stringi::stri_flatten(args, " "))
command = shQuote(command)
# command <- stringi::stri_replace_all_fixed(command, "\\$", "$")
sys.args <- c("-q", "myremotehost", command)
sys.cmd <- "ssh"
res <- suppressWarnings(system2(command = sys.cmd, args = sys.args, stdin = "", 
   stdout = TRUE, stderr = TRUE, wait = TRUE))
output <- as.character(res)
output

[1] "[bt] --BOF--"                                                  
[2] "[bt] /usr/local/lib/R/site-library/batchtools/bin/linux-helper"
[3] "[bt] --EOF--"                                                  
[4] ""  

sessionInfo()
R version 4.1.0 (2021-05-18)
Platform: x86_64-pc-linux-gnu (64-bit)
Running under: Ubuntu 20.04.2 LTS

[franzbischoff]

And now I'm here again "solving" the same issue... I forgot I've already had...

[jemus42]

Welp, guess I've found my people.

[jemus42]

I fixed it on my fork by just commenting out the check for the R version so shQuote is always used, but since I haven't done any real testing I don't know if that maybe breaks something else somehow

https://github.com/jemus42/batchtools/tree/ssh-shquote

(And since my batchtools experiment is ongoing, I can't hot-swap in a new batchtools version easily to test it in-situ 😬 )

[jemus42]

Hmm nevermind, I can construct the Worker with this fix but when actually trying to submitJobs or anything I get

Error in runOSCommand(self$script, args, nodename = self$nodename) :
  Assertion on 'sys.cmd' failed: Must be of type 'character', not 'NULL'.

..so this needs a proper fix.

[jemus42]

Maybe the fix wasn't that hard after all?

See #325