bigglm.ffdf not working

[egarpor]

When running the example of bigglm.ffdf:

> library(biglm)
> library(ff)
> 
> data(trees)
> x <- as.ffdf(trees)
> a <- bigglm(log(Volume)~log(Girth)+log(Height), 
+             data=x, chunksize=10, sandwich=TRUE)
Error in UseMethod("bigglm", data) : 
  no applicable method for 'bigglm' applied to an object of class "ffdf"

Error on ff 0.13.1 and biglm 0.9-2. My session:

> version
               _                           
platform       x86_64-apple-darwin17.0     
arch           x86_64                      
os             darwin17.0                  
system         x86_64, darwin17.0          
status                                     
major          4                           
minor          0.2                         
year           2020                        
month          06                          
day            22                          
svn rev        78730                       
language       R                           
version.string R version 4.0.2 (2020-06-22)
nickname       Taking Off Again    

[jwijffels]

Why did you decide not to add library(ffbase) in your code.

[egarpor]

Hi,

Thanks for the reply. I did:

> library(ffbase)
> library(biglm)
> library(ff)
> 
> data(trees)
> x <- as.ffdf(trees)
> a <- bigglm(log(Volume)~log(Girth)+log(Height), 
+             data=x, chunksize=10, sandwich=TRUE)
Error in UseMethod("bigglm", data) : 
  no applicable method for 'bigglm' applied to an object of class "ffdf"
> sessionInfo()
R version 4.0.2 (2020-06-22)
Platform: x86_64-apple-darwin17.0 (64-bit)
Running under: macOS Catalina 10.15.7

Matrix products: default
BLAS:   /System/Library/Frameworks/Accelerate.framework/Versions/A/Frameworks/vecLib.framework/Versions/A/libBLAS.dylib
LAPACK: /Library/Frameworks/R.framework/Versions/4.0/Resources/lib/libRlapack.dylib

locale:
[1] en_US.UTF-8/en_US.UTF-8/en_US.UTF-8/C/en_US.UTF-8/en_US.UTF-8

attached base packages:
[1] stats     graphics  grDevices utils     datasets  methods  
[7] base     

other attached packages:
[1] biglm_0.9-2   DBI_1.1.0     ffbase_0.13.1 ff_4.0.4     
[5] bit_4.0.4 

Besides, if I call bigglm.ffdf rather than bigglm, I got this:

> library(ffbase)
> library(biglm)
> library(ff)
> 
> data(trees)
> x <- as.ffdf(trees)
> a <- bigglm.ffdf(log(Volume)~log(Girth)+log(Height), 
+             data=x, chunksize=10, sandwich=TRUE)
Error in coef.bigqr(object$qr) : 
  NA/NaN/Inf in foreign function call (arg 5)

I had a code like that working in prior versions of ffbase. Now stopped working and is throwing that error, that's why I went to the help page.

Both issues also showing up in other machine:

> sessionInfo()
R version 4.0.3 (2020-10-10)
Platform: x86_64-pc-linux-gnu (64-bit)
Running under: Ubuntu 16.04.7 LTS

Matrix products: default
BLAS:   /usr/lib/atlas-base/atlas/libblas.so.3.0
LAPACK: /usr/lib/atlas-base/atlas/liblapack.so.3.0

locale:
 [1] LC_CTYPE=C.UTF-8       LC_NUMERIC=C          
 [3] LC_TIME=C.UTF-8        LC_COLLATE=C.UTF-8    
 [5] LC_MONETARY=C.UTF-8    LC_MESSAGES=C.UTF-8   
 [7] LC_PAPER=C.UTF-8       LC_NAME=C             
 [9] LC_ADDRESS=C           LC_TELEPHONE=C        
[11] LC_MEASUREMENT=C.UTF-8 LC_IDENTIFICATION=C   

attached base packages:
[1] stats     graphics  grDevices utils     datasets  methods  
[7] base     

other attached packages:
[1] biglm_0.9-2   DBI_1.1.0     ffbase_0.13.1 ff_4.0.4     
[5] bit_4.0.4    

loaded via a namespace (and not attached):
[1] compiler_4.0.3  tools_4.0.3     fastmatch_1.1-0

Best,
Eduardo

[jwijffels]

There seems to be 2 issues

1. biglm changed S3 generic bigglm to an S4 generic. That's the reason why just calling bigglm did not work but bigglm.ffdf does as ffbase exports explicitely bigglm.ffdf
2. issue with NA in foreign function calls seem to be caused by changes in bit::chunk. It looks like bit::chunk which is used by bigglm.ffdf in https://github.com/edwindj/ffbase/blob/master/pkg/R/bigglm.R#L35 no longer has argument by. @edwindj is that correct? In that case, would mean more issues

[egarpor]

FWIW, I have found success in downgrading bit, ff, and ffbase prior to bit's 4.0.0 version:

library(versions)
install.versions(pkgs = c("bit", "ff", "ffbase"), versions = c("1.1-15.2", "2.2-14.2", "0.12.8"))

These were the higher versions I manage to find for which the following example does not throw an error:

library(ffbase)
library(biglm)
library(ff)

data(trees)
x <- as.ffdf(trees)
a <- bigglm.ffdf(log(Volume)~log(Girth)+log(Height), data=x, chunksize=10, sandwich=TRUE)

My session:

> sessionInfo()
R version 4.0.2 (2020-06-22)
Platform: x86_64-apple-darwin17.0 (64-bit)
Running under: macOS  10.16

Matrix products: default
LAPACK: /Library/Frameworks/R.framework/Versions/4.0/Resources/lib/libRlapack.dylib

locale:
[1] en_US.UTF-8/en_US.UTF-8/en_US.UTF-8/C/en_US.UTF-8/en_US.UTF-8

attached base packages:
[1] stats     graphics  grDevices utils     datasets  methods   base     

other attached packages:
[1] biglm_0.9-2.1 DBI_1.1.0     ffbase_0.12.8 ff_2.2-14.2   bit_1.1-15.2 

loaded via a namespace (and not attached):
[1] compiler_4.0.2  tools_4.0.2     fastmatch_1.1-0

[dutangc]

Dear all,
I have the same issue with the latest version of ffbase / biglm. Is there any solution?

> sessionInfo()
R version 4.1.1 (2021-08-10)
Platform: x86_64-apple-darwin17.0 (64-bit)
Running under: macOS Big Sur 11.2.2
Matrix products: default
LAPACK: /Library/Frameworks/R.framework/Versions/4.1/Resources/lib/libRlapack.dylib

locale:
[1] fr_FR.UTF-8/fr_FR.UTF-8/fr_FR.UTF-8/C/fr_FR.UTF-8/fr_FR.UTF-8

attached base packages:
[1] stats     graphics  grDevices utils     datasets 
[6] methods   base     

other attached packages:
[1] ffbase_0.13.3 biglm_0.9-2.1 DBI_1.1.1     ff_4.0.4     
[5] bit_4.0.4     versions_0.3 

loaded via a namespace (and not attached):
 [1] compiler_4.1.1    htmltools_0.5.1.1 tools_4.1.1      
 [4] yaml_2.2.1        fastmatch_1.1-0   rmarkdown_2.8    
 [7] knitr_1.36        digest_0.6.28     xfun_0.27        
[10] rlang_0.4.12      evaluate_0.14    ```