When passed a tibble(), ds() always reports Warning: Unknown or uninitialised column: `distbegin`

[jjrob]

Unless you are doing a binned analysis, you should not include distbegin and distend columns in your data frame passed to ds(). But when you don't, ds() always reports warnings about those columns being missing. Here's an example:

> head(obs1)
# A tibble: 6 x 4
  object Sample.Label  size distance
   <int> <chr>        <dbl>    <dbl>
1 276831 16936673      1        368.
2 276844 16936692      2       3089.
3 276846 16936693      4.67     635.
4 276832 16936673      3        583.
5 276895 16936828      1       3640.
6 276847 16936694      2       2333.
> summary(obs1)
     object       Sample.Label            size          distance     
 Min.   :268824   Length:329         Min.   : 1.00   Min.   :   0.0  
 1st Qu.:271124   Class :character   1st Qu.: 1.00   1st Qu.: 574.2  
 Median :273335   Mode  :character   Median : 1.25   Median :1499.4  
 Mean   :273629                      Mean   : 2.14   Mean   :1980.6  
 3rd Qu.:276267                      3rd Qu.: 3.00   3rd Qu.:3072.5  
 Max.   :293257                      Max.   :15.00   Max.   :8230.6  
> ds(obs, truncation = 6000, key = "hr", adjustment = NULL)
Warning: Unknown or uninitialised column: `distbegin`.
Warning: Unknown or uninitialised column: `distend`.
Warning: Unknown or uninitialised column: `distbegin`.
Fitting hazard-rate key function
AIC= 5400.108
No survey area information supplied, only estimating detection function.

Distance sampling analysis object

Detection function:
 Hazard-rate key function 

Estimated abundance in covered region: 836.4272 

This problem occurs in the latest version of distance at the time of this writing, version 2.0.1. My guess is that it's caused by validation code in checkdata.R, such as this code that occurs at line 29:

  if(any(!is.null(data$distbegin), !is.null(data$distend)) && !all(!is.null(data$distbegin), !is.null(data$distend))){
    stop("You have provided either a 'distbegin' or 'distend' column in your dataset but not both. Please provide both or remove these and provide a distance column and use the cutpoint argument.", call. = FALSE)
  }

Checks like !is.null(data$distbegin) will report the warning if data does not contain a distbegin column. Here is a demonstration, in which I check first my obs1 data frame above for an existing column, distance, and then check it for an non-existing column foobar:

> is.null(obs1$distance)
[1] FALSE
> is.null(obs1$foobar)
Warning: Unknown or uninitialised column: `foobar`.
[1] TRUE

A better way to check for a missing column might be the base R hasName() function:

> !hasName(obs1, "foobar")
[1] TRUE

Am I missing something? If my understanding is correct, then this problem is affecting the entire Distance user community (except users performing binned analyses). It has come up recently on the email list, e.g. here, but was not diagnosed as a bug at that time.

[erex]

Good morning Jason. Thanks for the report and a suggested repair.

My brain is not fully functioning just yet, so this will require a bit more investigation. The discussion on the list you reference at the end of your message dates from November of 2024, but version 2.0.1 of the Distance package was released eight months later (July '25).

Virtually all of the analyses we describe in our workshop materials uses exact, rather than binned data. I would have expected to have encountered these warning messages prior to your report, given I conducted a workshop last month.

To double check, I rolled out the good old ducknest data set:

> library(Distance, quietly = TRUE)
> data(ducknest)
> names(ducknest)
[1] "Region.Label" "Area"         "Sample.Label" "Effort"      
[5] "object"       "distance"     "Study.Area"  
> cu <- convert_units("meter", "kilometer", "square kilometer")
> simple <- ds(ducknest, key = "hr", adjustment = NULL, convert_units=cu)
Fitting hazard-rate key function
AIC= 929.793
> sessionInfo("Distance")$otherPkgs$Distance$Version
[1] "2.0.1"

ducknest contains exact distances, and the call to ds() fails to generate the warning messages from your example.

As I say, this requires further digging.

[jjrob]

Thanks for your quick reply @erex. I can't dig much deeper for another week, but now suspect that in my case this is a tibble vs. data.frame difference, which would explain why you haven't see it or heard more about it (assuming most people are using data.frame).

Here's code showing what I mean:

> class(obs)
[1] "tbl_df"     "tbl"        "data.frame"
> . <- ds(obs, truncation = trunc_dist, key = "hr", adjustment = NULL)
Warning: Unknown or uninitialised column: `distbegin`.
Warning: Unknown or uninitialised column: `distend`.
Warning: Unknown or uninitialised column: `distbegin`.
Fitting hazard-rate key function
AIC= 5400.108
No survey area information supplied, only estimating detection function.
> obs2 <- as.data.frame(obs)
> class(obs2)
[1] "data.frame"
> . <- ds(obs2, truncation = trunc_dist, key = "hr", adjustment = NULL)
Fitting hazard-rate key function
AIC= 5400.108
No survey area information supplied, only estimating detection function.

If that's indeed the case, I would respectfully suggest you fix it on your side, unless you can convince the tidyverse folks that tibble needs to behave exactly like data.frame in this case. If they refuse to fix it and you guys are too busy, I can prepare a PR if you want (although it might take some time as I'm also busy).

[jjrob]

I edited the title to reflect this theory.

[erex]

Yes, I wondered whether the tibble was causing the issue, so I built this

> library(Distance, quietly = TRUE)
> data(ducknest)
> names(ducknest)
[1] "Region.Label" "Area"         "Sample.Label" "Effort"      
[5] "object"       "distance"     "Study.Area"  
> cu <- convert_units("meter", "kilometer", "square kilometer")
> simple <- ds(ducknest, key = "hr", adjustment = NULL)
Fitting hazard-rate key function
AIC= 929.793
> sessionInfo("Distance")$otherPkgs$Distance$Version
[1] "2.0.1"
> library(tidyverse, quietly = TRUE)
> tibbleduck <- as_tibble(ducknest)
> class(tibbleduck)
[1] "tbl_df"     "tbl"        "data.frame"
> tibbleds <- ds(tibbleduck, key = "hr", adjustment = NULL)
Fitting hazard-rate key function
AIC= 929.793

It doesn't appear that the tibble nature of the object is the problem.

You have indeed come across something we should fix, but we've not yet gotten to the cause of the issue. We'll keep it as an open issue.

[erex]

Final experiment:

convert ducknest to a tibble and remove fields such that only object, Sample.Label, size and distance remain. Running this through ds() still fails to generate warning message reported in original post.

data(ducknest)
names(ducknest)
cu <- convert_units("meter", "kilometer", "square kilometer")
simple <- ds(ducknest, key = "hr", adjustment = NULL, convert_units = cu)
sessionInfo("Distance")$otherPkgs$Distance$Version
library(tidyverse, quietly = TRUE)
tibbleduck <- as_tibble(ducknest)
class(tibbleduck)
tibbleds <- ds(tibbleduck, key = "hr", adjustment = NULL, convert_units = cu)
names(tibbleduck)
# remove fields and manufacture size, such that same fields are present as original post
tibbleduck$Region.Label <- NULL
tibbleduck$Effort <- NULL
tibbleduck$Area <- NULL
tibbleduck$Study.Area <- NULL
tibbleduck$size <- rpois(534, lambda=7)
tibbleds <- ds(tibbleduck, key = "hr", adjustment = NULL, convert_units = cu)

Original post shows structure of object obs1, but argument data= in call to ds() is obs. We don't see the content of obs, can you provide?

[lenthomas]

I'm happy to try to help with a fix, once we've identified exactly what the problem is!

[jjrob]

Thanks to both of you for trying to help!

Apologies for confusing you with the obs1 and obs in the original example. That is not the issue (but I know you want proof).

Before posting obs here, I want to try to explore the problem more and see if I can develop a repro for you with ducknest or otherwise diagnose the issue better. The problem does not repro on my side with a sterile R environment, just loading Distance and tidyverse, with either obs or ducknest. So I doubt it will repro with obs for you, and because I don't own the data in obs I'm reluctant to post it here where it will be freely available until the end of time.

I'm about to travel to a workshop that Len is also attending, and have to focus on preparation for that. We can potentially look at it together there. Also, I'll have more time to develop a better repro when I return next week.