Error in `computePathways` v3.1.2 on Spark/DatabaseConnector

[ob325]

Describe the bug
computePathways in new version v3.1.2 throws an error, but same code runs w/o error in 3.1.1.

Error occurs at this function call (new code as of 3.1.2):

TreatmentPatterns/R/computePathways.R

Line 149 in fb4437f

cdm[[cohortTableName]] <- dplyr::tbl(src = con, cohortTableName)

Error is:

Error in `db_query_fields.DBIConnection()`:
! Can't query fields.
ℹ Using SQL: SELECT * FROM `[cohort table name without schema]` `q01` WHERE (0 = 1)
Caused by error in `value[[3L]]()`:
! java.sql.SQLException: [Databricks][JDBCDriver](500051) ERROR processing query/statement. Error Code: 0, SQL state: 42P01, Query: SELECT *
F***, Error message from Server: org.apache.hive.service.cli.HiveSQLException: Error running query: [TABLE_OR_VIEW_NOT_FOUND] org.apache.spark.sql.catalyst.ExtendedAnalysisException: [TABLE_OR_VIEW_NOT_FOUND] The table or view `[cohort table name without schema]` cannot be found. Verify the spelling and correctness of the schema and catalog.

To Reproduce

Sort-of reprex below. Db schemas and table names censored (square braces)

library(dplyr)
library(DatabaseConnector)
library(TreatmentPatterns)

connectionDetails <- DatabaseConnector::createConnectionDetails(
  dbms = "spark",
  [...] 
)

cohorts <- tibble(
  cohortId = c(23015, 23150, 23151),
  cohortName = c('target1', 'event1', 'event2'),
  type = c('target', 'event', 'event')
)

# Confirm the cohort table does exist
tbl(connect(connectionDetails),
    I("[resultSchema.cohortTableName]")) %>%
  ncol()
# returns [1] 4

computePathways(
  description = 'test1',
  cohorts = cohorts,
  cohortTableName = "[cohortTableName]",
  connectionDetails = connectionDetails,
  cdmSchema = "[catalog.schema]",
  resultSchema = "[user writeable catalog.schema]",
  tempEmulationSchema = "[user writeable catalog.schema]",
  minEraDuration = 1,
  eraCollapseSize = 0,
  filterTreatments = "All",
  minPostCombinationDuration = 1,
  combinationWindow = 1,
  maxPathLength = 100
)

Expected behavior
computePathways return value w/o error

Screenshots
If applicable, add screenshots to help explain your problem.

Additional context

R version 4.5.1 (2025-06-13 ucrt)
Platform: x86_64-w64-mingw32/x64
Running under: Windows 11 x64 (build 26100)

Matrix products: default
  LAPACK version 3.12.1

locale:
[1] LC_COLLATE=English_United States.utf8  LC_CTYPE=English_United States.utf8    LC_MONETARY=English_United States.utf8 LC_NUMERIC=C                           LC_TIME=English_United States.utf8    

time zone: America/New_York
tzcode source: internal

attached base packages:
[1] stats     graphics  grDevices utils     datasets  methods   base     

other attached packages:
[1] TreatmentPatterns_3.1.2 DatabaseConnector_7.0.0 dplyr_1.1.4            

loaded via a namespace (and not attached):
 [1] bit_4.6.0          jsonlite_2.0.0     compiler_4.5.1     tidyselect_1.2.1   Rcpp_1.1.0         Andromeda_1.1.1    blob_1.2.4         stringr_1.5.2      urltools_1.7.3.1   tidyr_1.3.1        triebeard_0.4.1   
[12] R6_2.6.1           generics_0.1.4     backports_1.5.0    checkmate_2.3.3    tibble_3.3.0       DBI_1.2.3          pillar_1.11.1      rlang_1.1.6        stringi_1.8.7      CDMConnector_2.3.0 omopgenerics_1.3.6
[23] bit64_4.6.0-1      pkgload_1.4.1      cli_3.6.5          withr_3.0.2        magrittr_2.0.4     ROhdsiWebApi_1.3.3 rstudioapi_0.17.1  rJava_1.0-11       dbplyr_2.5.1       lifecycle_1.0.4    vctrs_0.6.5       
[34] glue_1.8.0         SqlRender_1.19.4   duckdb_1.4.1       purrr_1.1.0        tools_4.5.1        pkgconfig_2.0.3   

[anthonysena]

@ob325 - thanks for posting this issue and all of the details. I was just reviewing this as well and noticed that you are using DatabaseConnector v7.0.0 - could you try upgrading to DatabaseConnector v7.1.0 since the release notes mentions a fix:

Fixed getTableNames() for DataBricks when quoting (using backticks) the database and/or schema name.

It may be that TreatmentPatterns should require DatabaseConnector >= 7.1 in the DESCRIPTION.

[mvankessel-EMC]

@ob325 We noticed the same, using ODBC however. @ablack3 suggested setting useNativeQuery = FALSE. I assume you can do something similar for JDBC.