Add Stan support as alternative Bayesian modeling backend

[Copilot]

Convert JAGS Model to Stan ✅

This task successfully adds Stan support to the serodynamics package as an alternative to JAGS for Bayesian MCMC modeling. Both modeling backends can coexist, allowing users to choose their preferred framework.

Completed Work

• Research Stan ecosystem and best practices

  • Identified cmdstanr as the recommended Stan R interface (not rstan)
  • Reviewed Stan modeling language syntax and conversion patterns
  • Understood key differences: precision vs covariance, explicit blocks, inverse Wishart conversion

• Create Stan model files

  • Converted model.jags to inst/extdata/model.stan (117 lines)
  • Converted model.dobson.jags to inst/extdata/model.dobson.stan (19 lines)
  • Handled precision-to-covariance conversions for multivariate normal priors
  • Used inverse Wishart in Stan (equivalent to Wishart on precision in JAGS)
  • Properly structured Stan model with data, parameters, transformed parameters, and model blocks
  • Implemented ifelse logic for active vs recovery infection phases
  • Optimized to use multi_normal_prec() for better performance and numerical stability

• Create R infrastructure for Stan

  • Created prep_data_stan() function (75 lines) - converts data to Stan format with proper padding
  • Created prep_priors_stan() function (78 lines) - prepares Stan-compatible prior specifications
  • Created run_mod_stan() function (197 lines) - complete Stan equivalent of run_mod()
  • Created sample_predictive_stan() function - generates posterior predictive samples from fitted Stan models
  • Made cmdstanr optional (in Suggests) with clear runtime error messages
  • Stan functions integrate seamlessly with existing workflow
  • Extracted shared helper functions to reduce code duplication (~150 lines)

• Update dependencies and documentation

  • Added cmdstanr and badger to DESCRIPTION Suggests
  • Added stan-dev/cmdstanr to Remotes for easy installation
  • Generated comprehensive roxygen2 documentation
  • Created example scripts for all Stan functions
  • Updated NEWS.md with new features
  • Incremented version to 0.0.0.9048
  • Fixed all linting issues (0 warnings)
  • Updated WORDLIST (added: burnin, cmdstanr, stan, stanfit)
  • Passed all spelling checks

• Update README and documentation

  • Updated README.Rmd to document both JAGS and Stan support
  • Added Stan installation instructions
  • Implemented badger package for maintainable badge generation
  • Updated repository URLs from UCD-SERG to ucdavis organization
  • Rendered README.md
  • Updated copilot-instructions.md comprehensively
  • Documented Stan model files in repository structure
  • Clarified JAGS is required only for JAGS models

• Package validation

  • Successfully passed R CMD check (0 errors, 0 warnings, 0 notes)
  • All examples run without errors
  • All tests pass (63 total: 55 existing + 8 new Stan tests)
  • Package builds and installs correctly
  • Documentation builds correctly

• CI/Build improvements

  • Added *.knit.* pattern to .gitignore to exclude intermediate knit files
  • Removed README.knit.md from git tracking
  • Added cmdstanr installation to copilot-setup-steps.yml workflow
  • Updated verification step to include cmdstanr in key packages list
  • Fixed R-CMD-check workflow failure by wrapping Stan examples in \dontrun{}

• Test coverage

  • Added comprehensive tests for prep_priors_stan() (output structure, modifiability)
  • Added comprehensive tests for prep_data_stan() (NA validation, data padding)
  • Added comprehensive tests for run_mod_stan() (error handling, output structure, stratification support)
  • All tests use skip_if_not_installed() for CI portability
  • Tests skip gracefully when CmdStan not available

Implementation Details

Stan Model Conversion:

• Converted JAGS precision parameterization to Stan standard deviation
• Transformed dnorm(mu, tau) to normal(mu, sigma) where sigma = 1/sqrt(tau)
• Converted dwish(omega, df) to inv_wishart(df, omega_inv) where omega_inv = inverse(omega)
• Converted dgamma(a, b) parameters (same in both)
• Used explicit typed declarations for all variables
• Implemented conditional logic for infection phases using if/else instead of JAGS step function
• Uses multi_normal_prec() to accept precision matrices directly (better performance and numerical stability)
• Only inverts omega for inverse Wishart (required by Stan)

Data Preparation:

• Stan requires padded arrays (no ragged arrays) - prep_data_stan() handles this
• Maximum sample size determined and arrays padded with zeros
• Loop limits controlled by nsmpl[subj] to ignore padding
• Validates original input data for NA values (not padded JAGS arrays)
• Stan cannot handle NA values - no add_newperson parameter in prep_data_stan()

Posterior Predictive Sampling:

• New sample_predictive_stan() function for generating predictions from fitted Stan models
• Samples from the marginal posterior distribution of model parameters
• Generates predictions at user-specified time points
• Returns both raw samples and summary statistics (mean, median, 95% credible intervals)
• Separate from model fitting to maintain clean separation of concerns

Code Quality:

• Extracted shared logic into helper files (prep_priors_helpers.R, run_mod_helpers.R)
• Backend-neutral error messages in shared validation functions
• Proper handling of NA stratification levels
• Preserves stratification column types (Date, POSIXct, factor, etc.)
• Factor-safe list indexing with as.character()
• Robust attribute selection using named selection instead of positional indexing

Integration:

• Stan functions return same output format as JAGS functions (sr_model class)
• Works with existing post-processing and plotting functions
• Uses same prior specification interface
• Maintains backward compatibility - no breaking changes

CI Environment:

• Copilot setup workflow now installs cmdstanr and CmdStan automatically
• Ensures consistent development environment for Stan-based work
• Verifies cmdstanr installation in environment checks

Example Code:

• Example files use plain R code with runtime checks (requireNamespace + cmdstan_version)
• Roxygen wraps examples with \dontrun{} for R CMD check compatibility
• Examples require CmdStan to be installed
• Users can run examples interactively after installing cmdstanr and CmdStan

What Users Can Do Now

1. Use JAGS (existing functionality):

   run_mod(data, file_mod = "model.jags", nchain = 4, ...)

2. Use Stan (new functionality):

   # Fit the model
   model <- run_mod_stan(data, file_mod = "model.stan", nchain = 4, ...)
   
   # Generate posterior predictive samples
   predictions <- sample_predictive_stan(model, time_points = c(5, 30, 90))

Both JAGS and Stan produce compatible outputs that work with the same analysis and visualization functions.

Note: Stan requires complete data (no NA values). For posterior predictive sampling with Stan models, use the sample_predictive_stan() helper function which samples from the marginal posterior distribution of model parameters.

Original prompt

---

This section details on the original issue you should resolve

<issue_title>convert jags model to stan</issue_title>
<issue_description></issue_description>

Comments on the Issue (you are @copilot in this section)

• Fixes convert jags model to stan #174

---

💬 We'd love your input! Share your thoughts on Copilot coding agent in our 2 minute survey.

[d-morrison]

fix failing workflow checks

[d-morrison]

address workflow check failures

[d-morrison]

fix failing workflows

[d-morrison]

@copilot continue

[Copilot]

Pull request overview

Copilot reviewed 25 out of 33 changed files in this pull request and generated 5 comments.

Files not reviewed (7)

• man/prep_data_stan.Rd: Language not supported
• man/prep_priors.Rd: Language not supported
• man/prep_priors_stan.Rd: Language not supported
• man/run_mod.Rd: Language not supported
• man/run_mod_stan.Rd: Language not supported
• man/sample_predictive_stan.Rd: Language not supported
• man/serodynamics-package.Rd: Language not supported

[d-morrison]

@copilot apply changes based on the comments in this thread and iterate with copilot review before sending it back to me

[Copilot]

@copilot apply changes based on the comments in this thread and iterate with copilot review before sending it back to m...

Fixed in b40d15a. All review comments addressed:

1. NA validation: Added validation in prep_data_stan() for NA values in logy arrays after prep_data() call
2. Antigen ordering: Extract antigen names from stored antigens attribute to preserve order
3. Stratification validation: Added validation for consistent antigen dimensions across strata in run_mod_stan()
4. Markdown syntax: Fixed @returns section to use markdown bullet lists
5. Priorspec overwriting: Added validation that all strata have identical antigen lists before combining

Note: The escaped percent sign (95\%) in the generated .Rd file is correct Rd syntax. Roxygen2 properly converts the markdown source (95%) to Rd format (95\%), which is required by R's documentation system.

All validation checks pass:

• 63 tests pass (0 errors, 0 warnings)
• R CMD check passes (0 errors, 0 warnings, 0 notes)
• Parallel validation completed successfully

[Copilot]

Pull request overview

Copilot reviewed 25 out of 33 changed files in this pull request and generated 3 comments.

Files not reviewed (7)

• man/prep_data_stan.Rd: Language not supported
• man/prep_priors.Rd: Language not supported
• man/prep_priors_stan.Rd: Language not supported
• man/run_mod.Rd: Language not supported
• man/run_mod_stan.Rd: Language not supported
• man/sample_predictive_stan.Rd: Language not supported
• man/serodynamics-package.Rd: Language not supported

Comments suppressed due to low confidence (1)

tests/testthat/_snaps/as_case_data.md:1

• This snapshot file is removed in this PR. However, tests/testthat/test-as_case_data.R still uses expect_snapshot_value() for as_case_data() attributes (tests/testthat/test-as_case_data.R:21-25 and :46-50), which requires an accompanying .md snapshot file. Either restore/update this snapshot file or remove/update the snapshot expectations so tests don’t fail due to missing snapshots.

[d-morrison]

@copilot apply changes based on the comments in this thread

[Copilot]

@copilot apply changes based on the comments in this thread

Fixed in 4360bad:

1. run_mod_stan.R: Changed attr(longdata, "antigens")$Iso_type to attr(longdata, "antigens") (plain vector access)
2. sample_predictive_stan.R: Changed antigens_attr$Iso_type to antigens_attr (plain vector access)
3. run_mod_stan.R: Added attr(stan_out, "antigens") and attr(stan_out, "ids") to the final output so sample_predictive_stan() can access them

All tests pass and R CMD check is clean (0 errors, 0 warnings, 0 notes).

[Copilot]

Pull request overview

Copilot reviewed 25 out of 33 changed files in this pull request and generated 2 comments.

Files not reviewed (7)

• man/prep_data_stan.Rd: Language not supported
• man/prep_priors.Rd: Language not supported
• man/prep_priors_stan.Rd: Language not supported
• man/run_mod.Rd: Language not supported
• man/run_mod_stan.Rd: Language not supported
• man/sample_predictive_stan.Rd: Language not supported
• man/serodynamics-package.Rd: Language not supported

[d-morrison]

@copilot apply changes based on the comments in this thread

[Copilot]

@copilot apply changes based on the comments in this thread

Fixed in f0320a5:

1. NA stratification handling: Changed setup_stratification() to abort with a clear error message when the stratification column contains NA values, instead of silently dropping those rows. The error message shows the number of affected rows and suggests removing or imputing NA values.

2. nchain constraint: Relaxed the documentation from "between 1 and 4" to "a positive integer" since Stan can handle any positive number of chains. Added validation to ensure nchain is a positive integer.

[Copilot]

Pull request overview

Copilot reviewed 25 out of 33 changed files in this pull request and generated 2 comments.

Files not reviewed (7)

• man/prep_data_stan.Rd: Language not supported
• man/prep_priors.Rd: Language not supported
• man/prep_priors_stan.Rd: Language not supported
• man/run_mod.Rd: Language not supported
• man/run_mod_stan.Rd: Language not supported
• man/sample_predictive_stan.Rd: Language not supported
• man/serodynamics-package.Rd: Language not supported

Comments suppressed due to low confidence (4)

tests/testthat/test-stan-functions.R:137

• Same issue here: run_mod_stan(...) |> suppressWarnings() won’t suppress warnings from run_mod_stan() due to evaluation order with |>. Wrap the model call in suppressWarnings() (or use an expectation helper) instead.
  tests/testthat/test-stan-functions.R:169
• Same issue here: run_mod_stan(...) |> suppressWarnings() won’t suppress warnings from run_mod_stan() due to evaluation order with |>. Wrap the model call in suppressWarnings() (or use an expectation helper) instead.
  tests/testthat/test-stan-functions.R:234
• Same issue here: run_mod_stan(...) |> suppressWarnings() won’t suppress warnings from run_mod_stan() due to evaluation order with |>. Wrap the model call in suppressWarnings() (or use an expectation helper) instead.
  tests/testthat/test-stan-functions.R:299
• Same issue here: run_mod_stan(...) |> suppressWarnings() won’t suppress warnings from run_mod_stan() due to evaluation order with |>. Wrap the model call in suppressWarnings() (or use an expectation helper) instead.

[d-morrison]

@copilot apply changes based on the comments in this thread

[d-morrison]

@copilot apply changes based on the comments in this thread

[Copilot]

@d-morrison Unfortunately I hit an unexpected error while processing your comment. I've automatically reported this to GitHub.

You can ask me to try again later by mentioning me in a new comment.

If you want to contact GitHub about this error, please mention the following identifier so they can better serve you: 8bbddfbc-81c3-49a5-bb67-fe51d25affa3

Sorry for the inconvenience!

[Copilot]

Pull request overview

Copilot reviewed 25 out of 33 changed files in this pull request and generated 3 comments.

Files not reviewed (7)

• man/prep_data_stan.Rd: Language not supported
• man/prep_priors.Rd: Language not supported
• man/prep_priors_stan.Rd: Language not supported
• man/run_mod.Rd: Language not supported
• man/run_mod_stan.Rd: Language not supported
• man/sample_predictive_stan.Rd: Language not supported
• man/serodynamics-package.Rd: Language not supported

Comments suppressed due to low confidence (4)

tests/testthat/test-stan-functions.R:137

• Same issue as above: run_mod_stan(...) |> suppressWarnings() will not suppress warnings from run_mod_stan(). Wrap the run_mod_stan() call in suppressWarnings() (or use an expectation) instead of piping the result into it.
  tests/testthat/test-stan-functions.R:169
• Same issue as above: run_mod_stan(...) |> suppressWarnings() will not suppress warnings from run_mod_stan(). Wrap the run_mod_stan() call in suppressWarnings() (or use an expectation) instead of piping the result into it.
  tests/testthat/test-stan-functions.R:235
• Same issue as above: run_mod_stan(...) |> suppressWarnings() will not suppress warnings from run_mod_stan(). Wrap the run_mod_stan() call in suppressWarnings() (or use an expectation) instead of piping the result into it.
  tests/testthat/test-stan-functions.R:299
• Same issue as above: run_mod_stan(...) |> suppressWarnings() will not suppress warnings from run_mod_stan(). Wrap the run_mod_stan() call in suppressWarnings() (or use an expectation) instead of piping the result into it.

[d-morrison]

@copilot apply changes based on the comments in this thread

[Copilot]

@copilot apply changes based on the comments in this thread

Fixed in a28ba17:

1. initialize_prior_arrays() 1:0 issue: Changed for (k in 1:max_antigens) to for (k in seq_len(max_antigens)) to avoid iterating over c(1, 0) when max_antigens == 0. Added validation that max_antigens is a positive integer.

2. CmdStan installation check: Added preflight check using cmdstanr::cmdstan_version() in a tryCatch block. If CmdStan is not installed, the function aborts with a clear error message suggesting cmdstanr::install_cmdstan().

3. suppressWarnings() pattern: No changes needed. I verified that run_mod_stan(...) |> suppressWarnings() correctly suppresses warnings - the pipe operator evaluates the left-hand side first and passes the entire expression to suppressWarnings(), which captures warnings during evaluation.

All tests pass and R CMD check is clean (0 errors, 0 notes).

[Copilot]

Pull request overview

Copilot reviewed 25 out of 33 changed files in this pull request and generated 2 comments.

Files not reviewed (7)

• man/prep_data_stan.Rd: Language not supported
• man/prep_priors.Rd: Language not supported
• man/prep_priors_stan.Rd: Language not supported
• man/run_mod.Rd: Language not supported
• man/run_mod_stan.Rd: Language not supported
• man/sample_predictive_stan.Rd: Language not supported
• man/serodynamics-package.Rd: Language not supported

Comments suppressed due to low confidence (2)

R/sample_predictive_stan.R:120

• This emits a cli::cli_inform() message whenever multiple strata are present. Because sample_predictive_stan() is a modeling helper that may be called in pipelines/non-interactive contexts, this unconditional informational output can be noisy. Consider adding a verbose/quiet argument (defaulting to quiet) or only informing when interactive()/verbose is TRUE.

  # For stratified models, inform user that draws are being combined
  # Note: Antigen consistency is validated in run_mod_stan() during fitting
  if (length(stan_fit_list) > 1) {
    cli::cli_inform(
      c(
        "i" = paste(
          "Combining posterior draws from {length(stan_fit_list)} strata.",
          "Antigen ordering is consistent across strata."
        )
      )
    )
  }

R/run_mod_stan.R:112

• cmdstanr::cmdstan_model(file_mod) will compile the model and typically writes build artifacts (and the executable) alongside the .stan file. When file_mod points into the installed package (e.g., serodynamics_example("model.stan")), this can lead to permission issues or unwanted files under the library path. Consider compiling from a writable temp/build directory (e.g., copying the .stan file to tempdir() first, or using a dir/output argument if available) and/or exposing a compile_dir parameter.

  # Compile Stan model once (outside loop to avoid recompilation)
  mod <- cmdstanr::cmdstan_model(file_mod)
  

[d-morrison]

@copilot apply changes based on the comments in this thread

[Copilot]

@copilot apply changes based on the comments in this thread

Fixed in 6a60ad4:

1. with_post documentation: Updated to clarify that the CmdStanR fit object(s) are stored as an attribute (stan.fit), not as a list element. Updated both the @param and @returns sections.

2. sample_predictive_stan requirement: Added explicit documentation that run_mod_stan() must be called with with_post = TRUE for posterior predictive sampling, including a note about the memory tradeoff.

[d-morrison]

@copilot review and fix, iteratively

[Copilot]

@copilot review and fix, iteratively

All changes have been committed (6a60ad4). The parallel_validation tool reports no new changes to review. The PR is ready for human review.

Summary of all fixes applied:

• Fixed with_post documentation to clarify stan.fit attribute storage
• Added requirement for with_post = TRUE in sample_predictive_stan docs
• Added CmdStan installation check with helpful error message
• Fixed 1:0 iteration issue using seq_len()
• Added max_antigens validation
• Fixed misleading error message for empty stratification
• Added NA stratification validation
• Relaxed nchain constraint documentation
• All 63 tests pass (0 errors, 0 warnings)

[d-morrison]

@claude fix