Reuse cached corpus snapshot in direct rust fallback

Author: ealdent

docs/modernization-handoff.md

@@ -96,6 +96,7 @@ Delivered:
 - Rust session execution refactor to shared session corpus storage + borrowed orchestration helpers, eliminating deep corpus clone overhead on each session EM run
 - unified Rust session orchestration API (`run_em_on_session`) that applies settings + runs EM in one call inside Rust session orchestration
 - `Lda::Backends::Rust` now routes session-path EM through managed Rust session orchestration (`run_em_on_session_with_corpus`), leaving session reuse/recovery decisions in Rust; when session orchestration is unavailable it still prefers direct Rust non-session orchestration (`run_em_with_start_seed`) before legacy Ruby-side beta-input fallback (`run_em`)
+- direct non-session Rust orchestration now reuses the backend's cached Rust corpus snapshot instead of rebuilding corpus arrays from `@corpus` on each fallback invocation
 - Rust managed-session orchestration API (`run_em_on_session_with_corpus`) added to recreate missing sessions and run EM in one Rust call
 - Rust session lifecycle replacement API (`replace_corpus_session`) added so corpus reassignment can update existing Rust sessions in place (config reset + corpus swap) instead of Ruby-side drop/recreate
 - `Lda::Backends::Rust` now keeps session-based orchestration on the managed Rust path (`run_em_on_session_with_corpus`) even when sessions are dropped externally

docs/porting-strategy.md

@@ -63,6 +63,7 @@ Completed in `codex/experiment-ruby3-modernization`:
 - Rust session orchestration now runs on shared Rust-side corpus session data via borrowed execution helpers, avoiding deep corpus array cloning on each session EM call.
 - Unified Rust session API added (`run_em_on_session`) to apply settings and execute EM in one call inside Rust session orchestration.
 - `Lda::Backends::Rust` now prefers direct Rust non-session orchestration (`run_em_with_start_seed`) before legacy `run_em(initial_beta, ...)` compatibility fallback when a session path is unavailable.
+- Direct non-session Rust orchestration now reuses the backend's cached Rust corpus snapshot instead of rebuilding corpus arrays from `@corpus` on each fallback invocation.
 - Rust managed-session orchestration API added (`run_em_on_session_with_corpus`) to recreate missing sessions and execute EM in one Rust call.
 - Rust session lifecycle replacement API added (`replace_corpus_session`) so corpus reassignment can update existing Rust sessions in place (config reset + corpus swap) instead of Ruby-side drop/recreate.
 - `Lda::Backends::Rust` now routes session-path EM through `run_em_on_session_with_corpus`, leaving session reuse/recovery decisions in Rust and reducing fallback to non-session orchestration when sessions are externally dropped.

docs/rust-orchestration-guardrails.md

@@ -15,6 +15,7 @@ Current parity expectations:
 - Session-based orchestration paths (`run_em_on_session`, `run_em_on_session_with_start_seed`, `run_em_on_session_start`, `run_em_on_session_with_corpus`) must match direct non-session orchestration for equivalent settings/seeds.
 - `Lda::Backends::Rust` session-path EM should prefer the managed Rust session entrypoint (`run_em_on_session_with_corpus`) rather than branching in Ruby between session-only and recovery paths.
 - `Lda::Backends::Rust` non-session fallback should prefer Rust start-aware orchestration (`run_em_with_start_seed`) before legacy beta-input orchestration (`run_em`).
+- Direct non-session fallback should reuse the backend's cached Rust corpus snapshot rather than rebuilding corpus arrays from `@corpus` for each invocation.
 - Rust backend corpus/session lifecycle must not leak session count across corpus replacement.
 - Missing-session recovery in managed session orchestration (`run_em_on_session_with_corpus`) must recreate a usable session and keep parity with direct orchestration.
 - Corpus reassignment through Rust session replacement lifecycle (`replace_corpus_session`) must preserve stable session count and route subsequent EM runs over updated corpus data.

lib/lda-ruby/backends/rust.rb

@@ -176,31 +176,32 @@ def rust_orchestrated_em_with_session(start)
       def rust_orchestrated_em_with_start_seed(start)
         return false unless defined?(::Lda::RustBackend)
         return false unless ::Lda::RustBackend.respond_to?(:run_em_with_start_seed)
+        return false unless ensure_rust_corpus_snapshot
 
-        em_input = rust_em_corpus_input
-        return false if em_input.nil?
+        topics = Integer(num_topics)
+        return false unless topics.positive?
 
         random_seed = Integer(next_random_seed)
         output = ::Lda::RustBackend.run_em_with_start_seed(
           start.to_s,
-          em_input.fetch(:document_words),
-          em_input.fetch(:document_counts),
-          Integer(em_input.fetch(:topics)),
-          Integer(em_input.fetch(:terms)),
+          @rust_document_words,
+          @rust_document_counts,
+          topics,
+          Integer(@rust_corpus_terms),
           Integer(max_iter),
           Float(convergence),
           Integer(em_max_iter),
           Float(em_convergence),
           Float(init_alpha),
-          Float(em_input.fetch(:min_probability)),
+          MIN_PROBABILITY,
           random_seed
         )
 
         return false unless valid_rust_em_output?(
           output,
-          em_input.fetch(:document_lengths),
-          em_input.fetch(:topics),
-          em_input.fetch(:terms)
+          @rust_document_lengths,
+          topics,
+          Integer(@rust_corpus_terms)
         )
 
         beta_probabilities, beta_log, gamma, phi = output
@@ -349,6 +350,12 @@ def register_rust_corpus_session(previous_session_id = nil)
       end
 
       def ensure_rust_corpus_session
+        ensure_rust_corpus_snapshot
+      rescue StandardError
+        false
+      end
+
+      def ensure_rust_corpus_snapshot
         has_session_data = @rust_corpus_terms && @rust_document_lengths && @rust_document_words && @rust_document_counts
         return true if has_session_data
 

test/rust_orchestration_test.rb

@@ -667,6 +667,28 @@ def test_rust_backend_non_session_fallback_prefers_run_em_with_start_seed
     backend&.corpus = nil
   end
 
+  def test_rust_backend_direct_non_session_path_reuses_cached_corpus_snapshot
+    backend = Lda::Backends::Rust.new(random_seed: 1234)
+    backend.corpus = Lda::TextCorpus.new(FIXTURE_DOCUMENTS)
+    backend.verbose = false
+    backend.num_topics = @topics
+    backend.max_iter = @max_iter
+    backend.convergence = @convergence
+    backend.em_max_iter = @em_max_iter
+    backend.em_convergence = @em_convergence
+    backend.init_alpha = @init_alpha
+
+    backend.define_singleton_method(:rust_orchestrated_em_with_session) { |_start| false }
+    backend.define_singleton_method(:rust_em_corpus_input) do
+      raise "direct non-session path should reuse cached corpus snapshot"
+    end
+
+    backend.em("random")
+    assert_equal @topics, backend.gamma.first.size
+  ensure
+    backend&.corpus = nil
+  end
+
   def test_rust_backend_session_path_prefers_managed_session_entrypoint
     backend = nil
     rust_singleton = nil