test(llmrails): characterize public behavior (1/9)

[Pouyanpi]

Description

Adds characterization tests for the existing public LLMRails compatibility surface before implementation is moved.

Why

The stack needs a stable, reviewable contract for constructor visibility, update_llm, sync wrappers, and serialization behavior.

What Changed

• Adds public behavior coverage for LLMRails.
• Makes no production-code changes.

Review Guidance

The recorded compatibility baseline is maintained in #2122. Use the LLMRails decomposition review matrix to map each behavior to its exact test node. Merge #2122 first, then rebase this stack onto the updated develop branch.

Review whether these tests faithfully capture current behavior without over-specifying the later implementation.

This nine-part stack turns LLMRails into a public compatibility shell while moving owned behavior into focused modules. Review each PR only against its parent. Parts 2-6 intentionally stage behavior-preserving extraction; part 7 consolidates shared type contracts and logger naming; part 8 establishes the final package ownership; part 9 contains the explicit semantic follow-ups.

For parts 2-6, please distinguish stack-local correctness or compatibility problems from naming and placement concerns already resolved in parts 7-8. If an intermediate name or location is correct and mergeable at that point, prefer reviewing its final form in the resolving PR instead of requesting duplicate churn. Public API changes, behavior changes, import cycles, or an independently broken intermediate branch remain in scope wherever they appear. Much of the apparent added LOC is moved code plus focused tests; the corresponding implementation is removed or delegated from llmrails.py in the same PR.

Stack Position

Part 1 of 9.

• Previous: none, based on develop
• Next: #2124

Order	PR	Branch	Base
1	#2123	restack/llmrails-01-public-contract-tests	develop
2	#2124	restack/llmrails-02-startup	restack/llmrails-01-public-contract-tests
3	#2125	restack/llmrails-03-runtime-conversation	restack/llmrails-02-startup
4	#2126	restack/llmrails-04-generation	restack/llmrails-03-runtime-conversation
5	#2127	restack/llmrails-05-streaming	restack/llmrails-04-generation
6	#2128	restack/llmrails-06-checks-final	restack/llmrails-05-streaming
7	#2129	restack/llmrails-07-consistency	restack/llmrails-06-checks-final
8	#2130	restack/llmrails-08-boundary-ownership	restack/llmrails-07-consistency
9	#2131	restack/llmrails-09-followups	restack/llmrails-08-boundary-ownership

Verification

make test
poetry run pre-commit run --all-files

Rebase checkpoints: stack 2: 5202 passed, 178 skipped; stack 4: 5263 passed, 178 skipped; stack 5: 5284 passed, 178 skipped; final stack: 5320 passed, 178 skipped. Final pre-commit: all hooks passed. Base: develop at 3fc828b7f.

AI Assistance

• No AI tools were used.
• AI tools were used; a human reviewed and can explain every change (tool: Codex).

Checklist

• I've read the CONTRIBUTING guidelines.
• This PR links to a triaged issue assigned to me.
• My PR title follows the project commit convention.
• I've updated the documentation if applicable.
• I've added tests if applicable.
• I've noted any verification beyond CI and any checks I couldn't run.
• I did not update generated changelog files manually.
• I addressed all CodeRabbit, Greptile, and other review comments, or replied with why no change is needed.
• @mentions of the person or team responsible for reviewing proposed changes.

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.

📢 Thoughts on this report? Let us know!

[coderabbitai]

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: ca5279b5-35ce-4660-8015-d7ecd8c6872e

📥 Commits

Reviewing files that changed from the base of the PR and between 3fc828b and 7679002.

📒 Files selected for processing (1)

• tests/rails/llm/test_llmrails_public_contract.py

👮 Files not reviewed due to content moderation or server errors (1)

• tests/rails/llm/test_llmrails_public_contract.py

---

📝 Walkthrough

[!WARNING]

Walkthrough skipped

File diffs could not be summarized.

🚥 Pre-merge checks | ❌ 1

❌ Failed checks (1 inconclusive)

Check name	Status	Explanation	Resolution
Test Results For Major Changes	❓ Inconclusive	Custom check execution failed before a final verdict was produced.	Retry the review run. If this persists, inspect pre-merge custom-check logs for infrastructure or agent runtime failures.

✨ Finishing Touches

📝 Generate docstrings

• Create stacked PR
• Commit on current branch

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Commit unit tests in branch restack/llmrails-01-public-contract-tests

---

_{Comment @coderabbitai help to get the list of available commands.}

[greptile-apps]

Greptile Summary

This PR adds a new characterization test file (test_llmrails_public_contract.py) covering four aspects of the public LLMRails surface: constructor attribute visibility, update_llm sync semantics, sync-in-async-loop guards, and __getstate__ serialization — with no production code changes.

• Constructor & update_llm tests verify that rails.config, rails.llm, rails.runtime, rails.events_history_cache, and the three update_llm side-effects (self.llm, _llm_generation_actions.llm, runtime.registered_action_params[\"llm\"]) are jointly consistent after mutation.
• test_sync_wrappers_raise_when_called_from_async_loop pins each sync wrapper's RuntimeError message, including process_events's known erroneous message (which reads generate_events instead of process_events).
• test_getstate_serializes_config_only confirms the __getstate__ snapshot contains exactly {\"config\": rails.config}, ruling out leakage of events_history_cache or registered action params.

Confidence Score: 5/5

Safe to merge — adds tests only, no production code is touched.

The change is a single new test file with no production code modifications. All four test cases faithfully capture the current LLMRails behavior, including two known pre-existing issues (the copy-paste error message in process_events and deprecated-property access without warning guards) that were already flagged in the prior review thread. The tests will fail in the expected direction as later PRs in the stack fix those issues, which is the intended role of characterization tests.

No files require special attention; the only changed file is the new test module.

Important Files Changed

Filename	Overview
tests/rails/llm/test_llmrails_public_contract.py	New characterization test suite for LLMRails public surface; faithfully captures current behavior including a known copy-paste bug in the process_events error message, and accesses two deprecated properties (llm_generation_actions, explain_info) without pytest.warns guards — both already noted in the prior review thread.

Sequence Diagram

%%{init: {'theme': 'neutral'}}%%
sequenceDiagram
    participant Test
    participant LLMRails
    participant Runtime
    participant LLMGenActions as _llm_generation_actions

    Note over Test,LLMGenActions: test_constructor_keeps_public_state_visible
    Test->>LLMRails: LLMRails(config, llm)
    LLMRails-->>Test: "rails (config, llm, runtime, events_history_cache={}, explain_info=None)"

    Note over Test,LLMGenActions: test_update_llm_keeps_runtime_generation_actions_and_public_attr_in_sync
    Test->>LLMRails: update_llm(new_llm)
    LLMRails->>LLMRails: "self.llm = new_llm"
    LLMRails->>LLMGenActions: ".llm = new_llm"
    LLMRails->>Runtime: register_action_param("llm", new_llm)
    Test->>LLMRails: rails.llm
    LLMRails-->>Test: new_llm
    Test->>LLMRails: rails.llm_generation_actions.llm
    LLMRails-->>Test: new_llm
    Test->>Runtime: registered_action_params["llm"]
    Runtime-->>Test: new_llm

    Note over Test,LLMGenActions: test_sync_wrappers_raise_when_called_from_async_loop
    Test->>LLMRails: "generate(prompt="hi")"
    LLMRails->>LLMRails: check_sync_call_from_async_loop()
    LLMRails-->>Test: RuntimeError("sync `generate` inside async code")
    Test->>LLMRails: process_events([])
    LLMRails-->>Test: RuntimeError("sync `generate_events` inside async code") ⚠️ known bug

    Note over Test,LLMGenActions: test_getstate_serializes_config_only
    Test->>LLMRails: __getstate__()
    LLMRails-->>Test: "{"config": rails.config} (events_history_cache excluded)"

Loading

%%{init: {'theme': 'base', 'themeVariables': {"darkMode": true, "background": "#0d1117", "primaryColor": "#21262d", "primaryTextColor": "#e6edf3", "primaryBorderColor": "#8b949e", "lineColor": "#8b949e", "textColor": "#e6edf3", "edgeLabelBackground": "#161b22", "actorBkg": "#21262d", "actorBorder": "#8b949e", "actorTextColor": "#e6edf3", "actorLineColor": "#8b949e", "signalColor": "#8b949e", "signalTextColor": "#e6edf3", "noteBkgColor": "#373320", "noteBorderColor": "#d4a72c", "noteTextColor": "#f0e6c0", "labelBoxBkgColor": "#21262d", "labelBoxBorderColor": "#8b949e", "labelTextColor": "#e6edf3", "loopTextColor": "#e6edf3", "activationBkgColor": "#30363d", "activationBorderColor": "#8b949e"}}}%%
sequenceDiagram
    participant Test
    participant LLMRails
    participant Runtime
    participant LLMGenActions as _llm_generation_actions

    Note over Test,LLMGenActions: test_constructor_keeps_public_state_visible
    Test->>LLMRails: LLMRails(config, llm)
    LLMRails-->>Test: "rails (config, llm, runtime, events_history_cache={}, explain_info=None)"

    Note over Test,LLMGenActions: test_update_llm_keeps_runtime_generation_actions_and_public_attr_in_sync
    Test->>LLMRails: update_llm(new_llm)
    LLMRails->>LLMRails: "self.llm = new_llm"
    LLMRails->>LLMGenActions: ".llm = new_llm"
    LLMRails->>Runtime: register_action_param("llm", new_llm)
    Test->>LLMRails: rails.llm
    LLMRails-->>Test: new_llm
    Test->>LLMRails: rails.llm_generation_actions.llm
    LLMRails-->>Test: new_llm
    Test->>Runtime: registered_action_params["llm"]
    Runtime-->>Test: new_llm

    Note over Test,LLMGenActions: test_sync_wrappers_raise_when_called_from_async_loop
    Test->>LLMRails: "generate(prompt="hi")"
    LLMRails->>LLMRails: check_sync_call_from_async_loop()
    LLMRails-->>Test: RuntimeError("sync `generate` inside async code")
    Test->>LLMRails: process_events([])
    LLMRails-->>Test: RuntimeError("sync `generate_events` inside async code") ⚠️ known bug

    Note over Test,LLMGenActions: test_getstate_serializes_config_only
    Test->>LLMRails: __getstate__()
    LLMRails-->>Test: "{"config": rails.config} (events_history_cache excluded)"

Loading

_{Reviews (2): Last reviewed commit: "test(llmrails): characterize public comp..." | Re-trigger Greptile}

[github-actions]

PR merge guidance

@Pouyanpi thanks for the PR. GitHub is currently blocking merge for one or more repository requirements:

• 1 commit does not have a verified signature (7679002). Please sign the commits and force-push the updated branch.

Relevant guide:

• Signed commits: https://github.com/NVIDIA-NeMo/Guardrails/blob/develop/CONTRIBUTING.md#commit-signing
• Contribution guide: https://github.com/NVIDIA-NeMo/Guardrails/blob/develop/CONTRIBUTING.md

[Pouyanpi]

LLMRails decomposition behavioral review matrix

Purpose

This matrix is the behavioral-equivalence guide for reviewing the LLMRails
decomposition. It identifies the public contract each extraction must preserve
and the exact test that exercises it.

The expanded baseline is implemented in coverage PR #2122
and is intended to gate the #2123-#2131 decomposition stack.

The review invariant is: for the same input, the decomposition must preserve:

1. provider request bodies;
2. public output assembly, including return shapes and streamed chunks; and
3. errors, blocking decisions, state continuity, and short-circuits.

A covered behavior is not necessarily an ideal future API. Some tests
intentionally pin existing limitations so they cannot change accidentally during
the refactor.

Review protocol

1. Merge test: add public-API surface and server recorded tests #2122 and rebase the test(llmrails): characterize public behavior (1/9) #2123-fix(llmrails): track deferred startup and streaming followups (9/9) #2131 stack so the coverage is an ancestor.
2. Review each stacked PR against its parent branch, not directly against
   develop.
3. Run the affected matrix rows for each extraction PR.
4. On refactor(llmrails): fix package ownership boundaries (8/9) #2130, the final behavior-preserving ownership PR, run the full recorded
   replay with network blocked.
5. Review fix(llmrails): track deferred startup and streaming followups (9/9) #2131 separately: it contains intentional semantic follow-ups, so any
   matrix deviation must be identified and justified explicitly.
6. Do not rewrite cassettes while evaluating the decomposition. A cassette miss
   indicates a provider request change; a snapshot failure indicates a public
   output change. Either requires investigation.

Primary commands:

make replay-cassettes
make test WORKERS=1 TEST=tests/server/test_server_calls_with_state.py
poetry run pre-commit run --all-files

Existing behaviors intentionally pinned

• State is returned through GenerationResponse.state, not a tuple.
• In Colang 1.0, a named options.rails.input list currently behaves like
  True rather than selecting a subset.
• In Colang 1.0, options.llm_output=True currently leaves
  GenerationResponse.llm_output as None.
• Legacy streaming does not expose accumulated reasoning or tool-call deltas to
  the stream consumer.
• Closing the streaming-output-rail wrapper does not currently close its source
  generator.
• One Colang 2.x LLM-continuation scenario remains an expected xfail because it
  returns an intermediate timer/polling state.

A. Public entry-point contracts (38 scenarios)

ID	Contract to preserve	Entry point	Method	Decomposition review area	Evidence
A1.1	prompt only returns completion string	generate_async	REC	public shell; generation request/response	tests/recorded/rails/public_api/test_generate.py::test_openai_generate_async_public_contract
A1.2	messages only returns next-message dict	generate_async	REC	public shell; generation request/response	tests/recorded/rails/public_api/test_generate.py::test_nim_generate_async_public_contract
A1.3	options provided returns GenerationResponse shape	generate_async	REC	public shell; generation request/response	tests/recorded/rails/public_api/test_generate.py::test_openai_generate_async_options_returns_generation_response
A1.4	GenerationOptions object equals equivalent dict	generate_async	UNIT	public shell; generation request/response	tests/recorded/rails/public_api/test_options.py::test_generation_options_object_matches_dict_equivalent
A1.5	state round-trips through GenerationResponse.state and preserves continuity	generate_async	UNIT	public shell; generation request/response	tests/recorded/rails/public_api/test_state.py::test_dialog_generate_async_state_round_trips_across_turns
A1.6	streaming_handler passed to generate_async (not stream_async)	generate_async	UNIT	public shell; generation request/response	tests/integrations/langchain/test_streaming.py::test_streaming_generate_async_api
A1.7	neither prompt nor messages raises	generate_async	UNIT	public shell; generation request/response	tests/recorded/rails/public_api/test_generate.py::test_generate_async_without_prompt_or_messages_raises
A1.8	both prompt and messages raises	generate_async	UNIT	public shell; generation request/response	tests/recorded/rails/public_api/test_generate.py::test_generate_async_with_prompt_and_messages_raises
A1.9	context-role message injects context vars into request	generate_async	UNIT	public shell; generation request/response	tests/test_llm_rails_context_message.py::test_1
A1.10	event-role message (for example UserSilent) processed	generate_async	UNIT	public shell; generation request/response	tests/test_event_based_api.py::test_3
A1.11	multi-turn history (alternating user/assistant)	generate_async	UNIT	public shell; generation request/response	tests/test_event_based_api.py::test_3
A1.12	invalid model raises	generate_async	REC	public shell; generation request/response	tests/recorded/rails/public_api/test_generate.py::test_openai_generate_async_invalid_model_raises
A2.1	sync generate parity with async	generate	REC	public shell; generation request/response	tests/recorded/rails/public_api/test_generate.py::test_openai_generate_sync_public_contract
A2.2	sync generate inside a running event loop	generate	UNIT	public shell; generation request/response	tests/test_nest_asyncio.py::test_async_api_error
A3.1	token stream without metadata	stream_async	REC	streaming	tests/recorded/rails/public_api/test_stream.py::test_openai_stream_async_public_contract
A3.2	include_metadata yields metadata dicts	stream_async	REC	streaming	tests/recorded/rails/public_api/test_stream.py::test_stream_async_matches_recorded_chat_completion_metadata
A3.3	include_generation_metadata flag	stream_async	UNIT	streaming	tests/recorded/rails/public_api/test_stream.py::test_include_generation_metadata_matches_include_metadata
A3.4	external generator passed (rails over provided stream)	stream_async	UNIT	streaming	tests/recorded/rails/public_api/test_stream.py::test_streaming_output_rails_allowed
A3.5	streaming output rail allows	stream_async	UNIT	streaming	tests/recorded/rails/public_api/test_stream.py::test_streaming_output_rails_allowed
A3.6	streaming output rail blocks (full)	stream_async	UNIT	streaming	tests/recorded/rails/public_api/test_stream.py::test_streaming_output_rails_blocked
A3.7	streaming output rail allow-then-block boundary (buffer flush)	stream_async	UNIT	streaming	tests/recorded/rails/public_api/test_stream.py::test_streaming_output_rail_allows_then_blocks_at_buffer_boundary
A3.8	chunk_size / context_size / stream_first variations	stream_async	UNIT	streaming	tests/recorded/rails/public_api/test_stream.py::test_streaming_output_rail_buffer_configuration
A3.9	streaming requested but disabled raises	stream_async	UNIT	streaming	tests/recorded/rails/public_api/test_stream.py::test_streaming_output_rails_disabled_validation
A3.10	reasoning deltas are excluded from streamed text and are not currently surfaced	stream_async	REC	streaming	tests/recorded/rails/public_api/test_stream.py::test_nim_stream_async_reasoning_not_inlined_in_streamed_text
A3.11	tool-call deltas are accumulated but not surfaced in the legacy public stream	stream_async	UNIT	streaming	tests/recorded/rails/public_api/test_stream.py::test_stream_async_tool_call_deltas_are_not_surfaced
A3.12	input rail blocks before stream begins	stream_async	UNIT	streaming	tests/recorded/rails/public_api/test_stream.py::test_input_rail_blocks_before_stream_generation
A3.13	stream cancellation / early consumer break across pass-through and output-rail wrappers	stream_async	UNIT	streaming	tests/recorded/rails/public_api/test_stream_cancellation.py::test_consumer_break_closes_external_generator tests/recorded/rails/public_api/test_stream_cancellation.py::test_consumer_break_through_output_rail_does_not_close_source_generator
A4.1	check passes	check / check_async	UNIT	checks	tests/recorded/rails/public_api/test_check.py::test_check_async_public_contracts
A4.2	check blocks	check / check_async	UNIT	checks	tests/recorded/rails/public_api/test_check.py::test_check_async_public_contracts
A4.3	check empty messages passes	check_async	UNIT	checks	tests/recorded/rails/public_api/test_check.py::test_check_async_empty_messages_passes
A4.4	single-call check without io rails	check_async	UNIT	checks	tests/recorded/rails/public_api/test_dialog.py::test_single_call_check_async_without_io_rails
A4.5	RailsResult shape (status, content, rail)	check / check_async	UNIT	checks	tests/recorded/rails/public_api/test_check.py::test_check_async_public_contracts
A5.1	generate_events_async: events in, events out	generate_events_async	UNIT	runtime and conversation	tests/recorded/rails/public_api/test_events.py::test_generate_events_async_returns_event_stream
A5.2	process_events_async injects an external event and returns events plus State	process_events_async	UNIT	runtime and conversation	tests/recorded/rails/public_api/test_colang_v2.py::test_colang_v2_process_events_async_injects_event_and_returns_state
A5.3	process_events_async accepts a State object	process_events_async	UNIT	runtime and conversation	tests/cli/test_chat_v2x_integration.py::TestProcessEventsAsyncV2x::test_process_events_async_accepts_state_object
A6.1	explain().llm_calls after a generation	explain	REC	generation context and tracing	tests/recorded/rails/public_api/test_stream.py::test_stream_async_matches_recorded_chat_completion_metadata
A6.2	explain info does not leak across sequential calls	explain	UNIT	generation context and tracing	tests/recorded/rails/public_api/test_state.py::test_baseline_generate_async_log_does_not_leak_across_sequential_calls
A6.3	explain colang_history / llm-calls summary	explain	UNIT	generation context and tracing	tests/recorded/rails/public_api/test_options.py::test_colang_history_log_matches_explain_and_llm_summary

B. Cross-cutting contracts (36 scenarios)

ID	Contract to preserve	Method	Decomposition review area	Evidence
B1.1	options.rails.input False disables input rails	UNIT	generation context and response	tests/recorded/rails/public_api/test_options.py::test_input_rails_disabled_by_options_skips_input_rail
B1.2	options.rails.output False disables output rails	UNIT	generation context and response	tests/recorded/rails/public_api/test_options.py::test_output_rails_disabled_by_options_skips_output_rail
B1.3	options.rails.input name list behaves like True in Colang 1.0	UNIT	generation context and response	tests/recorded/rails/public_api/test_options.py::test_input_rails_name_list_behaves_like_true_in_colang_1
B1.4	options.rails.dialog False skips dialog rail	UNIT	generation context and response	tests/recorded/rails/public_api/test_options.py::test_dialog_disabled_by_options_skips_generation
B1.5	options.rails.retrieval toggle	UNIT	generation context and response	tests/recorded/rails/public_api/test_options.py::test_retrieval_rail_disabled_by_options_skips_retrieval_rail
B1.6	options.llm_params (temperature) reaches request body	REC	generation context and response	tests/recorded/rails/public_api/test_requests.py::test_openai_llm_params_generate_async_request
B1.7	options.output_vars True returns full context	UNIT	generation context and response	tests/recorded/rails/public_api/test_options.py::test_output_vars_true_returns_full_context
B1.8	options.output_vars list returns subset (relevant_chunks)	UNIT	generation context and response	tests/recorded/rails/public_api/test_options.py::test_output_vars_list_returns_subset
B1.9	options.llm_output=True remains None in the Colang 1.0 path	REC	generation context and response	tests/recorded/rails/public_api/test_generate.py::test_dialog_generate_async_llm_output_is_none
B1.10	log.activated_rails ordering and decisions	UNIT	generation context and response	tests/recorded/rails/public_api/test_options.py::test_activated_rails_ordering_and_decisions
B1.11	log.llm_calls details (prompt, completion, usage)	REC	generation context and response	tests/recorded/rails/public_api/test_generate.py::test_openai_generate_async_log_matches_recorded_chat_completion
B1.12	log.internal_events	UNIT	generation context and response	tests/recorded/rails/public_api/test_options.py::test_log_internal_events_populated
B1.13	log.colang_history	UNIT	generation context and response	tests/recorded/rails/public_api/test_options.py::test_colang_history_log_matches_explain_and_llm_summary
B1.14	log.stats token counts (durations excluded as non-deterministic)	REC	generation context and response	tests/recorded/rails/public_api/test_generate.py::test_openai_generate_async_log_stats_token_counts
B2.1	tracing enabled does not change outputs or request bodies	REC	tracing	tests/recorded/rails/public_api/test_tracing.py::test_openai_generate_async_traced_request_matches_untraced
B2.2	LLMCallInfo populated in log under tracing	REC	tracing	tests/recorded/rails/public_api/test_tracing.py::test_openai_generate_async_traced_request_matches_untraced
B2.3	OTel span tree structure and export	UNIT	tracing	tests/guardrails/test_iorails_telemetry.py::TestSpanHierarchy::test_span_tree_parent_child_links
B3.1	KB config retrieves relevant_chunks and injects into prompt	REC	startup, KB, and retrieval	tests/recorded/rails/public_api/test_kb.py::test_openai_kb_generate_async_retrieves_and_injects_chunks
B3.2	retrieved chunks returned via output_vars	REC	startup, KB, and retrieval	tests/recorded/rails/public_api/test_kb.py::test_openai_kb_generate_async_retrieves_and_injects_chunks
B3.3	embedding provider request-body fingerprint	REC	startup, KB, and retrieval	tests/recorded/rails/public_api/test_kb.py::test_openai_kb_generate_async_retrieves_and_injects_chunks
B4.1	state round-trip continuity (Colang 1.0)	UNIT	runtime, state, and server	tests/recorded/rails/public_api/test_state.py::test_dialog_generate_async_state_round_trips_across_turns
B4.2	state round-trip continuity (Colang 2.x)	UNIT	runtime, state, and server	tests/cli/test_chat_v2x_integration.py::TestProcessEventsAsyncV2x::test_process_events_async_accepts_state_object
B4.3	server-side state threading	UNIT	runtime, state, and server	tests/server/test_server_calls_with_state.py::test_colang_1_state_round_trip_continues_conversation
B5.1	main LLM tool_calls surfaced in GenerationResponse.tool_calls	REC	conversation and tool handling	tests/recorded/rails/public_api/test_tools.py::test_openai_generate_async_surfaces_tool_calls
B5.2	tool messages in history reach the request body	UNIT	conversation and tool handling	tests/recorded/rails/public_api/test_tools.py::test_tool_messages_reach_openai_request_body
B5.3	tool_input / tool_output rails	UNIT	conversation and tool handling	tests/test_input_tool_rails.py::TestInputToolRails::test_tool_input_validation_blocking tests/test_tool_output_rails.py::test_tool_output_rails_blocking
B6.1	reasoning_content surfaced in GenerationResponse	REC	generation request/response	tests/recorded/rails/public_api/test_generate.py::test_nim_generate_async_options_surfaces_reasoning_content
B6.2	enable_thinking param reaches request body	REC	generation request/response	tests/recorded/rails/public_api/test_requests.py::test_nim_llm_params_generate_async_request
B7.1	Colang 2.x basic generate_async	UNIT	runtime selection	tests/recorded/rails/public_api/test_colang_v2.py::test_colang_v2_generate_async_runs_deterministic_flow
B7.2	Colang 2.x rejects llm_output option	UNIT	runtime selection	tests/recorded/rails/public_api/test_colang_v2.py::test_colang_v2_generate_async_rejects_llm_output_option
B7.3	Colang 1.0 vs 2.x runtime selection	UNIT	runtime selection	tests/recorded/rails/public_api/test_colang_v2.py::test_runtime_selection_by_colang_version
B8.1	enable_rails_exceptions returns exception vs refusal text	REC	checks and response assembly	tests/recorded/rails/library/test_injection.py::test_injection_output_returns_exception_when_enabled
B9.1	provider error (5xx) propagates	REC	provider error handling	tests/recorded/rails/library/test_content_safety.py::test_content_safety_input_provider_error_raises
B9.2	provider 429 then retry behavior	UNIT	provider error handling	tests/llm/clients/test_openai_compatible.py::TestRetry::test_retries_on_429
B10.1	library file load order is deterministic across filesystems	UNIT	startup and rail execution	tests/test_llmrails.py::test_load_library_sorts_files_for_deterministic_overrides
B10.2	rail execution order is deterministic	REC	startup and rail execution	tests/recorded/rails/library/test_composition.py::test_input_self_check_runs_before_provider_rails

Broad parity backstop

tests/recorded/rails/public_api/test_parity.py::test_parity_generate_async_rail_assembly
provides a composed generate-path backstop. The targeted matrix above remains the
source of truth for identifying which contract failed and which decomposition
area needs review.

Final integration checklist

• Coverage PR test: add public-API surface and server recorded tests #2122 is present in the test(llmrails): characterize public behavior (1/9) #2123 stack base.
• Every extraction PR passes its affected matrix rows against its parent.
• make replay-cassettes passes with network blocked on refactor(llmrails): fix package ownership boundaries (8/9) #2130.
• Server state continuation passes across two HTTP requests.
• No cassette is rewritten to make the decomposition pass.
• Every intentional behavior change in fix(llmrails): track deferred startup and streaming followups (9/9) #2131 is identified against this matrix
  and reviewed explicitly.
• Pre-commit passes.

Baseline on #2122 at the time this matrix was prepared: 144 recorded tests
passed with 1 expected xfail; the server state module passed 8 tests.