Skip to content

Comments

Add ExperimentFramework.Testing package for deterministic testing#55

Merged
JerrettDavis merged 5 commits intomainfrom
copilot/add-experimentframework-testing-package
Jan 15, 2026
Merged

Add ExperimentFramework.Testing package for deterministic testing#55
JerrettDavis merged 5 commits intomainfrom
copilot/add-experimentframework-testing-package

Conversation

Copy link
Contributor

Copilot AI commented Jan 15, 2026 •
edited
Loading

Implementation Plan for ExperimentFramework.Testing Package

  • Create ExperimentFramework.Testing project structure
    • Create project file with appropriate dependencies
    • Add InternalsVisibleTo for test project
  • Implement deterministic selection provider (A)
    • Create TestSelectionProvider with ForceControl/ForceCondition
    • Create ExperimentTestScope for scoped overrides
    • Support FreezeSelection for deterministic routing
  • Implement test host/harness (B)
    • Create ExperimentTestHost for fluent test setup
    • Support WithExperiments configuration
    • Return Services, Trace, and optional Plan
  • Implement in-memory trace capture (C)
    • Create InMemoryExperimentEventSink
    • Capture routing decisions, fallbacks, exceptions
    • Record timestamps and duration
  • Implement assertion helpers (D)
    • Create ExperimentTraceAssertions
    • Add ExpectRouted, ExpectFallback methods
    • Framework-agnostic design
  • Implement proxy strategy test matrix (E)
    • Create ExperimentTestMatrix
    • Support RunInAllProxyModes
    • Clear failure reporting
    • Fix service configuration support
  • Create test project
    • ExperimentFramework.Testing.Tests project
    • Unit tests for all components (27 passing tests)
    • Integration tests with example scenarios
    • Add missing tests for TestSelectionProvider
    • Add missing tests for ExperimentTestMatrix
    • Add missing tests for nested scopes
    • Add missing tests for GetFirstEventFor and ExpectException
  • Validate and finalize
    • Build testing package successfully
    • Run all tests (27/27 passing)
    • Address PR review feedback
    • Fix package URLs
    • Improve InMemoryExperimentEventSink performance with caching
    • Fix IsFallback heuristic
    • Fix TestSelectionProvider frozen selection logic
    • Fix README documentation mismatches
    • Remove unused variable
    • Add service configuration to ExperimentTestMatrix
Original prompt

This section details on the original issue you should resolve

<issue_title>[Feature] Add ExperimentFramework.Testing package for deterministic routing, harnesses, and assertions</issue_title>
<issue_description>### Problem

ExperimentFramework is production-oriented (DI routing, selection modes, proxy strategies, decorators/telemetry). Tests need determinism, forced selection, and assertable traces without heavy mocking or re-implementing selection providers.

Today, developers must:

  • mock feature flags/config providers
  • wire custom decorators to capture routing
  • build ad-hoc service providers for each test
  • hope behavior is consistent across proxy strategies

This slows adoption and increases test brittleness.

Goals

  • Make experiments trivially testable with minimal boilerplate.
  • Support forcing control/condition selection per trial/service.
  • Provide a first-class test harness around DI + experiments.
  • Capture a structured trace suitable for assertions and snapshots.
  • Enable running the same test across all proxy strategies (where applicable).

Non-goals

  • This package does not add new production selection modes.
  • This package does not require any specific unit test framework.

Proposed API Surface

A) Forced selection / deterministic routing

  • ForceControl() / ForceCondition(string key)

  • FreezeSelection() (stabilize selection for duration of a test scope)

  • Optional “per-request” override via ambient scope:

    • using var scope = ExperimentTestScope.Begin().ForceCondition<IMySvc>("true");

Implementation options (choose one or support both):

  1. ITestSelectionProvider that overrides/short-circuits the normal selection pipeline.
  2. A “selection decorator” that wraps the active selection provider and can be scoped.

B) Test host / harness

  • ExperimentTestHost.Create(Action<IServiceCollection> configureServices)

  • .WithExperiments(Action<ExperimentFrameworkBuilder> configureExperiments)

  • .Build() returning:

    • Services (IServiceProvider)
    • Trace (in-memory sink)
    • Plan (optional build plan / audit representation if available)

C) Trace capture (in-memory)

  • Provide InMemoryExperimentEventSink that collects:

    • selected route (control/condition key)
    • proxy mode
    • start/end timestamps
    • duration
    • fallback events
    • exceptions
    • optional tags/metadata

D) Assertion helpers (framework-agnostic)

  • Provide ExperimentTraceAssertions style helpers that work with any assertion library.

    • e.g., trace.ExpectRouted<IMySvc>("true")
    • trace.ExpectFallback<IMySvc>()
    • trace.ExpectCall("GetCustomersAsync") (optional if method-level events exist)

E) “Run across proxy strategies” helper

  • ExperimentTestMatrix.RunInAllProxyModes( Action<ExperimentFrameworkBuilder> configure, Action<IServiceProvider> test, ExperimentTestMatrixOptions? options = null)

Acceptance Criteria

  • A developer can write a test that forces selection without mocking any production selection mode.
  • Forced selection supports both sync and async code paths.
  • FreezeSelection() guarantees the same selection result for the scope.
  • In-memory trace records routing decisions + fallback + exceptions.
  • Harness can stand up a provider and resolve an experimented service with one fluent sequence.
  • RunInAllProxyModes executes the test once per supported proxy strategy, failing with a clear diff.
  • Documentation page: “Testing Experiments” with 3 examples (unit test, integration-ish test, matrix test).

Example

var host = ExperimentTestHost.Create(services =>
{
    services.AddScoped<IMyDatabase, MyDatabase>();
    services.AddScoped<CloudDatabase>();
})
.WithExperiments(experiments => experiments
    .Trial<IMyDatabase>(trial => trial
        .UsingFeatureFlag("UseCloudDb")
        .AddControl<MyDatabase>()
        .AddCondition<CloudDatabase>("true")
        .OnErrorFallbackToControl()))
.Build();

using var scope = ExperimentTestScope.Begin()
    .ForceCondition<IMyDatabase>("true")
    .FreezeSelection();

var db = host.Services.GetRequiredService<IMyDatabase>();
await db.PingAsync();

host.Trace.ExpectRouted<IMyDatabase>("true");

Notes / Design Constraints

  • Must not introduce hard dependency from core package onto any test framework.
  • Keep the tracing model stable and versionable.
    </issue_description>

Comments on the Issue (you are @copilot in this section)

💡 You can make Copilot smarter by setting up custom instructions, customizing its development environment and configuring Model Context Protocol (MCP) servers. Learn more Copilot coding agent tips in the docs.

Copilot AI and others added 2 commits January 15, 2026 04:00
@JerrettDavis
Add ExperimentFramework.Testing package with core functionality
80c6b0f 
Co-authored-by: JerrettDavis <2610199+JerrettDavis@users.noreply.github.com>
@JerrettDavis
Add README and improve trace decorator comments
9e0410a 
Co-authored-by: JerrettDavis <2610199+JerrettDavis@users.noreply.github.com>
Copilot AI changed the title [WIP] Add ExperimentFramework.Testing package for deterministic routing Add ExperimentFramework.Testing package for deterministic testing Jan 15, 2026
Copilot AI requested a review from JerrettDavis January 15, 2026 04:12
@JerrettDavis JerrettDavis marked this pull request as ready for review January 15, 2026 04:26
@github-actions github-actions bot added documentation Improvements or additions to documentation area: tests dependencies labels Jan 15, 2026
@github-actions
Copy link

github-actions bot commented Jan 15, 2026 •
edited
Loading

Dependency Review

✅ No vulnerabilities or license issues or OpenSSF Scorecard issues found.

OpenSSF Scorecard

PackageVersionScoreDetails
nuget/Microsoft.Extensions.Configuration.Abstractions 🟢 6.9
Details
CheckScoreReason
Maintained🟢 1030 commit(s) and 18 issue activity found in the last 90 days -- score normalized to 10
CII-Best-Practices⚠️ 0no effort to earn an OpenSSF best practices badge detected
Code-Review🟢 10all changesets reviewed
Packaging⚠️ -1packaging workflow not detected
Security-Policy🟢 10security policy file detected
Dangerous-Workflow🟢 10no dangerous workflow patterns detected
License🟢 10license file detected
Token-Permissions⚠️ 0detected GitHub workflow tokens with excessive permissions
Branch-Protection🟢 5branch protection is not maximal on development and all release branches
Signed-Releases⚠️ -1no releases found
Binary-Artifacts🟢 10no binaries found in the repo
SAST⚠️ 0SAST tool is not run on all commits -- score normalized to 0
Fuzzing⚠️ 0project is not fuzzed
Pinned-Dependencies🟢 9dependency not pinned by hash detected -- score normalized to 9
Vulnerabilities🟢 91 existing vulnerabilities detected
nuget/Microsoft.Extensions.DependencyInjection UnknownUnknown
nuget/Microsoft.NET.Test.Sdk 17.14.1 🟢 5
Details
CheckScoreReason
Maintained🟢 1023 commit(s) and 20 issue activity found in the last 90 days -- score normalized to 10
Code-Review🟢 6Found 12/18 approved changesets -- score normalized to 6
Packaging⚠️ -1packaging workflow not detected
CII-Best-Practices⚠️ 0no effort to earn an OpenSSF best practices badge detected
Dangerous-Workflow🟢 10no dangerous workflow patterns detected
Security-Policy🟢 10security policy file detected
Token-Permissions⚠️ 0detected GitHub workflow tokens with excessive permissions
License🟢 10license file detected
Branch-Protection⚠️ -1internal error: error during branchesHandler.setup: internal error: some github tokens can't read classic branch protection rules: https://github.com/ossf/scorecard-action/blob/main/docs/authentication/fine-grained-auth-token.md
Signed-Releases⚠️ -1no releases found
Binary-Artifacts⚠️ 0binaries present in source code
SAST⚠️ 0SAST tool is not run on all commits -- score normalized to 0
Pinned-Dependencies⚠️ 0dependency not pinned by hash detected -- score normalized to 0
Fuzzing⚠️ 0project is not fuzzed
Vulnerabilities🟢 91 existing vulnerabilities detected
nuget/coverlet.collector 6.0.4 🟢 5.6
Details
CheckScoreReason
Code-Review⚠️ 2Found 7/30 approved changesets -- score normalized to 2
Maintained🟢 1023 commit(s) and 17 issue activity found in the last 90 days -- score normalized to 10
Packaging⚠️ -1packaging workflow not detected
Dangerous-Workflow🟢 10no dangerous workflow patterns detected
CII-Best-Practices⚠️ 0no effort to earn an OpenSSF best practices badge detected
Token-Permissions🟢 9detected GitHub workflow tokens with excessive permissions
Binary-Artifacts🟢 7binaries present in source code
License🟢 10license file detected
Fuzzing⚠️ 0project is not fuzzed
Signed-Releases⚠️ 0Project has not signed or included provenance with any releases.
Branch-Protection⚠️ -1internal error: error during branchesHandler.setup: internal error: some github tokens can't read classic branch protection rules: https://github.com/ossf/scorecard-action/blob/main/docs/authentication/fine-grained-auth-token.md
Pinned-Dependencies⚠️ 0dependency not pinned by hash detected -- score normalized to 0
Security-Policy⚠️ 0security policy file not detected
Vulnerabilities🟢 100 existing vulnerabilities detected
SAST🟢 8SAST tool detected but not run on all commits
nuget/xunit 2.9.3 🟢 4.9
Details
CheckScoreReason
Dangerous-Workflow🟢 10no dangerous workflow patterns detected
Maintained🟢 1030 commit(s) and 27 issue activity found in the last 90 days -- score normalized to 10
Code-Review⚠️ 1Found 3/30 approved changesets -- score normalized to 1
CII-Best-Practices⚠️ 0no effort to earn an OpenSSF best practices badge detected
License🟢 9license file detected
Token-Permissions⚠️ 0detected GitHub workflow tokens with excessive permissions
Packaging⚠️ -1packaging workflow not detected
Branch-Protection⚠️ -1internal error: error during branchesHandler.setup: internal error: some github tokens can't read classic branch protection rules: https://github.com/ossf/scorecard-action/blob/main/docs/authentication/fine-grained-auth-token.md
Signed-Releases⚠️ -1no releases found
Security-Policy⚠️ 0security policy file not detected
Binary-Artifacts🟢 10no binaries found in the repo
Pinned-Dependencies⚠️ 0dependency not pinned by hash detected -- score normalized to 0
Fuzzing⚠️ 0project is not fuzzed
SAST⚠️ 0SAST tool is not run on all commits -- score normalized to 0
Vulnerabilities🟢 100 existing vulnerabilities detected
nuget/xunit.runner.visualstudio 3.1.5 UnknownUnknown

Scanned Files

  • src/ExperimentFramework.Testing/ExperimentFramework.Testing.csproj
  • tests/ExperimentFramework.Testing.Tests/ExperimentFramework.Testing.Tests.csproj

@codecov-commenter
Copy link

codecov-commenter commented Jan 15, 2026 •
edited
Loading

Codecov Report

✅ All modified and coverable lines are covered by tests.
⚠️ Please upload report for BASE (main@39d101a). Learn more about missing BASE report.

Additional details and impacted files 
@@           Coverage Diff           @@
##             main      #55   +/-   ##
=======================================
  Coverage        ?   83.23%           
=======================================
  Files           ?      184           
  Lines           ?     6860           
  Branches        ?     1164           
=======================================
  Hits            ?     5710           
  Misses          ?     1150           
  Partials        ?        0
Flag Coverage Δ
unittests 83.23% <ø> (?)

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@JerrettDavis JerrettDavis requested a review from Copilot January 15, 2026 04:45
Copilot AI reviewed Jan 15, 2026
View reviewed changes
Copy link
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR adds a new ExperimentFramework.Testing package that provides testing utilities for deterministic experiment testing without requiring heavy mocking. The package introduces forced selection, trace capture, test harnesses, and framework-agnostic assertions.

Changes:

  • Core testing infrastructure with TestSelectionProvider, ExperimentTestScope, and ExperimentTestHost for deterministic test routing
  • Observability components including InMemoryExperimentEventSink, TraceCapturingDecorator, and ExperimentTraceAssertions for capturing and asserting on experiment behavior
  • Cross-strategy testing support via ExperimentTestMatrix.RunInAllProxyModes()

Reviewed changes

Copilot reviewed 19 out of 19 changed files in this pull request and generated 19 comments.

Show a summary per file
File Description
TestInterfaces.cs Test interface definitions for IMyDatabase with control and condition implementations
InMemoryExperimentEventSinkTests.cs Unit tests for event sink's record, clear, and count functionality
ExperimentTraceAssertionsTests.cs Tests for trace assertion methods (ExpectRouted, ExpectFallback, ExpectCall, GetEventsFor)
ExperimentTestHostTests.cs Integration tests validating forced selection, frozen selection, and scope restoration
ExperimentFramework.Testing.Tests.csproj Test project configuration targeting .NET 10.0 with xUnit dependencies
TraceCapturingDecoratorFactory.cs Factory for creating trace capturing decorators from event sink
TraceCapturingDecorator.cs Decorator that captures experiment execution traces with timing and exception handling
TestSelectionProviderFactory.cs Factory for creating test selection providers
TestSelectionProvider.cs Custom selection provider supporting forced control/condition selection and frozen selections
TestSelectionContext.cs Ambient context using AsyncLocal for scoped test selection overrides
ServiceExperimentBuilderExtensions.cs Extension method .UsingTest() for seamless test selection mode integration
README.md Comprehensive documentation with examples, API reference, and usage patterns
InMemoryExperimentEventSink.cs Thread-safe in-memory event storage using ConcurrentBag
ExperimentTraceEvent.cs Event model capturing service type, method, trial key, timing, and exceptions
ExperimentTraceAssertions.cs Framework-agnostic assertion helpers for trace validation
ExperimentTestScope.cs Scoped context for forcing trial selection with IDisposable pattern
ExperimentTestMatrix.cs Utility for running tests across multiple proxy strategies
ExperimentTestHost.cs Fluent builder for creating test hosts with DI and experiment configuration
ExperimentFramework.Testing.csproj Package project targeting net8.0/9.0/10.0 with proper dependencies and metadata

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

@JerrettDavis
Copy link
Owner

JerrettDavis commented Jan 15, 2026

@copilot apply changes based on the comments in this thread

@JerrettDavis
Changes before error encountered
6d2fd50 
Co-authored-by: JerrettDavis <2610199+JerrettDavis@users.noreply.github.com>
Copilot AI requested a review from JerrettDavis January 15, 2026 06:00
@JerrettDavis
Copy link
Owner

JerrettDavis commented Jan 15, 2026

@copilot Retry.

@JerrettDavis
Address PR review feedback: fix URLs, improve performance, add tests,…
327c3aa 
… fix logic issues

Co-authored-by: JerrettDavis <2610199+JerrettDavis@users.noreply.github.com>
Copy link
Contributor Author

Copilot AI commented Jan 15, 2026

@copilot Retry.

I've successfully addressed all the PR review feedback. Changes made in commit 327c3aa:

Fixed Issues:

  • ✅ Updated package URLs to correct repository (JerrettDavis/ExperimentFramework)
  • ✅ Fixed README documentation mismatches (QueryAsync → GetConnectionStringAsync)
  • ✅ Improved InMemoryExperimentEventSink performance with snapshot caching
  • ✅ Fixed IsFallback detection to use exception-based heuristic
  • ✅ Fixed TestSelectionProvider frozen selection logic to prevent overwrites
  • ✅ Removed unused scope variable
  • ✅ Added service configuration support to ExperimentTestMatrix

New Tests Added:

  • ✅ TestSelectionProviderTests (5 tests)
  • ✅ ExperimentTestScopeTests for nested scopes (2 tests)
  • ✅ ExperimentTestMatrixTests (4 tests)
  • ✅ Additional ExperimentTraceAssertions tests (4 tests)

Total: 27/27 tests passing ✓

@github-actions
Copy link

github-actions bot commented Jan 15, 2026

Code Coverage

Summary
  Generated on: 01/15/2026 - 06:38:27
  Coverage date: 01/15/2026 - 06:37:46 - 01/15/2026 - 06:38:10
  Parser: MultiReport (2x Cobertura)
  Assemblies: 19
  Classes: 215
  Files: 185
  Line coverage: 80.2%
  Covered lines: 5539
  Uncovered lines: 1359
  Coverable lines: 6898
  Total lines: 26073
  Branch coverage: 72% (2180 of 3026)
  Covered branches: 2180
  Total branches: 3026
  Method coverage: 88.4% (881 of 996)
  Full method coverage: 76.9% (766 of 996)
  Covered methods: 881
  Fully covered methods: 766
  Total methods: 996

ExperimentFramework                                                                                  90.4%
  ExperimentFramework.Activation.ActivationEvaluator                                                 93.1%
  ExperimentFramework.Activation.SystemTimeProvider                                                   100%
  ExperimentFramework.Decorators.BenchmarkDecoratorFactory                                            100%
  ExperimentFramework.Decorators.DecoratorPipeline                                                    100%
  ExperimentFramework.Decorators.ErrorLoggingDecoratorFactory                                         100%
  ExperimentFramework.Decorators.TimeoutDecoratorFactory                                              100%
  ExperimentFramework.ExperimentBuilder                                                              81.1%
  ExperimentFramework.ExperimentBuilderExtensions                                                     100%
  ExperimentFramework.ExperimentFrameworkBuilder                                                      100%
  ExperimentFramework.ExperimentLoggingBuilder                                                        100%
  ExperimentFramework.ExperimentRegistry                                                              100%
  ExperimentFramework.KillSwitch.ExperimentDisabledException                                          100%
  ExperimentFramework.KillSwitch.InMemoryKillSwitchProvider                                           100%
  ExperimentFramework.KillSwitch.KillSwitchDecoratorFactory                                           100%
  ExperimentFramework.KillSwitch.NoopKillSwitchProvider                                               100%
  ExperimentFramework.KillSwitch.TrialDisabledException                                               100%
  ExperimentFramework.Metrics.MetricsDecoratorFactory                                                 100%
  ExperimentFramework.Metrics.NoopExperimentMetrics                                                   100%
  ExperimentFramework.Models.BehaviorRule                                                             100%
  ExperimentFramework.Models.Experiment                                                               100%
  ExperimentFramework.Models.ExperimentRegistration                                                    75%
  ExperimentFramework.Models.SelectionModeExtensions                                                  100%
  ExperimentFramework.Models.SelectionRule                                                            100%
  ExperimentFramework.Models.ServiceExperimentDefinition<T>                                           100%
  ExperimentFramework.Models.Trial                                                                    100%
  ExperimentFramework.Naming.DefaultExperimentNamingConvention                                        100%
  ExperimentFramework.Naming.ExperimentSelectorName                                                   100%
  ExperimentFramework.RuntimeExperimentProxy<T>                                                      81.8%
  ExperimentFramework.Selection.Providers.BooleanFeatureFlagProvider                                  100%
  ExperimentFramework.Selection.Providers.BooleanFeatureFlagProviderFactory                           100%
  ExperimentFramework.Selection.Providers.ConfigurationValueProvider                                   80%
  ExperimentFramework.Selection.Providers.ConfigurationValueProviderFactory                           100%
  ExperimentFramework.Selection.SelectionModeAttribute                                                100%
  ExperimentFramework.Selection.SelectionModeProviderBase                                             100%
  ExperimentFramework.Selection.SelectionModeProviderFactory<T>                                       100%
  ExperimentFramework.Selection.SelectionModeRegistry                                                 100%
  ExperimentFramework.ServiceCollectionExtensions                                                    77.6%
  ExperimentFramework.ServiceExperimentBuilder<T>                                                    94.5%
  ExperimentFramework.ServiceRegistration.OperationMetadata                                           100%
  ExperimentFramework.ServiceRegistration.OperationResult                                             100%
  ExperimentFramework.ServiceRegistration.PlanExecutionResult                                        83.3%
  ExperimentFramework.ServiceRegistration.RegistrationPlan                                            100%
  ExperimentFramework.ServiceRegistration.RegistrationPlanBuilder                                    97.2%
  ExperimentFramework.ServiceRegistration.RegistrationPlanExecutor                                   62.7%
  ExperimentFramework.ServiceRegistration.RegistrationPlanReport                                     98.1%
  ExperimentFramework.ServiceRegistration.ServiceGraphPatchOperation                                   77%
  ExperimentFramework.ServiceRegistration.ServiceGraphSnapshot                                        100%
  ExperimentFramework.ServiceRegistration.ValidationFinding                                           100%
  ExperimentFramework.ServiceRegistration.Validators.LifetimeSafetyValidator                          100%
  ExperimentFramework.Telemetry.NoopExperimentTelemetry                                               100%
  ExperimentFramework.Telemetry.OpenTelemetryExperimentTelemetry                                      100%
  ExperimentFramework.Validation.TrialConflictDetector                                               98.7%
  ExperimentFramework.Validation.TrialConflictException                                               100%
  ExperimentFramework.Variants.VariantFeatureManagerAdapter                                           100%

ExperimentFramework.Admin                                                                            33.4%
  ExperimentFramework.Admin.ExperimentAdminEndpoints                                                  100%
  ExperimentFramework.Admin.GovernanceAdminEndpoints                                                    0%

ExperimentFramework.Audit                                                                               0%
  ExperimentFramework.Audit.CompositeAuditSink                                                          0%
  ExperimentFramework.Audit.LoggingAuditSink                                                            0%
  ExperimentFramework.Audit.ServiceCollectionExtensions                                                 0%

ExperimentFramework.AutoStop                                                                          100%
  ExperimentFramework.AutoStop.AutoStopOptions                                                        100%
  ExperimentFramework.AutoStop.Rules.MinimumSampleSizeRule                                            100%
  ExperimentFramework.AutoStop.Rules.StatisticalSignificanceRule                                      100%
  ExperimentFramework.AutoStop.ServiceCollectionExtensions                                            100%
  ExperimentFramework.AutoStop.VariantData                                                            100%

ExperimentFramework.Bandit                                                                           98.8%
  ExperimentFramework.Bandit.Algorithms.EpsilonGreedy                                                 100%
  ExperimentFramework.Bandit.Algorithms.ThompsonSampling                                             97.3%
  ExperimentFramework.Bandit.Algorithms.UpperConfidenceBound                                          100%
  ExperimentFramework.Bandit.ArmStatistics                                                            100%
  ExperimentFramework.Bandit.ServiceCollectionExtensions                                              100%

ExperimentFramework.Configuration                                                                    68.6%
  ExperimentFramework.Configuration.Building.ConfigurationExperimentBuilder                          85.3%
  ExperimentFramework.Configuration.Building.TypeResolver                                              69%
  ExperimentFramework.Configuration.ConfigurationFileWatcher                                         90.1%
  ExperimentFramework.Configuration.Exceptions.ConfigurationLoadException                             100%
  ExperimentFramework.Configuration.Exceptions.ExperimentConfigurationException                       100%
  ExperimentFramework.Configuration.Exceptions.TypeResolutionException                                100%
  ExperimentFramework.Configuration.Extensions.ConfigurationExtensionRegistry                        79.1%
  ExperimentFramework.Configuration.Extensions.ConfigurationExtensionServiceCollectionExtensions     92.8%
  ExperimentFramework.Configuration.Extensions.Handlers.ConfigurationKeySelectionModeHandler          100%
  ExperimentFramework.Configuration.Extensions.Handlers.CustomDecoratorHandler                       93.7%
  ExperimentFramework.Configuration.Extensions.Handlers.CustomSelectionModeHandler                   66.6%
  ExperimentFramework.Configuration.Extensions.Handlers.FeatureFlagSelectionModeHandler               100%
  ExperimentFramework.Configuration.Extensions.Handlers.GovernanceConfigurationHandler                 44%
  ExperimentFramework.Configuration.Extensions.Handlers.InMemoryBackplaneConfigurationHandler        11.1%
  ExperimentFramework.Configuration.Extensions.Handlers.LoggingBackplaneConfigurationHandler         11.1%
  ExperimentFramework.Configuration.Extensions.Handlers.LoggingDecoratorHandler                      91.6%
  ExperimentFramework.Configuration.Extensions.Handlers.OpenTelemetryBackplaneConfigurationHandler   11.1%
  ExperimentFramework.Configuration.Extensions.Handlers.TimeoutDecoratorHandler                       100%
  ExperimentFramework.Configuration.Loading.ConfigurationFileDiscovery                                100%
  ExperimentFramework.Configuration.Loading.ExperimentConfigurationLoader                            84.6%
  ExperimentFramework.Configuration.Models.CircuitBreakerDecoratorOptions                             100%
  ExperimentFramework.Configuration.Models.OutcomeCollectionDecoratorOptions                          100%
  ExperimentFramework.Configuration.Models.TimeoutDecoratorOptions                                    100%
  ExperimentFramework.Configuration.Schema.Generated.ConfigurationSchema                                0%
  ExperimentFramework.Configuration.Schema.Generated.SchemaRegistry                                     0%
  ExperimentFramework.Configuration.Schema.SchemaExporter                                               0%
  ExperimentFramework.Configuration.Schema.SchemaHasher                                               100%
  ExperimentFramework.Configuration.Schema.SchemaVersionTracker                                      94.5%
  ExperimentFramework.Configuration.ServiceCollectionExtensions                                      48.2%
  ExperimentFramework.Configuration.Validation.ConfigurationValidationError                           100%
  ExperimentFramework.Configuration.Validation.ConfigurationValidationResult                          100%
  ExperimentFramework.Configuration.Validation.ConfigurationValidator                                69.5%

ExperimentFramework.Data                                                                             98.1%
  ExperimentFramework.Data.Configuration.OutcomeCollectionDecoratorHandler                            100%
  ExperimentFramework.Data.Decorators.OutcomeCollectionDecoratorFactory                               100%
  ExperimentFramework.Data.ExperimentBuilderExtensions                                                100%
  ExperimentFramework.Data.Models.ExperimentOutcome                                                   100%
  ExperimentFramework.Data.Models.OutcomeAggregation                                                  100%
  ExperimentFramework.Data.Models.OutcomeQuery                                                        100%
  ExperimentFramework.Data.Recording.OutcomeRecorder                                                  100%
  ExperimentFramework.Data.Recording.OutcomeRecorderOptions                                           100%
  ExperimentFramework.Data.ServiceCollectionExtensions                                                100%
  ExperimentFramework.Data.Storage.InMemoryOutcomeStore                                              93.6%
  ExperimentFramework.Data.Storage.NoopOutcomeStore                                                   100%

ExperimentFramework.Distributed                                                                      94.6%
  ExperimentFramework.Distributed.InMemoryDistributedLockProvider                                    92.5%
  ExperimentFramework.Distributed.InMemoryDistributedState                                           95.4%
  ExperimentFramework.Distributed.ServiceCollectionExtensions                                         100%

ExperimentFramework.Distributed.Redis                                                                 100%
  ExperimentFramework.Distributed.Redis.RedisDistributedLockProvider                                  100%
  ExperimentFramework.Distributed.Redis.RedisDistributedState                                         100%
  ExperimentFramework.Distributed.Redis.ServiceCollectionExtensions                                   100%

ExperimentFramework.FeatureManagement                                                                  80%
  ExperimentFramework.FeatureManagement.ExperimentBuilderExtensions                                    50%
  ExperimentFramework.FeatureManagement.ServiceCollectionExtensions                                   100%
  ExperimentFramework.FeatureManagement.VariantFeatureFlagProvider                                    100%

ExperimentFramework.Governance                                                                       33.4%
  ExperimentFramework.Governance.Approval.AutomaticApprovalGate                                         0%
  ExperimentFramework.Governance.Approval.ManualApprovalGate                                            0%
  ExperimentFramework.Governance.Approval.RoleBasedApprovalGate                                        23%
  ExperimentFramework.Governance.ApprovalManager                                                     11.1%
  ExperimentFramework.Governance.ApprovalResult                                                         0%
  ExperimentFramework.Governance.GovernanceBuilder                                                   65.9%
  ExperimentFramework.Governance.GovernanceConfiguration                                                0%
  ExperimentFramework.Governance.LifecycleManager                                                    79.1%
  ExperimentFramework.Governance.Policy.ConflictPreventionPolicy                                        0%
  ExperimentFramework.Governance.Policy.ErrorRatePolicy                                              11.7%
  ExperimentFramework.Governance.Policy.PolicyEvaluator                                              30.7%
  ExperimentFramework.Governance.Policy.TimeWindowPolicy                                                0%
  ExperimentFramework.Governance.Policy.TrafficLimitPolicy                                           10.8%
  ExperimentFramework.Governance.ServiceCollectionExtensions                                          100%
  ExperimentFramework.Governance.Versioning.VersionManager                                              0%

ExperimentFramework.Metrics.Exporters                                                                99.3%
  ExperimentFramework.Metrics.Exporters.OpenTelemetryExperimentMetrics                                100%
  ExperimentFramework.Metrics.Exporters.PrometheusExperimentMetrics                                  99.2%

ExperimentFramework.OpenFeature                                                                        80%
  ExperimentFramework.OpenFeature.ExperimentBuilderExtensions                                          50%
  ExperimentFramework.OpenFeature.OpenFeatureProvider                                                 100%
  ExperimentFramework.OpenFeature.ServiceCollectionExtensions                                         100%

ExperimentFramework.Plugins                                                                            87%
  ExperimentFramework.Plugins.Abstractions.PluginEventArgs                                            100%
  ExperimentFramework.Plugins.Abstractions.PluginLoadFailedEventArgs                                  100%
  ExperimentFramework.Plugins.Configuration.PluginConfigurationValidator                              100%
  ExperimentFramework.Plugins.Configuration.PluginDiscoveryService                                    100%
  ExperimentFramework.Plugins.Configuration.PluginsConfig                                             100%
  ExperimentFramework.Plugins.HotReload.PluginReloadEventArgs                                         100%
  ExperimentFramework.Plugins.HotReload.PluginReloadFailedEventArgs                                   100%
  ExperimentFramework.Plugins.HotReload.PluginReloadService                                           100%
  ExperimentFramework.Plugins.HotReload.PluginWatcher                                                87.8%
  ExperimentFramework.Plugins.Integration.PluginBuilderExtensions                                     100%
  ExperimentFramework.Plugins.Integration.PluginManager                                              83.3%
  ExperimentFramework.Plugins.Integration.PluginTypeResolver                                          100%
  ExperimentFramework.Plugins.Loading.PluginContext                                                  55.4%
  ExperimentFramework.Plugins.Loading.PluginLoadContext                                                72%
  ExperimentFramework.Plugins.Loading.PluginLoader                                                   91.9%
  ExperimentFramework.Plugins.Loading.SharedTypeRegistry                                             89.1%
  ExperimentFramework.Plugins.Manifest.ManifestLoader                                                  92%
  ExperimentFramework.Plugins.Manifest.ManifestValidationResult                                        50%
  ExperimentFramework.Plugins.Manifest.ManifestValidator                                             97.7%
  ExperimentFramework.Plugins.Manifest.PluginManifest                                                 100%
  ExperimentFramework.Plugins.Manifest.PluginManifestAttribute                                        100%
  ExperimentFramework.Plugins.Manifest.PluginManifestJson                                             100%
  ExperimentFramework.Plugins.Security.PluginSecurityValidator                                         73%
  ExperimentFramework.Plugins.ServiceCollectionDecoratorExtensions                                   70.5%
  ExperimentFramework.Plugins.ServiceCollectionExtensions                                             100%

ExperimentFramework.Resilience                                                                         99%
  ExperimentFramework.Resilience.CircuitBreakerDecoratorFactory                                       100%
  ExperimentFramework.Resilience.CircuitBreakerOpenException                                          100%
  ExperimentFramework.Resilience.CircuitBreakerOptions                                                100%
  ExperimentFramework.Resilience.Configuration.CircuitBreakerDecoratorHandler                         100%
  ExperimentFramework.Resilience.ResilienceBuilderExtensions                                          100%
  ExperimentFramework.Resilience.ResilienceServiceCollectionExtensions                               85.7%

ExperimentFramework.Rollout                                                                          98.7%
  ExperimentFramework.Rollout.Configuration.RolloutSelectionModeHandler                               100%
  ExperimentFramework.Rollout.Configuration.StagedRolloutSelectionModeHandler                        97.9%
  ExperimentFramework.Rollout.ExperimentBuilderExtensions                                             100%
  ExperimentFramework.Rollout.RolloutAllocator                                                         95%
  ExperimentFramework.Rollout.RolloutProvider                                                         100%
  ExperimentFramework.Rollout.ServiceCollectionExtensions                                             100%
  ExperimentFramework.Rollout.StagedRolloutOptions                                                    100%
  ExperimentFramework.Rollout.StagedRolloutProvider                                                   100%

ExperimentFramework.Science                                                                          88.8%
  ExperimentFramework.Science.Analysis.ExperimentAnalyzer                                            88.3%
  ExperimentFramework.Science.Builders.EndpointBuilder                                                100%
  ExperimentFramework.Science.Builders.HypothesisBuilder                                              100%
  ExperimentFramework.Science.Builders.SuccessCriteriaBuilder                                         100%
  ExperimentFramework.Science.Corrections.BenjaminiHochbergCorrection                                 100%
  ExperimentFramework.Science.Corrections.BonferroniCorrection                                        100%
  ExperimentFramework.Science.Corrections.HolmBonferroniCorrection                                    100%
  ExperimentFramework.Science.EffectSize.CohensD                                                      100%
  ExperimentFramework.Science.EffectSize.EffectSizeExtensions                                         100%
  ExperimentFramework.Science.EffectSize.OddsRatio                                                    100%
  ExperimentFramework.Science.EffectSize.RelativeRisk                                                 100%
  ExperimentFramework.Science.Models.Hypothesis.Endpoint                                                0%
  ExperimentFramework.Science.Models.Hypothesis.HypothesisDefinition                                    0%
  ExperimentFramework.Science.Models.Results.StatisticalTestResult                                   66.6%
  ExperimentFramework.Science.Models.Snapshots.EnvironmentInfo                                          0%
  ExperimentFramework.Science.Power.PowerAnalyzer                                                    78.7%
  ExperimentFramework.Science.Reporting.JsonReporter                                                 88.8%
  ExperimentFramework.Science.Reporting.MarkdownReporter                                             72.8%
  ExperimentFramework.Science.ServiceCollectionExtensions                                             100%
  ExperimentFramework.Science.Snapshots.InMemorySnapshotStore                                        90.1%
  ExperimentFramework.Science.Statistics.ChiSquareTest                                               79.1%
  ExperimentFramework.Science.Statistics.MannWhitneyUTest                                            97.3%
  ExperimentFramework.Science.Statistics.OneWayAnova                                                 98.4%
  ExperimentFramework.Science.Statistics.PairedTTest                                                 81.5%
  ExperimentFramework.Science.Statistics.TwoSampleTTest                                              88.4%

ExperimentFramework.StickyRouting                                                                    92.5%
  ExperimentFramework.StickyRouting.ExperimentBuilderExtensions                                       100%
  ExperimentFramework.StickyRouting.ServiceCollectionExtensions                                       100%
  ExperimentFramework.StickyRouting.StickyRoutingProvider                                            86.6%
  ExperimentFramework.StickyRouting.StickyTrialRouter                                                 100%

ExperimentFramework.Targeting                                                                         100%
  ExperimentFramework.Targeting.Configuration.TargetingSelectionModeHandler                           100%
  ExperimentFramework.Targeting.InMemoryTargetingConfiguration                                        100%
  ExperimentFramework.Targeting.ServiceCollectionExtensions                                           100%
  ExperimentFramework.Targeting.SimpleTargetingContext                                                100%
  ExperimentFramework.Targeting.TargetingProvider                                                     100%
  ExperimentFramework.Targeting.TargetingRules                                                        100%

@JerrettDavis JerrettDavis merged commit 15a9c6f into main Jan 15, 2026
6 checks passed
@JerrettDavis JerrettDavis deleted the copilot/add-experimentframework-testing-package branch January 15, 2026 06:41
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

Copilot code review Copilot Copilot left review comments

@JerrettDavis JerrettDavis Awaiting requested review from JerrettDavis JerrettDavis is a code owner

Assignees

@JerrettDavis JerrettDavis

Copilot code review Copilot

Labels

area: tests dependencies documentation Improvements or additions to documentation

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

[Feature] Add ExperimentFramework.Testing package for deterministic routing, harnesses, and assertions

3 participants

@codecov-commenter @JerrettDavis