Expose programmatic test filtering via ITestFilter (avoid ClassInit cost for filtered-out tests)

[Evangelink]

Summary

Expose a public, programmatic test-filtering hook in MSTest so users can decide, per test case, whether to run / silently drop / skip-with-reason — without paying the cost of [AssemblyInitialize] or [ClassInitialize] for tests that will not run.

Motivation

Today users can:

• CLI filters (--filter, --filter-uid, --treenode-filter): metadata-only. Static. Set from outside the test process.
• [Ignore]: static at compile time.
• [GlobalTestInitialize] + Assert.Inconclusive: dynamic, but runs as part of the per-test TestInitialize phase, i.e. after the class's [ClassInitialize] has already executed. Internal teams that prototyped this approach found it too costly: ClassInitialize is paid even when every test in the class will end up inconclusive.

There is no in-process, programmatic way to filter tests before assembly/class init runs.

Related: #3590, #7160, #3528.

Proposed design

Public surface (one attribute, one interface, two value types)

Registration mirrors the existing AssemblyFixtureProviderAttribute:

namespace Microsoft.VisualStudio.TestTools.UnitTesting;

[AttributeUsage(AttributeTargets.Assembly, AllowMultiple = true, Inherited = false)]
public sealed class TestFilterProviderAttribute : Attribute
{
    public TestFilterProviderAttribute(Type filterType) => FilterType = filterType;
    public Type FilterType { get; }
}

public interface ITestFilter
{
    TestFilterResult Filter(TestFilterContext context);
}

public sealed class TestFilterContext
{
    public string FullyQualifiedName { get; }
    public string DisplayName { get; }
    public string TestClassName { get; }
    public string TestMethodName { get; }
    public IReadOnlyList<string> Categories { get; }
    // Multi-value-safe: VSTest traits can repeat keys
    public IReadOnlyList<KeyValuePair<string, string?>> Traits { get; }
    public int? Priority { get; }
    // Intentionally no MethodInfo / Type to keep the filter cheap (no type load,
    // no static ctor, no attribute construction for tests we are about to skip).
}

public readonly struct TestFilterResult
{
    public TestFilterAction Action { get; }
    public string? SkipReason { get; }

    // Properties for the no-arg cases — allocation-free on a readonly struct, reads naturally.
    public static TestFilterResult Run  { get; }   // = default(TestFilterResult); always include
    public static TestFilterResult Drop { get; }   // silently dropped, matches CLI --filter semantics

    // Method for the case that needs a parameter
    public static TestFilterResult Skip(string reason);
}

public enum TestFilterAction
{
    Run = 0,    // default(TestFilterResult).Action == Run (safe default if a user forgets)
    Drop,       // silent: no test result emitted, test count unaffected — same as --filter
    Skip,       // reported as Skipped with reason; appears in TRX / console / test count
}

Where it runs

Inside the same isolation host (AppDomain on .NET Framework) that runs tests, as the first step of UnitTestRunner.RunSingleTestAsync — before:

1. _typeCache.GetTestMethodInfo (no test type load),
2. IsTestMethodRunnable (no [Ignore] check),
3. RunAssemblyInitializeIfNeededAsync (no [AssemblyInitialize]),
4. GetResultOrRunClassInitializeAsync (no [ClassInitialize]).

So if every test in a class returns Drop or Skip(...), ClassInitialize for that class never runs. If every test in the assembly returns Drop/Skip, AssemblyInitialize never runs.

Semantics

Result	Test count	Result reported	Class/Assembly Init runs?
Run (default)	counts	as usual	yes, lazily, when first non-Drop test in that class is reached
Drop	not counted	nothing	no (if every test in the class is dropped)
Skip(reason)	counted	Skipped, reason flows to TRX/console	no (if every test in the class is skipped)

• Filter is always additive to the CLI filter (--filter / --filter-uid / --treenode-filter run first).
• Multiple [assembly: TestFilterProvider(typeof(...))] markers are allowed; filters are composed with AND (a test runs only if every filter returns Run).
• Filter exception → fails the test source with a clear "filter X threw" diagnostic; does not silently drop tests.
• Filter type is loaded and instantiated lazily, the first time a test is evaluated, and reused (single instance per assembly per test run).

Fully crafted end-to-end example

Suppose a team wants to:

• Skip tests tagged [TestCategory("Nightly")] unless RUN_NIGHTLY=1 is set.
• Skip tests on Linux when they have a trait Platform=Windows.
• Silently drop tests whose class name matches a per-developer "off" list configured in .testfilter.json, without polluting the test count.

// File: TestInfra/TestFilters.cs   (compiled into the test assembly)
using System.Runtime.InteropServices;
using System.Text.Json;
using Microsoft.VisualStudio.TestTools.UnitTesting;

[assembly: TestFilterProvider(typeof(MyOrg.Tests.TeamTestFilter))]

namespace MyOrg.Tests;

public sealed class TeamTestFilter : ITestFilter
{
    private static readonly bool s_runNightly =
        string.Equals(Environment.GetEnvironmentVariable("RUN_NIGHTLY"), "1", StringComparison.Ordinal);

    private static readonly bool s_isLinux =
        RuntimeInformation.IsOSPlatform(OSPlatform.Linux);

    // Loaded once. The provider type is instantiated lazily once per assembly per test run.
    private readonly HashSet<string> _droppedClasses = LoadDroppedClasses();

    public TestFilterResult Filter(TestFilterContext context)
    {
        // 1) Silently drop tests in classes a developer turned off locally.
        //    Drop = no count, no result, mirrors `--filter` semantics.
        if (_droppedClasses.Contains(context.TestClassName))
        {
            return TestFilterResult.Drop;
        }

        // 2) Skip Nightly tests unless explicitly opted in. Surface a reason
        //    so the report makes it obvious why the test was skipped.
        if (!s_runNightly && context.Categories.Contains("Nightly"))
        {
            return TestFilterResult.Skip("RUN_NIGHTLY=1 is required to run [TestCategory(\"Nightly\")] tests.");
        }

        // 3) Skip Windows-only tests on Linux.
        if (s_isLinux)
        {
            foreach (KeyValuePair<string, string?> trait in context.Traits)
            {
                if (trait.Key == "Platform" && trait.Value == "Windows")
                {
                    return TestFilterResult.Skip("Test requires Windows.");
                }
            }
        }

        // 4) Everything else runs normally. `Run` is the default, so we could also
        //    write `return default;` here.
        return TestFilterResult.Run;
    }

    private static HashSet<string> LoadDroppedClasses()
    {
        string path = Path.Combine(AppContext.BaseDirectory, ".testfilter.json");
        if (!File.Exists(path))
        {
            return new HashSet<string>(StringComparer.Ordinal);
        }

        var doc = JsonDocument.Parse(File.ReadAllText(path));
        return doc.RootElement.TryGetProperty("droppedClasses", out JsonElement arr)
            ? new HashSet<string>(arr.EnumerateArray().Select(e => e.GetString()!), StringComparer.Ordinal)
            : new HashSet<string>(StringComparer.Ordinal);
    }
}

// File: SomeTests.cs
using Microsoft.VisualStudio.TestTools.UnitTesting;

[TestClass]
public sealed class FastSuite
{
    [ClassInitialize]
    public static void Setup(TestContext _)
    {
        // This now does NOT run if `_droppedClasses` contains "MyOrg.Tests.FastSuite",
        // or if every test below is filtered out by the TeamTestFilter.
        ExpensiveDatabase.Migrate();
    }

    [TestMethod]
    public void Fast_Test() { /* ... */ }

    [TestMethod]
    [TestCategory("Nightly")]
    public void Long_Test() { /* ... */ }
}

When running with RUN_NIGHTLY unset on Linux:

• Fast_Test → runs (assuming FastSuite is not in the drop list).
• Long_Test → reported as Skipped with reason "RUN_NIGHTLY=1 is required to run [TestCategory(\"Nightly\")] tests.".

If FastSuite is in .testfilter.json's droppedClasses:

• Neither test appears in the count or in the report at all.
• ExpensiveDatabase.Migrate() does not run.

Non-goals (v1)

• No MethodInfo/Type exposure on TestFilterContext (would force type loading and defeat the perf point — opt-in metadata-only accessor can be added later behind a follow-up issue).
• No class-level [ClassFilter] attribute (per-test filter already prevents ClassInitialize from running when every test in the class is filtered).
• No "run after AssemblyInitialize" variant (defeats the perf goal and opens awkward lifecycle questions; revisit only on concrete demand).
• No "preview discovery" (dotnet test --list-tests) integration: the filter remains an execution-time decision, mirroring [Ignore]. List-tests still shows the full set.

Alternatives considered

1. Method-attribute on any static type ([GlobalTestFilter] on a method). Rejected: would force scanning every type in the assembly for the marker, breaking the existing TypeCache invariant that only [TestClass] types are inspected during discovery.
2. Discriminated-union return type via sealed class hierarchy. Rejected: per-test allocation in a hot path; the readonly struct + enum tag gives the same expressiveness with zero allocation.
3. Replace the CLI filter. Rejected: would break IDE / Test Explorer / dotnet test integration. Always additive.

Open questions

• Should we report the count of Drop-ed tests through a trace log to mitigate "silent test loss" risk? My current take: yes, info-level adapter trace only (no user-visible noise by default).
• Should multiple providers be composed AND or OR? My current take: AND (a test must be Run from every filter to run). OR would let one careless filter accidentally re-enable tests another filter disabled.

cc / fyi: #3590 #7160 #3528

[Evangelink]

Design update (reflected in #8896): the attribute is now AllowMultiple = false — a single [TestFilterProvider] per test assembly, instead of multiple with AND composition.

Rationale: with multiple filters there is no obvious right ordering across libraries, and any ordering we expose becomes public API forever. With a single entry point:

• No ordering question to standardize.
• No dedup / cross-assembly BFS during discovery (we only inspect the test assembly itself).
• Composition stays in user code, where authors can reference any library-provided ITestFilter types and call them in the order they choose.
• One decision point per test, no precedence rules to document.