test (docs) Guide for writing e2e tests that take summaries

packages/test/test-end-to-end-tests/README.md

@@ -1,5 +1,17 @@
 # @fluid-private/test-end-to-end-tests
 
+## Table of Contents
+
+-   [Introduction](#introduction)
+-   [How-to](#how-to)
+    -   [Writing Compat-Correct Tests](#writing-compat-correct-tests)
+    -   [Writing Tests That Take Summaries](#writing-tests-that-take-summaries)
+-   [Debugging](#debugging)
+-   ["Real Service" Tests](#real-service-tests)
+    -   [Endpoint names](#endpoint-names)
+
+## Introduction
+
 This package hosts end-to-end tests for the Fluid Framework.
 The tests are end-to-end in the sense that they construct, load, and orchestrate collaborative scenarios involving containers
 in much the same way a real application using Fluid would.
@@ -35,6 +47,12 @@ _"`SharedMap` import from `@fluidframework/map/internal` is restricted from bein
 
 For a worked example of an entire test file written this way, see [`sharedStringEndToEndTests.spec.ts`](src/test/sharedStringEndToEndTests.spec.ts). That same [directory](src/test) contains more complex examples too.
 
+### Writing Tests That Take Summaries
+
+Many e2e tests need to drive **summaries** on demand — configuring interactive containers so they don't summarize on their own, spinning up a dedicated summarizer, taking summaries at controlled points, and loading new containers or summarizers from a specific summary. Doing this incorrectly is a common source of flaky tests.
+
+**For the full pattern — the two container configurations, how to take and inspect a summary, how to load from a specific summary, and the rules that keep these tests deterministic — see [WritingTestsThatTakeSummaries.md](./WritingTestsThatTakeSummaries.md).**
+
 ## Debugging
 
 This package contains a VSCode workspace with launch targets for debugging e2e tests with common configurations.
@@ -53,7 +71,7 @@ The tests under the `real-service-tests` dir target a live production service li
 These are run via `npm run test:realsvc:mocha`, and are included in the CI such that a test failure doesn't
 fail the pipeline - since a service outage or network hiccup could cause a failure when no code defect is present.
 
-### Enpdoint names
+### Endpoint names
 
 When running tests against ODSP or R11s, be mindful of a second parameter/flag usually referred to as the "endpoint name".
 This should match the target environment you want to run against or the test driver might configure things in a way

packages/test/test-end-to-end-tests/WritingCompatCorrectTests.md

@@ -1,5 +1,27 @@
 # Writing Compat-Correct Tests
 
+## Table of Contents
+
+-   [Introduction](#introduction)
+-   [Why this matters](#why-this-matters)
+-   [Using `apis`](#using-apis)
+    -   [❌ Static import — DON'T](#-static-import--dont)
+    -   [✅ Through `apis` — DO](#-through-apis--do)
+    -   [DDS factories in a registry](#dds-factories-in-a-registry)
+    -   [Extending a DataObject](#extending-a-dataobject)
+    -   [Building a container runtime factory](#building-a-container-runtime-factory)
+    -   [Creating a Loader](#creating-a-loader)
+    -   [SharedTree schema and tree utilities](#sharedtree-schema-and-tree-utilities)
+    -   [Creating and loading containers](#creating-and-loading-containers)
+    -   [Two `Loader` instances](#two-loader-instances)
+    -   [Handling recently-added APIs](#handling-recently-added-apis)
+    -   [Type-only references](#type-only-references)
+-   [The lint rule](#the-lint-rule)
+    -   [Overriding the lint rule](#overriding-the-lint-rule)
+-   [Quick checklist](#quick-checklist)
+
+## Introduction
+
 This document explains how to write Fluid Framework end-to-end tests that correctly exercise compatibility across versions. Tests that bypass these patterns may silently pass under the current version and break when run against older or cross-client compat configurations.
 
 If you arrived here from an ESLint `@typescript-eslint/no-restricted-imports` error — typically of the form
@@ -215,7 +237,7 @@ describeCompat("Attach lifecycle", "FullCompat", (getTestObjectProvider, apis) =
 });
 ```
 
-## Handling recently-added APIs
+### Handling recently-added APIs
 
 Some `apis.dds.*` entries (for example, `SharedArray`, `SharedSignal`, and `SharedTree`) — and the matching `apis.dataRuntime.packages.*` packages — may be undefined when running against older compat versions whose Data Runtime didn't expose them.
 
@@ -272,7 +294,7 @@ This package's [`eslint.config.mts`](./eslint.config.mts) enforces the patterns
 - `src/test/benchmark/**` — benchmarks measure the current version's performance.
 - `src/test/migration-shim/**` — these tests intentionally target the current new `SharedTree` (the migration destination).
 
-## Overriding the lint rule
+### Overriding the lint rule
 
 The lint rule allows targeted overrides via `eslint-disable` with a comment explaining intent. The convention is one-line `eslint-disable-next-line` immediately above the import, prefaced by a short reason. Cases where this is appropriate:
 

packages/test/test-end-to-end-tests/WritingTestsThatTakeSummaries.md

@@ -0,0 +1,268 @@
+# Writing Tests That Take Summaries
+
+## Table of Contents
+
+-   [Introduction](#introduction)
+-   [Why a dedicated summarizer](#why-a-dedicated-summarizer)
+-   [How to configure the regular (interactive) containers](#how-to-configure-the-regular-interactive-containers)
+-   [How to configure and create the dedicated summarizer](#how-to-configure-and-create-the-dedicated-summarizer)
+-   [How to take a summary and inspect its result](#how-to-take-a-summary-and-inspect-its-result)
+    -   [Inspecting the summary result](#inspecting-the-summary-result)
+-   [How to load a new container or summarizer from a specific summary](#how-to-load-a-new-container-or-summarizer-from-a-specific-summary)
+    -   [Load a new interactive container from a summary](#load-a-new-interactive-container-from-a-summary)
+    -   [Load a new summarizer from a summary](#load-a-new-summarizer-from-a-summary)
+-   [The rules that keep these tests deterministic](#the-rules-that-keep-these-tests-deterministic)
+    -   [1. Use `syncSummarizer: true`](#1-use-syncsummarizer-true)
+    -   [2. Disable automatic summaries on every interactive container](#2-disable-automatic-summaries-on-every-interactive-container)
+    -   [3. Call `ensureSynchronized()` before every summary](#3-call-ensuresynchronized-before-every-summary)
+    -   [4. Use `summaryVersion` to chain loads](#4-use-summaryversion-to-chain-loads)
+    -   [5. Close one summarizer before starting another](#5-close-one-summarizer-before-starting-another)
+-   [A complete minimal example](#a-complete-minimal-example)
+-   [Reference: key imports](#reference-key-imports)
+
+## Introduction
+
+This document explains the pattern for writing end-to-end (e2e) tests that generate **summaries**. The canonical examples live under [`src/test/summarization/`](./src/test/summarization/).
+
+## Why a dedicated summarizer
+
+In production, summaries are produced by a single elected **summarizer client** - a non-interactive container that the runtime spins up in the background and generates summaries based on heuristics. In a test you want to control _exactly when_ a summary happens and _what_ goes into it, so you:
+
+1. Create your normal interactive container(s) with the runtime's automatic summarizer **disabled**, so nothing summarizes behind your back.
+2. Create a separate **summarizer container** that summarizes only when you call `summarizeNow` on it.
+3. Synchronize all clients before each summary so the summary is deterministic.
+
+This separation is the heart of the pattern. Everything below follows from it.
+
+## How to configure the regular (interactive) containers
+
+Every interactive container the test creates or loads must have the runtime's automatic summarizer turned **off**. Otherwise a background summary can race your `summarizeNow` calls and your assertions, making the test flaky.
+
+Disable it via `summaryConfigOverrides: { state: "disabled" }` in the container config:
+
+```ts
+const testContainerConfig: ITestContainerConfig = {
+	...
+	runtimeOptions: {
+		...
+		// The piece that matters for summarization: turn off the automatic summarizer.
+		summaryOptions: {
+			summaryConfigOverrides: { state: "disabled" },
+		},
+	},
+};
+```
+
+Use this config for **every** interactive container — both the one you create and any you later load from a summary (`provider.makeTestContainer(testContainerConfig)` / `provider.loadTestContainer(testContainerConfig, ...)`).
+
+## How to configure and create the dedicated summarizer
+
+The summarizer must _not_ inherit the `state: "disabled"` override from the interactive container's config; it needs to be able to summarize when asked. The simplest form lets `createSummarizer` apply a sensible default summary config for you (`state: "disableHeuristics"`, etc.):
+
+```ts
+const { summarizer } = await createSummarizer(provider, container);
+```
+
+If your test creates a custom `testContainerConfig` for interactive containers, the summarizer should resuse it otherwise config mimatch can lead to issues. It should however **supply a `disableHeuristics` summary config** or **clear its summary override**:
+
+```ts
+const summarizerContainerConfig: ITestContainerConfig = {
+	...testContainerConfig,
+	runtimeOptions: {
+		...testContainerConfig.runtimeOptions,
+		// Either supply a `disableHeuristics` config so the summarizer only summarizes on demand...
+		summaryOptions: { summaryConfigOverrides: { state: "disableHeuristics" } },
+		// ...or clear the override entirely and let createSummarizer apply its default config:
+		// summaryOptions: undefined,
+	},
+};
+```
+
+Either way, the rule is the same: the summarizer must not carry `state: "disabled"`.
+
+`createSummarizer(provider, container, config?, summaryVersion?, logger?)` returns `{ container, summarizer }`.
+The `summarizer` is the `ISummarizer` you call `summarizeNow` on and the returned `container` is the summarizer's own container (useful for reconnect/election below).
+
+> If your data store needs a custom registry/factory, use `createSummarizerFromFactory` instead - it takes the data store factory and (optionally) a container-runtime factory directly. See its uses in [`summaries.spec.ts`](./src/test/summarization/summaries.spec.ts).
+
+## How to take a summary and inspect its result
+
+Use the `summarizeNow` helper from `@fluidframework/test-utils/internal`. It drives the full submit → broadcast → ack/nack handshake, throws on failure, and returns a `SummaryInfo`:
+
+```ts
+interface SummaryInfo {
+	summaryTree: ISummaryTree; // the generated summary tree — inspect it for handles/blobs
+	summaryVersion: string;    // the acked summary handle — use this to load from this summary
+	summaryRefSeq: number;     // reference sequence number of this summary
+}
+```
+
+The minimal round looks like this:
+
+```ts
+// 1. Make whatever changes you want captured.
+dataObject.root.set("key", "value");
+
+// 2. Make sure every client has seen those ops before summarizing.
+await provider.ensureSynchronized();
+
+// 3. Summarize. summarizeNow throws if the summary fails.
+const { summaryTree, summaryVersion } = await summarizeNow(summarizer);
+```
+
+> You can call `summarizeOnDemand` on the `ISummarizer` directly and manage the submit, broadcast, ack / nack results independently.
+
+### Inspecting the summary result
+
+The result of the `summarizeNow` (or `summarizeOnDemand`) contains the generated summary tree (`ISummaryTree`).
+Tests can inspect the summary tree if needed. However, it should be careful to not rely on the summary tree structure as that can change.
+
+If you only care that summarizing succeeds, assert against the promise directly:
+
+```ts
+await assert.doesNotReject(summarizeNow(summarizer), "Summary should succeed");
+```
+
+## How to load a new container or summarizer from a specific summary
+
+`summaryVersion` is the key that ties everything together: pass it when loading to force a client to start from _that exact_ summary rather than the latest.
+
+### Load a new interactive container from a summary
+
+Pass the version through `LoaderHeader.version` to `loadTestContainer`:
+
+```ts
+const loaded = await provider.loadTestContainer(testContainerConfig, {
+	[LoaderHeader.version]: summaryVersion,
+});
+```
+
+This is how you validate that a summary round-trips: load a fresh container from the summary you just took and assert its state matches the source container.
+
+### Load a new summarizer from a summary
+
+Pass `summaryVersion` as the 4th argument to `createSummarizer`:
+
+```ts
+const { summarizer: summarizer2 } = await createSummarizer(
+	provider,
+	mainContainer,
+	undefined /* config */,
+	summaryVersion,
+);
+```
+
+A summarizer started from a given summary will produce its next summary incrementally on top of it.
+
+## The rules that keep these tests deterministic
+
+These are the things that, if skipped, make summarization tests flaky or wrong. Treat them as a checklist.
+
+### 1. Use `syncSummarizer: true`
+
+Get the provider with `getTestObjectProvider({ syncSummarizer: true })`. This ensures that when you call `provider.ensureSynchronized()`, the summarizer is also brought up to the latest state along with the other clients. Without it, `ensureSynchronized` does not wait for the summarizer, so a subsequent `summarizeNow` may run before the summarizer has processed your latest ops.
+
+```ts
+beforeEach("getTestObjectProvider", async function () {
+	provider = getTestObjectProvider({ syncSummarizer: true });
+});
+```
+
+### 2. Disable automatic summaries on every interactive container
+
+As covered above — `summaryConfigOverrides: { state: "disabled" }`. If a regular container is allowed to summarize, a background summary can land between your changes and your `summarizeNow`, and your assertions about what's in the summary become non-deterministic.
+
+### 3. Call `ensureSynchronized()` before every summary
+
+```ts
+await provider.ensureSynchronized();
+await summarizeNow(summarizer);
+```
+
+`summarizeNow` summarizes whatever the summarizer has processed _so far_. If you don't synchronize first, ops you just sent may not have reached the summarizer yet, and they'll silently be excluded from the summary. Always synchronize first.
+
+### 4. Use `summaryVersion` to chain loads
+
+When you load a container or summarizer to validate a summary, load it from that summary's `summaryVersion` (see above) — don't rely on "latest". On real services the latest summary may differ from the one you intend to test (or may have been replaced), so be explicit.
+
+### 5. Close one summarizer before starting another
+
+Two live summarizers fight over election and can interfere with each other. When you're done with a summarizer and want a fresh one (e.g. to load from a newer summary), **close the old one first**:
+
+```ts
+summarizer.close();
+const { summarizer: summarizer2 } = await createSummarizer(
+	provider,
+	mainContainer,
+	undefined,
+	summaryVersion,
+);
+await summarizeNow(summarizer2);
+```
+
+## A complete minimal example
+
+Putting it together — create, summarize, load-and-validate, then summarize from a new summarizer:
+
+```ts
+describeCompat("My summarization test", "NoCompat", (getTestObjectProvider) => {
+	const testContainerConfig: ITestContainerConfig = {
+		runtimeOptions: {
+			summaryOptions: { summaryConfigOverrides: { state: "disabled" } },
+		},
+	};
+
+	let provider: ITestObjectProvider;
+	beforeEach("getTestObjectProvider", async () => {
+		provider = getTestObjectProvider({ syncSummarizer: true });
+	});
+
+	it("round-trips a change through a summary", async () => {
+		// 1. Create an interactive container with auto-summaries disabled.
+		const container = await provider.makeTestContainer(testContainerConfig);
+		const dataObject = await getContainerEntryPointBackCompat<ITestFluidObject>(container);
+		await waitForContainerConnection(container);
+
+		// 2. Create a dedicated summarizer.
+		const { summarizer } = await createSummarizer(provider, container);
+
+		// 3. Make a change, synchronize, summarize.
+		dataObject.root.set("key", "value");
+		await provider.ensureSynchronized();
+		const { summaryVersion } = await summarizeNow(summarizer);
+
+		// 4. Load a fresh container from that exact summary and validate.
+		const loaded = await provider.loadTestContainer(testContainerConfig, {
+			[LoaderHeader.version]: summaryVersion,
+		});
+		const loadedObject = await getContainerEntryPointBackCompat<ITestFluidObject>(loaded);
+		assert.strictEqual(loadedObject.root.get("key"), "value");
+
+		// 5. Hand off to a new summarizer loaded from that summary.
+		summarizer.close();
+		const { summarizer: summarizer2 } = await createSummarizer(
+			provider,
+			container,
+			undefined,
+			summaryVersion,
+		);
+		await assert.doesNotReject(summarizeNow(summarizer2));
+	});
+});
+```
+
+## Reference: key imports
+
+All from `@fluidframework/test-utils/internal` unless noted:
+
+| Symbol | Purpose |
+|---|---|
+| `createSummarizer(provider, container, config?, summaryVersion?, logger?)` | Create a dedicated summarizer; returns `{ container, summarizer }`. |
+| `createSummarizerFromFactory(...)` | Same, when you need a custom data store / container-runtime factory. |
+| `summarizeNow(summarizer, reason?)` | Take an on-demand summary; returns `SummaryInfo`; throws on failure. |
+| `SummaryInfo` | `{ summaryTree, summaryVersion, summaryRefSeq }`. |
+| `ITestContainerConfig` | Container config — set `runtimeOptions.summaryOptions` here. |
+| `LoaderHeader.version` (from `@fluidframework/container-definitions/internal`) | Header to load a container from a specific summary. |
+| `ISummarizer` (from `@fluidframework/container-runtime/internal`) | The summarizer handle; `summarizer.close()` to release it. |
+| `provider.ensureSynchronized()` | Flush ops to all clients before summarizing. |
+| `getTestObjectProvider({ syncSummarizer: true })` | Provider configured for deterministic summaries. |

packages/test/test-end-to-end-tests/src/test/summarization/README.md

@@ -0,0 +1,18 @@
+# @fluidframework/test-end-to-end-tests/test/src/test/summarization
+
+End-to-end tests that exercise **summarization** — disabling the automatic summarizer on
+interactive containers, driving summaries on demand from a dedicated summarizer, inspecting
+the resulting summary, and loading containers/summarizers from a specific summary.
+
+## Writing a summarization test
+
+If you're here to learn how to write a test that takes summaries, read
+**[WritingTestsThatTakeSummaries.md](../../../WritingTestsThatTakeSummaries.md)**. It covers the
+container configurations to use, how to take and inspect a summary with `summarizeNow`, how to
+load from a specific `summaryVersion`, and the rules that keep these tests deterministic
+(`syncSummarizer: true`, calling `ensureSynchronized()` before summarizing, closing one
+summarizer before starting another, etc.).
+
+For working examples, see the specs in this folder — for instance
+[summarizeIncrementally.spec.ts](summarizeIncrementally.spec.ts) and
+[summaries.spec.ts](summaries.spec.ts).