diff --git a/packages/test/test-end-to-end-tests/README.md b/packages/test/test-end-to-end-tests/README.md index cf7844762ca..f27b6fae36a 100644 --- a/packages/test/test-end-to-end-tests/README.md +++ b/packages/test/test-end-to-end-tests/README.md @@ -1,5 +1,17 @@ # @fluid-private/test-end-to-end-tests +## Table of Contents + +- [Introduction](#introduction) +- [How-to](#how-to) + - [Writing Compat-Correct Tests](#writing-compat-correct-tests) + - [Writing Tests That Take Summaries](#writing-tests-that-take-summaries) +- [Debugging](#debugging) +- ["Real Service" Tests](#real-service-tests) + - [Endpoint names](#endpoint-names) + +## Introduction + This package hosts end-to-end tests for the Fluid Framework. The tests are end-to-end in the sense that they construct, load, and orchestrate collaborative scenarios involving containers in much the same way a real application using Fluid would. @@ -35,6 +47,12 @@ _"`SharedMap` import from `@fluidframework/map/internal` is restricted from bein For a worked example of an entire test file written this way, see [`sharedStringEndToEndTests.spec.ts`](src/test/sharedStringEndToEndTests.spec.ts). That same [directory](src/test) contains more complex examples too. +### Writing Tests That Take Summaries + +Many e2e tests need to drive **summaries** on demand — configuring interactive containers so they don't summarize on their own, spinning up a dedicated summarizer, taking summaries at controlled points, and loading new containers or summarizers from a specific summary. Doing this incorrectly is a common source of flaky tests. + +**For the full pattern — the two container configurations, how to take and inspect a summary, how to load from a specific summary, and the rules that keep these tests deterministic — see [WritingTestsThatTakeSummaries.md](./WritingTestsThatTakeSummaries.md).** + ## Debugging This package contains a VSCode workspace with launch targets for debugging e2e tests with common configurations. @@ -53,7 +71,7 @@ The tests under the `real-service-tests` dir target a live production service li These are run via `npm run test:realsvc:mocha`, and are included in the CI such that a test failure doesn't fail the pipeline - since a service outage or network hiccup could cause a failure when no code defect is present. -### Enpdoint names +### Endpoint names When running tests against ODSP or R11s, be mindful of a second parameter/flag usually referred to as the "endpoint name". This should match the target environment you want to run against or the test driver might configure things in a way diff --git a/packages/test/test-end-to-end-tests/WritingCompatCorrectTests.md b/packages/test/test-end-to-end-tests/WritingCompatCorrectTests.md index 735104886e4..4d51feac62c 100644 --- a/packages/test/test-end-to-end-tests/WritingCompatCorrectTests.md +++ b/packages/test/test-end-to-end-tests/WritingCompatCorrectTests.md @@ -1,5 +1,27 @@ # Writing Compat-Correct Tests +## Table of Contents + +- [Introduction](#introduction) +- [Why this matters](#why-this-matters) +- [Using `apis`](#using-apis) + - [❌ Static import — DON'T](#-static-import--dont) + - [✅ Through `apis` — DO](#-through-apis--do) + - [DDS factories in a registry](#dds-factories-in-a-registry) + - [Extending a DataObject](#extending-a-dataobject) + - [Building a container runtime factory](#building-a-container-runtime-factory) + - [Creating a Loader](#creating-a-loader) + - [SharedTree schema and tree utilities](#sharedtree-schema-and-tree-utilities) + - [Creating and loading containers](#creating-and-loading-containers) + - [Two `Loader` instances](#two-loader-instances) + - [Handling recently-added APIs](#handling-recently-added-apis) + - [Type-only references](#type-only-references) +- [The lint rule](#the-lint-rule) + - [Overriding the lint rule](#overriding-the-lint-rule) +- [Quick checklist](#quick-checklist) + +## Introduction + This document explains how to write Fluid Framework end-to-end tests that correctly exercise compatibility across versions. Tests that bypass these patterns may silently pass under the current version and break when run against older or cross-client compat configurations. If you arrived here from an ESLint `@typescript-eslint/no-restricted-imports` error — typically of the form @@ -215,7 +237,7 @@ describeCompat("Attach lifecycle", "FullCompat", (getTestObjectProvider, apis) = }); ``` -## Handling recently-added APIs +### Handling recently-added APIs Some `apis.dds.*` entries (for example, `SharedArray`, `SharedSignal`, and `SharedTree`) — and the matching `apis.dataRuntime.packages.*` packages — may be undefined when running against older compat versions whose Data Runtime didn't expose them. @@ -272,7 +294,7 @@ This package's [`eslint.config.mts`](./eslint.config.mts) enforces the patterns - `src/test/benchmark/**` — benchmarks measure the current version's performance. - `src/test/migration-shim/**` — these tests intentionally target the current new `SharedTree` (the migration destination). -## Overriding the lint rule +### Overriding the lint rule The lint rule allows targeted overrides via `eslint-disable` with a comment explaining intent. The convention is one-line `eslint-disable-next-line` immediately above the import, prefaced by a short reason. Cases where this is appropriate: diff --git a/packages/test/test-end-to-end-tests/WritingTestsThatTakeSummaries.md b/packages/test/test-end-to-end-tests/WritingTestsThatTakeSummaries.md new file mode 100644 index 00000000000..9b2fd681cf0 --- /dev/null +++ b/packages/test/test-end-to-end-tests/WritingTestsThatTakeSummaries.md @@ -0,0 +1,268 @@ +# Writing Tests That Take Summaries + +## Table of Contents + +- [Introduction](#introduction) +- [Why a dedicated summarizer](#why-a-dedicated-summarizer) +- [How to configure the regular (interactive) containers](#how-to-configure-the-regular-interactive-containers) +- [How to configure and create the dedicated summarizer](#how-to-configure-and-create-the-dedicated-summarizer) +- [How to take a summary and inspect its result](#how-to-take-a-summary-and-inspect-its-result) + - [Inspecting the summary result](#inspecting-the-summary-result) +- [How to load a new container or summarizer from a specific summary](#how-to-load-a-new-container-or-summarizer-from-a-specific-summary) + - [Load a new interactive container from a summary](#load-a-new-interactive-container-from-a-summary) + - [Load a new summarizer from a summary](#load-a-new-summarizer-from-a-summary) +- [The rules that keep these tests deterministic](#the-rules-that-keep-these-tests-deterministic) + - [1. Use `syncSummarizer: true`](#1-use-syncsummarizer-true) + - [2. Disable automatic summaries on every interactive container](#2-disable-automatic-summaries-on-every-interactive-container) + - [3. Call `ensureSynchronized()` before every summary](#3-call-ensuresynchronized-before-every-summary) + - [4. Use `summaryVersion` to chain loads](#4-use-summaryversion-to-chain-loads) + - [5. Close one summarizer before starting another](#5-close-one-summarizer-before-starting-another) +- [A complete minimal example](#a-complete-minimal-example) +- [Reference: key imports](#reference-key-imports) + +## Introduction + +This document explains the pattern for writing end-to-end (e2e) tests that generate **summaries**. The canonical examples live under [`src/test/summarization/`](./src/test/summarization/). + +## Why a dedicated summarizer + +In production, summaries are produced by a single elected **summarizer client** - a non-interactive container that the runtime spins up in the background and generates summaries based on heuristics. In a test you want to control _exactly when_ a summary happens and _what_ goes into it, so you: + +1. Create your normal interactive container(s) with the runtime's automatic summarizer **disabled**, so nothing summarizes behind your back. +2. Create a separate **summarizer container** that summarizes only when you call `summarizeNow` on it. +3. Synchronize all clients before each summary so the summary is deterministic. + +This separation is the heart of the pattern. Everything below follows from it. + +## How to configure the regular (interactive) containers + +Every interactive container the test creates or loads must have the runtime's automatic summarizer turned **off**. Otherwise a background summary can race your `summarizeNow` calls and your assertions, making the test flaky. + +Disable it via `summaryConfigOverrides: { state: "disabled" }` in the container config: + +```ts +const testContainerConfig: ITestContainerConfig = { + ... + runtimeOptions: { + ... + // The piece that matters for summarization: turn off the automatic summarizer. + summaryOptions: { + summaryConfigOverrides: { state: "disabled" }, + }, + }, +}; +``` + +Use this config for **every** interactive container — both the one you create and any you later load from a summary (`provider.makeTestContainer(testContainerConfig)` / `provider.loadTestContainer(testContainerConfig, ...)`). + +## How to configure and create the dedicated summarizer + +The summarizer must _not_ inherit the `state: "disabled"` override from the interactive container's config; it needs to be able to summarize when asked. The simplest form lets `createSummarizer` apply a sensible default summary config for you (`state: "disableHeuristics"`, etc.): + +```ts +const { summarizer } = await createSummarizer(provider, container); +``` + +If your test creates a custom `testContainerConfig` for interactive containers, the summarizer should resuse it otherwise config mimatch can lead to issues. It should however **supply a `disableHeuristics` summary config** or **clear its summary override**: + +```ts +const summarizerContainerConfig: ITestContainerConfig = { + ...testContainerConfig, + runtimeOptions: { + ...testContainerConfig.runtimeOptions, + // Either supply a `disableHeuristics` config so the summarizer only summarizes on demand... + summaryOptions: { summaryConfigOverrides: { state: "disableHeuristics" } }, + // ...or clear the override entirely and let createSummarizer apply its default config: + // summaryOptions: undefined, + }, +}; +``` + +Either way, the rule is the same: the summarizer must not carry `state: "disabled"`. + +`createSummarizer(provider, container, config?, summaryVersion?, logger?)` returns `{ container, summarizer }`. +The `summarizer` is the `ISummarizer` you call `summarizeNow` on and the returned `container` is the summarizer's own container (useful for reconnect/election below). + +> If your data store needs a custom registry/factory, use `createSummarizerFromFactory` instead - it takes the data store factory and (optionally) a container-runtime factory directly. See its uses in [`summaries.spec.ts`](./src/test/summarization/summaries.spec.ts). + +## How to take a summary and inspect its result + +Use the `summarizeNow` helper from `@fluidframework/test-utils/internal`. It drives the full submit → broadcast → ack/nack handshake, throws on failure, and returns a `SummaryInfo`: + +```ts +interface SummaryInfo { + summaryTree: ISummaryTree; // the generated summary tree — inspect it for handles/blobs + summaryVersion: string; // the acked summary handle — use this to load from this summary + summaryRefSeq: number; // reference sequence number of this summary +} +``` + +The minimal round looks like this: + +```ts +// 1. Make whatever changes you want captured. +dataObject.root.set("key", "value"); + +// 2. Make sure every client has seen those ops before summarizing. +await provider.ensureSynchronized(); + +// 3. Summarize. summarizeNow throws if the summary fails. +const { summaryTree, summaryVersion } = await summarizeNow(summarizer); +``` + +> You can call `summarizeOnDemand` on the `ISummarizer` directly and manage the submit, broadcast, ack / nack results independently. + +### Inspecting the summary result + +The result of the `summarizeNow` (or `summarizeOnDemand`) contains the generated summary tree (`ISummaryTree`). +Tests can inspect the summary tree if needed. However, it should be careful to not rely on the summary tree structure as that can change. + +If you only care that summarizing succeeds, assert against the promise directly: + +```ts +await assert.doesNotReject(summarizeNow(summarizer), "Summary should succeed"); +``` + +## How to load a new container or summarizer from a specific summary + +`summaryVersion` is the key that ties everything together: pass it when loading to force a client to start from _that exact_ summary rather than the latest. + +### Load a new interactive container from a summary + +Pass the version through `LoaderHeader.version` to `loadTestContainer`: + +```ts +const loaded = await provider.loadTestContainer(testContainerConfig, { + [LoaderHeader.version]: summaryVersion, +}); +``` + +This is how you validate that a summary round-trips: load a fresh container from the summary you just took and assert its state matches the source container. + +### Load a new summarizer from a summary + +Pass `summaryVersion` as the 4th argument to `createSummarizer`: + +```ts +const { summarizer: summarizer2 } = await createSummarizer( + provider, + mainContainer, + undefined /* config */, + summaryVersion, +); +``` + +A summarizer started from a given summary will produce its next summary incrementally on top of it. + +## The rules that keep these tests deterministic + +These are the things that, if skipped, make summarization tests flaky or wrong. Treat them as a checklist. + +### 1. Use `syncSummarizer: true` + +Get the provider with `getTestObjectProvider({ syncSummarizer: true })`. This ensures that when you call `provider.ensureSynchronized()`, the summarizer is also brought up to the latest state along with the other clients. Without it, `ensureSynchronized` does not wait for the summarizer, so a subsequent `summarizeNow` may run before the summarizer has processed your latest ops. + +```ts +beforeEach("getTestObjectProvider", async function () { + provider = getTestObjectProvider({ syncSummarizer: true }); +}); +``` + +### 2. Disable automatic summaries on every interactive container + +As covered above — `summaryConfigOverrides: { state: "disabled" }`. If a regular container is allowed to summarize, a background summary can land between your changes and your `summarizeNow`, and your assertions about what's in the summary become non-deterministic. + +### 3. Call `ensureSynchronized()` before every summary + +```ts +await provider.ensureSynchronized(); +await summarizeNow(summarizer); +``` + +`summarizeNow` summarizes whatever the summarizer has processed _so far_. If you don't synchronize first, ops you just sent may not have reached the summarizer yet, and they'll silently be excluded from the summary. Always synchronize first. + +### 4. Use `summaryVersion` to chain loads + +When you load a container or summarizer to validate a summary, load it from that summary's `summaryVersion` (see above) — don't rely on "latest". On real services the latest summary may differ from the one you intend to test (or may have been replaced), so be explicit. + +### 5. Close one summarizer before starting another + +Two live summarizers fight over election and can interfere with each other. When you're done with a summarizer and want a fresh one (e.g. to load from a newer summary), **close the old one first**: + +```ts +summarizer.close(); +const { summarizer: summarizer2 } = await createSummarizer( + provider, + mainContainer, + undefined, + summaryVersion, +); +await summarizeNow(summarizer2); +``` + +## A complete minimal example + +Putting it together — create, summarize, load-and-validate, then summarize from a new summarizer: + +```ts +describeCompat("My summarization test", "NoCompat", (getTestObjectProvider) => { + const testContainerConfig: ITestContainerConfig = { + runtimeOptions: { + summaryOptions: { summaryConfigOverrides: { state: "disabled" } }, + }, + }; + + let provider: ITestObjectProvider; + beforeEach("getTestObjectProvider", async () => { + provider = getTestObjectProvider({ syncSummarizer: true }); + }); + + it("round-trips a change through a summary", async () => { + // 1. Create an interactive container with auto-summaries disabled. + const container = await provider.makeTestContainer(testContainerConfig); + const dataObject = await getContainerEntryPointBackCompat(container); + await waitForContainerConnection(container); + + // 2. Create a dedicated summarizer. + const { summarizer } = await createSummarizer(provider, container); + + // 3. Make a change, synchronize, summarize. + dataObject.root.set("key", "value"); + await provider.ensureSynchronized(); + const { summaryVersion } = await summarizeNow(summarizer); + + // 4. Load a fresh container from that exact summary and validate. + const loaded = await provider.loadTestContainer(testContainerConfig, { + [LoaderHeader.version]: summaryVersion, + }); + const loadedObject = await getContainerEntryPointBackCompat(loaded); + assert.strictEqual(loadedObject.root.get("key"), "value"); + + // 5. Hand off to a new summarizer loaded from that summary. + summarizer.close(); + const { summarizer: summarizer2 } = await createSummarizer( + provider, + container, + undefined, + summaryVersion, + ); + await assert.doesNotReject(summarizeNow(summarizer2)); + }); +}); +``` + +## Reference: key imports + +All from `@fluidframework/test-utils/internal` unless noted: + +| Symbol | Purpose | +|---|---| +| `createSummarizer(provider, container, config?, summaryVersion?, logger?)` | Create a dedicated summarizer; returns `{ container, summarizer }`. | +| `createSummarizerFromFactory(...)` | Same, when you need a custom data store / container-runtime factory. | +| `summarizeNow(summarizer, reason?)` | Take an on-demand summary; returns `SummaryInfo`; throws on failure. | +| `SummaryInfo` | `{ summaryTree, summaryVersion, summaryRefSeq }`. | +| `ITestContainerConfig` | Container config — set `runtimeOptions.summaryOptions` here. | +| `LoaderHeader.version` (from `@fluidframework/container-definitions/internal`) | Header to load a container from a specific summary. | +| `ISummarizer` (from `@fluidframework/container-runtime/internal`) | The summarizer handle; `summarizer.close()` to release it. | +| `provider.ensureSynchronized()` | Flush ops to all clients before summarizing. | +| `getTestObjectProvider({ syncSummarizer: true })` | Provider configured for deterministic summaries. | diff --git a/packages/test/test-end-to-end-tests/src/test/summarization/README.md b/packages/test/test-end-to-end-tests/src/test/summarization/README.md new file mode 100644 index 00000000000..13465906fec --- /dev/null +++ b/packages/test/test-end-to-end-tests/src/test/summarization/README.md @@ -0,0 +1,18 @@ +# @fluidframework/test-end-to-end-tests/test/src/test/summarization + +End-to-end tests that exercise **summarization** — disabling the automatic summarizer on +interactive containers, driving summaries on demand from a dedicated summarizer, inspecting +the resulting summary, and loading containers/summarizers from a specific summary. + +## Writing a summarization test + +If you're here to learn how to write a test that takes summaries, read +**[WritingTestsThatTakeSummaries.md](../../../WritingTestsThatTakeSummaries.md)**. It covers the +container configurations to use, how to take and inspect a summary with `summarizeNow`, how to +load from a specific `summaryVersion`, and the rules that keep these tests deterministic +(`syncSummarizer: true`, calling `ensureSynchronized()` before summarizing, closing one +summarizer before starting another, etc.). + +For working examples, see the specs in this folder — for instance +[summarizeIncrementally.spec.ts](summarizeIncrementally.spec.ts) and +[summaries.spec.ts](summaries.spec.ts).