diff --git a/docs.json b/docs.json index a87eb70f66..b223b44f6a 100644 --- a/docs.json +++ b/docs.json @@ -715,7 +715,8 @@ "group": "Deploy and scale", "pages": [ "weave/guides/platform", - "weave/guides/platform/weave-self-managed" + "weave/guides/platform/weave-self-managed", + "weave/guides/platform/ingest-sampling" ] }, "weave/guides/platform/weave-projects", diff --git a/weave/guides/platform.mdx b/weave/guides/platform.mdx index 9a8d40751c..715c589bfb 100644 --- a/weave/guides/platform.mdx +++ b/weave/guides/platform.mdx @@ -12,7 +12,7 @@ Weave is available on the following deployment options: - **[W&B Multi-tenant Cloud](https://docs.wandb.ai/platform/hosting/hosting-options/multi_tenant_cloud):** A multi-tenant, fully managed platform deployed in W&B's Google Cloud Platform (Google Cloud) account in a North America region. - **[W&B Dedicated Cloud](https://docs.wandb.ai/platform/hosting/hosting-options/dedicated-cloud):** Generally available on AWS, Google Cloud, and Azure. -- **[Self-Managed instances](/weave/guides/platform/weave-self-managed):** For teams that prefer to host Weave independently, your W&B team provides guidance to evaluate deployment options. +- **[Self-Managed instances](/weave/guides/platform/weave-self-managed):** For teams that prefer to host Weave independently, your W&B team provides guidance to evaluate deployment options. Self-managed instances can use [ingest sampling](/weave/guides/platform/ingest-sampling) to control costs at high trace volume. ## Identity and access management diff --git a/weave/guides/platform/ingest-sampling.mdx b/weave/guides/platform/ingest-sampling.mdx new file mode 100644 index 0000000000..31af3e8d2c --- /dev/null +++ b/weave/guides/platform/ingest-sampling.mdx @@ -0,0 +1,79 @@ +--- +title: "Configure ingest sampling for self-managed Weave" +description: "Keep only a share of incoming traces to control storage and LLM scoring costs on a self-managed Weave instance" +keywords: ["ingest sampling", "sample rate", "self-managed", "trace server", "WEAVE_INGEST_SAMPLE_RATE", "dry run", "DogStatsD"] +--- + +Ingest sampling lets you control the share of traces that arrive at a [self-managed Weave instance](/weave/guides/platform/weave-self-managed) and drop the rest. This can help alleviate storage and processing costs for high-volume clusters by only keeping a specified percentage of traces. + +This guide walks you through how to enable the feature on your self-managed Weave instance. + + +Ingest sampling requires a Weave server version that includes the sampler, and it applies only to traffic from applications using the Weave Python SDK 0.53.0 or later. For details, see [Requirements and limitations](#requirements-and-limitations). + + +Ingest sampling is independent of client-side sampling. The `tracing_sample_rate` parameter in the `@weave.op` decorator still lets an individual traced function sample its own calls, but only a server-side rate applies across the whole deployment and can't be changed or ignored by individual clients. For more information, see [Control sampling rate](/weave/guides/tracking/ops#control-sampling-rate). + +## Why use ingest sampling + +By default, Weave keeps every trace that your applications send, and at higher volumes, maintaining complete records can become expensive. Because ingest sampling drops a trace on the server before it's stored or scored, your costs scale roughly with the share of traffic that you keep. + +You define the ingest sampling rate by setting a number between `0` and `1`: + +- `1.0` keeps all traces and sampling is off (default). +- `0.1` keeps 10% of traces and drops the rest. +- `0.0` drops all traces, except for evaluations. See [Evaluations are always kept](#evaluations-are-always-kept). + +## What ingest sampling applies to + +Ingest sampling applies only to traces emitted from the `@weave.op` decorator in your applications. These are the same traces that appear in the **Traces** tab of the Weave UI. The following traffic is never sampled and is always kept in full: + +- Evaluations. Weave keeps all evaluations because they are deliberate quality measurements, and their scores would be wrong if computed over partial data. +- Spans from agent tracing, which appear in the **Agents** tab. +- Traces sent with your own OpenTelemetry tooling to the raw OTel endpoint (`/otel/v1/traces`). Raw OpenTelemetry traffic carries no evaluation marker, so sampling it could silently drop evaluations. For more information, see [Send OpenTelemetry traces to Weave](/weave/guides/tracking/otel). +- Traffic from Weave SDK versions earlier than the supported versions. See [Requirements and limitations](#requirements-and-limitations). + +## How it works + +The server uses the trace's ID and [deterministic hash-based sampling](https://opentelemetry.io/docs/concepts/sampling/) to decide which traces to drop and keep. If a trace contains several calls that have the same `trace_id`, the server's verdict to keep or drop a trace extends to any nested calls that share the same trace ID. A trace is kept whole or dropped whole, never stored partway. + + +The verdict is stable only while the rate is fixed. A trace that's in flight when the rate changes can be split across the change. Change the sample rate during a low-traffic window. + + +## Configure ingest sampling + +To configure ingest sampling, set the following environment variables on the trace server deployment, for example through your Helm values: + +| Variable | Type | Default | Meaning | +| --- | --- | --- | --- | +| `WEAVE_INGEST_SAMPLE_RATE` | float | `1.0` | The share of traces to keep. An invalid or out-of-range value defaults back to `1.0`. | +| `WEAVE_INGEST_SAMPLE_DRY_RUN` | bool | `false` | When `true`, the server makes the keep or drop decision and counts what it would drop, but drops nothing. | + +Set `WEAVE_INGEST_SAMPLE_RATE` to the share of traces that you want to keep, for example `0.1`. + +### Optional: Preview with a dry run + +You can preview the effect of a sample rate before dropping anything using a dry run. Its only output is the server's sampling metrics, emitted as DogStatsD counters, for example to Datadog. If your deployment doesn't collect those metrics, a dry run has no visible effect, and you can skip it and set the rate directly. + +If you do collect the metrics, run with `WEAVE_INGEST_SAMPLE_RATE=0.1` and `WEAVE_INGEST_SAMPLE_DRY_RUN=true` for a period that covers representative peak traffic, such as one full day, to see how much traffic the server would drop and how much traffic isn't eligible for sampling. Then set `WEAVE_INGEST_SAMPLE_DRY_RUN=false` to begin dropping traces. + +Each counter carries a `route` tag that identifies the endpoint. The counters count individual call records, not whole traces, so ratios between them reflect message volume rather than trace counts. + +| Metric | What it counts | +| --- | --- | +| `ingest_sampling.seen` | Call records the sampler evaluated. | +| `ingest_sampling.evals_kept` | Evaluation calls kept. | +| `ingest_sampling.dropped` | Records dropped. Carries a `dry_run` tag. | +| `ingest_sampling.dropped_bytes` | Bytes dropped. Carries a `dry_run` tag. | +| `ingest_sampling.unsupported` | Traffic from clients too old to be sampled. | +| `ingest_sampling.parse_failures` | Messages received without a usable trace id. | + +## Limitations and expected behaviors + +Before you enable ingest sampling, review the following limitations and behaviors: + +- **SDK version requirements**: The server samples only traffic from the Weave Python SDK 0.53.0 or later, or the Weave TypeScript SDK 0.16.1 or later. These versions add the signals the server uses to group a trace's messages and recognize evaluation calls. Traffic from older SDKs isn't sampled and isn't rejected, so until your apps upgrade, the rate has no effect on their traffic. +- **Savings depend on your client mix**: Traffic that isn't sampled, such as traffic from older SDKs, agent tracing, and raw OpenTelemetry, doesn't reduce cost. If much of your traffic comes from those sources, savings are smaller than the rate implies. A dry run is the best way to measure this first. +- **Monitors and scoring**: A dropped trace is neither stored nor scored. If you rely on monitors covering 100% of traffic, account for this before you enable sampling. +- **No drop signal to the client**: A dropped trace receives a normal success response. The client isn't notified that its trace was dropped. diff --git a/weave/guides/platform/weave-self-managed.mdx b/weave/guides/platform/weave-self-managed.mdx index 9cbf1fd489..f5c1aa6517 100644 --- a/weave/guides/platform/weave-self-managed.mdx +++ b/weave/guides/platform/weave-self-managed.mdx @@ -1184,6 +1184,7 @@ kubectl get pods -n wandb | grep weave-trace ## Additional resources +- [Configure ingest sampling](/weave/guides/platform/ingest-sampling): Keep only a share of incoming traces to control storage and LLM scoring costs at high trace volume. - [Altinity ClickHouse Operator Documentation](https://docs.altinity.com/altinitykubernetesoperator/) - [ClickHouse Documentation](https://clickhouse.com/docs) - [W&B Weave Documentation](https://docs.wandb.ai/weave)