feat(core): typed serde, pagination, webhooks, tracing, and fixes

CHANGELOG.md

@@ -0,0 +1,100 @@
+# Changelog
+
+All notable changes to this project are documented in this file.
+
+The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/),
+and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
+
+## [Unreleased]
+
+A round of platform improvements to `dexpace-sdk-core`: new optional building
+blocks (typed serialization, webhook verification, pagination, two pipeline
+policies), tightened retry and tracing behaviour, and a batch of correctness
+fixes across bodies, SSE parsing, Digest auth, and error reporting. Everything
+lands in `core`; the transport packages are unchanged. No public symbol was
+removed, so existing code continues to work without modification.
+
+### Added
+
+- **Tristate values** (`serde.tristate`). A three-way type distinguishing
+  "set to a value", "explicitly set to null", and "absent", so partial updates
+  (PATCH-style payloads) can round-trip an explicit `null` without conflating it
+  with an omitted field.
+- **Typed model codec** (`serde.codec`). A small encode/decode layer over the
+  existing `Serde` protocol for converting between typed models and wire bytes,
+  built on the standard library only. This is the largest new surface and is
+  worth a careful read before depending on it.
+- **Webhook signature verification** (`http.webhooks`). Helpers to verify the
+  authenticity of inbound webhook payloads using constant-time comparison.
+- **Pagination** (`pagination`). A paginator abstraction with pluggable
+  next-page strategies, a `Link` header parser, and a page model, so list
+  endpoints can be iterated without each caller re-implementing cursor handling.
+- **Idempotency-key policy** (`pipeline.policies.idempotency`, plus its async
+  twin). Stamps a generated idempotency key onto retriable, non-idempotent
+  requests so safe automatic retries don't double-apply a side effect.
+- **Client-identity policy** (`pipeline.policies.client_identity`, plus its
+  async twin). Sets a consistent `User-Agent` / client-identity header derived
+  from the configured application id and SDK version.
+- **HTTP tracer** (`instrumentation.http_tracer`). An adapter-style tracer base
+  whose per-event methods default to no-ops, so a subclass overrides only the
+  events it cares about. Wired through the tracing policy for span emission.
+- **Log correlation** (`instrumentation.correlation`). A `contextvar`-backed
+  correlation id that flows through the pipeline and is attached to log records,
+  so logs from a single logical request can be tied together.
+
+### Changed
+
+- **Retry tuning** (`pipeline.policies.retry` / `async_retry`). More
+  configurable backoff and clearer rules for which responses and exceptions are
+  retried, including respecting `Retry-After`. The async retry path now observes
+  cancellation cleanly between attempts.
+- **Tracing and redirect policies** now emit tracer events and carry correlation
+  through redirects, with credentials stripped on cross-origin redirects.
+- **Default pipelines** (`pipeline.defaults`). The standard sync/async stacks now
+  assemble the new idempotency and client-identity policies alongside the
+  existing retry, redirect, logging, and tracing policies.
+- **Loggable bodies** (`http.request.loggable_request_body`,
+  `http.response.loggable_response_body`). Capture is bounded and repeatable
+  reads behave correctly; the byte cap is honoured on the tap without truncating
+  the primary write path.
+- **Error reporting** (`errors.http`). HTTP errors now expose whether they are
+  `retryable` and carry a bounded body snapshot for diagnostics, with the
+  snapshot capped so an error never holds an unbounded payload.
+
+### Fixed
+
+- **SSE parsing** (`http.sse.parser`) now strips a leading UTF-8 byte-order mark
+  and cleans up the async stream deterministically on cancellation or exit.
+- **Digest auth** (`http.auth.digest`) honours the server-advertised charset
+  when computing the digest, fixing authentication against servers that send
+  non-ASCII credentials.
+- **MediaType** (`http.common.media_type`) handles parameter parsing edge cases
+  (quoting, casing, and whitespace) more robustly.
+- **Async response cancellation** (`http.response.async_response`,
+  `async_response_body`). Cancelling an in-flight read now releases the
+  underlying resources instead of leaking them, and re-raises `CancelledError`
+  after cleanup.
+
+### Verified
+
+- `mypy --strict`, `ruff check`, `ruff format --check`, and `pytest` run in CI
+  across the supported Python matrix (3.12–3.14). New modules ship with tests
+  under each package's `tests/` tree, and `py.typed` continues to ship so
+  downstream type-checkers consume the annotations.
+
+### Honest scope boundaries
+
+The following were intentionally left out of this round and are **not** included:
+
+- **Default error map** — error classification beyond the `retryable`
+  flag and body snapshot was deferred; callers still map status codes to domain
+  errors themselves.
+- **`sendfile` fast-path** — file bodies are streamed via the existing
+  `iter_bytes` path; no zero-copy `sendfile` transport optimisation was added.
+- **MCP support** — no Model Context Protocol integration is included.
+- **Java SDK items** — the Java counterpart lives in a separate repository and
+  was out of scope here.
+- **Code generation** — no client/model code generation was added; all surfaces
+  in this release are hand-written.
+
+[Unreleased]: https://github.com/dexpace/python-sdk/compare/main...HEAD

packages/dexpace-sdk-core/src/dexpace/sdk/core/errors/http.py

@@ -5,8 +5,9 @@
 
 from __future__ import annotations
 
-from typing import TYPE_CHECKING, Any, Generic, TypeVar
+from typing import TYPE_CHECKING, Any, Final, Generic, TypeVar
 
+from ..http.response.loggable_response_body import LoggableResponseBody
 from .base import SdkError
 
 if TYPE_CHECKING:
@@ -26,6 +27,12 @@
 else:
     ModelT = TypeVar("ModelT")
 
+# Status codes for which a retry is worthwhile by default: request timeout,
+# rate limiting, and the transient 5xx family. Mirrors the retry policy's
+# ``_DEFAULT_STATUS_RETRIES`` so ``retryable`` and the policy agree out of
+# the box; callers can override per error via the ``retryable`` kwarg.
+_DEFAULT_RETRYABLE_STATUS: Final[frozenset[int]] = frozenset({408, 429, 500, 502, 503, 504})
+
 
 # UP046 wants PEP 695 ``class Foo[T = Any](...)`` form, but that syntax
 # requires Python 3.13+ at runtime; we still support 3.12.
@@ -49,12 +56,18 @@ class HttpResponseError(SdkError, Generic[ModelT]):  # noqa: UP046
         model: Optional deserialised body payload (set by consumer
             libraries when they parse the error body). Typed as
             ``ModelT | None``.
+        retryable: Whether retrying the request might succeed. Derived from
+            the response status by default (request timeout, rate limiting,
+            and transient 5xx are retryable) so the retry policy can read the
+            flag directly instead of re-deriving it; callers may override it
+            explicitly via the ``retryable`` constructor keyword.
     """
 
     status: Status | None
     reason: str | None
     response: _AnyResponse | None
     model: ModelT | None
+    retryable: bool
 
     def __init__(
         self,
@@ -70,17 +83,60 @@ def __init__(
             response: The HTTP response that triggered the error.
             **kwargs: Forwarded to ``SdkError`` (``error``,
                 ``continuation_token``). The ``model`` key is consumed
-                separately for caller-supplied deserialised bodies.
+                separately for caller-supplied deserialised bodies. The
+                ``retryable`` key, if given, overrides the status-derived
+                default (pass ``True``/``False`` to force it).
         """
         self.response = response
         self.status = response.status if response is not None else None
         self.reason = response.reason if response is not None else None
         self.model = kwargs.pop("model", None)
+        retryable_override = kwargs.pop("retryable", None)
+        self.retryable = (
+            self._status_is_retryable() if retryable_override is None else bool(retryable_override)
+        )
         if message is None:
             label = self.status.name if self.status is not None else "unknown"
             message = f"Operation returned a non-success status: {label}"
         super().__init__(message, **kwargs)
 
+    def _status_is_retryable(self) -> bool:
+        """Return whether this error's status is retryable by default.
+
+        Returns:
+            ``True`` when the captured status is one of the default
+            retryable codes, ``False`` when no status was captured.
+        """
+        return self.status is not None and int(self.status) in _DEFAULT_RETRYABLE_STATUS
+
+    def body_snapshot(self, max_bytes: int | None = None) -> bytes:
+        """Preview the error response body without consuming it.
+
+        Safe to call from logging and post-mortem paths: it never drains a
+        single-use stream. Bytes are only returned when the body has already
+        been captured for repeatable reads (a ``LoggableResponseBody``); for
+        any other body — or when no response/body is present — an empty
+        ``bytes`` is returned rather than destroying the payload.
+
+        Args:
+            max_bytes: If given, return at most this many bytes from the
+                front of the captured body. ``None`` returns the full
+                capture.
+
+        Returns:
+            The captured body bytes, optionally truncated to ``max_bytes``;
+            empty when no non-consuming preview is available.
+
+        Raises:
+            ValueError: If ``max_bytes`` is negative.
+        """
+        if max_bytes is not None and max_bytes < 0:
+            raise ValueError(f"max_bytes must be non-negative, got {max_bytes}")
+        body = self.response.body if self.response is not None else None
+        if isinstance(body, LoggableResponseBody):
+            return body.snapshot(max_bytes)
+        return b""
+
 
 class DecodeError(HttpResponseError[ModelT]):
     """The response body could not be decoded as the expected format."""

packages/dexpace-sdk-core/src/dexpace/sdk/core/http/auth/digest.py

@@ -107,6 +107,7 @@ def handle(
         realm = selected.parameters.get("realm", "")
         nonce = selected.parameters.get("nonce", "")
         opaque = selected.parameters.get("opaque")
+        charset = _select_charset(selected.parameters.get("charset"))
         qop = self._pick_qop(selected.parameters.get("qop"))
         if qop is None and "qop" in selected.parameters:
             # The server advertised qop but did not include ``auth``: we
@@ -125,6 +126,7 @@ def handle(
             nc=nc,
             cnonce=cnonce,
             qop=qop,
+            charset=charset,
         )
         header_value = _format_header(
             username=self._username,
@@ -188,9 +190,10 @@ def _compute_response(
         nc: str,
         cnonce: str,
         qop: str | None,
+        charset: str,
     ) -> str:
         def h(data: str) -> str:
-            return hasher(data.encode("utf-8")).hexdigest()
+            return hasher(data.encode(charset)).hexdigest()
 
         ha1 = h(f"{self._username}:{realm}:{self._password}")
         if algorithm.endswith("-SESS"):
@@ -202,6 +205,28 @@ def h(data: str) -> str:
         return h(f"{ha1}:{nonce}:{ha2}")
 
 
+def _select_charset(charset_param: str | None) -> str:
+    """Choose the encoding for credential hashing per RFC 7616 §3.4.
+
+    RFC 7616 defines exactly one valid ``charset`` value — ``UTF-8`` — which
+    a server advertises to request that ``username`` and ``password`` be
+    encoded as UTF-8 before hashing. When the directive is absent (or carries
+    any other value), the legacy RFC 2617 default of ISO-8859-1 applies.
+
+    Args:
+        charset_param: The raw ``charset`` directive from the challenge, or
+            ``None`` if the server did not send one. Matched case-insensitively
+            against ``UTF-8``.
+
+    Returns:
+        The Python codec name to pass to ``str.encode`` — ``"utf-8"`` when the
+        server advertised ``charset=UTF-8``, otherwise ``"iso-8859-1"``.
+    """
+    if charset_param is not None and charset_param.strip().upper() == "UTF-8":
+        return "utf-8"
+    return "iso-8859-1"
+
+
 def _request_uri(url: Url) -> str:
     """Compute the ``uri`` parameter — path plus query, per RFC 7616 §3.4.6."""
     path = url.path or "/"

packages/dexpace-sdk-core/src/dexpace/sdk/core/http/common/media_type.py

@@ -5,6 +5,7 @@
 
 from __future__ import annotations
 
+import codecs
 from collections.abc import Mapping
 from dataclasses import dataclass
 from typing import Self
@@ -75,9 +76,19 @@ def full_type(self) -> str:
 
     @property
     def charset(self) -> str | None:
-        """The ``charset`` parameter, or ``None`` if absent."""
+        """The ``charset`` parameter as a known codec name, or ``None``.
+
+        Returns ``None`` when the parameter is absent *or* names an encoding
+        that the Python codec registry does not recognise. Degrading an
+        unknown charset to ``None`` (rather than raising) lets callers fall
+        back to a default encoding instead of failing to decode a body.
+        """
         for key, value in self.parameters:
             if key == "charset":
+                try:
+                    codecs.lookup(value)
+                except (LookupError, ValueError):
+                    return None
                 return value
         return None
 

packages/dexpace-sdk-core/src/dexpace/sdk/core/http/common/streaming.py

@@ -13,7 +13,7 @@
 from collections.abc import AsyncIterable, AsyncIterator, Iterable, Iterator
 from typing import Any
 
-from ...errors import DeserializationError
+from ...errors.serialization import DeserializationError
 
 
 def iter_jsonl(chunks: Iterable[bytes]) -> Iterator[Any]:

packages/dexpace-sdk-core/src/dexpace/sdk/core/http/request/loggable_request_body.py

@@ -76,9 +76,26 @@ def iter_bytes(self, chunk_size: int = 64 * 1024) -> Iterator[bytes]:
                 self._tap.write(chunk[:remaining])
             yield chunk
 
-    def snapshot(self) -> bytes:
-        """Return an immutable copy of the captured bytes."""
-        return self._tap.getvalue()
+    def snapshot(self, max_bytes: int | None = None) -> bytes:
+        """Return an immutable copy of the captured bytes.
+
+        Args:
+            max_bytes: If given, copy at most this many bytes from the front
+                of the tap. A ``memoryview`` bounds the slice so no more than
+                ``max_bytes`` are ever materialised, even when the tap holds a
+                large payload. ``None`` returns the full tap.
+
+        Returns:
+            The captured bytes, optionally truncated to ``max_bytes``.
+
+        Raises:
+            ValueError: If ``max_bytes`` is negative.
+        """
+        if max_bytes is None:
+            return self._tap.getvalue()
+        if max_bytes < 0:
+            raise ValueError(f"max_bytes must be non-negative, got {max_bytes}")
+        return bytes(self._tap.getbuffer()[:max_bytes])
 
     @property
     def captured_size(self) -> int:

packages/dexpace-sdk-core/src/dexpace/sdk/core/http/response/async_response.py

@@ -12,6 +12,7 @@
 from ..common.headers import Headers
 from ..common.http_header_name import HttpHeaderName
 from ..common.protocol import Protocol
+from .async_response_body import _shielded_cleanup
 from .status import Status
 
 if TYPE_CHECKING:
@@ -38,9 +39,14 @@ class AsyncResponse:
     body: AsyncResponseBody | None = None
 
     async def close(self) -> None:
-        """Close the response body. Idempotent."""
+        """Close the response body. Idempotent.
+
+        When invoked from ``__aexit__`` while an ``asyncio.CancelledError`` is
+        propagating out of an ``async with`` block, the body close is shielded
+        so the transport handle is released before cancellation continues.
+        """
         if self.body is not None:
-            await self.body.close()
+            await _shielded_cleanup(self.body.close())
 
     async def __aenter__(self) -> Self:
         return self