fix: correctness, resource-handling, and documentation fixes across the SDK (#5)

PR: #5

Author: OmarAlJarrah

CHANGELOG.md

@@ -10,9 +10,11 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 A round of platform improvements to `dexpace-sdk-core`: new optional building
 blocks (typed serialization, webhook verification, pagination, two pipeline
 policies), tightened retry and tracing behaviour, and a batch of correctness
-fixes across bodies, SSE parsing, Digest auth, and error reporting. Everything
-lands in `core`; the transport packages are unchanged. No public symbol was
-removed, so existing code continues to work without modification.
+fixes across bodies, SSE parsing, Digest auth, and error reporting. Most of this
+lands in `core`; the transport adapters additionally get consistent connect- vs
+read-phase timeout classification and tighter resource release. The only removed
+public symbol is the unused `RetryConfig` (see Removed); existing code otherwise
+continues to work without modification.
 
 ### Added
 
@@ -41,6 +43,12 @@ removed, so existing code continues to work without modification.
 - **Log correlation** (`instrumentation.correlation`). A `contextvar`-backed
   correlation id that flows through the pipeline and is attached to log records,
   so logs from a single logical request can be tied together.
+- **Reconnecting SSE client** (`http.sse.connection`). `SseConnection` and
+  `AsyncSseConnection` resume an interrupted event stream by replaying the
+  `Last-Event-ID` header and reconnecting with jittered backoff that honours the
+  server's `retry:` hint. Built on the shared dispatch seam
+  (`pipeline.dispatch`), which lets both the SSE client and the paginator accept
+  either a pipeline or a bare send-callable.
 
 ### Changed
 
@@ -60,6 +68,20 @@ removed, so existing code continues to work without modification.
 - **Error reporting** (`errors.http`). HTTP errors now expose whether they are
   `retryable` and carry a bounded body snapshot for diagnostics, with the
   snapshot capped so an error never holds an unbounded payload.
+- **`HttpRange.suffix`** (`http.common.http_range`) now returns a public
+  `HttpRange` (carrying an `is_suffix` flag) instead of a private helper type,
+  so a `bytes=-N` suffix range composes with `HttpRange.format_many` alongside
+  ordinary ranges.
+- **`CallContext`** (`http.context`) is now an `abc.ABC`. It declares no
+  abstract methods, so existing subclasses are unaffected; the change only
+  prevents the base from being instantiated directly.
+
+### Removed
+
+- **`RetryConfig`** (`pipeline` / `pipeline.step.config`). It was exported but
+  never wired into the retry policy, so it configured nothing; `RetryPolicy`'s
+  constructor is the real configuration surface. Code that imported
+  `RetryConfig` should configure `RetryPolicy` directly.
 
 ### Fixed
 

CLAUDE.md

@@ -43,13 +43,18 @@ bodies are modelled as typed Pythonic abstractions instead.
   `from_form` / `from_stream` / `from_iter` / `from_file`. `ResponseBody`
   exposes `iter_bytes` / `bytes` / `string`. Single-use bodies (stream /
   iter) raise `RuntimeError` on second consumption — call `to_replayable()`
-  before the first send if retries are needed.
+  before the first send if retries are needed. `AsyncRequestBody` /
+  `AsyncResponseBody` are the async twins (`aiter_bytes`), and
+  `MultipartField` / `MultipartRequestBody` build `multipart/form-data`
+  payloads.
 - **Body capture for logging uses `BytesIO`.** `LoggableRequestBody` mirrors
   writes into a `BytesIO` tap; `LoggableResponseBody` caches drained bytes
   for repeatable reads. Both honour a configurable byte cap.
 - **Thread-safety where stated.** `ContextStore` is safe under concurrent
-  use; individual bodies and streams are not. Per-context lookups rely on
-  CPython's GIL for atomic dict ops and use a lock only for check-and-set.
+  use; individual bodies and streams are not. Every store operation
+  (`get` / `put` / `set` / `remove`) acquires a `threading.Lock`, so the
+  guarantee survives free-threaded CPython (PEP 703) and runtimes without
+  atomic dict ops rather than relying on the GIL.
 - **Public API is narrow.** Helpers and concrete adapter classes are
   module-private (leading underscore). The public surface for each subpackage
   is what its `__init__.py` re-exports.
@@ -103,14 +108,20 @@ python-sdk/
     │   │   │   ├── common/                 # Headers, HttpHeaderName, MediaType,
     │   │   │   │                           # Protocol, Url, QueryParams, ETag,
     │   │   │   │                           # HttpRange, RequestConditions,
-    │   │   │   │                           # common_media_types
-    │   │   │   ├── request/                # Request, RequestBody, FileRequestBody,
-    │   │   │   │                           # LoggableRequestBody, Method
-    │   │   │   ├── response/               # Response, ResponseBody,
-    │   │   │   │                           # LoggableResponseBody, Status
+    │   │   │   │                           # common_media_types; pagination.py
+    │   │   │   │                           # (ItemPaged/Pager + async twins),
+    │   │   │   │                           # streaming.py (jsonl/chunked-frame iters)
+    │   │   │   ├── request/                # Request, RequestBody, AsyncRequestBody,
+    │   │   │   │                           # FileRequestBody, LoggableRequestBody,
+    │   │   │   │                           # MultipartField/MultipartRequestBody, Method
+    │   │   │   ├── response/               # Response, AsyncResponse, ResponseBody,
+    │   │   │   │                           # AsyncResponseBody, LoggableResponseBody,
+    │   │   │   │                           # Status
     │   │   │   ├── context/                # CallContext, DispatchContext,
     │   │   │   │                           # RequestContext, ExchangeContext,
     │   │   │   │                           # ContextStore
+    │   │   │   ├── sse/                     # Server-Sent Events parser + connection
+    │   │   │   ├── webhooks/                # webhook signature verification
     │   │   │   └── auth/                   # TokenCredential, BearerTokenPolicy,
     │   │   │                               # BasicAuthPolicy, KeyCredentialPolicy,
     │   │   │                               # ChallengeHandler (Basic/Digest/Composite),
@@ -119,19 +130,24 @@ python-sdk/
     │   │   │   │                           # Stage, StagedPipelineBuilder, defaults,
     │   │   │   │                           # sans-io + transport runners under the hood
     │   │   │   │
-    │   │   │   ├── policies/               # retry, redirect, logging, tracing,
-    │   │   │   │                           # set_date (+ async twins)
-    │   │   │   └── step/                   # PipelineStep, StepMetadata, RetryConfig
+    │   │   │   ├── policies/               # redirect, idempotency, retry, set_date,
+    │   │   │   │                           # client_identity, logging, tracing
+    │   │   │   │                           # (async twins only for the first five)
+    │   │   │   └── step/                   # PipelineStep, StepMetadata
     │   │   ├── client/                     # HttpClient + AsyncHttpClient Protocols
     │   │   ├── config/                     # Configuration
     │   │   ├── serde/                      # Serde, Serializer, Deserializer Protocols
     │   │   ├── errors/                     # SDK-level exception hierarchy
     │   │   ├── instrumentation/            # InstrumentationContext, Span, Tracer,
-    │   │   │                               # TracingScope, noops
+    │   │   │                               # TracingScope, noops, metrics,
+    │   │   │                               # correlation, client_logger, http_tracer,
+    │   │   │                               # identifiers, log_level, url_redactor
+    │   │   ├── pagination/                  # Page, Paginator, link-header + strategy
     │   │   └── util/                       # clock, proxy helpers
     │   └── tests/                          # pytest suite — auth/, config/, context/,
     │                                       # errors/, http/, instrumentation/,
-    │                                       # pipeline/, serde/, sse/, util/
+    │                                       # pagination/, pipeline/, serde/, sse/,
+    │                                       # util/, webhooks/
     ├── dexpace-sdk-http-stdlib/            # reference stdlib transports:
     │   │                                   # UrllibHttpClient, AsyncioHttpClient
     │   └── src/dexpace/sdk/http/stdlib/
@@ -143,6 +159,10 @@ python-sdk/
         └── src/dexpace/sdk/http/requests/
 ```
 
+Community-health and tooling files (`CHANGELOG.md`, `CONTRIBUTING.md`,
+`SECURITY.md`, `CODE_OF_CONDUCT.md`, `conftest.py`, `tools/`) are elided from
+the tree above.
+
 Every transport package depends on `dexpace-sdk-core` and adapts its HTTP
 library to the `HttpClient` / `AsyncHttpClient` Protocols. Namespace
 packaging (no `__init__.py` at `src/dexpace/`, `src/dexpace/sdk/`, or
@@ -185,12 +205,16 @@ Layered, bottom-up:
    evict on `CallContext.close()`.
 4. **`pipeline`** — `Policy` (and `AsyncPolicy`) wrap the downstream chain;
    `Pipeline` / `AsyncPipeline` run an ordered set of policies grouped into
-   `Stage`s via `StagedPipelineBuilder`. Shipped policies: retry, redirect,
-   logging, tracing, set-date (each with an async twin under
-   `pipeline/policies/`). `default_pipeline()` / `default_async_pipeline()`
-   assemble the standard stack. The lower-level `pipeline/step/PipelineStep`
-   Protocol (`(input, context) -> output`) plus `StepMetadata` / `RetryConfig`
-   remain for custom composition.
+   `Stage`s via `StagedPipelineBuilder`. Shipped policies: redirect,
+   idempotency, retry, set-date, client-identity, logging, tracing. Async
+   twins under `pipeline/policies/` exist only for redirect, idempotency,
+   retry, set-date, and client-identity; logging and tracing are sync-only.
+   `default_pipeline()` / `default_async_pipeline()` assemble the standard
+   stack in the order redirect → idempotency → retry → set-date →
+   client-identity → [auth] → logging → tracing (the async pipeline omits
+   logging and tracing). The lower-level `pipeline/step/PipelineStep` Protocol
+   (`(input, context) -> output`) plus `StepMetadata` remain for custom
+   composition.
 5. **`client/HttpClient`** — single-method Protocol
    (`execute(request) -> Response`). Transport is **not** provided by `core`;
    the `dexpace-sdk-http-*` packages (stdlib, httpx, aiohttp, requests) each

README.md

@@ -79,8 +79,9 @@ with UrllibHttpClient() as client, client.execute(request) as response:
 ### A configured pipeline
 
 `default_pipeline()` returns a `StagedPipelineBuilder` pre-wired with the
-canonical policy stack (redirect, retry, set-date, logging, tracing). Add
-authentication and adjust whatever the defaults get wrong for you:
+canonical policy stack (redirect, idempotency, retry, set-date,
+client-identity, logging, tracing). Add authentication and adjust whatever
+the defaults get wrong for you:
 
 ```python
 from dexpace.sdk.core.http.auth import BearerTokenPolicy
@@ -112,7 +113,8 @@ from dexpace.sdk.core.http.request import RequestBody
 RequestBody.from_stream(open("payload.bin", "rb"))
 RequestBody.from_iter([b"chunk-1", b"chunk-2"])
 
-# Replayable (transports may use zero-copy sendfile)
+# Replayable; a transport could special-case file bodies (e.g. zero-copy
+# sendfile), though none of the shipped transports do so today
 RequestBody.from_file("upload.bin")
 
 # Convert any single-use body into a replayable one before retrying
@@ -130,8 +132,9 @@ the way back up. The terminal policy hands the request to an `HttpClient`
 transport.
 
 ```
-caller → Pipeline → REDIRECT → RETRY → SET_DATE → AUTH → LOGGING → POST_LOGGING → HttpClient → wire
-                     (pillar)  (pillar)            (pillar) (pillar)
+caller → Pipeline → REDIRECT → POST_REDIRECT → RETRY → POST_RETRY → [AUTH] → LOGGING → POST_LOGGING → HttpClient → wire
+                     (pillar)   idempotency    (pillar)  set-date    (pillar) (pillar)  tracing
+                                                          client-identity
 ```
 
 Ordering is governed by `Stage`, an `IntEnum` whose values sit 100 apart so
@@ -169,12 +172,14 @@ Bottom-up, the layers are:
 | `http.common`       | `Headers`, `HttpHeaderName`, `MediaType`, `Protocol`, `Url`, `QueryParams`, `ETag`, `HttpRange`, `RequestConditions`, paging primitives |
 | `http.context`      | `CallContext` → `DispatchContext` → `RequestContext` → `ExchangeContext` chain, `ContextStore`           |
 | `http.auth`         | `BearerTokenPolicy`, `BasicAuthPolicy`, `KeyCredentialPolicy`, `DigestChallengeHandler`, RFC 7235 challenge parser, `TokenCache` |
-| `http.sse`          | `SseParser` for Server-Sent Events streams                                                               |
+| `http.sse`          | `SseParser`, plus reconnecting `SseConnection` / `AsyncSseConnection` (Last-Event-ID replay + backoff)   |
+| `http.webhooks`     | `WebhookVerifier`, `InvalidWebhookSignatureError` — HMAC signature verification with timestamp tolerance  |
+| `pagination`        | `Page`, `Paginator` / `AsyncPaginator`, `PaginationStrategy` (`CursorStrategy`, `PageNumberStrategy`, `LinkHeaderStrategy`) |
 | `pipeline`          | `Pipeline`, `AsyncPipeline`, `Policy` ABC, `Stage` enum, `StagedPipelineBuilder`, `default_pipeline()`   |
-| `pipeline.policies` | `RetryPolicy`, `RedirectPolicy`, `SetDatePolicy`, `LoggingPolicy`, `TracingPolicy` (+ async twins)       |
+| `pipeline.policies` | `RedirectPolicy`, `IdempotencyPolicy`, `RetryPolicy`, `SetDatePolicy`, `ClientIdentityPolicy`, `LoggingPolicy`, `TracingPolicy` (async twins for all but logging/tracing) |
 | `client`            | `HttpClient` and `AsyncHttpClient` Protocols                                                             |
 | `serde`             | `Serde`, `Serializer`, `Deserializer` Protocols + `JsonSerde` reference impl                             |
-| `instrumentation`   | `ClientLogger`, `UrlRedactor`, `Tracer`, `Span`, `InstrumentationContext`, noop singletons               |
+| `instrumentation`   | `ClientLogger`, `UrlRedactor`, `Tracer`, `Span`, `InstrumentationContext`, `contextvars` correlation helpers, noop singletons |
 | `errors`            | `SdkError` hierarchy: `ServiceRequestError`, `ServiceResponseError`, `HttpResponseError[ModelT]`, …      |
 | `util`              | `Clock`, `AsyncClock`, `ProxyOptions`                                                                    |
 | `config`            | `Configuration` (layered env-var + override lookup) + `ConfigurationBuilder`                             |
@@ -203,7 +208,16 @@ Bottom-up, the layers are:
   OpenTelemetry-compatible spans via `TracingPolicy`, URL redaction with
   allowlisted query parameters, and capped body capture for diagnostics.
 - **Server-Sent Events.** A WHATWG-compliant `SseParser` with a bounded
-  line buffer.
+  line buffer, plus reconnecting `SseConnection` / `AsyncSseConnection`
+  that resume with `Last-Event-ID` and honour server `retry:` backoff.
+- **Pagination.** A top-level `pagination` package: `Page`, sync and async
+  `Paginator`s that iterate item-by-item or page-by-page, and pluggable
+  `PaginationStrategy` (cursor, page-number, and `Link`-header).
+- **Webhooks.** `WebhookVerifier` checks HMAC signatures with a timestamp
+  tolerance and constant-time comparison, raising
+  `InvalidWebhookSignatureError` on mismatch.
+- **Correlation.** `contextvars`-based trace/span propagation so the
+  idempotency and client-identity policies and logging share one id.
 - **A lean core.** `dexpace-sdk-core` carries a single runtime dependency
   (`furl`, which backs `Url` parsing); each transport adapter adds exactly
   one HTTP library.
@@ -220,8 +234,8 @@ uv sync
 ```
 
 ```bash
-uv run pytest -q                 # 646 tests across 5 packages
-uv run mypy --strict             # type-check (171 source files)
+uv run pytest -q                 # run the full test suite across 5 packages
+uv run mypy --strict             # type-check every package under strict mode
 uv run ruff check                # lint
 uv run ruff format --check       # formatting gate
 ```

docs/architecture.md

@@ -17,7 +17,13 @@ plug in a concrete transport via the `HttpClient` Protocol.
          │   - Pipeline      │          │   - BearerToken     │
          │   - Policy        │          │   - KeyCredential   │
          │   - PipelineStep  │          │   - BasicAuth       │
-         │   - retry, log,   │          │   - TokenCache      │
+         │   - redirect,     │          │   - TokenCache      │
+         │     idempotency,  │          │                     │
+         │     retry,        │          │                     │
+         │     set-date,     │          │                     │
+         │     client-       │          │                     │
+         │     identity,     │          │                     │
+         │     logging,      │          │                     │
          │     tracing       │          │                     │
          └──────────┬────────┘          └─────────────────────┘
                     │
@@ -32,9 +38,13 @@ plug in a concrete transport via the `HttpClient` Protocol.
                     │
          ┌──────────▼─────────────────────────────────────────┐
          │   client/  HttpClient + AsyncHttpClient            │
+         │   - Protocols only; transports plug in here        │
+         └──────────┬─────────────────────────────────────────┘
+                    │
+         ┌──────────▼─────────────────────────────────────────┐
+         │   dexpace-sdk-http-stdlib (separate distribution)  │
          │   - UrllibHttpClient (sync reference)              │
          │   - AsyncioHttpClient (async reference)            │
-         │   - real transports plug in here                   │
          └────────────────────────────────────────────────────┘
 ```
 
@@ -67,5 +77,5 @@ The Java port has an `IoProvider` / `Buffer` / `Source` / `Sink` layer
 (a port of Okio). In Python, `bytes` / `bytearray` / `memoryview` /
 `BytesIO` / `BinaryIO` already cover the same surface idiomatically.
 Bodies use `iter_bytes(chunk_size)` for streaming and ordinary stdlib
-primitives for everything else. See `to-implement.md` for the design
-rationale.
+primitives for everything else. See the "Things That Will Bite You"
+section in `CLAUDE.md` for the design rationale.

docs/auth.md

@@ -33,9 +33,10 @@ All concrete credentials redact secrets in their `__repr__`.
 - Calls `AccessTokenInfo.needs_refresh()` before each request; refreshes
   proactively when `refresh_on` has passed or `expires_on - leeway`
   (default 300 s) has been reached.
-- On a 401 response with a `WWW-Authenticate` header, invalidates the
-  cached token and calls `on_challenge(request, response)`. Override
-  `on_challenge` in a subclass to handle CAE / claims-challenge flows.
+- On any 401 response, invalidates the cached token. If the response
+  also carries a `WWW-Authenticate` header, then calls
+  `on_challenge(request, response)`. Override `on_challenge` in a
+  subclass to handle CAE / claims-challenge flows.
 
 ## Token cache