feat!: add mcp conformance test infra (#585); add SSE reconnection with retry support (#596) #585
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -7,67 +7,166 @@ on: | |
| push: | ||
| branches: [ main ] | ||
|
|
||
| env: | ||
| JAVA_VERSION: '21' | ||
| JAVA_DISTRIBUTION: temurin | ||
| NODE_VERSION: '22' | ||
|
|
||
|
|
||
| concurrency: | ||
| group: ${{ github.workflow }}-${{ github.event.pull_request.number || github.ref }} | ||
| # Cancel only when the run is NOT on `main` branch | ||
| cancel-in-progress: ${{ github.ref != 'refs/heads/main' }} | ||
|
|
||
| jobs: | ||
| server: | ||
| runs-on: ubuntu-latest | ||
| name: Conformance Server Tests | ||
| timeout-minutes: 20 | ||
|
|
||
| steps: | ||
| - uses: actions/checkout@v6 | ||
|
|
||
| - name: Set up JDK | ||
| uses: actions/setup-java@v5 | ||
| with: | ||
| java-version: ${{ env.JAVA_VERSION }} | ||
| distribution: ${{ env.JAVA_DISTRIBUTION }} | ||
|
|
||
| - name: Setup Gradle | ||
| uses: gradle/actions/setup-gradle@v5 | ||
| with: | ||
| cache-read-only: ${{ github.ref != 'refs/heads/main' }} | ||
|
|
||
| - name: Build | ||
| run: ./gradlew :conformance-test:installDist | ||
|
|
||
| - name: Start server | ||
| run: | | ||
| MCP_PORT=3001 conformance-test/build/install/conformance-test/bin/conformance-test & | ||
| for i in $(seq 1 30); do | ||
| if curl -s -o /dev/null http://localhost:3001/mcp; then | ||
| echo "Server is ready" | ||
| break | ||
| fi | ||
| sleep 1 | ||
| done | ||
|
|
||
| - name: Run conformance tests | ||
| uses: modelcontextprotocol/conformance@v0.1.15 | ||
|
Contributor
There was a problem hiding this comment. Let's run the same run-conformance.sh here so it is consistent with the local testing.
Contributor
Author
There was a problem hiding this comment. doen't work for |
||
| with: | ||
| mode: server | ||
| url: http://localhost:3001/mcp | ||
| suite: active | ||
| node-version: ${{ env.NODE_VERSION }} | ||
| expected-failures: ./conformance-test/conformance-baseline.yml | ||
|
|
||
| client: | ||
| runs-on: ubuntu-latest | ||
| name: "Conformance Client Tests: ${{ matrix.scenario }}" | ||
| timeout-minutes: 20 | ||
|
|
||
| strategy: | ||
| fail-fast: false | ||
| matrix: | ||
| scenario: | ||
| - initialize | ||
| - tools_call | ||
| - elicitation-sep1034-client-defaults | ||
| - sse-retry | ||
|
|
||
| steps: | ||
| - uses: actions/checkout@v6 | ||
|
|
||
| - name: Set up JDK | ||
| uses: actions/setup-java@v5 | ||
| with: | ||
| java-version: ${{ env.JAVA_VERSION }} | ||
| distribution: ${{ env.JAVA_DISTRIBUTION }} | ||
|
|
||
| - name: Setup Gradle | ||
| uses: gradle/actions/setup-gradle@v5 | ||
| with: | ||
| cache-read-only: ${{ github.ref != 'refs/heads/main' }} | ||
|
|
||
| - name: Build | ||
| run: ./gradlew :conformance-test:installDist | ||
|
|
||
| - name: Run conformance tests | ||
| uses: modelcontextprotocol/conformance@v0.1.15 | ||
| with: | ||
| mode: client | ||
| command: conformance-test/build/install/conformance-test/bin/conformance-client | ||
| scenario: ${{ matrix.scenario }} | ||
| node-version: ${{ env.NODE_VERSION }} | ||
| expected-failures: ./conformance-test/conformance-baseline.yml | ||
|
|
||
| auth: | ||
| runs-on: ubuntu-latest | ||
| name: Conformance Auth Tests | ||
| timeout-minutes: 20 | ||
|
|
||
| steps: | ||
| - uses: actions/checkout@v6 | ||
|
|
||
| - name: Set up JDK | ||
| uses: actions/setup-java@v5 | ||
| with: | ||
| java-version: ${{ env.JAVA_VERSION }} | ||
| distribution: ${{ env.JAVA_DISTRIBUTION }} | ||
|
|
||
| - name: Setup Gradle | ||
| uses: gradle/actions/setup-gradle@v5 | ||
| with: | ||
| cache-read-only: ${{ github.ref != 'refs/heads/main' }} | ||
|
|
||
| - name: Build | ||
| run: ./gradlew :conformance-test:installDist | ||
|
|
||
| - name: Run conformance tests | ||
| uses: modelcontextprotocol/conformance@v0.1.15 | ||
| with: | ||
| mode: client | ||
| command: conformance-test/build/install/conformance-test/bin/conformance-client | ||
| suite: auth | ||
| node-version: ${{ env.NODE_VERSION }} | ||
| expected-failures: ./conformance-test/conformance-baseline.yml | ||
|
|
||
| auth-scenarios: | ||
| runs-on: ubuntu-latest | ||
| name: "Conformance Auth Scenario: ${{ matrix.scenario }}" | ||
| timeout-minutes: 20 | ||
|
|
||
| strategy: | ||
| fail-fast: false | ||
| matrix: | ||
| scenario: | ||
| - auth/client-credentials-jwt | ||
| - auth/client-credentials-basic | ||
| - auth/cross-app-access-complete-flow | ||
|
|
||
| steps: | ||
| - uses: actions/checkout@v6 | ||
|
|
||
| - name: Set up JDK | ||
| uses: actions/setup-java@v5 | ||
| with: | ||
| java-version: ${{ env.JAVA_VERSION }} | ||
| distribution: ${{ env.JAVA_DISTRIBUTION }} | ||
|
|
||
| - name: Setup Gradle | ||
| uses: gradle/actions/setup-gradle@v5 | ||
| with: | ||
| cache-read-only: ${{ github.ref != 'refs/heads/main' }} | ||
|
|
||
| - name: Build | ||
| run: ./gradlew :conformance-test:installDist | ||
|
|
||
| - name: Run conformance tests | ||
| uses: modelcontextprotocol/conformance@v0.1.15 | ||
| with: | ||
| mode: client | ||
| command: conformance-test/build/install/conformance-test/bin/conformance-client | ||
| scenario: ${{ matrix.scenario }} | ||
| node-version: ${{ env.NODE_VERSION }} | ||
| expected-failures: ./conformance-test/conformance-baseline.yml | ||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1 @@ | ||
| /results/ |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,130 @@ | ||
| # MCP Conformance Tests | ||
|
|
||
| Conformance tests for the Kotlin MCP SDK. Uses the external | ||
| [`@modelcontextprotocol/conformance`](https://www.npmjs.com/package/@modelcontextprotocol/conformance) | ||
| runner (pinned to **0.1.15**) to validate compliance with the MCP specification. | ||
|
|
||
| ## Prerequisites | ||
|
|
||
| - **JDK 17+** | ||
| - **Node.js 18+** and `npx` (for the conformance runner) | ||
| - **curl** (used to poll server readiness) | ||
|
|
||
| ## Quick Start | ||
|
|
||
| Run **all** suites (server, client core, client auth) from the project root: | ||
|
|
||
| ```bash | ||
| ./conformance-test/run-conformance.sh all | ||
| ``` | ||
|
|
||
| ## Commands | ||
|
|
||
| ``` | ||
| ./conformance-test/run-conformance.sh <command> [extra-args...] | ||
| ``` | ||
|
|
||
| | Command | What it does | | ||
| |---------------|--------------------------------------------------------------------------------------| | ||
| | `list` | [List scenarios available in MCP Conformance Test Framework][list-scenarios-command] | | ||
| | `server` | Starts the Ktor conformance server, runs the server test suite against it | | ||
| | `client` | Runs the client test suite (`initialize`, `tools_call`, `elicitation`, `sse-retry`) | | ||
| | `client-auth` | Runs the client auth test suite (20 OAuth scenarios) | | ||
| | `all` | Runs all three suites sequentially | | ||
|
|
||
| Any `[extra-args]` are forwarded to the conformance runner (e.g. `--verbose`). | ||
|
|
||
| ## What the Script Does | ||
|
|
||
| 1. **Builds** the module via `./gradlew :conformance-test:installDist` | ||
| 2. For `server` — starts the conformance server on `localhost:3001`, polls until ready | ||
| 3. Invokes `npx @modelcontextprotocol/conformance@0.1.15` with the appropriate arguments | ||
| 4. Saves results to `conformance-test/results/<command>/` | ||
| 5. Cleans up the server process on exit | ||
| 6. Exits non-zero if any suite fails | ||
|
|
||
| ## Environment Variables | ||
|
|
||
| | Variable | Default | Description | | ||
| |------------|---------|---------------------------------| | ||
| | `MCP_PORT` | `3001` | Port for the conformance server | | ||
|
|
||
| ## Project Structure | ||
|
|
||
| ``` | ||
| conformance-test/ | ||
| ├── run-conformance.sh # Single entry point script | ||
| ├── conformance-baseline.yml # Expected failures for known SDK limitations | ||
| └── src/main/kotlin/.../conformance/ | ||
| ├── ConformanceServer.kt # Ktor server entry point (StreamableHTTP, DNS rebinding, EventStore) | ||
| ├── ConformanceClient.kt # Scenario-based client entry point (MCP_CONFORMANCE_SCENARIO routing) | ||
| ├── ConformanceTools.kt # 18 tool registrations | ||
| ├── ConformanceResources.kt # 5 resource registrations (static, binary, template, watched, dynamic) | ||
| ├── ConformancePrompts.kt # 5 prompt registrations (simple, args, image, embedded, dynamic) | ||
| ├── ConformanceCompletions.kt # completion/complete handler | ||
| ├── InMemoryEventStore.kt # EventStore impl for SSE resumability (SEP-1699) | ||
| └── auth/ # OAuth client for 20 auth scenarios | ||
| ├── registration.kt # Scenario handler registration | ||
| ├── utils.kt # Shared utilities: JSON instance, constants, extractOrigin() | ||
| ├── discovery.kt # Protected Resource Metadata + AS Metadata discovery | ||
| ├── pkce.kt # PKCE code verifier/challenge generation + AS capability check | ||
| ├── tokenExchange.kt # Token endpoint interaction (exchange code, error handling) | ||
| ├── authCodeFlow.kt # Main Authorization Code flow handler (runAuthClient + interceptor) | ||
| ├── scopeHandling.kt # Scope selection strategy + step-up 403 handling | ||
| ├── clientRegistration.kt # Client registration logic (pre-reg, CIMD, dynamic) | ||
| ├── JWTScenario.kt # Client Credentials JWT scenario | ||
| ├── basicScenario.kt # Client Credentials Basic scenario | ||
| └── crossAppAccessScenario.kt # Cross-App Access (SEP-990) scenario | ||
| ``` | ||
|
|
||
| ## Test Suites | ||
|
|
||
| ### Server Suite | ||
|
|
||
| Tests the conformance server against all server scenarios: | ||
|
|
||
| | Category | Scenarios | | ||
| |-------------|-------------------------------------------------------------------------------------------------------------------------------------| | ||
| | Lifecycle | initialize, ping | | ||
| | Tools | text, image, audio, embedded, multiple, progress, logging, error, sampling, elicitation, dynamic, reconnection, JSON Schema 2020-12 | | ||
| | Resources | list, read-text, read-binary, templates, subscribe, dynamic | | ||
| | Prompts | simple, with-args, with-image, with-embedded-resource, dynamic | | ||
| | Completions | complete | | ||
| | Security | DNS rebinding protection | | ||
|
|
||
| ### Client Core Suite | ||
|
|
||
| | Scenario | Description | | ||
| |---------------------------------------|-----------------------------------------------| | ||
| | `initialize` | Connect, list tools, close | | ||
| | `tools_call` | Connect, call `add_numbers(a=5, b=3)`, close | | ||
| | `elicitation-sep1034-client-defaults` | Elicitation with `applyDefaults` capability | | ||
| | `sse-retry` | Call `test_reconnection`, verify reconnection | | ||
|
|
||
| ### Client Auth Suite | ||
|
|
||
| 17 OAuth Authorization Code scenarios + 2 Client Credentials scenarios (`jwt`, `basic`) + 1 Cross-App Access scenario = 20 total. | ||
|
|
||
| > [!NOTE] | ||
| > Auth scenarios are implemented using Ktor's `HttpClient` plugins (`HttpSend` interceptor, | ||
| > `ktor-client-auth`) as a standalone OAuth client. They do not use the SDK's built-in auth support. | ||
|
|
||
| ## Known SDK Limitations | ||
|
|
||
| 8 scenarios are expected to fail due to current SDK limitations (tracked in [ | ||
| `conformance-baseline.yml`](conformance-baseline.yml). | ||
|
|
||
| | Scenario | Suite | Root Cause | | ||
| |---------------------------------------|--------|--------------------------------------------------------------------------------------------------------------------------------------------------------| | ||
| | `tools-call-with-logging` | server | Notifications from tool handlers have no `relatedRequestId`; transport routes them to the standalone SSE stream instead of the request-specific stream | | ||
| | `tools-call-with-progress` | server | *(same as above)* | | ||
| | `tools-call-sampling` | server | *(same as above)* | | ||
| | `tools-call-elicitation` | server | *(same as above)* | | ||
| | `elicitation-sep1034-defaults` | server | *(same as above)* | | ||
| | `elicitation-sep1330-enums` | server | *(same as above)* | | ||
| | `resources-templates-read` | server | SDK does not implement `addResourceTemplate()` with URI pattern matching; resources are looked up by exact URI | | ||
| | `elicitation-sep1034-client-defaults` | client | SDK does not fill in `default` values from the elicitation request schema before sending the response | | ||
|
|
||
| These failures reveal SDK gaps and are intentionally not fixed in this module. | ||
|
|
||
| [list-scenarios-command]: https://github.com/modelcontextprotocol/conformance/tree/main?tab=readme-ov-file#list-available-scenarios |
Oops, something went wrong.
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Observation: Conformance tests are no longer being run on different platforms.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
yeap, it's jvm