Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
72 changes: 69 additions & 3 deletions src/Microsoft.Agents.A365.DevTools.MockToolingServer/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -8,9 +8,10 @@ The following mock server definitions are included out of the box:

| Server | File | Description |
|--------|------|-------------|
| `mcp_CalendarTools` | `mocks/mcp_CalendarTools.json` | Calendar operations (createEvent, listEvents, getSchedule, findMeetingTimes, etc.) |
| `mcp_MailTools` | `mocks/mcp_MailTools.json` | Email operations (SendEmail, SendEmailWithAttachments, etc.) |
| `mcp_MeServer` | `mocks/mcp_MeServer.json` | User/directory operations (listUsers, getMyProfile, getManager, etc.) |
| `mcp_CalendarTools` | `mocks/mcp_CalendarTools.json` | Calendar operations (ListEvents, CreateEvent, FindMeetingTimes, AcceptEvent, etc.) |
| `mcp_MailTools` | `mocks/mcp_MailTools.json` | Email operations (SendEmailWithAttachments, SearchMessages, FlagEmail, ReplyToMessage, etc.) |
| `mcp_MeServer` | `mocks/mcp_MeServer.json` | User/directory operations (GetMyDetails, GetUserDetails, GetManagerDetails, etc.) |
| `mcp_KnowledgeTools` | `mocks/mcp_KnowledgeTools.json` | Federated knowledge operations (configure_federated_knowledge, query_federated_knowledge, etc.) |

### mcp_MeServer Tools

Expand All @@ -34,6 +35,71 @@ Tools for email operations including `SendEmail`, `SendEmailWithAttachments`, an

---

## Fidelity Contract

### What the mock guarantees

Every tool exposed by a real M365 MCP server is present in the corresponding mock with the same name, same casing, and same required input fields. This ensures that agents developed against the mock will not encounter missing-tool or schema-mismatch errors when switched to a real server.

Comment on lines +40 to +43
Copy link

Copilot AI Feb 28, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This section claims the mock matches the real server’s “required input fields”, but the current CI fidelity tests only check tool names and several mock inputSchema definitions differ from snapshots (e.g., MailTools SearchMessages, MeServer GetUserDetails). Either tighten the fidelity contract/tests to validate schemas/required fields, or adjust this documentation to reflect what is actually enforced.

Copilot uses AI. Check for mistakes.
### What the mock does not guarantee

The mock does **not** provide real data, real authentication, or real side effects. Responses are rendered from templates and are not backed by Microsoft Graph or any live service.

### Snapshot-based verification

The `snapshots/` directory contains authoritative tool catalogs captured from real M365 MCP servers. Each snapshot file records the tool names, descriptions, and input schemas as they exist on the real server at the time of capture.

To verify that mock definitions match the real server contracts locally:

```bash
dotnet test --filter "FullyQualifiedName~MockToolFidelityTests"
```

To refresh snapshots when real servers change (requires M365 credentials):

```bash
$env:MCP_BEARER_TOKEN = a365 develop get-token --output raw
MCP_UPDATE_SNAPSHOTS=true dotnet test --filter "FullyQualifiedName~MockToolSnapshotCaptureTests"
```

---

## Keeping Mocks Current

### When to update snapshots

- When real M365 MCP servers add, rename, or remove tools
- Before a release, to confirm mocks still match production
- When agent tests pass locally against mocks but fail against a real environment

### How to update

1. Obtain a bearer token using the CLI:

```pwsh
# With an agent project present:
$env:MCP_BEARER_TOKEN = a365 develop get-token --output raw

# Without an agent project — pass app ID and explicit scopes:
$env:MCP_BEARER_TOKEN = a365 develop get-token --app-id <your-app-id> --scopes McpServers.Mail.All McpServers.Calendar.All McpServers.Me.All McpServers.Knowledge.All --output raw
```

2. Run the snapshot capture tests to write updated snapshot files:

```bash
MCP_UPDATE_SNAPSHOTS=true dotnet test --filter "FullyQualifiedName~MockToolSnapshotCaptureTests"
```

3. After updating snapshots, update the corresponding mock JSON files in `mocks/` to match any new or changed tools.

4. Run fidelity tests to confirm coverage:

```bash
dotnet test --filter "FullyQualifiedName~MockToolFidelityTests"
```

---

# How to mock notifications for custom activities

## Prerequisites
Expand Down
38 changes: 38 additions & 0 deletions src/Microsoft.Agents.A365.DevTools.MockToolingServer/design.md
Original file line number Diff line number Diff line change
Expand Up @@ -35,6 +35,11 @@ flowchart TB

subgraph Storage["File Storage"]
MocksDir["mocks/<br/>*.json files"]
SnapshotsDir["snapshots/<br/>Authoritative tool catalogs<br/>from real M365 servers"]
end

subgraph Testing["Fidelity Testing"]
FidelityTests["MockToolFidelityTests<br/>Asserts mock coverage<br/>against snapshots"]
end

subgraph Endpoints["HTTP Endpoints"]
Expand All @@ -51,6 +56,8 @@ flowchart TB
ServerClass --> MockToolExecutor
MockToolExecutor --> IMockToolStore
ServerClass --> Endpoints
FidelityTests --> SnapshotsDir
FidelityTests --> MocksDir
```

---
Expand Down Expand Up @@ -202,6 +209,37 @@ The `FileMockToolStore` uses `FileSystemWatcher` to detect changes to mock defin

---

## Fidelity Contract

The mock server maintains a contract with the real M365 MCP servers through a two-layer design.

### Two-layer design

- **`mocks/`** (behavior layer) — Contains mock tool definitions including response templates, simulated delay, and error simulation configuration. This is what the mock server loads at runtime.
- **`snapshots/`** (contract layer) — Contains authoritative tool catalogs captured from real M365 MCP servers. Each snapshot records the exact tool names, descriptions, and input schemas as they exist on the real server. Snapshots are the source of truth for what the mock must cover.

### CI enforcement

`MockToolFidelityTests` loads each snapshot file alongside the corresponding mock definition file and asserts that:

- Every tool present in the snapshot exists in the mock (same name, same casing)
- Every enabled tool in the mock exists in the snapshot (no phantom tools with unverified names)

These tests run as part of the standard test suite. Snapshots where `capturedAt` is `"UNPOPULATED"` are skipped — they do not block CI until real data has been captured.

### The `capturedAt` freshness indicator

Each snapshot file includes a `capturedAt` field set to an ISO 8601 UTC timestamp at time of capture. The value `"UNPOPULATED"` indicates the file is a placeholder that has never been verified against a real server. Fidelity tests skip for UNPOPULATED snapshots.

### Update process

1. **Detect drift** — Run `MockToolSnapshotCaptureTests` with `MCP_BEARER_TOKEN` set to query live M365 MCP servers and compare against snapshots. Tests fail with a clear diff if the real server has changed.
2. **Refresh snapshots** — Re-run with `MCP_UPDATE_SNAPSHOTS=true` to write updated snapshot files to disk.
3. **Update mocks** — Add, rename, or remove tools in the corresponding `mocks/` JSON files to match the refreshed snapshots.
4. **Verify** — Run `MockToolFidelityTests` (no credentials required) to confirm all mock files satisfy their snapshot contracts.

---

## Usage

### Starting the Server
Expand Down
Loading
Loading