bug: mock MCP tool names have drifted from real M365 MCP server contracts

[pratapladhani]

Problem

The mock tool definitions in mocks/*.json no longer match the tool catalogs
exposed by the real M365 MCP servers. Agents developed and tested against the
mocks will encounter tool-not-found or schema-mismatch errors when switched to
a real M365 environment.

There is currently no mechanism to detect or prevent this drift.

Evidence

The following mismatches were found by querying the real M365 MCP server endpoints
directly via the tools/list JSON-RPC method:

mcp_CalendarTools

• All 9 tool names use camelCase in the mock but PascalCase on the real server
  (e.g., createEvent → CreateEvent, deleteEvent → DeleteEventById)
• 3 tools in the mock do not exist on the real server: getEvent, getOrganization, getSchedule
• 4 tools on the real server are missing from the mock: TentativelyAcceptEvent,
  ForwardEvent, GetUserDateAndTimeZoneSettings, GetRooms

mcp_MeServer

• All 5 tool names are wrong (e.g., getMyProfile → GetMyDetails,
  listUsers → GetMultipleUsersDetails, getUser → GetUserDetails)

mcp_MailTools

• All 20 tool names carry an incorrect Async suffix
  (e.g., SendEmailWithAttachmentsAsync instead of SendEmailWithAttachments)
• The tool FlagEmail is missing entirely

mcp_KnowledgeTools

• The file contains 3 disabled placeholder entries with invented names.
  The real server exposes 5 enabled tools with snake_case names.

Impact

Any agent that relies on tool names as registered with the mock will silently
break when pointed at the real M365 environment. This makes local mock-based
development unreliable as a proxy for real behavior.

Expected behavior

Mock tool names, casing, and required input parameters should exactly match
what the real M365 MCP servers expose, and there should be a way to verify
and maintain that alignment over time.

[github-actions]

[AUTO-TRIAGE] Team Assistant Triage

This issue has been automatically analyzed and triaged.

AI Decision Reasoning: Type: Classified as 'bug' because the issue describes a mismatch between mock tool definitions and real server contracts, leading to broken functionality when transitioning from mock to real environments.
Priority: Assigned P1 because the issue causes silent failures in agents relying on the mocks, making local development unreliable and potentially blocking critical workflows.
Copilot: This issue involves aligning mock JSON definitions with real server contracts, which requires querying the real M365 MCP server, analyzing discrepancies, and updating multiple JSON files. It also involves understanding the server's schema and ensuring future alignment, which is beyond Copilot's capabilities. The issue spans multiple files and requires external API interaction and schema validation, making it unsuitable for Copilot.
Assignment: Grant Harris has backend expertise, which aligns well with the issue's focus on API and server contracts. Additionally, he has a lower contribution count compared to the tech lead, making him a suitable choice for balancing workload.
Labels: Applied labels bug, P1 based on issue type and priority

Labels and assignee have been applied based on this analysis.

[github-actions]

[AUTO-TRIAGE] Results

Classification: bug
Priority: P1
Confidence: 100.0%

Recommended Assignee: @gwharris7

Analysis:
type rationale: Classified as 'bug' because the issue describes a mismatch between mock tool definitions and real server contracts, leading to broken functionality when transitioning from mock to real environments.

priority rationale: Assigned P1 because the issue causes silent failures in agents relying on the mocks, making local development unreliable and potentially blocking critical workflows.

copilot rationale: This issue involves aligning mock JSON definitions with real server contracts, which requires querying the real M365 MCP server, analyzing discrepancies, and updating multiple JSON files. It also involves understanding the server's schema and ensuring future alignment, which is beyond Copilot's capabilities. The issue spans multiple files and requires external API interaction and schema validation, making it unsuitable for Copilot.

assignment rationale: Grant Harris has backend expertise, which aligns well with the issue's focus on API and server contracts. Additionally, he has a lower contribution count compared to the tech lead, making him a suitable choice for balancing workload.

labels rationale: Applied labels bug, P1 based on issue type and priority

[SUGGESTION] Suggested Approach

1. Update the mock tool definitions in mocks/*.json to match the real M365 MCP server contracts. Use the tools/list JSON-RPC method to query the real server endpoints and align tool names, casing, and schemas. Ensure all missing tools are added and incorrect or placeholder entries are removed.
2. Implement a validation script in the scripts directory that periodically queries the real M365 MCP server endpoints and compares the responses against the mock definitions in mocks/*.json. This script can use the requests or httpx library (already listed in autoTriage/requirements.txt) for HTTP requests.
3. Add unit tests in src/Tests/Microsoft.Agents.A365.DevTools.Cli.Tests/Models to verify that mock tool definitions match the real server contracts. These tests can use a snapshot of the real server responses for comparison and ensure that any drift is flagged during CI runs.
4. Introduce a CI workflow in .github/workflows/ci.yml to automatically run the validation script and unit tests against the mock definitions whenever changes are made to the mocks directory. This ensures alignment is maintained over time and prevents drift from being introduced accidentally.
5. Document the alignment process in the docs directory, including steps for querying the real M365 MCP server, updating the mock definitions, and running validation scripts. Provide examples of common mismatches and how to resolve them to assist contributors in maintaining consistency.

---

Generated by TeamAssistant Auto-Triage powered by GitHub Models

[github-actions]

SLA Escalation Alert

This P1 issue has exceeded its SLA threshold.

Metric	Value
Priority	P1
SLA Threshold	48 hours
Time Since Update	61.9 hours
Current Assignees	@gwharris7

Action: Adding Tech Lead @sellakumaran as assignee for immediate attention.

cc: @ajmfehr

---

Generated by AutoTriage SLA Escalation