Skip to content

Add telemetry for chat quota notification banners#321793

Merged
pwang347 merged 5 commits into
microsoft:mainfrom
rfeltis:rfeltis/consumption-checkpoint-telemetry
Jun 19, 2026
Merged

Add telemetry for chat quota notification banners#321793
pwang347 merged 5 commits into
microsoft:mainfrom
rfeltis:rfeltis/consumption-checkpoint-telemetry

Conversation

@rfeltis

@rfeltis rfeltis commented Jun 17, 2026

Copy link
Copy Markdown
Contributor

Closes #320996

Summary

Adds telemetry to track when users see and interact with Copilot chat consumption quota checkpoint notification banners.

Changes

  • Telemetry events added:

    • chat.quotaNotificationShown: Emitted when a quota banner actually renders in the UI
    • chat.quotaNotificationDismissed: Emitted when user clicks the dismiss (X) button
    • chat.quotaNotificationActionInvoked: Emitted when user clicks an action button (e.g., 'Manage Budget')
  • Implementation details:

    • Added lifecycle events to ChatInputNotificationService (onDidShow, onDidAction, onDidDismiss)
    • Wired telemetry through real widget render path in ChatInputNotificationWidget
    • Implemented deduplication logic to prevent duplicate 'shown' events for the same notification
    • All events include GDPR-compliant classifications with SystemMetaData and FeatureInsight

Testing

  • Added integration test (chatQuotaNotification.integrationTest.ts) using production services:
    • Real ChatInputNotificationService (singleton)
    • Real ChatInputNotificationWidget
    • Real TelemetryService with test appender
    • DOM interaction (clicks on action/dismiss buttons)
  • Integration test passed: 3 passing
  • Unit tests passed: 47 passing

pm run precommit passed (hygiene clean)

Validation

  • ✅ TypeScript compilation clean for changed files
  • ✅ Integration test verifies telemetry through real UI pipeline
  • ✅ Unit tests verify telemetry data structure and deduplication
  • ✅ Hygiene checks passed

Samples of each banner being displayed:
image

image image image image

Sample of one event being dismissed:
04-found-dismissed-telemetry-in-output

Sample of CTA click:
04-found-action-telemetry-in-output

Closes microsoft#320996

Adds telemetry events to track when users see and interact with
consumption quota checkpoint notification banners:
- chat.quotaNotificationShown: Emitted when banner renders
- chat.quotaNotificationDismissed: Emitted when user dismisses
- chat.quotaNotificationActionInvoked: Emitted when user clicks action

Implementation adds lifecycle events (onDidShow, onDidAction, onDidDismiss)
to ChatInputNotificationService and wires telemetry through the real
widget render path with deduplication logic.

Includes integration test using production services and DOM interactions
to verify telemetry flows through the real pipeline.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot AI review requested due to automatic review settings June 17, 2026 16:38

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Note

Copilot was unable to run its full agentic suite in this review.

Adds explicit “shown” and “action invoked” signals to chat input notifications so ChatQuotaNotificationContribution can log quota-notification telemetry on real UI render, dismiss, and action interactions (plus unit + integration coverage).

Changes:

  • Extended IChatInputNotificationService with onDidShow / onDidAction and corresponding showNotification / actionNotification APIs.
  • Updated ChatInputNotificationWidget to emit “shown” on render and “action invoked” on action clicks.
  • Added quota-notification telemetry wiring + new integration tests that exercise the real widget DOM path.
Show a summary per file
File Description
src/vs/workbench/contrib/chat/browser/chatQuotaNotification.ts Logs shown/dismissed/action telemetry for the quota notification using new service events.
src/vs/workbench/contrib/chat/browser/widget/input/chatInputNotificationService.ts Adds show/action events + tracking state needed to avoid duplicate “shown” emissions.
src/vs/workbench/contrib/chat/browser/widget/input/chatInputNotificationWidget.ts Emits show/action signals from the actual render and click paths.
src/vs/workbench/contrib/chat/test/browser/chatQuotaNotification.test.ts Updates mocks and adds unit tests for quota notification telemetry behaviors.
src/vs/workbench/contrib/chat/test/browser/chatQuotaNotification.integrationTest.ts New integration tests verifying telemetry via the real widget DOM interactions.
src/vs/workbench/contrib/chat/test/browser/agentSessions/agentHostPermissionUiContribution.test.ts Updates fake notification service to satisfy new interface members.

Copilot's findings

  • Files reviewed: 6/6 changed files
  • Comments generated: 2

Comment thread src/vs/workbench/contrib/chat/test/browser/chatQuotaNotification.test.ts Outdated
- Add validation that commandId exists in notification actions before emitting event
- Guard mock service methods against deleted state to match real service behavior

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

@rfeltis rfeltis left a comment

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

All review comments have been addressed in commit 7b366de.

@rfeltis rfeltis marked this pull request as ready for review June 17, 2026 21:21
Giuspepe
Giuspepe previously approved these changes Jun 18, 2026
benvillalobos
benvillalobos previously approved these changes Jun 18, 2026
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@rfeltis rfeltis dismissed stale reviews from benvillalobos and Giuspepe via c51cb18 June 19, 2026 16:26

@pwang347 pwang347 left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks great, thanks!

@pwang347 pwang347 merged commit 8c08e04 into microsoft:main Jun 19, 2026
25 checks passed
@vs-code-engineering vs-code-engineering Bot added this to the 1.126.0 milestone Jun 19, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Add telemetry for consumption checkpoint banners

6 participants