-
Notifications
You must be signed in to change notification settings - Fork 11
test(examples): citations e2e coverage + fix ag-ui citation delivery #774
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Merged
Merged
Changes from all commits
Commits
Show all changes
2 commits
Select commit
Hold shift + click to select a range
File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,50 @@ | ||
| // SPDX-License-Identifier: MIT | ||
| import { test, expect } from '@playwright/test'; | ||
| import { sendPromptAndWait } from './test-helpers'; | ||
|
|
||
| // Twin of examples/chat's citations spec. AG-UI delivers citations differently | ||
| // (the backend surfaces them as state.citations[messageId], which the ag-ui | ||
| // adapter mirrors and bridgeCitationsState() maps onto Message.citations) — but | ||
| // the rendered surface is the shared @threadplane/chat markers/preview/panel, so | ||
| // the assertions match. The graph runs the REAL search_documents tool; its | ||
| // no-match fallback returns the first 3 corpus docs, so the set is deterministic: | ||
| // ng-signals-overview, ng-signals-rxjs, ng-control-flow. | ||
| const PROMPT = 'cite your sources on angular signals'; | ||
|
|
||
| test('inline citation markers render as resolved pills', async ({ page }) => { | ||
| const bubble = await sendPromptAndWait(page, PROMPT); | ||
| const markers = bubble.locator('.chat-citation-marker'); | ||
| await expect(markers.first()).toBeVisible(); | ||
| expect(await markers.count()).toBeGreaterThanOrEqual(3); | ||
| await expect(bubble.locator('a.chat-citation-marker').first()).toHaveAttribute('href', /angular\.dev/); | ||
| await expect(bubble.locator('.chat-citation-marker--unresolved')).toHaveCount(0); | ||
| }); | ||
|
|
||
| test('sources panel is collapsed by default and expands to the cited sources', async ({ page }) => { | ||
| const bubble = await sendPromptAndWait(page, PROMPT); | ||
|
|
||
| await expect(bubble.locator('.chat-citations')).toBeVisible(); | ||
| await expect(bubble.locator('.chat-citations__count')).toHaveText('3'); | ||
|
|
||
| await expect(bubble.locator('.chat-citations__header')).toBeVisible(); | ||
| await expect(bubble.locator('.chat-citations__list')).toHaveCount(0); | ||
|
|
||
| await bubble.locator('.chat-citations__header').click(); | ||
| const cards = bubble.locator('.chat-citations-card'); | ||
| await expect(cards).toHaveCount(3); | ||
| await expect(cards.nth(0)).toContainText('Signals'); | ||
| await expect(cards.nth(1)).toContainText('RxJS interop'); | ||
| await expect(cards.nth(2)).toContainText('control flow'); | ||
| await expect(cards.nth(0)).toHaveAttribute('href', /angular\.dev\/guide\/signals/); | ||
| }); | ||
|
|
||
| test('focusing a marker opens the portaled provenance preview card', async ({ page }) => { | ||
| const bubble = await sendPromptAndWait(page, PROMPT); | ||
|
|
||
| await bubble.locator('a.chat-citation-marker').first().focus(); | ||
|
|
||
| const preview = page.locator('.chat-citation-preview'); | ||
| await expect(preview).toBeVisible(); | ||
| await expect(preview.locator('.chat-citation-preview__domain')).toContainText('angular.dev'); | ||
| await expect(preview.locator('.chat-citation-preview__open')).toBeVisible(); | ||
| }); | ||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,21 @@ | ||
| { | ||
| "fixtures": [ | ||
| { | ||
| "match": { "userMessage": "cite your sources on angular signals", "hasToolResult": true }, | ||
| "response": { | ||
| "content": "Angular signals are a reactivity primitive that tracks reads and notifies consumers on change [^ng-signals-overview]. They interoperate with RxJS through `toSignal()` and `toObservable()` [^ng-signals-rxjs], and modern templates express reactive UI with built-in control flow like `@if` and `@for` [^ng-control-flow]. Signals pair naturally with that control flow for local state [^ng-signals-overview]." | ||
| } | ||
| }, | ||
| { | ||
| "match": { "userMessage": "cite your sources on angular signals" }, | ||
| "response": { | ||
| "toolCalls": [ | ||
| { | ||
| "name": "search_documents", | ||
| "arguments": { "query": "authoritative overview of angular signals reactivity and control flow" } | ||
| } | ||
| ] | ||
| } | ||
| } | ||
| ] | ||
| } |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change | ||||
|---|---|---|---|---|---|---|
| @@ -0,0 +1,53 @@ | ||||||
| // SPDX-License-Identifier: MIT | ||||||
| import { test, expect } from '@playwright/test'; | ||||||
| import { sendPromptAndWait } from './test-helpers'; | ||||||
|
|
||||||
| // Matches fixtures/citations.json. The graph runs the REAL search_documents | ||||||
| // tool; its no-match fallback returns the first 3 corpus docs, so the citation | ||||||
| // set is deterministic: ng-signals-overview, ng-signals-rxjs, ng-control-flow. | ||||||
| const PROMPT = 'cite your sources on angular signals'; | ||||||
|
|
||||||
| test('inline citation markers render as resolved pills', async ({ page }) => { | ||||||
| const bubble = await sendPromptAndWait(page, PROMPT); | ||||||
| const markers = bubble.locator('.chat-citation-marker'); | ||||||
| await expect(markers.first()).toBeVisible(); | ||||||
| // 4 inline references across 3 distinct sources. | ||||||
| expect(await markers.count()).toBeGreaterThanOrEqual(3); | ||||||
|
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Same issue as the ag-ui twin — raw
Suggested change
|
||||||
| // Resolved markers with a URL render as anchors linking out; none unresolved. | ||||||
| await expect(bubble.locator('a.chat-citation-marker').first()).toHaveAttribute('href', /angular\.dev/); | ||||||
| await expect(bubble.locator('.chat-citation-marker--unresolved')).toHaveCount(0); | ||||||
| }); | ||||||
|
|
||||||
| test('sources panel is collapsed by default and expands to the cited sources', async ({ page }) => { | ||||||
| const bubble = await sendPromptAndWait(page, PROMPT); | ||||||
|
|
||||||
| await expect(bubble.locator('.chat-citations')).toBeVisible(); | ||||||
| await expect(bubble.locator('.chat-citations__count')).toHaveText('3'); | ||||||
|
|
||||||
| // Collapsed by default: header shows, list is absent. | ||||||
| await expect(bubble.locator('.chat-citations__header')).toBeVisible(); | ||||||
| await expect(bubble.locator('.chat-citations__list')).toHaveCount(0); | ||||||
|
|
||||||
| // Expand → three detail cards in citation order. | ||||||
| await bubble.locator('.chat-citations__header').click(); | ||||||
| const cards = bubble.locator('.chat-citations-card'); | ||||||
| await expect(cards).toHaveCount(3); | ||||||
| await expect(cards.nth(0)).toContainText('Signals'); | ||||||
| await expect(cards.nth(1)).toContainText('RxJS interop'); | ||||||
| await expect(cards.nth(2)).toContainText('control flow'); | ||||||
| // Cards are links to the source (the card element itself is the <a>). | ||||||
| await expect(cards.nth(0)).toHaveAttribute('href', /angular\.dev\/guide\/signals/); | ||||||
| }); | ||||||
|
|
||||||
| test('focusing a marker opens the portaled provenance preview card', async ({ page }) => { | ||||||
| const bubble = await sendPromptAndWait(page, PROMPT); | ||||||
|
|
||||||
| // Focus (deterministic across pointer types) opens the preview. | ||||||
| await bubble.locator('a.chat-citation-marker').first().focus(); | ||||||
|
|
||||||
| // The preview is portaled to the body-level overlay container, not the bubble. | ||||||
| const preview = page.locator('.chat-citation-preview'); | ||||||
| await expect(preview).toBeVisible(); | ||||||
| await expect(preview.locator('.chat-citation-preview__domain')).toContainText('angular.dev'); | ||||||
| await expect(preview.locator('.chat-citation-preview__open')).toBeVisible(); | ||||||
| }); | ||||||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,21 @@ | ||
| { | ||
| "fixtures": [ | ||
| { | ||
| "match": { "userMessage": "cite your sources on angular signals", "hasToolResult": true }, | ||
| "response": { | ||
| "content": "Angular signals are a reactivity primitive that tracks reads and notifies consumers on change [^ng-signals-overview]. They interoperate with RxJS through `toSignal()` and `toObservable()` [^ng-signals-rxjs], and modern templates express reactive UI with built-in control flow like `@if` and `@for` [^ng-control-flow]. Signals pair naturally with that control flow for local state [^ng-signals-overview]." | ||
| } | ||
| }, | ||
| { | ||
| "match": { "userMessage": "cite your sources on angular signals" }, | ||
| "response": { | ||
| "toolCalls": [ | ||
| { | ||
| "name": "search_documents", | ||
| "arguments": { "query": "authoritative overview of angular signals reactivity and control flow" } | ||
| } | ||
| ] | ||
| } | ||
| } | ||
| ] | ||
| } |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Raw
expect(await markers.count())doesn't auto-retry. If markers render fractionally after the first one is visible this can flake. The fixture has 4 inline references across 3 sources — prefer the retrying form: