Fix/image render#116
Open
Kaguya-19 wants to merge 5 commits into
Open
Conversation
Add marshalMcpContent() to convert MCP TextContent/ImageContent blocks into PilotDeck text/image result types, replacing the old single json blob. This enables MCP tool screenshots (e.g. Playwright) to render inline in the chat UI. Co-authored-by: Cursor <cursoragent@cursor.com>
Extract image sub-blocks from tool_result content during JSONL session replay and attach them as WebMessage.images. Fixes screenshots disappearing on page refresh or session reload. Co-authored-by: Cursor <cursoragent@cursor.com>
Add projectName prop to <Markdown> and resolveImageSrc() helper that rewrites relative paths (e.g. ./foo.png) to the backend file-serving endpoint /api/projects/:name/files/content. Wire projectName into all Markdown usages across V1 and V2 chat views. Co-authored-by: Cursor <cursoragent@cursor.com>
… models Add downgradeUnsupportedContent() that replaces image/pdf/audio blocks with descriptive text placeholders before assertContentSupported runs. Prevents unsupported_modality turn failures when a text-only model receives MCP screenshot results in tool_result content. Co-authored-by: Cursor <cursoragent@cursor.com>
Contributor
There was a problem hiding this comment.
Pull request overview
This PR focuses on improving inline image rendering in the chat UI (including tool outputs like screenshots) and making multimodal handling more robust when a target model does not support certain media inputs.
Changes:
- Pass
projectNameinto the sharedMarkdownrenderer so relative Markdown image paths can be resolved via the project files content API. - Project tool-result image blocks into web-facing messages so tool screenshots/images can render inline in chat.
- Add a pre-flight “downgrade unsupported content” step that replaces unsupported media blocks with text placeholders before validating/sending requests.
Reviewed changes
Copilot reviewed 8 out of 8 changed files in this pull request and generated 3 comments.
Show a summary per file
| File | Description |
|---|---|
| ui/src/components/chat/view/subcomponents/MessageComponent.tsx | Passes projectName into Markdown renders so relative images can be resolved in the V1 chat UI. |
| ui/src/components/chat/view/subcomponents/Markdown.tsx | Adds projectName prop and custom img renderer that rewrites relative image URLs to the project files content API. |
| ui/src/components/chat-v2/MessageRowV2.tsx | Passes projectName into Markdown renders so relative images can be resolved in the V2 chat UI. |
| src/web/server/readSessionMessages.ts | Includes tool_result image blocks in WebMessage.images so tool images can be surfaced to the UI. |
| src/model/request/validateModelRequest.ts | Runs the new downgrade step before asserting multimodal constraints. |
| src/model/protocol/multimodal.ts | Implements downgradeUnsupportedContent() to replace unsupported media blocks with text placeholders. |
| src/model/index.ts | Re-exports downgradeUnsupportedContent from the model package. |
| src/mcp/runtime/PluginToToolBridge.ts | Maps MCP ContentBlock[] (text/image) into PilotDeck tool result content so images can flow through to the chat UI. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Comment on lines
+121
to
+126
| function resolveImageSrc(src: string | undefined, projectName: string | undefined): string | undefined { | ||
| if (!src || !projectName) return src; | ||
| if (src.startsWith('data:') || src.startsWith('http://') || src.startsWith('https://')) return src; | ||
| const cleaned = src.replace(/^\.\//, ''); | ||
| return `/api/projects/${encodeURIComponent(projectName)}/files/content?path=${encodeURIComponent(cleaned)}`; | ||
| } |
Comment on lines
1
to
4
| import type { CanonicalModelRequest, ModelConfig, ModelDefinition, ProviderConfig } from "../protocol/canonical.js"; | ||
| import { ModelRequestError } from "../protocol/errors.js"; | ||
| import { assertContentSupported } from "../protocol/multimodal.js"; | ||
| import { assertContentSupported, downgradeUnsupportedContent } from "../protocol/multimodal.js"; | ||
|
|
Comment on lines
41
to
46
|
|
||
| downgradeUnsupportedContent(request.messages, model.multimodal); | ||
|
|
||
| for (const message of request.messages) { | ||
| assertContentSupported(message.content, model.multimodal); | ||
| } |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
前端图片显示&流式显示优化