Enhance chat backdrop with new tab background and effects

CHANGELOG.md

@@ -7,6 +7,7 @@ This changelog was generated from the repository Git history and release tags. V
 ## [Unreleased]
 
 ### Fixed
+- Replies cut off by the model's output limit retain their partial text, show a localized notice asking you to request the rest, and end as partial rather than successful. This applies to streaming and non-streaming answers in Chrome and Firefox, without automatically retrying the request or replaying browser actions.
 - Vision mode can attach more than two files in one task. Each file goes through its own upload hand-off, and a run used to stop after two hand-offs even when both uploads worked: the third file, such as the last zip of a three-asset GitHub release, was refused with "file uploads are not available in vision mode", and the agent asked you to turn Vision mode off. Now only hand-offs that end without attaching a file count toward the limit of two. When that limit is reached, the agent says the earlier attempts attached nothing instead of blaming vision mode. Chrome and Firefox.
 - Vision mode attaches each file with the first upload attempt. After a click on an upload control hands the upload to the element tools, the agent uses the file input the click named, but WebBrain's page check refused that first attempt when the input was hidden, as on a GitHub release form, because it had not seen that input itself. The agent then had to list the page's file inputs and send the same upload again, two wasted steps per file. The hand-off now records that exact input before the agent's next step, so the first attempt attaches the file. A file input the agent guessed is still refused. Chrome and Firefox.
 

docs/newtab.md

@@ -68,6 +68,29 @@ its own tool calls stay in the conversation it sees. Page tools are never offere
 (WebGPU) model stays tool-free, and the standalone window itself is unchanged. The first-open pin
 coachmark is left for the first real side panel, since it points at the panel's header.
 
+The full-page chat is a centred column (up to 880 px; a narrower window is all chat) on the
+background the new tab was showing, so asking feels like the new tab opening up rather than a jump
+to another page. Right before navigating, the new tab writes a description of that background to
+its own tab's `sessionStorage` (`wbTabChatBackdrop`, see
+`newtab/src/features/background/api/chatBackdrop.js`). Both pages are on the extension's origin and
+`sessionStorage` is per tab, so the chat finds it on its first paint, keeps it across reloads, and
+no other tab sees it. `src/ui/tab-chat-backdrop.js` (identical in both extensions, a classic script
+in the page's `<head>` like the theme bootstrap) reads and checks it before first paint:
+
+- A bundled or remote picture goes by its address (only this extension's own files or http(s)).
+  An uploaded one goes by its id in the new tab's `MueCustomBackgrounds` database, which the chat
+  reads only if it already exists, so a multi-MB upload is never copied.
+- A colour or gradient goes as its CSS value. The browser must accept it and it may not contain
+  `url()`.
+- Blur, brightness and the effect filter carry over, with bounded numbers, for pictures and
+  colours alike. They're applied in the new tab's order: blur and the effect first, then the
+  darkening.
+- Videos are not carried over. Neither is anything when backgrounds are turned off, or when the
+  chat was opened some other way. The column then sits on a plain, slightly darker surround.
+
+The column itself stays solid, so answers are never drawn over the picture. Back still shows a new
+random photo on the new tab, as any new tab load does.
+
 1. In the submit handler (no `await` before it, to keep the user gesture) the page
    writes `{id, tabId, text, mode, createdAt}` to extension storage key
    `wbNewtabHandoff:<tabId>` (one key per tab, so different tabs cannot overwrite each

newtab/src/features/agent/handoff.js

@@ -17,7 +17,12 @@
  *     still applies when it is on, and the box warns about it before the user submits);
  *     if Act is not allowed for the panel's current provider or scope, the text is left
  *     in the composer, unsent.
+ *
+ * Ask also leaves the background this new tab is showing for the chat to open on
+ * (features/background/api/chatBackdrop.js).
  */
+import { saveChatBackdrop } from '../background/api/chatBackdrop.js';
+
 // One key per tab, so concurrent submissions from several new tab pages can never
 // overwrite each other. Mirrors src/*/src/ui/newtab-handoff.js (checked in test/newtab.mjs).
 export const HANDOFF_PREFIX = 'wbNewtabHandoff:';
@@ -198,9 +203,16 @@ export const TAB_CHAT_PATH = 'src/ui/sidepanel.html?mode=ask&standalone=true&tab
  * Ask: store the task for this tab, then load the full-page chat in it. The chat page
  * claims the record addressed to its own tab, as the side panel would. No user gesture is
  * needed (it is a navigation, not sidePanel.open), so the write is awaited first: leaving
- * the page before it lands could lose the task.
+ * the page before it lands could lose the task. The background on screen is left for the
+ * chat (`rememberBackdrop`) right before leaving.
  */
-export function openAskChat({ text }, api = extApi(), tab = currentTab, navigate = (url) => window.location.assign(url)) {
+export function openAskChat(
+  { text },
+  api = extApi(),
+  tab = currentTab,
+  navigate = (url) => window.location.assign(url),
+  rememberBackdrop = saveChatBackdrop,
+) {
   if (!api?.storage?.local?.set || !api?.runtime?.getURL || tab?.id == null) {
     return Promise.reject(new Error('Extension API unavailable'));
   }
@@ -216,6 +228,11 @@ export function openAskChat({ text }, api = extApi(), tab = currentTab, navigate
   }
   return stored.then(
     async () => {
+      try {
+        rememberBackdrop();
+      } catch {
+        // cosmetic only: the chat then opens on its plain surround
+      }
       try {
         navigate(api.runtime.getURL(TAB_CHAT_PATH));
       } catch (error) {

newtab/src/features/background/Background.jsx

@@ -5,6 +5,9 @@ import { useBackgroundState } from './hooks/useBackgroundState';
 import { useBackgroundLoader } from './hooks/useBackgroundLoader';
 import { useBackgroundRenderer, useBackgroundFilters, useBackgroundOverlayFilters } from './hooks/useBackgroundRenderer';
 import { useBackgroundEvents } from './hooks/useBackgroundEvents';
+import { setShownBackground } from './api/chatBackdrop';
+
+import { useEffect } from 'react';
 
 import './scss/index.scss';
 
@@ -23,6 +26,8 @@ export default function Background() {
     if (failedUrl === backgroundData.url && !backgroundData.video) updateBackground({ video: true });
   });
   useBackgroundEvents(backgroundData, refreshBackground);
+  // Remembered for the full-page chat Ask opens in this tab, which keeps the same background.
+  useEffect(() => setShownBackground(backgroundData), [backgroundData]);
 
   return (
     <>

newtab/src/features/background/api/backgroundLoader.js

@@ -110,6 +110,8 @@ async function getCustomBackground(isOffline) {
     photoInfo: {
       hidden: true,
       blur_hash: selected.blurHash || null,
+      // Lets the full-page chat show the same upload without copying it (api/chatBackdrop.js).
+      ...(Number.isInteger(selected.id) ? { customId: selected.id } : {}),
     },
   };
 

newtab/src/features/background/api/chatBackdrop.js

@@ -0,0 +1,106 @@
+/**
+ * The background this new tab is showing, handed to the full-page chat Ask turns the tab into
+ * (openAskChat in features/agent/handoff.js), so the chat opens on the same picture instead of
+ * a new random one or a flat page.
+ *
+ * It is written to this tab's sessionStorage just before the tab navigates. Both pages belong to
+ * the extension's origin and sessionStorage is per tab, so the chat finds it on its first paint,
+ * keeps it across reloads, and no other tab sees it. The chat reads and checks it in
+ * src/*\/src/ui/tab-chat-backdrop.js (CHAT_BACKDROP_KEY must match there; test/newtab.mjs checks).
+ *
+ * Only a description goes across, never the media itself: a bundled or remote picture by its
+ * address, an uploaded one by its id in the custom-background database (it can be several MB),
+ * a colour or gradient by its CSS value, plus the blur/brightness/filter settings. Videos are not
+ * carried over; the chat then uses its plain surround.
+ */
+export const CHAT_BACKDROP_KEY = 'wbTabChatBackdrop';
+export const CHAT_BACKDROP_VERSION = 1;
+
+// Mirrors the choices in options/optionTypes/backgroundImageEffects.js.
+const FILTERS = new Set(['grayscale', 'sepia', 'invert', 'saturate', 'contrast']);
+const MAX_CSS_VALUE = 300;
+
+let shown = null;
+
+/** Called by Background whenever the background on screen changes. */
+export function setShownBackground(data) {
+  shown = data ? { ...data, photoInfo: data.photoInfo ? { ...data.photoInfo } : data.photoInfo } : null;
+}
+
+const readSetting = (storage, key) => {
+  try {
+    return storage?.getItem?.(key) ?? null;
+  } catch {
+    return null;
+  }
+};
+
+const clampNumber = (value, min, max, fallback) => {
+  const number = Number(value);
+  return Number.isFinite(number) ? Math.min(max, Math.max(min, number)) : fallback;
+};
+
+function effects(storage) {
+  const filter = readSetting(storage, 'backgroundFilter');
+  return {
+    blur: clampNumber(readSetting(storage, 'blur') ?? 0, 0, 100, 0),
+    brightness: clampNumber(readSetting(storage, 'brightness') ?? 100, 0, 100, 100),
+    ...(FILTERS.has(filter)
+      ? { filter, filterAmount: clampNumber(readSetting(storage, 'backgroundFilterAmount') ?? 100, 0, 100, 100) }
+      : {}),
+  };
+}
+
+// Colour backgrounds are stored as a declaration ("background: #123456;" or a gradient).
+function backgroundValue(style) {
+  const match = /^\s*background\s*:\s*([^;]+?)\s*;?\s*$/i.exec(String(style || ''));
+  const value = match?.[1] ?? '';
+  if (!value || value.length > MAX_CSS_VALUE || /url\s*\(/i.test(value)) return null;
+  return value;
+}
+
+/**
+ * The description of `data` the chat can paint, or null when there is nothing to carry over
+ * (backgrounds turned off, a video, nothing shown yet).
+ */
+export function chatBackdropSnapshot(
+  data = shown,
+  { storage = globalThis.localStorage, base = globalThis.location?.href } = {},
+) {
+  if (!data || readSetting(storage, 'background') === 'false' || data.video) return null;
+
+  const url = String(data.url || '');
+  if (url) {
+    const colour = typeof data.photoInfo?.colour === 'string' && data.photoInfo.colour.length <= 64
+      ? { colour: data.photoInfo.colour }
+      : {};
+    const customId = Number(data.photoInfo?.customId);
+    if (/^(data|blob):/i.test(url)) {
+      // An uploaded picture: too large to copy, and a blob address dies with this page.
+      if (!Number.isInteger(customId) || customId < 0) return null;
+      return { v: CHAT_BACKDROP_VERSION, customId, ...colour, ...effects(storage) };
+    }
+    let href;
+    try {
+      href = new URL(url, base).href;
+    } catch {
+      return null;
+    }
+    return { v: CHAT_BACKDROP_VERSION, image: href, ...colour, ...effects(storage) };
+  }
+
+  // The new tab's effect overlay covers colours too, so their effects go along as well.
+  const background = backgroundValue(data.style);
+  return background ? { v: CHAT_BACKDROP_VERSION, background, ...effects(storage) } : null;
+}
+
+/** Writes (or clears) this tab's snapshot. Never throws: the chat simply opens without it. */
+export function saveChatBackdrop(session = globalThis.sessionStorage, snapshot = chatBackdropSnapshot()) {
+  try {
+    if (snapshot) session.setItem(CHAT_BACKDROP_KEY, JSON.stringify(snapshot));
+    else session.removeItem(CHAT_BACKDROP_KEY);
+    return Boolean(snapshot);
+  } catch {
+    return false;
+  }
+}

package.json

@@ -31,7 +31,7 @@
     "test:attachment-drop": "node test/attachment-drop.mjs",
     "test:workflow-editor": "node --test test/workflow-editor.mjs",
     "test:security": "node test/security/injection-corpus.mjs",
-    "test:cloud-bridge-approval": "node --test test/cloud-bridge-approval.mjs test/cloud-handoff.mjs test/bot-memory.mjs test/bot-memory-click-guard.mjs test/bot-memory-registries.mjs test/bot-memory-cloud-controller.mjs test/cloud-conversation-isolation.mjs test/cloud-run-outcomes.mjs test/cloud-act-completion.mjs test/cloud-model-recovery.mjs test/private-results.mjs",
+    "test:cloud-bridge-approval": "node --test test/ask-output-limit.mjs test/cloud-bridge-approval.mjs test/cloud-handoff.mjs test/bot-memory.mjs test/bot-memory-click-guard.mjs test/bot-memory-registries.mjs test/bot-memory-cloud-controller.mjs test/cloud-conversation-isolation.mjs test/cloud-run-outcomes.mjs test/cloud-act-completion.mjs test/cloud-model-recovery.mjs test/private-results.mjs",
     "test:discord": "node --test test/discord-guard.mjs",
     "test:toolbar-guard": "node test/rich-text-toolbar-guard.mjs",
     "test:pdf-read": "node test/pdf-read.mjs",