Update yutori-computer-use templates for n1-latest (#113)

## Summary
- Update yutori templates (Python + TypeScript) to the `n1-latest` model
API, which uses OpenAI-compatible `tool_calls` format instead of the
previous custom format
- Remove Playwright computer mode in favor of a single Computer Controls
path (Playwright is still used for `goto_url` in kiosk mode)
- Add kiosk mode option that hides browser chrome and uses Playwright
for navigation
- Rename `max_tokens` to `max_completion_tokens`, update viewport to
1280x800, simplify the sampling loop and tool mapping

## Test plan
All 13 action types were tested against live Kernel browser sessions for
both Python and TypeScript implementations (27 tests each, all passing):

| Action | Status |
|---|---|
| `left_click`, `double_click`, `triple_click`, `right_click` | Pass |
| `hover`, `drag` | Pass |
| `type` (basic, `clear_before_typing`, `press_enter_after`) | Pass |
| `key_press` (single key, combos, modifier mapping) | Pass |
| `scroll` (up, down, left, right) | Pass |
| `goto_url`, `go_back`, `refresh`, `wait` | Pass |
| Error handling (unknown action, missing required fields, invalid
direction) | Pass |


Made with [Cursor](https://cursor.com)

<!-- CURSOR_SUMMARY -->
---

> [!NOTE]
> **Medium Risk**
> Moderate risk because it rewires the agent loop/tool protocol and
action schema for both Python and TypeScript templates, which can break
runtime behavior if the `tool_calls` or coordinate/action mapping
differs from expectations.
> 
> **Overview**
> Updates the `yutori-computer-use` Python and TypeScript templates to
use Yutori `n1-latest`’s OpenAI-compatible `tool_calls` flow (tool
result messages keyed by `tool_call_id`) instead of the prior
JSON-in-content + `observation` approach.
> 
> Simplifies execution to a single Computer Controls path by removing
the Playwright “mode”, adding an optional `kiosk`/`kioskMode` that
launches the browser in kiosk mode and uses Playwright only for
`goto_url`, and updating action/schema details (e.g., `coordinates` +
new click variants, `max_completion_tokens`, default viewport
`1280x800`, WebP screenshot encoding via `Pillow`/`sharp`). QA docs were
updated to reflect the new Yutori invocation patterns, and `.gitignore`
now excludes `.cursor/plans/`.
> 
> <sup>Written by [Cursor
Bugbot](https://cursor.com/dashboard?tab=bugbot) for commit
c3757ab. This will update automatically
on new commits. Configure
[here](https://cursor.com/dashboard?tab=bugbot).</sup>
<!-- /CURSOR_SUMMARY -->

---------

Co-authored-by: Cursor <cursoragent@cursor.com>

Author: dprevoznik

.cursor/commands/qa.md

@@ -60,8 +60,6 @@ Here are all valid language + template combinations:
 | typescript | claude-agent-sdk       | ts-claude-agent-sdk | ts-claude-agent-sdk | Yes            | ANTHROPIC_API_KEY              |
 | typescript | yutori-computer-use    | ts-yutori-cua     | ts-yutori-cua         | Yes            | YUTORI_API_KEY                 |
 
-> **Note:** The `yutori-computer-use` template supports two modes: `computer_use` (default, full VM screenshots) and `playwright` (viewport-only screenshots via CDP). Both modes should be tested.
-
 | python     | sample-app             | py-sample-app     | python-basic          | No             | -                              |
 | python     | gemini-computer-use    | py-gemini-cua     | python-gemini-cua     | Yes            | GOOGLE_API_KEY                 |
 | python     | captcha-solver         | py-captcha-solver | python-captcha-solver | No             | -                              |
@@ -72,9 +70,7 @@ Here are all valid language + template combinations:
 | python     | claude-agent-sdk       | py-claude-agent-sdk | py-claude-agent-sdk | Yes            | ANTHROPIC_API_KEY              |
 | python     | yutori-computer-use    | py-yutori-cua     | python-yutori-cua     | Yes            | YUTORI_API_KEY                 |
 
-> **Yutori Modes:**
-> - `computer_use` (default): Uses Kernel's Computer Controls API with full VM screenshots
-> - `playwright`: Uses Playwright via CDP WebSocket for viewport-only screenshots (optimized for n1 model)
+> **Yutori:** Test both default browser and `"kiosk": true` (uses Playwright for goto_url when kiosk is enabled).
 
 ### Create Commands
 
@@ -275,8 +271,8 @@ kernel invoke ts-magnitude mag-url-extract --payload '{"url": "https://en.wikipe
 kernel invoke ts-openai-cua cua-task --payload '{"task": "Go to https://news.ycombinator.com and get the top 5 articles"}'
 kernel invoke ts-gemini-cua cua-task --payload '{"query": "Go to https://www.magnitasks.com, Click the Tasks option in the left-side bar, and move the 5 items in the To Do and In Progress items to the Done section of the Kanban board. You are done successfully when the items are moved.", "record_replay": true}'
 kernel invoke ts-claude-agent-sdk agent-task --payload '{"task": "Go to https://news.ycombinator.com and get the top 3 stories"}'
-kernel invoke ts-yutori-cua cua-task --payload '{"query": "Go to https://www.magnitasks.com, Click the Tasks option in the left-side bar, and drag the 5 items in the To Do and In Progress columns to the Done section of the Kanban board. You are done successfully when the items are dragged to Done. Do not click into the items.", "record_replay": true, "mode": "computer_use"}'
-kernel invoke ts-yutori-cua cua-task --payload '{"query": "Go to https://www.magnitasks.com, Click the Tasks option in the left-side bar, and drag the 5 items in the To Do and In Progress columns to the Done section of the Kanban board. You are done successfully when the items are dragged to Done. Do not click into the items.", "record_replay": true, "mode": "playwright"}'
+kernel invoke ts-yutori-cua cua-task --payload '{"query": "Go to https://www.magnitasks.com, Click the Tasks option in the left-side bar, and drag the 5 items in the To Do and In Progress columns to the Done section of the Kanban board. You are done successfully when the items are dragged to Done. Do not click into the items.", "record_replay": true}'
+kernel invoke ts-yutori-cua cua-task --payload '{"query": "Go to https://www.magnitasks.com, Click the Tasks option in the left-side bar, and drag the 5 items in the To Do and In Progress columns to the Done section of the Kanban board. You are done successfully when the items are dragged to Done. Do not click into the items.", "record_replay": true, "kiosk": true}'
 
 # Python apps
 kernel invoke python-basic get-page-title --payload '{"url": "https://www.google.com"}'
@@ -287,8 +283,8 @@ kernel invoke python-openai-cua cua-task --payload '{"task": "Go to https://news
 kernel invoke python-openagi-cua openagi-default-task -p '{"instruction": "Navigate to https://agiopen.org and click the What is Computer Use? button"}'
 kernel invoke py-claude-agent-sdk agent-task --payload '{"task": "Go to https://news.ycombinator.com and get the top 3 stories"}'
 kernel invoke python-gemini-cua cua-task --payload '{"query": "Go to https://www.magnitasks.com, Click the Tasks option in the left-side bar, and move the 5 items in the To Do and In Progress items to the Done section of the Kanban board. You are done successfully when the items are moved.", "record_replay": true}'
-kernel invoke python-yutori-cua cua-task --payload '{"query": "Go to https://www.magnitasks.com, Click the Tasks option in the left-side bar, and drag the 5 items in the To Do and In Progress columns to the Done section of the Kanban board. You are done successfully when the items are dragged to Done. Do not click into the items.", "record_replay": true, "mode": "computer_use"}'
-kernel invoke python-yutori-cua cua-task --payload '{"query": "Go to https://www.magnitasks.com, Click the Tasks option in the left-side bar, and drag the 5 items in the To Do and In Progress columns to the Done section of the Kanban board. You are done successfully when the items are dragged to Done. Do not click into the items.", "record_replay": true, "mode": "playwright"}'
+kernel invoke python-yutori-cua cua-task --payload '{"query": "Go to https://www.magnitasks.com, Click the Tasks option in the left-side bar, and drag the 5 items in the To Do and In Progress columns to the Done section of the Kanban board. You are done successfully when the items are dragged to Done. Do not click into the items.", "record_replay": true}'
+kernel invoke python-yutori-cua cua-task --payload '{"query": "Go to https://www.magnitasks.com, Click the Tasks option in the left-side bar, and drag the 5 items in the To Do and In Progress columns to the Done section of the Kanban board. You are done successfully when the items are dragged to Done. Do not click into the items.", "record_replay": true, "kiosk": true}'
 ```
 
 ## Step 7: Automated Runtime Testing (Optional)
@@ -313,8 +309,8 @@ If the human agrees, invoke each template use the Kernel CLI and collect results
 | ts-openai-cua     | ts-openai-cua         |         |       |
 | ts-gemini-cua     | ts-gemini-cua         |         |       |
 | ts-claude-agent-sdk | ts-claude-agent-sdk |         |       |
-| ts-yutori-cua     | ts-yutori-cua         |         | mode: computer_use |
-| ts-yutori-cua     | ts-yutori-cua         |         | mode: playwright |
+| ts-yutori-cua     | ts-yutori-cua         |         | default |
+| ts-yutori-cua     | ts-yutori-cua         |         | kiosk: true |
 | py-sample-app     | python-basic          |         |       |
 | py-captcha-solver | python-captcha-solver |         |       |
 | py-browser-use    | python-bu             |         |       |
@@ -323,8 +319,8 @@ If the human agrees, invoke each template use the Kernel CLI and collect results
 | py-openagi-cua    | python-openagi-cua    |         |       |
 | py-claude-agent-sdk | py-claude-agent-sdk |         |       |
 | py-gemini-cua     | python-gemini-cua     |         |       |
-| py-yutori-cua     | python-yutori-cua     |         | mode: computer_use |
-| py-yutori-cua     | python-yutori-cua     |         | mode: playwright |
+| py-yutori-cua     | python-yutori-cua     |         | default |
+| py-yutori-cua     | python-yutori-cua     |         | kiosk: true |
 
 Status values:
 - **SUCCESS**: App started and returned a result

.gitignore

@@ -37,6 +37,9 @@ report.[0-9]_.[0-9]_.[0-9]_.[0-9]_.json
 
 # Finder (MacOS) folder config
 .DS_Store
+
+# Cursor
+.cursor/plans/
 kernel
 
 # QA testing directories

pkg/templates/python/yutori-computer-use/README.md

@@ -20,7 +20,7 @@ kernel deploy main.py --env-file .env
 ## Usage
 
 ```bash
-kernel invoke python-yutori-cua cua-task --payload '{"query": "Navigate to https://example.com and describe the page"}'
+kernel invoke python-yutori-cua cua-task --payload '{"query": "Go to https://www.magnitasks.com, Click the Tasks option in the left-side bar, and drag the 5 items in the To Do and In Progress columns to the Done section of the Kanban board. You are done successfully when the items are dragged to Done. Do not click into the items."}'
 ```
 
 ## Recording Replays
@@ -35,19 +35,44 @@ kernel invoke python-yutori-cua cua-task --payload '{"query": "Navigate to https
 
 When enabled, the response will include a `replay_url` field with a link to view the recorded session.
 
+## Kiosk mode
+
+Prefer **non-kiosk mode** by default and when the agent is expected to switch domains via URL. Use **kiosk (`"kiosk": true`)** when: (1) you're recording sessions and want a cleaner UI in the replay, or (2) you're automating on a single website and the combination of the complex site layout and browser chrome (address bar, tabs) may confuse the agent. 
+
+Note: In kiosk mode the agent may still try to use the address bar to enter URLs; it's not available, so it will eventually use `goto_url`, but those attempts may result in slowdown of the overall session.
+
+Default (non-kiosk):
+
+```bash
+kernel invoke python-yutori-cua cua-task --payload '{"query": "Navigate to https://example.com, then navigate to ign.com and describe the page"}'
+```
+
+With kiosk (single-site or recording):
+
+```bash
+kernel invoke python-yutori-cua cua-task --payload '{"query": "Enter https://example.com in the search box and then describe the page.", "kiosk": true}'
+```
+
 ## Viewport Configuration
 
-Yutori n1 recommends a **1280×800 (WXGA, 16:10)** viewport for best grounding accuracy. Kernel's closest supported viewport is **1200×800 at 25Hz**, which this template uses by default.
+Yutori n1 recommends a **1280×800 (WXGA, 16:10)** viewport for best grounding accuracy.
 
-> **Note:** n1 outputs coordinates in a 1000×1000 relative space, which are automatically scaled to the actual viewport dimensions. The slight width difference (1200 vs 1280) should have minimal impact on accuracy.
+> **Note:** n1 outputs coordinates in a 1000×1000 relative space, which are automatically scaled to the actual viewport dimensions.
 
 See [Kernel Viewport Documentation](https://www.kernel.sh/docs/browsers/viewport) for all supported configurations.
 
-## n1 Supported Actions
+## Screenshots
+
+Screenshots are automatically converted to WebP format for better compression across multi-step trajectories, as recommended by Yutori.
+
+## n1-latest Supported Actions
 
 | Action | Description |
 |--------|-------------|
-| `click` | Left mouse click at coordinates |
+| `left_click` | Left mouse click at coordinates |
+| `double_click` | Double-click at coordinates |
+| `triple_click` | Triple-click at coordinates |
+| `right_click` | Right mouse click at coordinates |
 | `scroll` | Scroll page in a direction |
 | `type` | Type text into focused element |
 | `key_press` | Send keyboard input |
@@ -57,7 +82,6 @@ See [Kernel Viewport Documentation](https://www.kernel.sh/docs/browsers/viewport
 | `refresh` | Reload current page |
 | `go_back` | Navigate back in history |
 | `goto_url` | Navigate to a URL |
-| `stop` | End task with final answer |
 
 ## Resources