fix metadata discovery

README.md

@@ -328,46 +328,38 @@ mb transform-tag delete 5 --yes
 
 ## Databases
 
-Read warehouse metadata from `/api/database`. The `db` group exposes the full database list, the per-database record, schema and table inspection, the two manual-sync triggers, and (rarely useful) full-warehouse rollup endpoints.
+Read warehouse metadata from `/api/database`. The `db` group exposes the full database list, the per-database record with optional table/field hydration, schema and table inspection, and the two manual-sync triggers.
 
 `db` is aliased to `database`.
 
-> **Agent traversal:** prefer the granular path — `db list` → `db schemas <db-id>` → `db schema-tables <db-id> <schema>` → `table get <table-id> --include fields`. On a real warehouse (dozens of schemas, hundreds of tables, dozens of fields per table) the rollup commands (`db metadata`, `db get --include tables.fields`, `db list --include tables`) return megabytes of JSON and exhaust the agent context. Reach for them only on small/dev warehouses where you know the size up front.
+> **Agent traversal — the hydration ladder:** start with `db get <db-id> --include tables`, the compact table map (id, name, schema, description per table) — one call that fits most databases. Pick the relevant tables, then fetch fields per table with `table fields <table-id>` (bounded: a table has at most a few hundred fields). `--include tables.fields` is the full rollup — small databases only. When output outgrows the `--max-bytes` cap, the error message names the next command down the ladder. On warehouses with hundreds of tables, traverse by schema (`db schemas <db-id>` → `db schema-tables <db-id> <schema>`) or find tables by name (`mb search <term> --models table --db-id <db-id>`).
 
 ### `mb db list`
 
 ```sh
 mb db list
 mb db list --json
 mb db list --saved --json
-mb db list --include tables --full --json   # rollup: every db with its full table list
+mb db list --include tables --json   # every db with its compact table map
 ```
 
-| Flag                | Description                                                                                                                                                                                                          |
-| ------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
-| `--include <which>` | Hydrate related entities. Currently only `tables` is supported (each database is returned with its `tables`). On real warehouses this returns hundreds of table records per db — use the granular traversal instead. |
-| `--saved`           | Include the Saved Questions virtual database in the list. The virtual db has id `-1337` and no `engine`.                                                                                                             |
+| Flag                | Description                                                                                                                                                                             |
+| ------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
+| `--include <which>` | Hydrate related entities. Currently only `tables` is supported (each database is returned with its compact `tables`). To map a single warehouse, prefer `db get <id> --include tables`. |
+| `--saved`           | Include the Saved Questions virtual database in the list. The virtual db has id `-1337` and no `engine`.                                                                                |
 
 ### `mb db get <id>`
 
 ```sh
 mb db get 1
 mb db get 1 --json
-mb db get 1 --include tables --full --json          # rollup: db + every table (compact)
-mb db get 1 --include tables.fields --full --json   # rollup: db + every table + every field
+mb db get 1 --include tables --json          # + compact table map (fits most databases)
+mb db get 1 --include tables.fields --json   # + every field of every table (small databases only)
 ```
 
-| Flag                | Description                                                                                                                                                                                                                                                                       |
-| ------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
-| `--include <which>` | Hydrate related entities. One of `tables` or `tables.fields`. `tables.fields` returns every column of every table in the database in one response — only safe on small/dev warehouses. For a real warehouse use `db schemas` → `db schema-tables` → `table get --include fields`. |
-
-### `mb db metadata <id>`
-
-Equivalent to `GET /api/database/:id/metadata`: a single database with all its tables and fields rolled up in one response. This is the largest read in the `db` group — on a real warehouse the response will exceed the agent context. Use only when you know the database is small (a seeded dev instance, a sample db, a freshly-bootstrapped test fixture). For agent-driven introspection on a real warehouse, walk `db schemas` → `db schema-tables` → `table get --include fields` instead.
-
-```sh
-mb db metadata 1 --json --full --max-bytes 0
-```
+| Flag                | Description                                                                                                                                                                                                                                                                                                     |
+| ------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
+| `--include <which>` | Hydrate related entities. `tables` is the compact table map — the recommended first call for schema discovery. `tables.fields` is the full rollup and fits only small databases; on anything larger, take the map and fetch fields per table with `table fields <table-id>`, or traverse by schema (see above). |
 
 ### `mb db schemas <id>`
 
@@ -430,7 +422,7 @@ mb table list --db-id 1 --json
 
 ### `mb table get <id>`
 
-Returns the basic table record (no fields). Pass `--include fields` to route through `/api/table/:id/query_metadata` so the response carries the table's columns compact-projected as `fields` — this is the default agent path for field introspection. Use `mb table fields <id>` if you only want the fields as a list envelope, or `mb table metadata <id>` when you also need FKs and dimensions hydrated.
+Returns the basic table record (no fields). Pass `--include fields` to route through `/api/table/:id/query_metadata` so the response carries the table's columns compact-projected as `fields` — this is the default agent path for field introspection (the response also carries FK targets and dimensions under `--full`). Use `mb table fields <id>` if you only want the fields as a list envelope.
 
 ```sh
 mb table get 42
@@ -451,14 +443,6 @@ mb table fields 42
 mb table fields 42 --json
 ```
 
-### `mb table metadata <id>`
-
-`GET /api/table/:id/query_metadata`: the table with its fields, FKs, dimensions, segments, and measures all hydrated. Heavier than `table get --include fields` — reach for it only when you actually need the FK / dimension / segment / measure data.
-
-```sh
-mb table metadata 42 --json --full --max-bytes 0
-```
-
 ### `mb table update <id>`
 
 Patch a table (`PUT /api/table/:id`). Body fields: `display_name`, `description`, `caveats`, `points_of_interest`, `entity_type`, `visibility_type`, `field_order`, `show_in_getting_started`. Pass the body via `--body`, `--file`, or stdin (exactly one).
@@ -1525,7 +1509,7 @@ mb skills path                              # absolute paths for direct Read
 mb skills path core                         # one path
 ```
 
-`mb skills get` honors the shared `--max-bytes` list cap. With the default 65 536 cap, `--all` will return only the first skill and emit a truncation notice — pass `--max-bytes 0` to dump every skill in one envelope.
+`mb skills get` honors the shared `--max-bytes` list cap. With the default 24 576 cap, `--all` will return only the first skill and emit a truncation notice — pass `--max-bytes 0` to dump every skill in one envelope.
 
 Bundled skills:
 

package.json

@@ -1,6 +1,6 @@
 {
   "name": "@metabase/cli",
-  "version": "0.1.18",
+  "version": "0.2.0",
   "description": "Metabase CLI",
   "license": "AGPL-3.0",
   "repository": {

skill-data/core/SKILL.md

@@ -59,22 +59,25 @@ Every list/get verb supports the same output flags:
 - `--json` — emit the full JSON envelope, safe for `jq`. Default is human-readable text.
 - `--full` — include every field (the compact projection is the default, and is the agent-facing contract).
 - `--fields a,b.c.d` — project specific dot-paths. Mutually exclusive with `--full`. **Paths are relative to each `data[]` item on list verbs, and to the root on single-item verbs.** So it's `--fields id,name` on `… list` / `database schema-tables` (`data.id` and `data[].id` both fail with `unknown field path: "data.id"`), and `--fields id,name,display` on `card get`, `--fields data.rows` on `mb query` (whose `data` is an object).
-- `--max-bytes <n>` — cap **list** output size (drops trailing items, sets `truncated`). Default 65536; `0` disables. Single-item commands (`get`, `metadata`) never truncate — when their output exceeds the cap they throw a `ConfigError` (exit 2: "output is N bytes, over the M-byte --max-bytes cap…"); raise `--max-bytes` or narrow with `--fields`.
+- `--max-bytes <n>` — cap **list** output size (drops trailing items, sets `truncated`). Default 24576 (sized to fit under agent-harness tool-output limits); `0` disables. Single-item commands (`get`) never truncate — when their output exceeds the cap they throw a `ConfigError` (exit 2: "output is N bytes, over the M-byte --max-bytes cap; …") whose tail names the remedy: on schema-shaped commands it is the exact narrower command to run instead — follow it rather than raising the cap.
+- JSON output is a single line when stdout is piped (pretty-printed only at a TTY) — always parse it, never scrape by line position.
 
-List envelope shape:
+List envelope shape (metadata first, so counts and the truncation marker survive if a consumer cuts the tail):
 
 ```json
 {
-  "data": [
-    /* items */
-  ],
   "returned": 10,
   "total": 42,
   "limit": 50,
-  "truncated": false
+  "truncated": { "reason": "max_bytes", "bytes": 123456 },
+  "data": [
+    /* items */
+  ]
 }
 ```
 
+`truncated` is present only when the cap dropped trailing items.
+
 `total` is best-effort and may be `null` (empty / permissions-filtered collections, or `--limit` early-stop); use `returned` for the count you got and `data.length` for the rendered slice.
 
 ## Body input (create / update / run)
@@ -118,8 +121,8 @@ mb transform --help --json | jq -r '.commands[].command'  # verbs under "transfo
 
 Routine verb shapes (list / get / create / update), every flag, and output schemas live in each command's `--help` (add `--json` for output schemas). Below is only what help does _not_ tell you: footguns and non-obvious behaviors.
 
-- **db traversal vs. rollup.** Default to granular: `database list` → `database schemas <db-id>` → `database schema-tables <db-id> <schema>` → `table get <table-id> --include fields`. The rollup endpoints (`database get --include tables.fields`, `database metadata <db-id>`) pull megabytes and blow the context window on any real warehouse — use them only on a small/dev db. `sync-schema` / `rescan-values` queue async work and return `{status:"ok"}` immediately; `sync-schema --wait` blocks until `initial_sync_status: complete`.
-- **table fields.** `table get` never returns fields on its own — pass `--include fields` (compact) or use `table fields <id>` (list envelope). `table metadata <id>` adds FKs + dimensions (heavier). `table update` patches table-level metadata only; physical columns aren't editable.
+- **db traversal: the hydration ladder.** Start with `database get <db-id> --include tables` — the compact table map (id, name, schema, description per table), one call that fits most databases. Pick the relevant tables, then `table fields <table-id>` per table (bounded: fields are per-table). `--include tables.fields` is the full rollup — small databases only; when either outgrows the cap the error message names the next command down the ladder. Hundreds of tables? Traverse by schema (`database schemas <db-id>` → `database schema-tables <db-id> <schema>`) or look tables up by name (`search <term> --models table --db-id <db-id> --limit 10`). `sync-schema` / `rescan-values` queue async work and return `{status:"ok"}` immediately; `sync-schema --wait` blocks until `initial_sync_status: complete`.
+- **table fields.** `table get` never returns fields on its own — pass `--include fields` (compact; the underlying query_metadata response also carries FK targets and dimensions, visible under `--full`) or use `table fields <id>` (list envelope). `table update` patches table-level metadata only; physical columns aren't editable.
 - **field has no `list`.** Fields are per-table — get them via `table get <id> --include fields`. Never enumerate fields across a whole db (context blow-up). `field summary` is live cardinality `{field_id, count, distincts}`; `field values` is the cached distinct set (`has_more_values: true` ⇒ truncated cache). `field update` patches metadata only (`base_type` isn't editable) — this is where you set a column's `semantic_type` or foreign-key target.
 - **upload (CSV → tables).** `upload csv --file <path>` creates a new table + model (prints `{model_id, table_id}`); `upload append <table-id>` / `upload replace <table-id> --file <path>` add to / overwrite a table **previously created by upload** (columns must match). The destination db+schema is admin-configured, not per-call — check with `mb setting get uploads-settings --json` (`db_id: null` ⇒ uploads off/unconfigured; needs admin to read). `--collection <id|root>` only sets the model's collection. Max 50 MB. Errors: **"The uploads database is not configured."** = no db has uploads enabled; **"Uploads are not enabled."** = the append/replace target isn't an uploaded table.
 - **card.** `dataset_query` is the **flat** `mbql/query` value, not a legacy `{type:"query",query:…}` envelope (→ `mbql`). `--export-format csv|xlsx` streams the raw export (pipe to a file), bypassing the JSON envelope. `archive` is the only delete; unarchive with `update --body '{"archived":false}'`. `visualization_settings` keys are scoped by `display` and aren't pre-flighted — see `visualization`.

skill-data/data-workflow/SKILL.md

@@ -92,7 +92,7 @@ Don't make the user name a _stage_ — but do find out _where their data lives_
 
 **Ask before you crawl.** If you don't already know which database/schema/table the user means, ask — one plain question short-circuits a dozen tool calls. The asymmetry: if they name a **database**, ask which **schema**; if they name a **table**, ask which **database**. "If you don't know, no problem — I'll look" is the fallback, not the opener.
 
-**When you do crawl,** use `core`'s cheap, narrowest-first ladder (never whole-warehouse rollups): `mb db list` → `db schemas <id>` → `db schema-tables <id> <schema>` → `table list [--db-id]` → `table fields <id>` (or `table metadata <id>` for FK targets and dimensions — heavier). Have a _name_ rather than a tree to walk? `mb search <query> [--models] [--db-id]`. Need to know what's in a column? `mb field summary <id>` (counts) and `field values <id>` (sample values). If a database looks freshly connected or an expected table is missing, offer `mb db sync-schema <id> --wait` before concluding it doesn't exist.
+**When you do crawl,** use `core`'s cheap, narrowest-first ladder (never whole-warehouse rollups): `mb db list` → `db schemas <id>` → `db schema-tables <id> <schema>` → `table list [--db-id]` → `table fields <id>` (or `table get <id> --include fields --full` for FK targets and dimensions). Have a _name_ rather than a tree to walk? `mb search <query> [--models] [--db-id]`. Need to know what's in a column? `mb field summary <id>` (counts) and `field values <id>` (sample values). If a database looks freshly connected or an expected table is missing, offer `mb db sync-schema <id> --wait` before concluding it doesn't exist.
 
 **Read the shape to pick a stage.** Raw, normalized, SaaS-synced tables (many tables, coded columns, `*_field`/`*_choice` lookups)? → **build clean tables** first. Already wide, clean, human-readable ones? Then it depends on the goal:
 

src/commands/context.test.ts

@@ -107,8 +107,8 @@ describe("resolveCommonFlags — fields CSV parsing", () => {
 });
 
 describe("resolveCommonFlags — maxBytes parsing", () => {
-  it("uses default 65536 when omitted", () => {
-    expect(resolveCommonFlags({}, { isTty: true }).maxBytes).toBe(65536);
+  it("uses default 24576 when omitted", () => {
+    expect(resolveCommonFlags({}, { isTty: true }).maxBytes).toBe(24576);
   });
 
   it("parses '0' as 0 (disables cap)", () => {
@@ -144,7 +144,7 @@ describe("resolveCommonFlags — full result shape", () => {
       format: "text",
       full: false,
       fields: undefined,
-      maxBytes: 65536,
+      maxBytes: 24576,
       url: undefined,
       apiKey: undefined,
       profile: undefined,