fix: update Zerobus Ingest skill for SDK v1.1.0 breaking API changes (#291)

* fix: update Zerobus Ingest skill for SDK v1.1.0 breaking API changes

SDK v1.1.0 introduces breaking changes from v0.2.x that cause skill-generated
code to fail at runtime. All changes verified through E2E testing.

Key fixes:
- Constructor: positional args -> keyword args (host=, unity_catalog_url=)
- Ingest pattern: ingest_record(json.dumps(record)) + flush()
- SDK version: >=0.2.0 -> >=1.0.0
- Status: Public Preview -> GA (Feb 2026)
- Add serverless compute limitation and REST API alternative
- Add explicit table grants requirement (Error 4024)
- Fix typos: "speficfied" -> "specified", "Workslfow" -> "Workflow"

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* revert constructor to positional style (matches official docs)

The ZerobusSdk constructor works fine with positional args. The internal
parameter names changed in v1.1.0 but positional passing still works.
Reverted to match the official Databricks documentation.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>

Author: yogesh-dbx

databricks-skills/databricks-zerobus-ingest/1-setup-and-authentication.md

@@ -76,21 +76,25 @@ GRANT MODIFY, SELECT ON TABLE my_catalog.my_schema.my_events TO `<service-princi
 
 **Tip:** For broader access (e.g., writing to multiple tables in a schema), grant `MODIFY` and `SELECT` at the schema level instead.
 
+**Important:** For Zerobus, always grant explicit table-level `MODIFY` and `SELECT` permissions in addition to catalog/schema access. Schema-level inherited grants may not be sufficient for the OAuth `authorization_details` flow used by Zerobus.
+
 ---
 
 ## 4. Install the SDK
 
 ### Python (3.9+)
 
 ```bash
-pip install databricks-zerobus-ingest-sdk
+pip install databricks-zerobus-ingest-sdk>=1.0.0
 ```
 
 Or with a virtual environment:
 ```bash
-uv pip install databricks-zerobus-ingest-sdk
+uv pip install databricks-zerobus-ingest-sdk>=1.0.0
 ```
 
+**Note:** The Zerobus SDK cannot be pip-installed on Databricks serverless compute. Use classic compute clusters, or use the [Zerobus REST API](https://docs.databricks.com/aws/en/ingestion/zerobus-rest-api) (Beta) for notebook-based ingestion without the SDK.
+
 ### Java (8+)
 
 Maven:

databricks-skills/databricks-zerobus-ingest/2-python-client.md

@@ -355,4 +355,4 @@ stream.wait_for_offset(offset)
 | `ingest_records_nowait(records)` | None | No | Max batch throughput |
 | `wait_for_offset(offset)` | None | Yes (until ACK) | Durability confirmation |
 | `flush()` | None | Yes (until sent) | Ensure all buffered records are sent |
-| `ingest_record(record)` | RecordAcknowledgment | No | **Deprecated** — use `ingest_record_offset` |
+| `ingest_record(record)` | RecordAcknowledgment | No | Primary method in SDK v1.1.0+; pass `json.dumps(record)` for JSON |

databricks-skills/databricks-zerobus-ingest/SKILL.md

@@ -7,7 +7,7 @@ description: "Build Zerobus Ingest clients for near real-time data ingestion int
 
 Build clients that ingest data directly into Databricks Delta tables via the Zerobus gRPC API.
 
-**Status:** Public Preview (currently free; Databricks plans to introduce charges in the future)
+**Status:** GA (Generally Available since February 2026; billed under Lakeflow Jobs Serverless SKU)
 
 **Documentation:**
 - [Zerobus Overview](https://docs.databricks.com/aws/en/ingestion/zerobus-overview)
@@ -37,7 +37,7 @@ Zerobus Ingest is a serverless connector that enables direct, record-by-record d
 | Schema generation from UC table | Any | Protobuf | [4-protobuf-schema.md](4-protobuf-schema.md) |
 | Retry / reconnection logic | Any | Any | [5-operations-and-limits.md](5-operations-and-limits.md) |
 
-If not speficfied, default to python.
+If not specified, default to python.
 
 ---
 
@@ -46,7 +46,7 @@ If not speficfied, default to python.
 These libraries are essential for ZeroBus data ingestion:
 
 - **databricks-sdk>=0.85.0**: Databricks workspace client for authentication and metadata
-- **databricks-zerobus-ingest-sdk>=0.2.0**: ZeroBus SDK for high-performance streaming ingestion
+- **databricks-zerobus-ingest-sdk>=1.0.0**: ZeroBus SDK for high-performance streaming ingestion
 - **grpcio-tools**
 These are typically NOT pre-installed on Databricks. Install them using `execute_databricks_command` tool:
 - `code`: "%pip install databricks-sdk>=VERSION databricks-zerobus-ingest-sdk>=VERSION"
@@ -85,6 +85,7 @@ See [1-setup-and-authentication.md](1-setup-and-authentication.md) for complete
 ## Minimal Python Example (JSON)
 
 ```python
+import json
 from zerobus.sdk.sync import ZerobusSdk
 from zerobus.sdk.shared import RecordType, StreamConfigurationOptions, TableProperties
 
@@ -95,8 +96,8 @@ table_props = TableProperties(table_name)
 stream = sdk.create_stream(client_id, client_secret, table_props, options)
 try:
     record = {"device_name": "sensor-1", "temp": 22, "humidity": 55}
-    offset = stream.ingest_record_offset(record)
-    stream.wait_for_offset(offset)
+    stream.ingest_record(json.dumps(record))
+    stream.flush()
 finally:
     stream.close()
 ```
@@ -115,7 +116,7 @@ finally:
 
 ---
 
-You must always follow all the steps in the Workslfow
+You must always follow all the steps in the Workflow
 
 ## Workflow
 0. **Display the plan of your execution**
@@ -129,8 +130,10 @@ You must always follow all the steps in the Workslfow
 ---
 
 ## Important
-- Never install local packages 
+- Never install local packages
 - Always validate MCP server requirement before execution
+- **Serverless limitation**: The Zerobus SDK cannot pip-install on serverless compute. Use classic compute clusters, or use the [Zerobus REST API](https://docs.databricks.com/aws/en/ingestion/zerobus-rest-api) (Beta) for notebook-based ingestion without the SDK.
+- **Explicit table grants**: Service principals need explicit `MODIFY` and `SELECT` grants on the target table. Schema-level inherited permissions may not be sufficient for the `authorization_details` OAuth flow.
 
 ---
 
@@ -173,7 +176,7 @@ When execution fails:
 Databricks provides Spark, pandas, numpy, and common data libraries by default. **Only install a library if you get an import error.**
 
 Use `execute_databricks_command` tool:
-- `code`: "%pip install databricks-zerobus-ingest-sdk>=0.2.0"
+- `code`: "%pip install databricks-zerobus-ingest-sdk>=1.0.0"
 - `cluster_id`: "<cluster_id>"
 - `context_id`: "<context_id>"
 
@@ -193,7 +196,7 @@ The timestamp generation must use microseconds for Databricks.
 - **gRPC + Protobuf**: Zerobus uses gRPC as its transport protocol. Any application that can communicate via gRPC and construct Protobuf messages can produce to Zerobus.
 - **JSON or Protobuf serialization**: JSON for quick starts; Protobuf for type safety, forward compatibility, and performance.
 - **At-least-once delivery**: The connector provides at-least-once guarantees. Design consumers to handle duplicates.
-- **Durability ACKs**: Each ingested record returns an offset. Use `wait_for_offset(offset)` to confirm durable write. ACKs indicate all records up to that offset have been durably written.
+- **Durability ACKs**: Each ingested record returns a `RecordAcknowledgment`. Use `flush()` to ensure all buffered records are durably written, or use `wait_for_offset(offset)` for offset-based tracking.
 - **No table management**: Zerobus does not create or alter tables. You must pre-create your target table and manage schema evolution yourself.
 - **Single-AZ durability**: The service runs in a single availability zone. Plan for potential zone outages.
 
@@ -210,6 +213,8 @@ The timestamp generation must use microseconds for Databricks.
 | **Throughput limits hit** | Max 100 MB/s and 15,000 rows/s per stream. Open multiple streams or contact Databricks. |
 | **Region not supported** | Check supported regions in [5-operations-and-limits.md](5-operations-and-limits.md). |
 | **Table not found** | Ensure table is a managed Delta table in a supported region with correct three-part name. |
+| **SDK install fails on serverless** | The Zerobus SDK cannot be pip-installed on serverless compute. Use classic compute clusters or the REST API (Beta) from notebooks. |
+| **Error 4024 / authorization_details** | Service principal lacks explicit table-level grants. Grant `MODIFY` and `SELECT` directly on the target table — schema-level inherited grants may be insufficient. |
 
 ---