Initial commit

Signed-off-by: teodordelibasic-db <teodor.delibasic@databricks.com>

Author: teodordelibasic-db

.gitignore

@@ -19,6 +19,9 @@ buildNumber.properties
 .settings/
 .project
 .classpath
+.metals/
+.bsp/
+.bazelbsp/
 
 # OS
 .DS_Store

CHANGELOG.md

@@ -1,17 +1,96 @@
-# Version changelog
+# Changelog
 
-## Release v0.1.0
+All notable changes to the Databricks Zerobus Java SDK will be documented in this file.
 
-Initial release of the Databricks Zerobus Ingest SDK for Java.
+The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/),
+and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
 
-### API Changes
+## [Unreleased]
 
-- Added `ZerobusSdk` class for creating ingestion streams
-- Added `ZerobusStream` class for managing stateful gRPC streams
-- Added `RecordAcknowledgment` for blocking until record acknowledgment
-- Added `TableProperties` for configuring table schema and name
-- Added `StreamConfigurationOptions` for stream behavior configuration
-- Added `ZerobusException` and `NonRetriableException` for error handling
-- Added `StreamState` enum for tracking stream lifecycle
-- Added utility methods in `ZerobusSdkStubUtils` for gRPC stub management
-- Support for Java 8 and higher
+### Added
+
+#### Native Rust Backend (JNI Migration)
+- **Complete architecture rewrite**: The SDK now uses JNI (Java Native Interface) to call a high-performance Rust implementation instead of pure Java gRPC calls
+- Native library is automatically loaded from the classpath or system library path
+- Improved performance through zero-copy data passing and optimized async handling
+
+#### New APIs
+
+**Offset-Based Ingestion API** - High-throughput alternative to CompletableFuture-based API:
+- `ZerobusStream.ingestRecordOffset(IngestableRecord)` - Returns offset immediately without future allocation
+- `ZerobusStream.ingestRecordOffset(RecordType)` - Convenience overload for Protocol Buffer messages
+- `ZerobusStream.ingestRecordsOffset(Iterable)` - Batch ingestion returning last offset
+- `ZerobusStream.waitForOffset(long)` - Block until specific offset is acknowledged
+
+**JSON Record Support**:
+- `IngestableRecord` interface - Unified interface for all record types
+- `JsonRecord` class - JSON string wrapper implementing IngestableRecord
+- `ProtoRecord<T>` class - Protocol Buffer wrapper implementing IngestableRecord
+- `RecordType` enum - Specifies stream serialization format (`PROTO` or `JSON`)
+- `StreamConfigurationOptions.setRecordType(RecordType)` - Configure stream for JSON or Proto records
+- Both record types work with `ingestRecord()` and `ingestRecordOffset()` methods
+
+**Batch Operations**:
+- `ZerobusStream.ingestRecords(Iterable)` - Ingest multiple records with single acknowledgment
+- `ZerobusStream.getUnackedBatches()` - Get unacknowledged records preserving batch grouping
+- `EncodedBatch` class - Represents a batch of encoded records
+
+**Arrow Flight Support** (Experimental):
+- `ZerobusArrowStream` class - High-performance columnar data ingestion
+- `ArrowTableProperties` class - Table configuration with Arrow schema
+- `ArrowStreamConfigurationOptions` class - Arrow stream configuration
+- `ZerobusSdk.createArrowStream()` - Create Arrow Flight streams
+- `ZerobusSdk.recreateArrowStream()` - Recover failed Arrow streams
+
+**New Callback Interface**:
+- `AckCallback` interface with `onAck(long offsetId)` and `onError(long offsetId, String message)`
+- More detailed error information than the deprecated Consumer-based callback
+
+### Changed
+
+- `ZerobusSdk` now requires native library to be available at runtime
+- `ZerobusStream.close()` now throws `ZerobusException` instead of being silent on errors
+- `ZerobusArrowStream.close()` also throws `ZerobusException`
+- Improved error messages with more context from the Rust backend
+
+### Deprecated
+
+- `ZerobusStream.ingestRecord(RecordType)` - Use `ingestRecordOffset()` instead for better performance. The offset-based API avoids CompletableFuture allocation overhead.
+- `ZerobusStream.ingestRecord(IngestableRecord)` - Use `ingestRecordOffset()` instead.
+- `ZerobusStream.ingestRecords(Iterable)` - Use `ingestRecordsOffset()` instead.
+- `ZerobusStream.getState()` - Stream state is no longer exposed by the native backend. Returns `OPENED` or `CLOSED` only.
+- `ZerobusStream.getUnackedRecords()` - Returns empty iterator. Use `getUnackedBatches()` or `getUnackedRecordsRaw()` instead.
+- `StreamConfigurationOptions.Builder.setAckCallback(Consumer<IngestRecordResponse>)` - Use `setAckCallback(AckCallback)` instead.
+- `ZerobusSdk.setStubFactory()` - gRPC stub factory is no longer used with native backend. Throws `UnsupportedOperationException`.
+
+### Removed
+
+- Direct gRPC implementation - All network communication now goes through the Rust native library
+- Internal gRPC stub management classes (still present but unused)
+
+### Fixed
+
+- Memory leaks in long-running streams through proper native resource management
+- Race conditions in concurrent record ingestion
+- Improved backpressure handling through native implementation
+
+### Performance
+
+- **Reduced latency**: Direct native calls avoid gRPC Java overhead
+- **Lower memory footprint**: No CompletableFuture allocation for offset-based API
+- **Better throughput**: Optimized Rust async runtime handles network I/O
+- **Reduced GC pressure**: Fewer intermediate object allocations
+
+## [0.1.0] - Initial Release
+
+### Added
+
+- Initial SDK release with Protocol Buffer record ingestion
+- `ZerobusSdk` - Main entry point for creating streams
+- `ZerobusStream` - Stream for ingesting Protocol Buffer records
+- `TableProperties` - Table configuration
+- `StreamConfigurationOptions` - Stream configuration with builder pattern
+- `ZerobusException` and `NonRetriableException` - Exception hierarchy
+- `GenerateProto` tool - Generate Protocol Buffer schemas from Unity Catalog
+- Blocking and non-blocking ingestion examples
+- Comprehensive README documentation