Fix tests

Author: aclark4life

.evergreen/README.md

@@ -66,19 +66,23 @@ pip install shrub.py
 
 ## Test Coverage
 
-The Rust extension currently passes **78% of BSON tests** (69/88 tests):
+The Rust extension currently passes **100% of BSON tests** (60 tests: 58 passing + 2 skipped):
 
 ### Passing Tests
 - Basic BSON encoding/decoding
-- All BSON types (ObjectId, DateTime, Decimal128, Regex, Code, etc.)
-- Binary data handling
+- All BSON types (ObjectId, DateTime, Decimal128, Regex, Binary, Code, Timestamp, etc.)
+- Binary data handling (including UUID with all representation modes)
 - Nested documents and arrays
 - Exception handling (InvalidDocument, InvalidBSON, OverflowError)
 - Error message formatting with document property
+- Datetime clamping and timezone handling
+- Custom classes and codec options
+- Buffer protocol support (bytes, bytearray, memoryview, array, mmap)
+- Unicode decode error handlers
+- BSON validation (document structure, string null terminators, size fields)
 
-### Known Limitations (22% of tests)
-- **Datetime edge cases** (9 tests) - Datetime clamping and timezone handling
-- **Advanced features** (8 tests) - Custom classes, UUID, buffer protocol, codec options
+### Skipped Tests
+- **2 tests** - Require optional numpy dependency
 
 ## Platform Support
 
@@ -89,15 +93,24 @@ The Rust extension is tested on:
 
 ## Performance
 
-The Rust extension provides comparable performance to the C extension with the benefit of:
-- Memory safety guarantees
-- Easier maintenance and debugging
+The Rust extension is currently **slower than the C extension** for both encoding and decoding:
+- Simple encoding: **0.84x** (16% slower than C)
+- Complex encoding: **0.21x** (5x slower than C)
+- Simple decoding: **0.42x** (2.4x slower than C)
+- Complex decoding: **0.29x** (3.4x slower than C)
+
+The main bottleneck is **Python FFI overhead** - creating Python objects from Rust incurs significant performance cost.
+
+**Benefits of Rust implementation:**
+- Memory safety guarantees (prevents buffer overflows and use-after-free bugs)
+- Easier maintenance and debugging with strong type system
 - Cross-platform compatibility via Rust's toolchain
+- 100% test compatibility with C extension
+
+**Recommendation:** C extension remains the default and recommended choice. The Rust extension demonstrates feasibility and correctness but is not yet performance-competitive for production use.
 
 ## Future Work
 
-- Complete datetime clamping implementation
-- Add codec options support
-- Implement custom class handling
-- Add UUID support
+- Performance optimization (type caching, reduce FFI overhead)
 - Performance benchmarking suite
+- Additional BSON type optimizations

README.md

@@ -266,15 +266,25 @@ python your_script.py
 
 ### Implementation Status
 
-**✅ Complete (100% test pass rate - 88/88 tests):**
-- All BSON types (ObjectId, DateTime, Decimal128, Regex, Binary, Code, Timestamp, MinKey/MaxKey, DBRef, etc.)
+**✅ Complete (100% test pass rate - 60 tests, 58 passing + 2 skipped):**
+
+**Encoding (Python → BSON bytes):**
+- Direct implementation for: Double, String, Document, Array, Binary, ObjectId, Boolean, DateTime, Null, Regex, Int32, Timestamp, Int64, Decimal128
+- Converts Python types to BSON using the Rust `bson` library
 - Full codec_options support (document_class, tz_aware, uuid_representation, datetime_conversion, etc.)
-- UUID encoding/decoding (all representation modes)
+- UUID encoding (all representation modes)
 - Datetime clamping and conversion modes
-- Unicode decode error handlers
-- BSON validation and error formatting
+- Key validation (checks for `$` prefix, `.` characters, null bytes)
 - Buffer protocol support
 
+**Decoding (BSON bytes → Python):**
+- Fast-path direct byte reading for: Double (0x01), String (0x02), Document (0x03), Array (0x04), Boolean (0x08), Null (0x0A), Int32 (0x10), Int64 (0x12)
+- Fallback to Rust `bson` library for: Binary (0x05), ObjectId (0x07), DateTime (0x09), Regex (0x0B), DBPointer (0x0C), Symbol (0x0E), Code (0x0F), Timestamp (0x11), Decimal128 (0x13), and other types
+- BSON validation (document structure, string null terminators, size fields)
+- Proper error messages matching C extension format
+- Unicode decode error handlers
+- Field name tracking for error reporting in nested structures
+
 ### Performance Results
 
 **Current Performance (vs C extension):**
@@ -283,12 +293,13 @@ python your_script.py
 - Simple decoding: **0.42x** (2.4x slower than C)
 - Complex decoding: **0.29x** (3.4x slower than C)
 
-**Implementation Status:**
+**Architecture:**
 - ✅ Hybrid encoding strategy (fast path for PyDict, `items()` for other mappings)
-- ✅ Direct buffer writing with `doc.to_writer()`
+- ✅ Direct buffer writing with `doc.to_writer()` for nested documents
 - ✅ Efficient `_id` field ordering at top level
-- ✅ Direct byte reading for decoding (single-pass bytes → Python dict)
-- ✅ 100% test pass rate (88/88 tests)
+- ✅ Direct byte reading for common types (single-pass bytes → Python dict)
+- ✅ Fallback to Rust `bson` library for less common types
+- ✅ 100% test pass rate (60 tests: 58 passing + 2 skipped for optional numpy dependency)
 
 **Performance Analysis:**
 

bson/_rbson/src/lib.rs

@@ -2408,19 +2408,15 @@ fn decode_datetime(
         let utc_module = py.import("bson.tz_util")?;
         let utc = utc_module.getattr("utc")?;
 
-        // Use datetime.fromtimestamp(seconds, tz=utc)
-        let kwargs = [("tz", utc)].into_py_dict(py)?;
-        let dt = datetime_class.call_method("fromtimestamp", (seconds,), Some(&kwargs))?;
-
-        // Add microseconds if needed
-        let dt_final = if microseconds != 0 {
-            let timedelta_class = datetime_module.getattr("timedelta")?;
-            let kwargs = [("microseconds", microseconds)].into_py_dict(py)?;
-            let delta = timedelta_class.call((), Some(&kwargs))?;
-            dt.call_method1("__add__", (delta,))?
-        } else {
-            dt
-        };
+        // Construct datetime from epoch using timedelta to avoid platform-specific limitations
+        // This works on all platforms including Windows for dates outside fromtimestamp() range
+        let epoch = datetime_class.call1((1970, 1, 1, 0, 0, 0, 0, utc))?;
+        let timedelta_class = datetime_module.getattr("timedelta")?;
+
+        // Create timedelta for seconds and microseconds
+        let kwargs = [("seconds", seconds), ("microseconds", microseconds)].into_py_dict(py)?;
+        let delta = timedelta_class.call((), Some(&kwargs))?;
+        let dt_final = epoch.call_method1("__add__", (delta,))?;
 
         // Convert to local timezone if tzinfo is provided in codec_options
         if let Some(opts) = codec_options {
@@ -2464,27 +2460,15 @@ fn decode_datetime(
         Ok(dt_final.into())
     } else {
         // Return naive datetime (no timezone)
-        let timezone_module = py.import("datetime")?;
-        let timezone_class = timezone_module.getattr("timezone")?;
-        let utc = timezone_class.getattr("utc")?;
-
-        let kwargs = [("tz", utc)].into_py_dict(py)?;
-        let dt = datetime_class.call_method("fromtimestamp", (seconds,), Some(&kwargs))?;
-
-        // Remove timezone to make it naive
-        let kwargs = [("tzinfo", py.None())].into_py_dict(py)?;
-        let naive_dt = dt.call_method("replace", (), Some(&kwargs))?;
-
-        // Add microseconds if needed
-        if microseconds != 0 {
-            let timedelta_class = datetime_module.getattr("timedelta")?;
-            let kwargs = [("microseconds", microseconds)].into_py_dict(py)?;
-            let delta = timedelta_class.call((), Some(&kwargs))?;
-            let dt_with_micros = naive_dt.call_method1("__add__", (delta,))?;
-            Ok(dt_with_micros.into())
-        } else {
-            Ok(naive_dt.into())
-        }
+        // Construct datetime from epoch using timedelta to avoid platform-specific limitations
+        let epoch = datetime_class.call1((1970, 1, 1, 0, 0, 0, 0))?;
+        let timedelta_class = datetime_module.getattr("timedelta")?;
+
+        // Create timedelta for seconds and microseconds
+        let kwargs = [("seconds", seconds), ("microseconds", microseconds)].into_py_dict(py)?;
+        let delta = timedelta_class.call((), Some(&kwargs))?;
+        let naive_dt = epoch.call_method1("__add__", (delta,))?;
+        Ok(naive_dt.into())
     }
 }