💾 Introduce raw RDB loading falling back to sanitised RDB load strategy#51
Open
💾 Introduce raw RDB loading falling back to sanitised RDB load strategy#51
Conversation
…rom_utf8_unchecked - Bump redis-module and redis-module-macros to 2.0.4 in both Cargo.toml files - Replace todo!() in aof_rewrite with empty stub + explanatory comment - Replace from_utf8_unchecked + save_string with raw::save_slice in rdb_save
- Add 01-01-SUMMARY.md - Update STATE.md with decisions, metrics, session - Update ROADMAP.md phase 1 progress - Mark SAFE-01, SAFE-02, UPGR-01 complete in REQUIREMENTS.md
- Enable chrono serde feature in workspace - Add serde dependency to redical_ical - Custom Serialize/Deserialize for Tzid (chrono_tz::Tz lacks serde) - Derive Serialize, Deserialize on PositiveNegative enum
- Values: Text, Integer, Float, Date, Time, Duration, ClassValue, Reltype, DateTime, ValueType, List<T>, Recur, Frequency, WeekDay, WeekDayNum - Content line: ContentLineParam, ContentLineParams, ContentLine - Properties: UID, LastModified, Calendar, DTStart, DTEnd, Duration, RRule, ExRule, RDate, ExDate, Categories, LocationType, Class, Geo, RelatedTo, Passive, EventProperty, EventProperties - build_ical_param! macro generates types with serde derives - No query-only or index types modified
- SUMMARY, STATE, ROADMAP, REQUIREMENTS updated - SERD-01, SERD-02, SERD-05 marked complete
…notations - KeyValuePair, GeoPoint, EventOccurrenceOverride derive serde - ScheduleProperties derives serde, parsed_rrule_set skipped - IndexedProperties, PassiveProperties derive serde - Event derives serde, 5 computed index fields skipped - Calendar derives serde, 5 computed index fields skipped - indexes_active NOT skipped (source state)
- test_calendar_bincode_round_trip: serialize/deserialize/rebuild/equality - test_empty_calendar_bincode_round_trip: empty calendar round-trip - Full serde derive chain verified end-to-end
- SUMMARY.md with task commits and decisions - STATE.md advanced to plan 2 of phase 2 - ROADMAP.md and REQUIREMENTS.md updated
- RDBCalendarDump with version, raw_dump, dump fields - Two bincode round-trip tests (with/without version)
- BUILD_VERSION const from option_env!(GIT_SHA) - rdb_save now writes envelope with bincode raw_dump + iCal fallback - All existing tests pass including integration
- SUMMARY, STATE, ROADMAP, REQUIREMENTS updated
- Envelope (RDBCalendarDump) attempted first, legacy RDBCalendar on failure - Fast path gated on BUILD_VERSION match, wrapped in catch_unwind - Both bincode deser and rebuild_indexes inside catch_unwind scope - iCal fallback from envelope.dump on fast-path failure/version mismatch - Appropriate log levels: debug (success), warning (fallback), notice (legacy)
- Test envelope with None version (iCal fallback within envelope path) - Test legacy bytes (bare RDBCalendar deserialization) - Test corrupted raw_dump in envelope (fallback to iCal from dump) - Add log wrapper module to safely no-op redis logging in test mode
- test_helpers.rs with override-enriched Calendar builder and fixture_path helper - mod.rs load_tests now imports shared helper instead of inline fn - All 3 existing load_tests pass with enriched calendar
- #[ignore]-gated generate_fixtures test in rdb_data.rs - rdb_calendar_legacy.bin: bare RDBCalendar bincode bytes with override - rdb_calendar_dump_mismatch.bin: RDBCalendarDump with mismatched version
- SUMMARY.md with execution results - STATE.md advanced to phase 4 plan 1 - ROADMAP.md and REQUIREMENTS.md updated
- Legacy fixture loads correctly via load_legacy (TEST-04) - Mismatch fixture falls back to iCal via load_from_envelope (TEST-05) - Envelope serialize/deserialize round-trip produces identical Calendar (TEST-06)
Fixes bug where RDB envelope loading did not rebuild parsed rrule set, breaking queries because event instances could not be extrapolated. Adds validation step before index rebuild in Event and updates all call sites and comments to reflect the new name.
Changes: * `RDBCalendarDump` → `RDBCalendarEnvelope`, `.dump` → `.logical_dump` * `load_from_dump_envelope` → `load_from_envelope` * `load_from_legacy_ical_dump` → `load_from_logical_dump` * fixture files renamed accordingly.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Currently RediCal dumps all
CalendarRDB key data as a structured serialized iCal payload (the "logical dump") which is then re-parsed on load. This is great because it is version agnostic and robust, the major downside of this is that it is slow to load/rehydrate largeCalendarkey values.The changes in this PR work to improve on this by wrapping this logical dump in an envelope (
RDBCalendarEnvelope) alongside a raw bincode serialization of theCalendar. If the RediCal build version recorded in the envelope lines up with the current version, it deserializes theraw dump directly (much faster), falling back to the logical dump if there are any issues.
This PR also introduces GSD planning artefacts (experimentally used in implementing this feature).