Skip to content

Comments

feat(hyperswarm): add negentropy sync with sqlite store#1176

Open
Bushstar wants to merge 31 commits intomainfrom
bush/hypr-rbsr
Open

feat(hyperswarm): add negentropy sync with sqlite store#1176
Bushstar wants to merge 31 commits intomainfrom
bush/hypr-rbsr

Conversation

@Bushstar
Copy link
Contributor

@Bushstar Bushstar commented Feb 13, 2026

This PR introduces Negentropy-based anti-entropy sync for the Hyperswarm mediator to replace connect-time full-history sync between upgraded nodes. The goal is to keep peer sync fast and bounded as operation history grows, while remaining backward compatible with legacy peers.

Why this change

  • Previous behavior shared large/full operation history on peer connect, which increasingly blocked mediator throughput.
  • Negentropy lets peers exchange compact set differences (have/need) instead of bulk history.

What was added

  • Negentropy protocol flow in Hyperswarm mediator:
    • new message flow for reconciliation and transfer (neg_open, neg_msg, neg_close, ops_req, ops_push)
    • per-peer sync session state (mode, rounds, pending ids, activity)
  • Backward compatibility:
    • capability negotiation in ping
    • upgraded↔upgraded uses negentropy
    • upgraded↔legacy falls back to existing legacy sync path
  • Persistent sync store:
    • SQLite-backed ordered store for operation sync index
    • in-memory store for tests
    • mediator persists only operations accepted by gatekeeper
  • Bootstrap on upgrade/startup:
    • if sync-store is empty, one-time backfill from gatekeeper exportBatch
    • startup depends on successful bootstrap to avoid partial sync state
    • wipe and bootstrap again if op store has >1% drift from gatekeeper
  • Gatekeeper contract extension:
    • importBatch result now includes rejectedIndices
    • mediator filters index-rejected items before persistence pipeline
    • accepted-only persistence is finalized using gatekeeper processing results

Performance safeguards

  • Removed full DB export for upgraded peers (legacy only).
  • Added adapter freshness controls to avoid repeated rebuild work:
    • dirty flag + built-at age check
    • single in-flight rebuild guard
    • background prebuild when store changes and no active session
  • Kept sync concurrency conservative (single active negentropy session).

@Bushstar Bushstar closed this Feb 13, 2026
@Bushstar Bushstar reopened this Feb 13, 2026
@Bushstar Bushstar marked this pull request as ready for review February 24, 2026 11:01
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants