Skip to content

feat: Implement persistent storage #108

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Draft
SoarinSkySagar wants to merge 11 commits into
base: devnet-5
Choose a base branch
from
+1,122 −15
Draft

feat: Implement persistent storage #108

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
11 commits
Select commit Hold shift + click to select a range
6acb243
feat: init database crate
SoarinSkySagar Jun 27, 2026
19c6afd
chore: pull database as a grandine dep and create storage crate
SoarinSkySagar Jun 27, 2026
57d0438
feat: add leanspec consts for db
SoarinSkySagar Jun 27, 2026
e0233e0
feat: add databases for each KV pair defined in spec
SoarinSkySagar Jun 28, 2026
4376cdd
feat: add Storage impl with builderplate functions from spec
SoarinSkySagar Jun 29, 2026
2a25e9e
feat: storage spec API functions
SoarinSkySagar Jun 30, 2026
efe3e0a
feat: implement libmdbx with custom compression per database instance
SoarinSkySagar Jul 2, 2026
2da086b
chore: tests (all passing)
SoarinSkySagar Jul 2, 2026
25902c6
chore: combine database and storage
SoarinSkySagar Jul 3, 2026
eaffc22
feat: fully spec-compliant storage API
SoarinSkySagar Jul 3, 2026
5f1fdb8
chore: privatized Database and removed DatabaseMode
SoarinSkySagar Jul 3, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
180 changes: 166 additions & 14 deletions lean_client/Cargo.lock
View file
Open in desktop

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

12 changes: 11 additions & 1 deletion lean_client/Cargo.toml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -7,7 +7,8 @@ members = [
"http_api",
"metrics",
"networking",
"spec_test_fixtures",
"spec_test_fixtures",
"storage",
"validator",
"xmss",
]
Expand Down Expand Up @@ -229,6 +230,7 @@ http_api = { path = "./http_api" }
metrics = { path = "./metrics" }
networking = { path = "./networking" }
spec_test_fixtures = { path = "./spec_test_fixtures" }
storage = { path = "./storage" }
validator = { path = "./validator" }
xmss = { path = "./xmss" }

Expand All @@ -237,12 +239,15 @@ async-trait = "0.1"
axum = "0.8.8"
bitvec = "1.0.1"
bls = { git = "https://github.com/grandinetech/grandine", package = "bls", features = ["blst"], rev = "64afdee3c6be79fceffb66933dcb69a943f3f1ae" }
bytesize = { version = '2', features = ['serde'] }
clap = { version = "4", features = ["derive"] }
database = { git = "https://github.com/grandinetech/grandine", package = "database", rev = "64afdee3c6be79fceffb66933dcb69a943f3f1ae" }

@ArtiomTr ArtiomTr Jun 30, 2026

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

the database crate from grandine is a bit flawed, I would suggest implementing your own. Specifically, current database impl forces snappy compression for all values - this is inefficient in some cases (e.g., if we save slot -> state_root indexes, there is no point of compressing state_root, as it is pure entropy), and not optimal in others (e.g., for blobs you probably want to use something more compressing, like zstd). Thus, forcing single compression algorithm for all values wasn't a good idea. Also, for cases where performance matters, looks like there are currently compression algorithms that are both more performant & give better compression ratios than snappy - like lz4.

@SoarinSkySagar SoarinSkySagar Jun 30, 2026

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

okay, I will be implementing my own database crate on top of this next

derive_more = "2.1.1"
discv5 = "0.10.2"
enr = { version = "0.13", features = ["k256"] }
eth_ssz = { package = "ethereum_ssz", version = "0.10.0" }
ethereum-types = "0.14"
fs-err = "3"
futures = "0.3"
features = { git = "https://github.com/grandinetech/grandine", rev = "64afdee3c6be79fceffb66933dcb69a943f3f1ae" }
git-version = "0.3"
Expand All @@ -255,6 +260,7 @@ rec_aggregation = { git = "https://github.com/leanEthereum/leanVM.git", rev = "e
backend = { git = "https://github.com/leanEthereum/leanVM.git", rev = "e2592df4e30fdddbbf8ae26a333116c68cec7026" }
leansig = { git = "https://github.com/leanEthereum/leanSig", branch = "devnet4" }
leansig_wrapper = { git = "https://github.com/leanEthereum/leanVM.git", rev = "e2592df4e30fdddbbf8ae26a333116c68cec7026" }
libmdbx = { git = "https://github.com/paradigmxyz/reth.git", package = "reth-libmdbx", rev = "6f8e7258f4733279080e4bd8345ce50538a40d6e" }
libp2p = { git = "https://github.com/libp2p/rust-libp2p.git", rev = "91e8931e275bcd1c72791d18b09fea8b77209baf", default-features = false, features = [
'dns',
'gossipsub',
Expand All @@ -268,6 +274,7 @@ libp2p = { git = "https://github.com/libp2p/rust-libp2p.git", rev = "91e8931e275
'yamux'
] }
libp2p-identity = { version = "0.2.13", features = ["secp256k1"] }
lz4_flex = "0.13"
dedicated_executor = { path = "dedicated_executor" }
num-bigint = "0.4"
num-traits = "0.2"
Expand All @@ -291,6 +298,7 @@ snap = "1.1"
ssz = { git = "https://github.com/grandinetech/grandine", package = "ssz", rev = "64afdee3c6be79fceffb66933dcb69a943f3f1ae" }
ssz-types = "0.3"
itertools = "0.14"
tempfile = "3"
test-generator = "0.3.1"
thiserror = "2"
tikv-jemallocator = { version = "0.6", features = ["stats", "unprefixed_malloc_on_supported_platforms"] }
Expand All @@ -305,6 +313,7 @@ try_from_iterator = { git = "https://github.com/grandinetech/grandine", package
typenum = "1.19"
yamux = "0.12"
zeroize = "1.8"
zstd = "0.13.3"

[package]
name = "lean_client"
Expand All @@ -330,6 +339,7 @@ networking = { workspace = true }
num_cpus = { workspace = true }
parking_lot = { workspace = true }
ssz = { workspace = true }
storage = { workspace = true }
tokio = { workspace = true }
tracing = { workspace = true }
tracing-subscriber = { workspace = true }
Expand Down
Loading