Skip to content

feat: Support Store S3 #520

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
speeddragon wants to merge 7 commits into
base: edge
Choose a base branch
from
+1,409 −3
Open

feat: Support Store S3 #520

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
7 commits
Select commit Hold shift + click to select a range
0b91c72
feat: Add S3 storage backend
shyba Oct 10, 2025
841b020
style: code cleanup
nikooo777 Oct 11, 2025
a5a7bd6
test: Add S3 store to test suites
speeddragon Oct 14, 2025
4d2edf1
impr: Replace hackney with gun for S3 HTTP client
speeddragon Oct 22, 2025
0a1c1b4
refactor: Simplify S3 store by removing links and groups
speeddragon Oct 29, 2025
c95a6c1
feat: Add sharding support to S3 store
speeddragon Oct 31, 2025
db9b0d7
fix: Configure erlcloud for S3 profile release
speeddragon Nov 7, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -87,6 +87,7 @@ HyperBEAM supports several optional build profiles that enable additional featur

- `genesis_wasm`: Enables Genesis WebAssembly support
- `rocksdb`: Enables RocksDB storage backend (adds RocksDB v1.8.0 dependency)
- `s3`: Enables S3 storage backend
- `http3`: Enables HTTP/3 support via QUIC protocol


Expand Down
14 changes: 14 additions & 0 deletions rebar.config
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -41,6 +41,17 @@
{d, 'ENABLE_ROCKSDB', true}
]}
]},
{s3, [
{deps, [
{erlcloud, {git, "https://github.com/erlcloud/erlcloud.git",
{ref, "fe58402b40b93a20176d12b4bc211ea6a5b0d915"}}
}
]},
{erl_opts, [
{d, 'ENABLE_S3', true}
]},
{relx, [{release, {'hb', "0.0.1"}, [hb, b64fast, cowboy, gun, luerl, prometheus, prometheus_cowboy, elmdb, erlcloud]}]}
]},
{http3, [
{deps, [
{quicer, {git, "https://github.com/emqx/quic.git",
Expand All @@ -56,6 +67,9 @@
{add, cowboy, [{erl_opts, [{d, 'COWBOY_QUICER', 1}]}]},
{add, gun, [{erl_opts, [{d, 'GUN_QUICER', 1}]}]}
]}
]},
{test, [
{deps, [{meck, "1.1.0"}]}
]}
]}.

Expand Down
16 changes: 13 additions & 3 deletions src/hb_store.erl
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -440,7 +440,7 @@ test_stores() ->
}
]
}
] ++ rocks_stores().
] ++ rocks_stores() ++ s3_stores().

-ifdef(ENABLE_ROCKSDB).
rocks_stores() ->
Expand All @@ -453,16 +453,26 @@ rocks_stores() ->
-else.
rocks_stores() -> [].
-endif.

-ifdef(ENABLE_S3).
s3_stores() ->
[(hb_store_s3:default_test_opts())#{
<<"benchmark-scale">> => 0.01
}].
-else.
s3_stores() -> [].
-endif.
generate_test_suite(Suite) ->
generate_test_suite(Suite, test_stores()).
generate_test_suite(Suite, Stores) ->
hb:init(),
application:ensure_all_started(hb),
Copy link
Author

@speeddragon speeddragon Oct 24, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is necessary to use hb_http_client in hb_store_s3 requests.

lists:map(
fun(Store = #{<<"store-module">> := Mod}) ->
{foreach,
fun() ->
hb_store:start(Store)
hb_store:start(Store),
% If the test fails, the store isn't cleared.
hb_store:reset(Store)
end,
fun(_) ->
hb_store:reset(Store)
Expand Down
Loading