Skip to content

Replace BlockTreeDb's leveldb-based storage with flat data files #165

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
sedited wants to merge 23 commits into
base: master
Choose a base branch
from
Open

Replace BlockTreeDb's leveldb-based storage with flat data files #165

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
23 commits
Select commit Hold shift + click to select a range
96002e7
bench: add shell.nix
willcl-ark Feb 10, 2025
389e1cd
bench: add uv + python deps
willcl-ark Feb 10, 2025
e158b83
bench: add benchmark ci workflows
willcl-ark Feb 10, 2025
b007bdb
guix: build static
willcl-ark Feb 10, 2025
4324642
doc: add benchcoin docs
willcl-ark Feb 10, 2025
13b3c22
remove legacy assumeutxo bench
willcl-ark Feb 13, 2025
a21d232
use *instrumented for flame runs
willcl-ark Feb 13, 2025
20912e7
add uninstrumented run
willcl-ark Feb 13, 2025
b328214
include instrumentation in name to avoid conflicts
willcl-ark Feb 14, 2025
4263ca2
allow failing source guix profile
willcl-ark Feb 14, 2025
e24bbc1
use github guix mirror (faster)
willcl-ark Mar 11, 2025
48a1e06
Ignore speedup of instrumented runs
l0rinc Mar 13, 2025
4635af7
remove nightly upstream sync
willcl-ark Mar 17, 2025
89780f7
Add commit id to the plots to make sure they're not overwritten
l0rinc Mar 19, 2025
a923692
Plot coins_cache_vs_height instead of coins_cache_vs_time
l0rinc Apr 7, 2025
0f3964f
Add vertical lines for major protocol upgrades if this is a height-ba…
l0rinc Apr 7, 2025
367b57e
kernel: Add headerstorage module
sedited May 2, 2025
bec55dd
fuzz: Use BlockTreeStore in block index fuzz test
sedited May 3, 2025
edb6b64
blockstorage: Replace BlockTreeDB with BlockTreeStore
sedited May 3, 2025
9084bfd
kernel: Remove block tree db params
sedited May 2, 2025
fe6fff5
kernel: Add assumed header store to chainparams
sedited May 3, 2025
b9505c0
blockstorage: Remove BlockTreeDB dead code
sedited May 4, 2025
2291916
Add write-ahead log
sedited Jun 4, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions .github/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
../doc/benchcoin.md
177 changes: 177 additions & 0 deletions .github/workflows/benchmark.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,177 @@
name: Benchmark
on:
pull_request:
branches:
- master
jobs:
build-binaries:
runs-on: [self-hosted, linux, x64]
env:
NIX_PATH: nixpkgs=channel:nixos-unstable
BASE_SHA: ${{ github.event.pull_request.base.sha }}
steps:
- name: Checkout repo
uses: actions/checkout@v4
with:
fetch-depth: 1
- name: Fetch base commit
run: |
echo "CHECKOUT_COMMIT=$(git rev-parse HEAD)" >> "$GITHUB_ENV"
git fetch --depth=1 origin ${{ github.event.pull_request.base.sha }}
- name: Setup ccache
run: |
mkdir -p /data/ccache
export CCACHE_DIR=/data/ccache
export CCACHE_MAXSIZE=50G
ccache -M 50G
ccache -s
- name: Build both binaries
env:
CCACHE_DIR: /data/ccache
run: |
mkdir -p ${{ runner.temp }}/binaries/base
mkdir -p ${{ runner.temp }}/binaries/head
nix-shell --command "just build-assumeutxo-binaries-guix $BASE_SHA $CHECKOUT_COMMIT"
cp binaries/base/bitcoind ${{ runner.temp }}/binaries/base/bitcoind
cp binaries/head/bitcoind ${{ runner.temp }}/binaries/head/bitcoind
- name: Upload binaries
uses: actions/upload-artifact@v4
with:
name: bitcoind-binaries
path: ${{ runner.temp }}/binaries/
uninstrumented:
needs: build-binaries
strategy:
matrix:
include:
- network: mainnet
name: mainnet-default-uninstrumented
timeout: 600
datadir_path: /data/pruned-840k
dbcache: 450
- network: mainnet
name: mainnet-large-uninstrumented
timeout: 600
datadir_path: /data/pruned-840k
dbcache: 32000
runs-on: [self-hosted, linux, x64]
timeout-minutes: ${{ matrix.timeout }}
env:
NIX_PATH: nixpkgs=channel:nixos-unstable
ORIGINAL_DATADIR: ${{ matrix.datadir_path }}
BASE_SHA: ${{ github.event.pull_request.base.sha }}
steps:
- name: Checkout repo
uses: actions/checkout@v4
with:
fetch-depth: 1
- name: Download binaries
uses: actions/download-artifact@v4
with:
name: bitcoind-binaries
path: ${{ runner.temp }}/binaries
- name: Set binary permissions
run: |
chmod +x ${{ runner.temp }}/binaries/base/bitcoind
chmod +x ${{ runner.temp }}/binaries/head/bitcoind
- name: Fetch base commit
run: |
echo "CHECKOUT_COMMIT=$(git rev-parse HEAD)" >> "$GITHUB_ENV"
git fetch --depth=1 origin ${{ github.event.pull_request.base.sha }}
- name: Run AssumeUTXO ${{ matrix.network }}
env:
TMP_DATADIR: "${{ runner.temp }}/base_datadir"
BINARIES_DIR: "${{ runner.temp }}/binaries"
run: |
env
mkdir -p "$TMP_DATADIR"
nix-shell --command "just run-${{ matrix.network }}-ci $BASE_SHA $CHECKOUT_COMMIT $TMP_DATADIR $ORIGINAL_DATADIR ${{ runner.temp }}/results.json ${{ matrix.dbcache }} $BINARIES_DIR"
- uses: actions/upload-artifact@v4
with:
name: result-${{ matrix.name }}
path: "${{ runner.temp }}/results.json"
- name: Write GitHub and runner context files
env:
GITHUB_CONTEXT: ${{ toJSON(github) }}
RUNNER_CONTEXT: ${{ toJSON(runner) }}
run: |
mkdir contexts
echo "$GITHUB_CONTEXT" | nix-shell -p jq --command "jq 'del(.token)' > contexts/github.json"
echo "$RUNNER_CONTEXT" > contexts/runner.json
- name: Upload context metadata as artifact
uses: actions/upload-artifact@v4
with:
name: run-metadata-${{ matrix.name }}
path: ./contexts/
instrumented:
needs: build-binaries
strategy:
matrix:
include:
- network: mainnet
name: mainnet-default-instrumented
timeout: 600
datadir_path: /data/pruned-840k
dbcache: 450
- network: mainnet
name: mainnet-large-instrumented
timeout: 600
datadir_path: /data/pruned-840k
dbcache: 32000
runs-on: [self-hosted, linux, x64]
timeout-minutes: ${{ matrix.timeout }}
env:
NIX_PATH: nixpkgs=channel:nixos-unstable
ORIGINAL_DATADIR: ${{ matrix.datadir_path }}
BASE_SHA: ${{ github.event.pull_request.base.sha }}
steps:
- name: Checkout repo
uses: actions/checkout@v4
with:
fetch-depth: 1
- name: Download binaries
uses: actions/download-artifact@v4
with:
name: bitcoind-binaries
path: ${{ runner.temp }}/binaries
- name: Set binary permissions
run: |
chmod +x ${{ runner.temp }}/binaries/base/bitcoind
chmod +x ${{ runner.temp }}/binaries/head/bitcoind
- name: Fetch base commit
run: |
echo "CHECKOUT_COMMIT=$(git rev-parse HEAD)" >> "$GITHUB_ENV"
git fetch --depth=1 origin ${{ github.event.pull_request.base.sha }}
- name: Run AssumeUTXO ${{ matrix.network }}
env:
TMP_DATADIR: "${{ runner.temp }}/base_datadir"
BINARIES_DIR: "${{ runner.temp }}/binaries"
run: |
env
mkdir -p "$TMP_DATADIR"
nix-shell --command "just run-${{ matrix.network }}-ci-instrumented $BASE_SHA $CHECKOUT_COMMIT $TMP_DATADIR $ORIGINAL_DATADIR ${{ runner.temp }}/results.json ${{ matrix.dbcache }} ${{ runner.temp }}/pngs $BINARIES_DIR"
- uses: actions/upload-artifact@v4
with:
name: result-${{ matrix.name }}
path: "${{ runner.temp }}/results.json"
- uses: actions/upload-artifact@v4
with:
name: pngs-${{ matrix.name }}
path: "${{ runner.temp }}/pngs/*.png"
- uses: actions/upload-artifact@v4
with:
name: flamegraph-${{ matrix.name }}
path: "**/*-flamegraph.svg"
- name: Write GitHub and runner context files
env:
GITHUB_CONTEXT: ${{ toJSON(github) }}
RUNNER_CONTEXT: ${{ toJSON(runner) }}
run: |
mkdir contexts
echo "$GITHUB_CONTEXT" | nix-shell -p jq --command "jq 'del(.token)' > contexts/github.json"
echo "$RUNNER_CONTEXT" > contexts/runner.json
- name: Upload context metadata as artifact
uses: actions/upload-artifact@v4
with:
name: run-metadata-${{ matrix.name }}
path: ./contexts/
Loading