Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
51 changes: 51 additions & 0 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -11,3 +11,54 @@ The files are organized by year. The `latest_asmap.dat` at the root of the proje
## Tools for analysis of maps

Users can further verify the integrity of the data with some of the [analysis commands](https://github.com/bitcoin/bitcoin/blob/master/contrib/asmap/README.md) available in `contrib/asmap/asmap-tool.py` (part of Bitcoin Core). There is also a ASMap health check log printed every 24 hours by Bitcoin Core nodes.

## Attestations

ASmap files can be attested to via PGP. In this repo:
- `attestations/<year>/<run epoch>/<signer>`: each ASmap from a collaborative run is run at a specific epoch (Unix timestamp), which serves to identify a given run.
- `SHA256SUMS`: hashes of the `final_result.txt`, filled and unfilled encoded ASmap
- `SHA256SUMS.asc`: detached PGP signature over the `SHA256SUMS` file
- `builder-keys/<signer>.gpg`: signer keys

The attestation file should contain three lines, the first for the final result, the second for the filled ASmap and the third for the unfilled ASmap, for example (assuming `EPOCH=1700000000000`):
```
cc199d5de04add6b5c2d95a72610c8a1a7b1f41fe01bd2b4c6db17795856aa31 final_result.txt
1146cbba8719cf3988d377df579667f68f97d2376d67755beb1e38194e196cfc 1700000000000_asmap_filled.dat
1c20ea2dee306af0a3ab4eaefaabe1e4c23a1c4256e60639e7ba48b2bbe56f24 1700000000000_asmap_unfilled.dat
```

### Script Usage

#### Attesting

To attest to an ASmap, you must have:
- the result file `final_result.txt` containing the ASmap in text format
- encoded the file via [`asmap-tool.py`](https://github.com/bitcoin/bitcoin/tree/master/contrib/asmap) as both filled and unfilled versions
- the Unix timestamp associated with the ASmap run
- a PGP key added to the `builder-keys` dir in this repo

Attesting to an ASmap output:
```bash
env SIGNER=<gpg-key-name>\
ASMAP_TXT=<path/to/final_result.txt>\
ENCODED_FILLED=<path/to/filled.dat>\
ENCODED_UNFILLED=<path/to/unfilled.dat>\
EPOCH=<unix_timestamp>\
./asmap-attest
```

This will add a `SHA256SUMS` file and a `SHA256SUMS.asc` file under the `<EPOCH>/<SIGNER>` folder.

#### Verifying

Verifying attestations in this repo:
```bash
./asmap-verify
```
or, for a specific epoch,
```bash
env EPOCH=177000000 \
./asmap-verify
```

This will print out verifications for the relevant attestations in the `attestations` dir.
248 changes: 248 additions & 0 deletions asmap-attest

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nanonit in 5a80d79:
If you need to re-touch - would expand the commit description to include the justification for removal of the NO_SIGN=1 example (fitting it on a typical terminal screen).

Original file line number Diff line number Diff line change
@@ -0,0 +1,248 @@
#!/usr/bin/env bash
export LC_ALL=C
set -e -o pipefail

################
# Required non-builtin commands should be invocable
################

check_tools() {
for cmd in "$@"; do
if ! command -v "$cmd" > /dev/null 2>&1; then
echo "ERR: This script requires that '$cmd' is installed and available in your \$PATH"
exit 1
fi
done
}

check_tools cat env basename mkdir sha256sum mktemp diff date

GPG=${GPG:-gpg}

if [ -z "$NO_SIGN" ]; then
# shellcheck disable=SC2206
GPG_ARRAY=($GPG)
check_tools "${GPG_ARRAY[0]}"
fi

################
# Usage
################

cmd_usage() {
cat <<EOF
Synopsis:

env SIGNER=<gpg-key-name> \\
ASMAP_TXT=<path/to/final_result.txt> \\
ENCODED_FILLED=<path/to/filled.dat> \\
ENCODED_UNFILLED=<path/to/unfilled.dat> \\
EPOCH=<unix_timestamp> \\
[ NO_SIGN=1 ] \\
./asmap-attest

Example:

env SIGNER=satoshi \\
ASMAP_TXT=/home/user/kartograf/out/1772726379/final_result.txt \\
ENCODED_FILLED=/home/user/kartograf/out/1772726379/filled.dat \\
ENCODED_UNFILLED=/home/user/kartograf/out/1772726379/unfilled.dat \\
EPOCH=1772726379 \\
./asmap-attest

Outputs:
\$PWD/2026/attestations/1772726379/satoshi/SHA256SUMS (attestation)
\$PWD/2026/attestations/1772726379/satoshi/SHA256SUMS.asc (signature)

Environment variables:

SIGNER GPG key name used for signing
ASMAP_TXT Path to the asmap text file (e.g. final_result.txt)
ENCODED_FILLED Path to the filled encoded binary (.dat)
ENCODED_UNFILLED Path to the unfilled encoded binary (.dat)
EPOCH Unix timestamp to identify the run
NO_SIGN If set and non-empty, skip GPG signing
GPG Override the gpg binary (default: gpg)

EOF
}

################
# Required env vars should be non-empty
################

if [ -z "$SIGNER" ] || [ -z "$ASMAP_TXT" ] || [ -z "$ENCODED_FILLED" ] || [ -z "$ENCODED_UNFILLED" ] || [ -z "$EPOCH" ]; then

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I've now also tested asmap-attest & asmap-verify.

Remarks for another PR:

  • Would also be nice if one could set EPOCH and get ASMAP_TXT+ENCODED_FILLED+ENCODED_UNFILLED calculated to default locations.

  • It's annoying that SIGNER is required even for NO_SIGN=1.

  • The output from asmap-verify comparing a file against itself is pointless:

Files /home/hodlinator/asmap-data/attestations/2026/1772726400/Hodlinator/SHA256SUMS and /home/hodlinator/asmap-data/attestations/2026/1772726400/Hodlinator/SHA256SUMS are identical

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It's annoying that SIGNER is required even for NO_SIGN=1.

the reason it's like that is that we always want to know a valid signer is usable from the builder-keys. The NO_SIGN=1 just means "don't actually do the signing". It's a dry-run, but we do need to have a Signer set for it to be meaningful.

Would also be nice if one could set EPOCH and get ASMAP_TXT+ENCODED_FILLED+ENCODED_UNFILLED calculated to default locations.

yes it would, but I'm not sure what to provide as a useful default. The epoch is used in the asmap-data directory structure. We could find the final_result.txt location if we know the user's kartograf dir, but its not guaranteed the user put the encoded output files in that same dir (since the encoding script runs from bitcoin/contrib/asmap). It would be a bit of a blind guess imo. If you have any suggestions for defaults I'd take them.

The output from asmap-verify comparing a file against itself is pointless:

yes, actually that output is useless in general, i'll have only return differences

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Agree running without a SIGNER is of debatable value. Spawned #54, lets continue there as far as asmap-attest goes.

cmd_usage
exit 1
fi

################
# Input files should exist and be readable
################

for var_name in ASMAP_TXT ENCODED_FILLED ENCODED_UNFILLED; do
file_path="${!var_name}"
if [ ! -f "$file_path" ]; then
cat << EOF
ERR: The specified ${var_name} does not exist or is not a regular file:

'$file_path'

EOF
exit 1
fi
if [ ! -r "$file_path" ]; then
cat << EOF
ERR: The specified ${var_name} is not readable:

'$file_path'

EOF
exit 1
fi
done

################
# The SIGNER should have a corresponding .gpg key in builder-keys/
################

signer_key="builder-keys/${SIGNER}.gpg"
if [ ! -f "$signer_key" ]; then
cat << EOF
ERR: No builder key found for signer '${SIGNER}':

'$signer_key'

Hint: The SIGNER name must match a .gpg file under builder-keys/ in the
asmap.sigs repository (without the .gpg extension). Available signers:

EOF
if [ -d "builder-keys" ]; then
for f in "builder-keys"/*.gpg; do
if [ -f "$f" ]; then
key_name="$(basename "$f" .gpg)"
echo " ${key_name}"
fi
done
else
echo " (builder-keys directory not found)"
fi
echo ""
exit 1
fi

################
# The GPG key should be usable
################

if [ -z "$NO_SIGN" ] && ! ${GPG} --dry-run --list-secret-keys "${SIGNER}" >/dev/null 2>&1; then
echo "ERR: GPG can't seem to find any secret key named '${SIGNER}'"
exit 1
fi

################
# Helper: derive year from unix epoch
################

year_from_epoch() {
local epoch="$1"
if date -d "@${epoch}" +%Y >/dev/null 2>&1; then
# GNU date
date -d "@${epoch}" +%Y
elif date -r "${epoch}" +%Y >/dev/null 2>&1; then
# BSD / macOS date
date -r "${epoch}" +%Y
else
echo "ERR: Cannot determine year from epoch='${epoch}'. Is 'date' available?" >&2
exit 1
fi
}

################
# Derive year from EPOCH
################

YEAR="$(year_from_epoch "$EPOCH")"

################
# Set up output directory
################

signer_dir="${PWD}/attestations/${YEAR}/${EPOCH}/${SIGNER}"
mkdir -p "$signer_dir"

##############
## Attest ##
##############

txt_basename="$(basename "$ASMAP_TXT")"
filled_basename="$(basename "$ENCODED_FILLED")"
unfilled_basename="$(basename "$ENCODED_UNFILLED")"

txt_hash="$(sha256sum "$ASMAP_TXT" | cut -d' ' -f1)"
filled_hash="$(sha256sum "$ENCODED_FILLED" | cut -d' ' -f1)"
unfilled_hash="$(sha256sum "$ENCODED_UNFILLED" | cut -d' ' -f1)"

sha256sums_file="$signer_dir/SHA256SUMS"

temp_sha256sums="$(mktemp)"
cat > "$temp_sha256sums" <<EOF
${txt_hash} ${txt_basename}
${filled_hash} ${filled_basename}
${unfilled_hash} ${unfilled_basename}
EOF

if [ -e "$sha256sums_file" ]; then
if diff -u "$sha256sums_file" "$temp_sha256sums" >/dev/null 2>&1; then
echo "A SHA256SUMS file already exists for signer '${SIGNER}' and epoch '${EPOCH}' and is up-to-date."
rm -f "$temp_sha256sums"
else
diff -u "$sha256sums_file" "$temp_sha256sums" || true
cat << EOF
--

ERR: A SHA256SUMS file already exists for signer '${SIGNER}' and epoch '${EPOCH}' and attests
differently. See the diff above for details.

Hint: You may wish to remove the existing attestation and its signature by
invoking:

rm '${sha256sums_file}'{,.asc}

Then try running this script again.

EOF
rm -f "$temp_sha256sums"
exit 1
fi
else
mv "$temp_sha256sums" "$sha256sums_file"
echo "SHA256SUMS written to '$sha256sums_file'"
fi
echo ""

##############
## Sign ##
##############

if [ -z "$NO_SIGN" ]; then
if [ ! -e "$sha256sums_file".asc ]; then
${GPG} --detach-sign \
--digest-algo sha256 \
--local-user "$SIGNER" \
--armor \
--output "$sha256sums_file".asc "$sha256sums_file"
else
echo "Signature '${sha256sums_file}.asc' already exists."
fi
else
echo "Not signing SHA256SUMS as \$NO_SIGN is set"
fi

echo ""
echo "Done. Results for epoch '${EPOCH}', signer '${SIGNER}':"
echo " ${sha256sums_file} (attestation)"
if [ -z "$NO_SIGN" ] && [ -e "$sha256sums_file".asc ]; then
echo " ${sha256sums_file}.asc (signature)"
fi

Loading