BIPs: SwiftSync Specification by rustaceanrob · Pull Request #2152 · bitcoin/bips

rustaceanrob · 2026-05-06T12:00:18Z

SwiftSync is a protocol for clients to parallelize initial block download, based on the original writeup.

murchandamus

Just a quick first glance, but could you please break your text into shorter lines? That makes it easier to leave review and track what changed between commits. Either 100 or 120 characters per line seems to work well enough.

jonatack · 2026-05-06T17:17:58Z

FWIW, I don't mind the unbroken lines and even prefer them. Avoids rejigging line lengths to keep them consistent when updating or having lines with very different lengths.

danielabrozzoni

I did an initial pass and left some comments. I read the BIPs in the commit order (block undo -> histfile -> swiftsync) and it was pretty easy to follow.

jurraca

some writing nits but overall the concept is clear enough.

murchandamus · 2026-05-12T23:02:49Z

Thanks for the review, @danielabrozzoni and @jurraca, as well as the quick turnaround @rustaceanrob. I notice that this pull request is still marked as a Draft PR. Are you still planning significant changes? If your submission is ready for another BIP Editor review, please mark the PR as "ready for review".

rustaceanrob · 2026-05-14T11:46:21Z

I will keep these as a draft as the hintsfile format is subject to change.

Roasbeef · 2026-05-23T02:08:44Z

+| Amount                   | 64 bit unsigned integer       | Defined above | Satoshi denominated value                                                                                         |
+### Messages
+
+#### MSG_GET_SPENT_COINS


Is the idea that a peer would issue this request for every block in the chain? If we assume mainnet at height, and a 150 ms round trip time, then a peer would spend nearly 80 hours just downloading this undo data.

You may want to consider a batched variant, similar to the way messages like getheaders works.

We've found that bandwidth throughput is the limiting factor when downloading blocks in parallel. Not all spent coins have to be downloaded if a client keeps a cache, as this document describes. In the batched variant, the cache is not possible and the bandwidth requirement increases significantly.

Roasbeef · 2026-05-23T02:10:50Z

+
+| Field             | Value      |
+| :----------------- | :---------- |
+| `NODE_BLOCK_UNDO` | `1 << ???` |


Rationale should be added for the choice of a new node version over the more common place (as of the past few years) exchange of a sendX message during the version handshake.

IMO a version makes sense here, as it can be used to filter out peers upfront that support sending this undo data over the network.

I opted for a BIP-434 feature message, which has a similar mechanism for the sendX

Roasbeef · 2026-05-23T02:11:41Z

+
+#### MSG_SPENT_COINS
+
+`MSG_SPENT_COINS` defines the data structure for inputs of a block.


Probably not really y'all's intended use case, but if you optionally make it possible to include merkle proofs for the set of coins, then this message can be used to obtain a proof that an output was spent in a given block.

It would actually also be useful for BIP 157+158 peers, as the final version that shipped includes the script spent (instead of the outpoint), which means that if you're using the filters to find a block where a given script has been spent, you need to make some assumptions about what the prev script is for a given transaction.

The most recent response on this mailing list post mentions commitment to the UTXO set as part of the block header. There are additional ways to do this outside of a soft fork as well, i.e. utreexo proofs. For now I think it best to leave this unspecified in this version of the message while the community shares ideas, but I do think this is interesting.

rustaceanrob · 2026-06-03T17:07:52Z

Wrapped text and standardized formatting with mdformat, will address outstanding feedback soon.

murchandamus · 2026-06-03T20:25:41Z

Great, thanks! I’ll give it a read when you’re done with that.

hintsfile.js implements the SwiftSync hintsfile (bitcoin/bips#2152 'Hints for unspent coins'): per-block unspent output indices encoded with Elias-Fano (CompactSize(n) || CompactSize(m) || L || H; low bits LSB-first, unary gap high bits), plus the 'UTXO' magic/version/height/vector container. This is the ONE cross-compatibility artifact (per Somsen), so it's validated byte-for-byte against the BIP's own elias_fano.json vectors — all three match, plus round-trips, edge cases, and a container round-trip. Exported from index + package exports (./hintsfile).

…matches BIP vectors undo.js implements the full-validation spent-coin data (bitcoin/bips#2152 'Peer sharing of block spent coins'): Core's CompressAmount/DecompressAmount, the reconstructable-script prefix table (P2PKH/P2SH/P2PK/P2WPKH/P2WSH/P2TR/raw), the height code (height<<1|coinbase), and a spent-coin record. Amount + script compression validated byte-for-byte against the BIP's compressed_amount.json and reconstructable_script.json vectors (+ round-trips). Also factored CompactSize/concat into varint.js, shared by hintsfile + undo. Full suite: 21 pass / 0 fail.

rustaceanrob · 2026-06-22T08:56:21Z

Given there are a few clients that have started implementations of SwiftSync, and new hintsfile encodings may simply increment the file version, I am moving these out of draft. Some outstanding comments addressed, others require some additional thought.

edilmedeiros

Thanks for documenting the protocol in this draft.

Did a deep dive together with the guys from @vinteumorg and left many comments concerning conceptual aspects of the BIPs. I have many editing suggestions, but left them for a second round after the higher-level aspects are more mature.

edilmedeiros · 2026-06-24T19:28:21Z

+in Bitcoin Core, and reasonable for most clients to hold directly in memory. This encoding represents elements in $2n +
+n \\lceil \\log_2(m/n) \\rceil$ bits, which is within a reasonable bound of the theoretical optimum.
+
+Partitioning the hints by block is an intuitive choice, and allows for efficient random access of hints. Groupings of


Partitioning the hints by block is an intuitive choice, and allows for efficient random access of hints.

I don't see how this can be true: the bistream has a header (magic, version, height) followed by a sequence of EliasFano items, each of which are composed by N, M (fixed size info), L, H (variable size info).

So, imagine I have a hintsfile. To find data for block k, one do need to (minimally) process data for block 1 to discover the size of the first EliasFano item (because of the variable size parts). Then, block 2 and so forth, until the intended block target. This would be true if the EliasFano items were fixed size among all blocks to allow for offset arithmetic, but the amount of padding would be prohibitively high.

Thus, the hints payload works more like a List<EliasFano> (requires sequential access) than a vector<EliasFano> (allows random access). Of course, the decoder could create an index of offsets, but this is not only an implementation detail, but also something that will add to the required resources to process the hintsfile.

Good catch, the original version had a header section, but was removed as it could be reconstructed as you described. I will remove that note.

I wonder if having more blocks taken together will not improve compression sensibly (I'll experiment with it). We can add a counter in the bitstream to allow the encoder to choose it freely (at the cost of having more bytes in the bitstream) but we potentially gain:

Less H, L pairs in the bitstream.

More data to feed to the Elias-fano process (tends to push it closer to the theoretical entropy).

johnnyasantoss

I participated with @edilmedeiros in the review and had a few concept concerns, most linked to the undo data, its tradeoff (the worst case scenario seems to be really bad) and the optional 5-block index.

johnnyasantoss · 2026-06-24T21:42:39Z

+The lifetime, or interval between creation and spending height, of the coins on the Bitcoin blockchain demonstrate an
+empirical phenomena that the majority of coins are spent within 100 blocks. In fact, approximately 41 percent of coins
+are spent within 10 blocks at the time of writing[^1]. Clients may leverage this to reduce the bandwidth required to
+fetch undo data by using an in-memory cache. For example, a client may store coins that were created in a 5 block
+window, and request only coins that are older than this height via the `cutoff` filter. This results in a significant
+bandwidth reduction at the cost of a cache that can be set dynamically by the client depending on available memory.


This cutoff cache optimization seems to nudge implementors back to sequentially processing blocks with the added burden of requesting extra data over the wire.

Also with the current messages I still need to get the data for the block (even if there's only one unspent cache miss?), right?
If that's true, wouldn't a parameter for inputs of interest (delta encoded index) help here?

At 150ms RTT * 955233 blocks that's ~39.8 hrs of round-trip latency for uncached requests alone, before counting download time (as Roasbeef noted in an earlier review). It seems to me that the cache mitigates this but at the cost of reintroducing the very sequentiality it aims to eliminate.

Is this understanding correct?

seems to nudge implementors back to sequentially processing blocks

Caching requires sequential processing, but you can have multiple sequential threads in parallel.

added burden of requesting extra data over the wire

You're going to have to request the undo data regardless for non-assumevalid SwiftSync - it is not related to caching.

I still need to get the data for the block (even if there's only one unspent cache miss?), right?

It seems to me that the cache mitigates [round-trip latency]

Caching does not prevent the need for requesting undo data. You can safely assume pretty much every block has cache misses. No cache missses is equivalent to having the full UTXO set (and impossible with multiple sequential threads), which defeats the point.

round-trip latency for uncached requests

I have no strong opinion on batching, but round-trip latency won't add up sequentially if requests are sent out in parallel.

Concrete example: Let's say you're starting another sequential thread from block height 1001 and you intend to cache the last 5 blocks worth of outputs. For the first block you'd request the full undo data. For block 1002 until 1005 you'd request everything created until block height 1000. From height 1006 onwards your 5-block window starts to shift so you'd request everything created until block height 1001, and so on.

All this data can be requested in parallel. As long as your caching strategy is not based on what you witnessed during the previous block, at no point do you have to wait for one block to finish processing before requesting the data for upcoming blocks.

murchandamus

I read the first document "Peer sharing of block spent coins". Given my prior knowledge it’s pretty clear what’s going on, but I think people reading about the topic for the first time could use more context in some passages. I noticed a couple sections with potential for improvement.

The Abstract and first sentences of the Motivation are a bit repetitive.
For the Definitions and Data Structures sections, I could have used a little more context. What will I be shown? Why? How is the table to be read? What do the columns mean?

murchandamus · 2026-06-26T23:39:41Z

+
+## Motivation
+
+A current limitation of IBD is that it must be done sequentially. This is a result of the height, coinbase flag, input


A current limitation of IBD is that it must be done sequentially.

Given the postulation or existence of alternative syncing models, that feels a bit loaded. Maybe mention that this specifically refers to Bitcoin Core or alternatively consider something along the lines of: "The common approach to IBD is to process blocks sequentially as that ensures the existence of TXO details when input validation requires them to be available."

This is a result of the height, coinbase flag, input script, and amount of the block inputs being omitted from the data committed to by proof of work in the current block

This is jumping several steps from the prior statement at once. Maybe you could segue that a bit more, e.g., by mentioning that fields you introduce are TXO details, before going into them being only implicitly or not at all committed to by transaction inputs, before explaining how that makes it impossible to verify what is provided by a peer.

murchandamus · 2026-06-26T23:48:14Z

+script, and amount of the block inputs being omitted from the data committed to by proof of work in the current block,
+and, thus, this data cannot be trusted if received over the wire naively. Using the SwiftSync protocol, a client is able
+to verify the correctness of this data, even if served by a potentially untrusted party. This allows a significant
+improvement in IBD performance, as block downloads may be done in parallel.


This is a bit imprecise: block download is always done in parallel, it’s just validation that is sequential. Do you mean that block validation can be parallelized?

murchandamus · 2026-06-26T23:55:50Z

+#### Height Code
+
+When validating a block, a client must confirm coinbase outputs are mature, which is given by the height of the coin.
+The height and coinbase flag are encoded as a 32 bit integer. To encode the height and flag, binary left shift the


Maybe you could add a footnote why both are stored in one data structure, and/or mention here that even with sacrificing one bit, heights up to 2,147,483,647 can be expressed and 30,000+ years of blocks is plenty planning horizon? :p

murchandamus · 2026-06-27T00:22:52Z

+window, and request only coins that are older than this height via the `cutoff` filter. This results in a significant
+bandwidth reduction at the cost of a cache that can be set dynamically by the client depending on available memory.


Ah cool. I was missing this context above when cutoff was introduced.

I added a short note when introducing the request message that the cutoff field is motivated in the rationale section.

murchandamus · 2026-06-27T00:25:52Z

+11gb reduction in bandwidth is achieved. The application of `VARINT` as opposed to `CompactSize` offers a further
+reduction of 4gb, however the `VARINT` primitive is currently a Bitcoin Core implementation detail. Reusing existing


Given the confusing terminology in regard to CompactSize and VARINT and Bitcoin Core, you probably want to define these terms more concretely.

murchandamus reviewed May 6, 2026

View reviewed changes

Comment thread bip-xxxx-swiftsync.md

Comment thread bip-xxxx-swiftsync.md Outdated

murchandamus added New BIP PR Author action required Needs updates, has unaddressed review comments, or is otherwise waiting for PR author labels May 6, 2026

jonatack changed the title ~~SwiftSync Specification~~ BIP drafts: SwiftSync Specification May 6, 2026

danielabrozzoni reviewed May 7, 2026

View reviewed changes

jurraca reviewed May 8, 2026

View reviewed changes

Comment thread bip-xxxx-block-undo.md Outdated

Comment thread bip-xxxx-block-undo.md Outdated

Comment thread bip-xxxx-block-undo.md Outdated

rustaceanrob force-pushed the swiftsync-bips branch 2 times, most recently from 92093e1 to f4cd99a Compare May 10, 2026 09:31

Roasbeef reviewed May 23, 2026

View reviewed changes

rustaceanrob mentioned this pull request May 31, 2026

Implementation of SwiftSync bitcoin/bitcoin#34004

Closed

yancyribbens reviewed May 31, 2026

View reviewed changes

Comment thread bip-xxxx-swiftsync.md Outdated

rustaceanrob force-pushed the swiftsync-bips branch from f4cd99a to e4f8172 Compare June 3, 2026 17:06

rustaceanrob force-pushed the swiftsync-bips branch from e4f8172 to e5ec578 Compare June 22, 2026 08:53

rustaceanrob changed the title ~~BIP drafts: SwiftSync Specification~~ BIPs: SwiftSync Specification Jun 22, 2026

rustaceanrob marked this pull request as ready for review June 22, 2026 08:54

rustaceanrob force-pushed the swiftsync-bips branch 5 times, most recently from 24ae4e7 to ecbda2a Compare June 23, 2026 09:00

jonatack removed the PR Author action required Needs updates, has unaddressed review comments, or is otherwise waiting for PR author label Jun 23, 2026

edilmedeiros reviewed Jun 24, 2026

View reviewed changes

johnnyasantoss reviewed Jun 24, 2026

View reviewed changes

rustaceanrob force-pushed the swiftsync-bips branch 13 times, most recently from 27c42b6 to 2606bd8 Compare June 26, 2026 08:52

murchandamus reviewed Jun 27, 2026

View reviewed changes

rustaceanrob force-pushed the swiftsync-bips branch 2 times, most recently from dfab2a7 to 473007d Compare June 27, 2026 12:18

rustaceanrob added 3 commits June 27, 2026 14:35

BIP ???: Block spent coins over P2P

e80a351

BIP ???: Unspent outputs hintsfile

32925f2

BIP ???: SwiftSync specification

5bd069f

rustaceanrob force-pushed the swiftsync-bips branch from 473007d to 5bd069f Compare June 27, 2026 12:39


		#### MSG_SPENT_COINS

		`MSG_SPENT_COINS` defines the data structure for inputs of a block.


		## Motivation

		A current limitation of IBD is that it must be done sequentially. This is a result of the height, coinbase flag, input

		window, and request only coins that are older than this height via the `cutoff` filter. This results in a significant
		bandwidth reduction at the cost of a cache that can be set dynamically by the client depending on available memory.

		11gb reduction in bandwidth is achieved. The application of `VARINT` as opposed to `CompactSize` offers a further
		reduction of 4gb, however the `VARINT` primitive is currently a Bitcoin Core implementation detail. Reusing existing

Uh oh!

Conversation

rustaceanrob commented May 6, 2026

Uh oh!

murchandamus left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

jonatack commented May 6, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

danielabrozzoni left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

jurraca left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

murchandamus commented May 12, 2026

Uh oh!

rustaceanrob commented May 14, 2026

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

rustaceanrob Jun 22, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

rustaceanrob Jun 23, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

rustaceanrob commented Jun 3, 2026

Uh oh!

murchandamus commented Jun 3, 2026

Uh oh!

rustaceanrob commented Jun 22, 2026

Uh oh!

edilmedeiros left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

jonatack commented May 6, 2026 •

edited

Loading

rustaceanrob Jun 22, 2026 •

edited

Loading

rustaceanrob Jun 23, 2026 •

edited

Loading

RubenSomsen Jun 25, 2026 •

edited

Loading