High-level thoughts & directions for BB project RFCs

[yiannisbot]

Bitswap is a simple file exchange protocol. Given that it was not designed to be a content routing protocol, its feature-set was kept to a minimum, which is probably the right thing to do. I believe there is lots that can be done to improve its performance and this project is certainly on the right direction, but I think we should categorise carefully the nature of changes we're proposing in order to be able to evaluate the significance of the results we get out of the study.

I'm presenting some thoughts on the RFCs currently being worked on. The current set of RFCs can be split in two categories: i) those that improve content fetching, and ii) those that improve content discovery.

Content Fetching RFCs

RFC|BB|L2-03A: Use of compression and adjustable block size
RFC|BB|L2-07: Request minimum piece size and content protocol extension
RFC|BB|L12-01: Bitswap/Graphsync exchange messages extension and transmission choice

I suggest that the first two RFCs above get integrated under the umbrella of the third one. The third RFC is the most sophisticated one IMO. The reasoning is that it is difficult to set global standards/values for optimisation, e.g., all content transferred by Bitswap is compressed or all applications use the same block/chunk size. These decisions depend largely on the application and our approach should be to provide to applications all the knobs they need in order to optimise for their own case.

Content Discovery RFCs

RFC|BB|L1-04: Track WANT Message for future queries
RFC|BB|L1-02: TTLs for rebroadcasting WANT messages

These RFCs go towards the content routing approach. I suggest that we do implement and evaluate their performance, but we should target situations where Bitswap is used within the remit of some specific application, i.e., not globally, say within the global IPFS network. My fear is that doing content discovery with a simple protocol won't scale as the network size increases and we'll end up re-inventing the wheel by optimising towards directions that DHTs or pubsub protocols do out of the box.

Consider the following: assuming each node has ~1k connections: with a TTL=1 if every node sends a WANT message to all of its connections then we effectively flood the network in it's current size - the message will be propagated to 1M nodes (larger than the entire IPFS network right now). It might actually be ok to do flooding with the current network size in order to improve content discovery performance (this would be something cool to evaluate and propose), but this won't take us far as the network size increases (i.e., too much overhead).

Furthermore, the evaluation here needs to take into account the popularity of content. For instance, we should not try to optimise for content that has only one copy in the entire network. There is a tradeoff here between the request span we impose and the success rate this returns. In other words, given we now span across the connections of many peers, what we could do is limit the number of peers each node is reaching out to. We could envision a scheme where the requestor node is sending the request to 75% of its peers, then those nodes are sending to 50% of their peers and the final hop nodes send to 25% of their own peers. Again, these percentages can be tunable by the application. The applications are incentivised to set the values right in order to avoid overloading their own users.

On the "symmetric vs asymmetric" routing options of RFC|BB|L1-02, I would suggest that we evaluate both, as they can serve different purposes. For instance, "symmetric" routing can improve privacy out of the box (it "emulates" an onion-routing-like approach), which we should not disregard. There is the IPFS principle of not storing content into nodes that have not requested it, which we should not forget. However, we can work around it in several ways, e.g., as opt-in. We should also not forget that Bitswap is a protocol of its own, so it can possibly deviate slightly, without removing the options that IPFS actually requires.

[adlrocha]

The current set of RFCs can be split in two categories: i) those that improve content fetching, and ii) those that improve content discovery.

Agree, and I think this is closely related to the layered architecture we identified in the project's related work and included in the RFCs ids, with your permission I include it here for the sake of completeness

• Layer 0 - Data Structure: Layer responsible for determining the structure of the content. It determines how the data is stored, the coding scheme used to represent the data, and the specific architecture followed to store content in the network.
• Layer 1 - Content Discovery and Announcement: It specifies the mechanisms for the discovery of content in the network. It determines how to conform the requests and the messages to be sent to find content in the network, as well as the announcements of seen/stored content to other nodes.
• Layer 2 - Negotiation and Content Transmission: Along with the discovery of content, there may optionally be a negotiation phase so that once the content has been discovered its transmission is “formally” requested. This layer implements schemes to negotiate the request of content to the most suitable peers available as well as the actual transmission of blocks from the network. It negotiates and opens a transmission channel between the requester node and the provider of the content. In some implementations this layer may be embedded in layer 1.

suggest that the first two RFCs above get integrated under the umbrella of the third one. The third RFC is the most sophisticated one IMO. The reasoning is that it is difficult to set global standards/values for optimisation, e.g., all content transferred by Bitswap is compressed or all applications use the same block/chunk size.

Totally agree! Also, as you suggest (and we already discussed) the implementation of this RFC opens the door to the exploration of a greater gamut of embedded protocols within Bitswap as it offers flexibility in the discovery and transmission through negotiation.

RFC|BB|L1-04: Track WANT Message for future queries
RFC|BB|L1-02: TTLs for rebroadcasting WANT messages

As shown in the initial results of RFC|BB|L1-04, this RFCs can speed-up the discovery of popular content. One could say that Bitswap is "competing" against the DHT in the disovery of content. For rare content, the DHT will definitely be more efficient, but for (i) popular content, and (ii) large files (comprised of a huge number of blocks) we can potentially (i) get content faster with Bitswap compared to the DHT and/or (ii) make the DHT's job easier (by letting it know in advance the rare blocks it needs to discover to efficiently fulfil the full request).

[petar]

@aschmahmann

This division into two categories makes perfect sense. These two layers are bitswap/graphsync-specific in my mind.
To complete the picture, I would introduce a third layer which lives outside of bitswap/graphsync, to get:

• Content Fetching layer is narrowly about optimizing a fetch between two bitswap nodes that have already found each other.

• Content Discovery layer is about mechanisms for discovering content within the bitswap/graphsync network and protocol.
  In particular, this layer is dealing with techniques for gossiping and retaining information across bitswap/graphsync nodes,
  without resorting to external systems (like the DHT). When content discovery, as defined, fails it definitionally resorts to the
  content routing layer, next.

• Content Routing layer is a separate system that performs two tasks generically: provide (peer for CID) and find (peer for CID).
  It's the content routing layer's discretion how these two operations are implemented to maximize the probability that find succeeds
  if there was a preceding provide.

Of particular note here is how content discovery and content routing interact. Content discovery attempts to
find the CIDs that Content Routing couldn't. They are two separate and complementary software systems and protocols.
One is embodied in the bitswap/graphsync protocol, the other in the content-routing protocol over the DHT.

Note: Our current implementation does not have an actual "Content Routing" software component, as defined here.
Instead, applications directly (and usually ad-hoc) write to the DHT itself. For instance, bitswap currently resorts to the DHT,
not to a content routing layer.

We want to move away from this and introduce a dedicated content routing component (and protocol, in fact),
which will contain the sophistication of how to utilize the DHT to implement a high-quality content routing API.
Quite a lot can be done in this layer (caching, delegation, piggybacking, etc.) and none of it is bitswap/graphsync specific, hence justifying its own category.