Comparison to IPFS

[hsribei]

I'm trying to compare IPFS to hyperlog (and ipfs-hyperlog), but I'm having a hard time wrapping my head around everything. Doesn't IPFS also have a replication layer? Is hyperlog a kind of IPFS without persistence? Could you contrast the two and comment on how (if) they fit together, please? Sorry if any of this sounds stupid, I don't know what I'm talking about :) Thanks a lot!

[hackergrrl]

Awesome question! The differences are subtle, so it's great you asked. :)

I'll try my best here -- maybe @mafintosh can help me out.

---

Both hyperlog and IPFS store and replicate merkle dags: immutable graphs of data. Beyond that, each have some different properties:

Storage

Hyperlog and IPFS are both storage backend agnostic, though the local filesystem is usually used. In this sense, hyperlog is just as persistent as IPFS.

Replication

(the interesting part)

Using replication streams, Hyperlog users form a network of peers which replicate each log's full merkle dag to each other. You can form any network topology you'd like with hyperlog, since it only exposes a replication stream and leaves connecting it to things up to the downstream consumer.

IPFS operates at a higher level: it dictates the network topology, which peers to bootstrap to, and what replication occurs. IPFS nodes have a want list and a have list of blocks. Blocks are just data blobs and aren't merkle dags (kind of like the raw data in a hyperlog node). Replication happens at the block level in IPFS, and only based on wants/haves that peers broadcast. This differs from hyperlog's model of replicating full merkle state to all peers.

This difference in replication gives Hyperlog a superpower that IPFS does not have: since all merkle dags are replicated to all replication streams as they are available, peers can learn about new nodes in the graph without needing to [know to] ask for it. IPFS operates on a want list, so unless you know the hash of the new content added to the dag, you have no way to (quickly) discover it.

It's this power of realtime merkle dag replication that ipfs-hyperlog comes in: it lets you form a secondary (likely smaller, but not necessarily) network of peers that exchange merkle dags in realtime. The ipfs- prefix is there because it uses the same hashing mechanism as IPFS, making the merkle dags produced by it binary compatible with IPFS. This has all sorts of powerful uses for distributed applications, like p2p realtime text editors or chat!

[mafintosh]

I think @noffle's comment is pretty spot on - there are big technical differences because they try to solve different replication problems.

To me hyperlog is also more of a replication module (single responsibility - batteries not included) where as I would clasify ipfs as more of a replication application / network / framework (not in a bad way).

I think a good example of this is that this module doesn't couple any network. It simply exposes a replication stream that its up to you, the user, to wire up to other peers using some sort of swarm / transport.

[hsribei]

Thanks for the clarifications, that was very helpful. So I can use hyperlog in a single-user app basically as a sort of in-memory git repository (content-addressable redux alternative even?), then if I want to hook users together I have to wire them up to a transport myself and hyperlog only deals with sending the right messages over it so that logs converge, is that about right? Again, thanks for the extended answers.

[mafintosh]

@obvio171 sounds about right although it persists to a leveldb so doesn't have to be in memory

[gritzko]

@mafintosh Do I understand it right that "hyperlog" is what literature calls "reliable causal broadcast"?