[Proposed Feature] Integrate with BitTorrent

[alukach]

Description of Feature:

Note

I don't really know much about BitTorrent protocol so this is a bit of a wild idea that warrants much deeper research before any implementation would take place

Source Cooperative could expose its products via the torrent protocol. I believe Source would serve as both the tracker and the seeder for each product. A bit of research suggests that if the torrents are created correctly, consumers could optionally download just portions of a product rather than a product in its entirety.

What value is this feature adding to Source Cooperative?

This would make it easier for other users to mirror the products for easier decentralized resilience of datasets in a simple opt-in manner.

---

@jedsundwall curious if you've thought about this

[diegoripley]

This is a great initiative. I am currently implementing this exact architecture at Data for Canada.

I believe this hybrid approach (using HTTP web seeding alongside traditional P2P) is the correct path for resilient open data distribution. Here are a few notes on how I am approaching this, which might help scope this feature for Source Cooperative:

1. Proof of Concept & Strategy
I have already integrated this into my dissemination strategy. You can see a high-level overview of how the pieces fit together here:

• Data for Canada Dissemination Strategy

2. Precedence
There is strong precedence for this model in the academic and cloud sectors.

• Amazon S3 famously provided native BitTorrent support (allowing any S3 object to be fetched via torrent) for over a decade. You can view the archived documentation for that feature here: https://documentation.help/s3-dg-20060301/S3TorrentRetrieve.html
• Academic Torrents has successfully used this model to distribute large research datasets for years: https://academictorrents.com/. Datasets regularly are supported by various mirrors (ex. Zenodo, Internet Archive, etc.).

3. Current Laboratory Work
I am actively benchmarking this setup right now. You can follow the technical implementation details in my decentralized distribution labs repo:

• Data for Canada Decentralized Infrastructure Distribution (d4c-infra-distribution)
• Specifically, see the Benchmark for HTTP Web Seeding where I am testing performance: Issue #5

4. Tracker Implementation
If you decide to move forward, I would be happy to discuss how I set up the BitTorrent tracker to effectively direct peers to one another. I believe Source acting as both tracker and HTTP web seed (GetRight style) is the sweet spot for reliability.

Recommended Resources
For a deeper dive into the specific mechanism that makes this work (hybrid HTTP/Torrent downloading), I recommend looking into BEP 19 (WebSeed).

• Reading: BitTorrent Enhancement Proposal 19 - WebSeed
• Video: A Deep Dive Into The BitTorrent Protocol

[jedsundwall]

Very much in favor of this. Am I right that the costs for this should be negligible? I remember S3 doing this but didn't realized it had been deprecated!

A few thoughts:

• My guess is that this only works on a per-object basis, so we would create torrent endpoints for each object?
• This makes me wonder if we should also consider a feature that could TAR up (potentially using Amazon S3 Tar Tool) relatively small repos to make them torrent able.
• This also makes me think that we should offer more robust support for .zip and .tar objects. We now have a great .zip viewer, and I'm wondering if we could allow people to stream objects out of .zips and .tars, and download them, and torrent them.

[diegoripley]

I am doing what I am doing with < $100 USD, so yes, negligible.

A few Thoughts of Mine

• You can either have it per object, or you could even do folder of files as well. My preference is to do per object, due to some limitations of how Zenodo distributes their HTTP files.
  • Here is an Internet Archive dataset as an example. Click on "TORRENT", and it will download a single torrent file, but a folder of files will be available through the torrent, using HTTP seeding.
  • There are a few more factors that you will need to take into account before implementation. For example, what if a user overwrites the current object. You will need to regenerate the torrent and have a way of notifying your users so they stop their old torrents of the file
• A BitTorrent tracker, properly tuned, can handle millions of peers, with minimal hardware. You can either make the torrent "private", so that peers have to go through your tracker, or you can set it up in a way that allows communication between peers, even if your tracker is down.
• The only thing to consider is that WebSeed uses range requests to download the data from HTTP servers, so it should be taken into consideration if you care about overall number of requests being made to Source Cooperative. If you are it pointing to your data proxy, that would be ideal, as you could implement caching and limit S3 costs.
• You can fine-tune the "piece size" (see here for more information) to lessen the number of requests made to Source Cooperative

Wild Idea

• There is WebTorrent that could be used by people viewing the dataset via a browser. You can set it up so that they all share the file with each other....or at least the pieces that they have 😂.

[diegoripley]

Update.

I have added sample torrents to https://source.coop/dataforcanada/ca-orthoimagery-labs . Here is the Bash script I used to create them. It uses both Source Coop and the Data for Canada infrastructure at the same time, plus a couple of "smart nodes". It uses a BitTorrent tracker hosted by my own infrastructure, but you put all sorts of information into a magnet link and dynamically generate them to suit your needs 😉.

Here is a screenshot of it in action.

Personal Computer

This is my personal computer with 2.5GBit/200MBit.

Location: Tottenham, ON, CA.

Smart Node 01

This is from a shared VPS with 50GBit/50GBit, 950GB NVME.

Location: Toronto, ON, CA

Smart Node 02

This is from a shared VPS with 50GBit/50GBit, 950GB NVME.

Location: Amsterdam, NL

[diegoripley]

Hey all, since there's a help wanted tag, I thought I would share some resources/helpful links for deploying, in order of what I think might be of use to you based on my current understanding of your organization:

• Dragonfly - if BitTorrent is a hard sell, you could use this. It is hosted by the Cloud Native Computing Foundation. Apache License.
• aquatic - a Rust UDP/TCP(HTTP) BitTorrent and WebTorrent tracker. Apache License.
• torrust-actix - a Rust UDP/TCP(HTTP) BitTorrent tracker. Persistence can be done via SQLite3, MySQL, or PostgreSQL. Swagger UI built-in, and optinal caching via Redis. MIT License.
• simple-torrent-tracker - A simple BitTorrent tracker (HTTP) for Cloudflare workers. Just sharing this one to demonstrate that you could get really creative with this.

I will share any updates here as I continue my exploration of these technologies! 😄

[diegoripley]

Update. I have used the "industry standard" for BitTorrent (libtorrent to generate torrents, backed by HTTP seeding, more information available at dataforcanada/d4c-infra-distribution#12 (comment) , and current code available at dataforcanada/d4c-infra-distribution@6944839 . libtorrent has bindings in multiple languages, including Rust.

I am currently working on the magnet link generation, which is a way of sharing a torrent without an actual physical file. My specific use-case is to dynamically generate magnet URIs to target specific peering requirements. For example, we can create a magnet URI to target University students with HTTP URLs that are hosted by their CS department, this way the University network is not overloaded by making requests outside of it. The image below demonstrates the torrent working and downloading from multiple providers (including Source Cooperative).

P.S.: libtorrent supports WebTorrent https://feross.org/libtorrent-webtorrent/ , so you could "seed" with your libtorrent backend, and use WebTorrent to utilize web browsers (AKA your end-users) to share pieces of the datasets they are viewing with each other 🤯.

[diegoripley]

Here is something that I am using to guide me in this issue, amazing read, and it aligns really well with Source Cooperative's goals.

https://hackmd.io/wKKm4cIDR6a9kYwZ3srVFg?view

[diegoripley]

I did a brief experiment with Magnet Links (a fancy way of saying "let me download your data without a .torrent file"), and this is what I have so far.

See work at 02_generate_torrent_magnet.py

Notes From The Experiment

While magnet links supports downloading from HTTP servers, my current understanding is that you cannot encode the metadata of the files you are downloading (asides from a few attributes), so you will need a peer to get the other information.

This torrent was started with Magnet Links, but it is effectively stalled since the client does not have information about the file itself, asides the hash.

This is the torrent, but this time there is a peer

Future

I plan on applying everything I have learned in attempt of creating a KISS data catalogue (see here) for more details.

[diegoripley]

I experimented with creating a torrent file for a dataset with 35,993 files to see if it libtorrent could handle it. I tuned the piece size to 64MiB, so the dataset will only be ~76,000 pieces. All tests passed with small subset of the data, and I hope to replicate with the entire dataset once the ca-on_geospatial-ontario-2024A000235_d4c-datapkg-orthoimagery_2024_16cm_v0.0.1-beta raw dataset is finished uploading to Source Cooperative and I am done generating the PMTiles package for it (ca-on_geospatial-ontario-2024A000235_d4c-datapkg-orthoimagery_2024_16cm_v0.0.1-beta.pmtiles), I will connect this up to HTTP seeding, pointing to Source Cooperative, and seeding with a node in Amsterdam, Netherlands. I believe it will be a good case-study on the "smart node" feature I am going for.

https://github.com/dataforcanada/d4c-infra-distribution/blob/faeb967cb1b1bbf97cf685bcf6c1cb9d986dd59f/scripts/02_libtorrent_experiments/03_generate_torrent_large_number_files.py

[diegoripley]

I have finally published a derivative data product that uses Source Cooperative as a mirror, and uses some of my project's infrastructure to share the file as well. You can find more information about the dataset at. I posted it there since there are a TON of geeky data people there, which is one of my target audiences.

https://academictorrents.com/details/adb5741cdcb9352848cc80c976629b44720a04c2/tech

[diegoripley]

Update. I have pivoted into a more general P2P strategy and I am currently looking into libp2p/IPFS; they are a bit like Django, batteries included.

I might continue doing BitTorrent as the technology is still in use and my infrastructure hosts only support it, but I hope to move away from it. What's great about these two tech is that you can also run them on any device (including web browsers), which would enable my dream dissemination strategy.