Implement an IPFS compatible CID function in Elixir using Multihash SHA256

[nelsonic]

This issue/epic is dedicated exclusively to How i.e. implementation

For the reasoning behind Why we are writing this code, please see: #1
If anything is unclear as to Why, please comment on that issue.
(we all need to be 100% clear on Why we are doing this work if you aren't, Ask Questions!)

Todo

• Read the JavaScript Implementation of CID: https://github.com/multiformats/js-cid
  to understand how it works. If you have questions, please ask them in: https://github.com/dwyl/learn-ipfs/issues

  • try out any examples in the docs on localhost and try to see if re-ordering elements in an Object or nested Array produces a different CID.
  • Document the process of trying out the JS cid implementation.
    (i.e. this is for everyone to learn, not just the one person!)

• Read the "not working" Elixir version: https://github.com/nocursor/ex-cid

  • See if it's "salvageable" or if we need to write our own implementation from scratch.

  (obviously my preference is to use a module someone else has invested time in,
  to avoid "fragmenting" the ecosystem, but if the author does not respond to issues requesting help, then I am not left with much choice ... 😢)

• Implement an "offline" version of CID in Elixir that produces the exact same CID as the JS version.

  • cid of a String should always be the same for a given string.
  • cid of a Map should work regardless of the order of content.

  The way I did this was to order the keys of the Map but we need to check the JS implementation to ensure that's how they have done it.

  • What other types of content do we want/need to create CIDs for? (can we add them later?!)

MVP CID v1

for our MVP we only need a sha2-256 hash in Base58BTC which is URL-safe.
For this we can use the code from https://github.com/multiformats/ex_multihash (which is maintained) and https://github.com/nocursor/b58 (which is unresponsive) respectively.

• Write comprehensive doctests that demonstrate that the code works as expected.
• Create beginner-friendly examples. (we can split this out into separate repos later!)
• Re-org the Readme of this repo so the "theory" & reasoning is lower down
  • and the "get started in 1 minute" is at the top.
• Publish to Hex.pm
• PR to https://github.com/multiformats/multibase

@RobStallion I apologise for using the expression "deep dive" to describe this quest.
I sometimes forget that certain expressions aren't widely used outside of the tech community. 😕
(and sadly, we don't yet have a comprehensive glossary of common tech expressions/terms ...)
https://en.wikipedia.org/wiki/Deep_diving just means "diving to a depth beyond the norm".
i.e. diving into the code/examples/docs beyond the superficial.
We need to deeply understand everything about IPFS CIDs so that we can create compatible Elixir-lang implementation that we can use to create cids for all our content.
Deeply understanding the JS/Go-lang CID code requires "Deep Work" youtu.be/ZD7dXfdDPfg
This is a prereq to writing a CMS, a Learning Platform and a Time/Task Management app!
i.e. this is a "foundational piece" for the entire future of @dwyl (#NoPressure...)

Relevant Reading

• general background Content addressing - what is that and why? learn-ipfs#6
• docs: https://docs.ipfs.io/guides/concepts/cid/
• post: https://pascalprecht.github.io/posts/content-identifiers-in-ipfs

Help Wanted

We really need help on getting this package built, documented and shipped so we can move forward with our "stack" dwyl/technology-stack#67 and "roadmap" https://github.com/dwyl/product-roadmap
If you have the curiosity, energy and time to help, please comment below! (Thanks!)

[RobStallion]

Have been looking into CIDs and IPFS. See here for all thoughts captured.

CIDs are made up of codecs and multihashes.

Multihashes themselves are self describing hashes (e.g. they contain information about the hashing algorithm that was used to hash the data). See this comment for an example of a multihash in elixir.

In order to create our own CIDs it appears we needs to be able to create a multihash (something that we have been able to do with ex_multihash and a codec.

Looking into codecs now to get a better understanding of what exactly they are. They appear to have a similar role in a CID as the hash_type in a multihash.

[RobStallion]

I have been able to recreate the steps listed in this article. This shows that I can at least create the same hashes at CIDv0 myself using the command line.

I do not fully understand what additional data that IPFS is adding to the data that you store on it...

$ echo "Hello World" | ipfs add -n
$ added QmWATWQ7fVPP2EFGu71UkfnqhYXDYH566qy47CnJDgvs8u QmWATWQ7fVPP2EFGu71UkfnqhYXDYH566qy47CnJDgvs8u

$ ipfs block get QmWATWQ7fVPP2EFGu71UkfnqhYXDYH566qy47CnJDgvs8u | sed -n l
$ $
$ \022\b\002\022\fHello World$
$ \030\f$

As you can see above, I have just added Hello World text to a file in IPFS but when I log the file from IPFS it shows more than just Hello World now.

However, I do not think that we need to fully understand exactly what extra data IPFS is adding right now. We should (hopefully) be able to use a library that will handle this step for us (otherwise we are just reimplementing the IPFS logic ourselves which doesn't seem logical)

The code above only relates to CIDv0.

CIDv1
<mb><version><mc><mh>

CIDv0
<mh>

mb = multibase prefix
version = CID version
mc = multicodec-packed-content-type
mh = multihash-content-address

CIDv0 is only the last part of CIDv1.

Next step

Try to recreate the steps to create a CIDv0 hash taken in the command line, but in elixir.