Support depending on Haskell libraries instead of pkg-depends

[hsyl20]

Description

This MR allows the use of Haskell crypto libraries (blst-clib, sodium-clib, secp256k1-clib) instead of globally installed ones.

Pro:

• building from sources becomes very easy even on weird systems (e.g. Windows' WSL) and https://github.com/IntersectMBO/cardano-base/blob/master/INSTALL.md#option-3-building-from-source could be modified to be only:

cabal update
cabal build all
cabal test all

• we have more control on our dependencies (e.g. secp256k1's enabled modules), even without using nix

Con:

• the Cabal libraries are experimental: I've replaced the upstream build systems with a .cabal file, and in some cases (libsodium...) it wasn't trivial. Maybe they regress in some aspects for now (perf, portability...)

I'm opening this MR because it's already been helpful to someone on Windows. It might become the recommended way to build cardano-base in the future but I'd like to have some benchmarks to ensure that performance doesn't regress while switching to the cabal-built packages.

Checklist

• Benchmark for regressions
• Commit sequence broadly makes sense and commits have useful messages
• New tests are added if needed and existing tests are updated
• All visible changes are prepended to the latest section of a CHANGELOG.md for the affected packages.
  New section is never added with the code changes. (See RELEASING.md)
• When applicable, versions are updated in .cabal and CHANGELOG.md files according to the
  versioning process.
• The version bounds in .cabal files for all affected packages are updated.
  If you change the bounds in a cabal file, that package itself must have a version increase. (See RELEASING.md)
• Self-reviewed the diff

[lehins]

I am not sure I understand the reasoning for adding this complexity.

• First of all we can't use this approach with cardano-node as it is today, since none of these packages are on Hackage/CHaP and we can't be using SRPs
• None of these packages are maintained by IOG or Intersect
• Most importantly, however, they are forks of original libraries, which means we/someone at https://github.com/haskell-cryptography would have to maintain forks of those crypto packages.

I don't understand what kind of control do we need for cardano-node that we don't already have today?

In my opinion using this approach adds a weak point, instead of being an improvement, since cryptographic indirection is maintained by a third party. We can trust downstream C dependencies like libsodium, blst and secp256k. But if we switch to using forks, even if the original C code doesn't change, we need to constantly keep an eye on it and do an audit every time we need to use a new release. From that perspective in my opinion it is a regression, not an improvement.

To sum it up. I am personally very much against this approach, since:

• it doesn't buy us anything new (aside form helping someone on Windows, which should not be our main goal anyways)
• it increases overhead and complexity due to the need to maintain those forks, without IOG or Intersect having total control of them either. And if we did, why should we maintain them and update for every release of crypto dependency?

I would really to see some strong motivating factors for introducing such indirection. Also I'd like hear some input on this from @angerman, @perturbing and @disassembler cause that is serious change to how we deal with cryptography and we shouldn't take it lightly.

[hsyl20]

I should have specified that we're interested in doing this for Plinth, not for cardano-node.

For Plinth, this approach reduces installation complexity (users don't have to install the crypto libs themselves). And we care about Windows users being able to use Plinth.

None of these packages are maintained by IOG or Intersect

Most importantly, however, they are forks of original libraries, which means we/someone at https://github.com/haskell-cryptography would have to maintain forks of those crypto packages.

I agree that maintaining forks is less than ideal.

• About control: I've put the packages under the haskell-cryptography umbrella (that I joined for this purpose) to share maintenance with other interested parties, maybe we could keep them in the IOHK repo instead. If we pin a version we trust in our cabal files, then it won't upgrade to a version we haven't checked and approved.
• The forks are lightweight as we're just adding cabal support and we don't touch the domain code. Maybe we could add some automated way to check that building with cabal vs building with whatever they use produces an equivalent binary to some degree. As we don't need the timing guarantees provided by the libraries for Plinth, maybe just checking we get semantically valid outputs is enough?
• We already ask users to install a fork of libsodium (cf https://github.com/IntersectMBO/cardano-base/blob/master/INSTALL.md) that is lagging several years behind upstream, so the situation isn't totally new.

[perturbing]

I appreciate the goal to improve DevX (@rober-m), as indeed plinth building is nontrivial if you are not tech-savvy.

That said, I agree with @lehins that it is not really ideal how one should maintain this (and who is responsible).

What is the impact if such an addition is not merged? Are there no alternatives?

We already ask users to install a fork of libsodium (cf https://github.com/IntersectMBO/cardano-base/blob/master/INSTALL.md) that is lagging several years behind upstream, so the situation isn't totally new.

Also note that since that was written, the VRF that is on our fork has been merged into libsodium itself. So you can either use the old fork or any new version of libsodium now.

[lehins]

I should have specified that we're interested in doing this for Plinth, not for cardano-node.

This is the most important piece of information that was needed for this PR, which clarifies it quite a bit. I would like to ask you to document the cabal flag as such. Something along the lines: "This flag is used for Plinth users and should not be used when compiling cardano-node" Could you also please add a link to the Plinth source code where it will be used. This will be quite important for future maintenace, since right now I can't even find Plinth source code 🤷

We already ask users to install a fork of libsodium (cf https://github.com/IntersectMBO/cardano-base/blob/master/INSTALL.md) that is lagging several years behind upstream, so the situation isn't totally new.

Exactly, that situation serves as a proof that we should try and avoid doing that in the future. I've created a ticket to abandon custom version of libsodium: #636

The forks are lightweight as we're just adding cabal support and we don't touch the domain code.

This still doesn't give me confidence when comes to using those for cardano-node. Fork is still a fork. It has different set of people that has access to the code that the original maintainers of crypto packages, which as I pointed out earlier creates a week link.

maybe we could keep them in the IOHK repo instead.

If these packages are not going to be used for cardano-node, it's fine to not have them in IOG or Interesect, but I am not sure what are the requirements of Plinth are.
One thing we do need for sure is to release them somewhere, either hackage or CHaP (preferably hackage, since they live in haskell-cryptography. We can't have any SRPs in our repo, because that would make cardano-crypto-class unbuildable when used from CHaP and with that flag on.

I am still quite skeptical about this approach, maybe Plint should create a lightweight fork of cardano-crypto-class and depend on that instead? For now, however, I am not going to appose merging it, since it will have no affect on cardano-node, which is my main concern.

[lehins]

Having a flag like this that is untested on CI is a not good.
IF we are to proceed with this, we definitely need at least one CI action that runs all the tests for cardano-crypto-class and cardano-crypto-praos with use-haskell-clibs flag on

[lehins]

Suggested change

	package secp256k1-clib
	-- These configuration flags are only needed when `cardano-crypto-class` is used with `use-haskell-clibs`
	package secp256k1-clib

[hsyl20]

Thanks a lot for the feedback! I've removed the most controversial bits and updated the comments.

However I agree with:

I am still quite skeptical about this approach, maybe Plint should create a lightweight fork of cardano-crypto-class and depend on that instead? For now, however, I am not going to appose merging it, since it will have no affect on cardano-node, which is my main concern.

And:

Having a flag like this that is untested on CI is a not good.
IF we are to proceed with this, we definitely need at least one CI action that runs all the tests for cardano-crypto-class and cardano-crypto-praos with use-haskell-clibs flag on

So maybe our best way forward is to have Plinth use its own branch of cardano-base with this minimal patch applied (it already has its own cabal.project with the required flags and SRPs). In which case we can close this PR.

Another thing I'm wondering is: does Plinth actually need any of the features provided by the C libs? If it doesn't, then maybe we could add a flag to make them optional in cardano-crypto-class/praos (no idea how possible it is). It would be the simplest: no fork to deal with at all.

I'll discuss this with the Plinth team.

[rober-m]

Thank you, @perturbing, for pinning me. Dealing with Plinth is significantly more painful than with alternatives, so anything that improves devx is very welcome! @zliu41 @SeungheonOh, what do you think about:

So maybe our best way forward is to have Plinth use its own branch of cardano-base with this minimal patch applied (it already has its own cabal.project with the required flags and SRPs).

Somewhat related issue: IntersectMBO/plutus#7145