Show how to use `opam-nix` to add dependencies

[brendanzab]

opam-nix looks like a neat tool to read Opam files, and pull down dependencies. Thanks @balsoft for letting me know about it! It would be cool to demo how this could be used in this repo.

It would be good to consider if it's still a good idea to let Dune drive the generation of hello.opam. It's already kind of janky how I check that the generated file is up to date, requiring people to re-run dune build in the dev shell to keep it up-to date. I'd want to ensure that the process of adding a new dependency, then re-building the package is smooth. Perhaps we should just get rid of the generate_opam_files and package specification and let Opam drive pulling down the packages?

[brendanzab]

I guess one of the issues I have is that I don't know how regenerate the opam file without calling dune build. But if I need to add a new dependency, and that dependency will be added by using the Opam file, then I'd assume the initial dune build would fail to run?

[balsoft]

I guess one of the issues I have is that I don't know how regenerate the opam file without calling dune build

I think dune build hello.opam should work and not require any dependencies.

As for the duplication issue, I think it can be approached from the other direction. IIUC we can generate hello.opam using just dune and ocaml without any networking required, so we can do so on the fly using IFD, and then use the existing opam-nix functionality to call the resulting opam package. I'll see if this is as easy as I think it is tomorrow (hopefully)

[balsoft]

Actually, couldn't restrain myself and did it today: tweag/opam-nix@178c26e

If you remove hello.opam from this repository, then (buildDuneProject "hello" ./. {}).hello builds it successfully.

[brendanzab]

Oh, thanks so much, that was fast! Yeah removing hello.opam might work nicely. I'll probably also want to add some instructions for a raw Opam+Dune workflow, as I want to support that as well - not everyone uses Nix and I want it to be usable with the traditional tooling people know.

So yeah, I've been trying to understand how to use opam-nix but I'm finding the API and examples a bit hard to wrap my head around? I don't remember having the same level of difficulties with naersk (even if some things still confused me at first).

I've jotted down some of my first impressions – trying to capture my clueless, beginner state. They're a bit raw but if you're up for it I'm happy to post them either here, or in private, but I'm aware that I lack lots of context and don't know and your existing constraints/challenges with the project.

[balsoft]

Please do! opam-nix is still being developed, and I'll update documentation as I do so. For now, I think the example for buildDuneProject should be relevant for you.

[brendanzab]

Ah yeah I was more stuggling actually getting to a state where I could call buildDuneProject.

At first I just tried adding opam-nix.url = "github:tweag/opam-nix" and then calling:

{
  checks = {
    hello = (opam-nix.buildDuneProject "hello" ocaml-src {
      doCheck = true;
    }).hello;
  };
}

But yeah of course that didn't work because I'd misread the example 😅. So I tried to look for some full examples.

This is my attempt at trying to get something going based on the README (apologies for the trainwreck):

• Looking at the README I'm intially looking for an example.
• There's a list of them which is nice! I think it might be better if it was in the readme itself, but I can live.
• When I click on the first example, labelled in the readme as "Building a package from opam-repository" it looks very different from what I'm used to from the flake setup, which is initially disorienting.
• I'm not sure what the parameters are for, so I skimmed past those.
• For some reason opam-nix = inputs.self.lib.${pkgs.system}; is being bound - like I'm confused why the system packages are being called opam-nix?
• What is scope, and why is it being returned from the nix file?
• I'm also confused as to what is being shown in the example. Where is the package being built, as was claimed in the name of the example?

Looking through the other examples was similarly confusing, so instead I tried looking at the templates for more help.

• First I looked at the simple template. Simple sounds good!
• The inputs seem pretty standard
• I'm confused why only [ "x86_64-linux" ] is iterated over - wouldn't it be better to use eachDefaultSystem?
• Why are we using legacyPackages in the flake?
• The let binding is rather large and it was hard to understand how it was scoped, especially with the two attribute sets passed to opam-nix.queryToScope (which perhaps I'm not used to reading).
• Again we have opam-nix = inputs.opam-nix.lib.${system}; - for some reason we are not using opam-nix directly, that was bound in the parameter of the outputs, instead acessing it with inputs.opam-nix?
• We have the scope thing again - I'm not sure what that is for. What is a query, and why am I converting it to a scope?
• Assigning the package to null as a way to get the current version seems uhh, ok (if a little ugly), but why do I need to do that when it should be reading the version from my opam file?
• Being able to set the base compiler seems pretty neat!
• setting the default package seems reasonable, still seems kind of weird to be using legacyPackages - perhaps this is something I'm unaware of as a newcomer to nix?

I'm very confused about what this is meant to do:

let
  # ...
  overlay = self: super:
    {
      # Your overrides go here
    };
in scope.overrideScope' overlay;

why the overrideScope'? is there an older deprecated version of overrideScope?

I tried to go on, then reading the concepts in the README, in an attempt to answer some confusions I had, but I was already kind of reaching my limit in terms of information overload.

[brendanzab]

[balsoft]

Examples were originally intended for prototyping the interface and testing the capabilities of the builder, and only then they became actual examples. I have refactored them to be flakes, so it should be a bit easier to understand.

I'm confused why only [ "x86_64-linux" ] is iterated over - wouldn't it be better to use eachDefaultSystem?

For now, only Linux x86_64 is supported, so I only iterate on that. In the future, I may also support macos, but for now there's no point in providing outputs that don't work.

Assigning the package to null as a way to get the current version seems uhh, ok (if a little ugly), but why do I need to do that when it should be reading the version from my opam file?

It was using an outdated API, I have updated the local template. Now the resulting scope will automatically include latest versions of all packages from your project. As for the simple template, it reads the entire opam-repository and so you have to tell it which packages from there you want to resolve for. There's no single "opam file" to read the versions from.

Why are we using legacyPackages in the flake?

legacyPackages is the the only correct output format for nixpkgs scopes. It allows you to nicely build individual dependencies of your package, with the correct switched versions.

[balsoft]

And, as for your example, try

{
  checks = {
    hello = (opam-nix.lib.${system}.buildDuneProject "hello" ocaml-src { }).hello.overrideAttrs;
  };
}

[brendanzab]

For now, only Linux x86_64 is supported, so I only iterate on that. In the future, I may also support macos, but for now there's no point in providing outputs that don't work.

Ahh, I am on macOS - will that mean I'll need to wait for that to be supported?

[brendanzab]

Btw, checked out the examples and yeah, they seem a bit more approachable now which is super nice! :)

[balsoft]

Ahh, I am on macOS - will that mean I'll need to wait for that to be supported?

Yep. I don't have a mac device to test on. I imagine it wouldn't be too difficult to support, basically need to remove some things which were hardcoded to linux and add a depexts map for macos.

[balsoft]

@brendanzab Hey, now it should somewhat work on macOS! External dependency handling is not as good as with Linux yet, but you can help by updating https://github.com/tweag/opam-nix/blob/main/overlays/external/homebrew.nix .

[brendanzab]

Hey, just wondering, do you have any idea how to install packages off Opam that aren't in the nix packages repository? Would I have to invoke opam manually for that?

[balsoft]

The entire idea of opam-nix is to build packages which aren't in nixpkgs :)