Support for GHCup (support for multiple package databases)

[hasufell]

I'm currently rewriting large parts of GHCup, which will enable to add new compilers and tools to it without changing the code. Instead we support an "install specification" in the GHCup metadata (such as executing a Makefile with specific arguments or copying files): haskell/ghcup-hs#1315

The invariant that must be satisfied is as follows: anything that GHCup installs into ~/.ghcup/ must be immutable. The tool must not during runtime add any new files there. This is true for GHC, because the package database it ships with is separate from the "user package database" that gets used when you e.g. cabal install things.

I guess MicroHS needs to support reading from two package databases to make this work as well.

[augustss]

I will make sure that it can do that.

[AlexandreTunstall]

The ability to read from multiple package databases would also help in Nixpkgs.

At the moment, we generate a single package database by symbolically linking all of a package's transitive dependencies in a single directory.
And we also wrap the mhs binary with a script to add a -a argument for each direct dependency.
This setup is a bit fragile, and we could do away with it by instead pointing mhs+mcabal to all of the dependencies' individual database directories.

Another related thing we need to do in Nixpkgs is install to a completely empty package database.
We currently hack around the inability to do so by deleting the aforementioned symlinks after mcabal install.

[augustss]

I'm happy to make changes to make your life easier. Can you make a list of exactly what you need?

[AlexandreTunstall]

a list of exactly what you need

I think that's a bit hard to describe, so here's a description of what we're trying to achieve, and the specifics of how microhs can enable it are up to you.

Each package is built into its own package database containing only that package and nothing more.

For package builds and shell environments, we would like a way to give microhs and microcabal the list of package databases so that it knows where to find them (as requested in the original post).
And we would also like a way to tell microhs and microcabal where to install the current package, separately from how we tell them where to find dependencies, since the latter paths are immutable.

For example: if we start out with an empty package database, where base and a few internal things are installed, this database lives in the same Nix store path as the microhs binary (/nix/store/...-microhs-0.15.4.0/lib/mcabal/mhs-0.15.4.0).
When we build containers, we need to write the output to a new path (/nix/store/...-containers-0.8/lib.mcabal/mhs-0.15.4.0).
Then, when we build mtl, we need to tell microhs and microcabal to look in both of the aforementioned paths for the base and containers packages, while writing the output to yet another path (/nix/store/...-mtl-2.3.2/lib/mcabal/mhs-0.15.4.0).

As I understand from hasufell's description, this is very similar to GHCup's need to configure an immutable package search path alongside the existing mutable one. I believe that the main difference is that I'm asking to generalise hasufell's request to an arbitrary number of "read" databases (where the compiler looks for packages), where one of them is also considered the "write" database (where the compiler installs new packages).

(If I'm interfering with discussion of the original issue, please let me know and we can continue discussion for Nixpkgs in a separate issue.)

[hasufell]

Just to be clear: I have no game in nix and my requirements are rather different.

I want MicroHs to behave just like ghc/cabal under cabal v1-install. These are my exact requirements:

• the package database that MicroHs ships with is kind of "immutable"/global (see next point)
• installing packages via mcabal installs by default into a user package database automatically (e.g. ~/.mhs/packages)
• I don't have to specify what database to use or install into: both databases are visible by default
  • otherwise I'd have to create wrapper scripts, which I don't want to

That implies:

• we actually can tell mcabal to install into the global database... as that is what we have to do to create a proper distribution of MicroHs. But it's not the default.

[angerman]

My understanding of what @hasufell wants is:

cabal-install :: PkgDB -> PkgDB -> PkgDB -> Pkg
cabal-install globaldb userdb userdb pkg

Whereas what @AlexandreTunstall suggests is:

cabal-install :: [PkgDB] -> PkgDB -> Pkg
cabal-install (global:user:_)@pkgdbs targetdb pkg

E.g. @hasufell just wants to split the global and user db, keeping the global immutable. Whereas @AlexandreTunstall wants to in the extreme split each package into their own db and especially be able to specify the target db to install a package into.

[augustss]

Here is what I suggest. Comments, please.

Finding modules NOT YET IMPLEMENTED

A module M is located using the sourcePath and the packageDbPath.
Both of the paths are a list of directories and are searched left-to-right.
This lookup is used both import and modules given on the command line.

Note: If the module name given on the command line has suffix .hs or .lhs
it is used as a file name, rather than a module name.

First, the the sourcePath is used. For each directory in the
path:

• the module name M is appended with . changed to /.
• then the suffixes .hs, .hsc, and .lhs are tried in order
• if the file exists it is assumed to be the source for module M

Second, if no source file is found, then each directory in the
packageDbPath is tried. If the module is found, the corresponing
package is loaded and the compiled module 'M' from that package is used.

Setting the sourcePath

The default sourcePath is just the current directory, ..
The path can be cleared by using the -i command line flag.
To append to the path use -iPATH, where PATH is a colon (:)
separated list of directories.

Setting the packageDbPath

The default packageDbPath is take from the mhs.config file
in the section [mhs] and key packageDbPath.
If this does not exists the defaults path is empty.

In case this does not exists the path defaults to ~/.mcabal/mhs-VERSION,
where ~ will be replaced by the home directory and VERSION is the
compiler version (as displayed by mhs --numeric-version).

The directories in packageDbPath can start with a '%', and in this
case the value of the key packageDbPrefix is prepended to the path.

The path can be cleared by using the -a command line flag.
To append to the path use -aPATH, where PATH is a colon (':')
separated list if directories.

Installing packages

The default packageDb for installing packages is given by the
key packageDbDefaultInstall.
If absent, it defaults to ~/.mcabal/mhs-VERSION.

The packageDb directory can also be given directly with the -Q
flag used to install.

The mcabal command can install to a particular package db by using
--install=PATH.

Examples

Status Quo

All packages go to the same place.

Compile a module with mhs Foo.

mhs.config:

[mhs]
packageDbPath=~/.mcabal/mhs-0.15.5.0
packageDbDefaultInstall=~/.mcabal/mhs-0.15.5.0
...

GhcUp

Separate system and user package dbs.

Compile a module with mhs Foo.

mhs.config:

[mhs]
packageDbPath=~/.mcabal/system:~/.mcabal-user
packageDbDefaultInstall=~/.mcabal-user
...

Nix

A system package db and then a separate db for each package.

Compile a module with mhs -a%transformers:%mtl Foo.

[mhs]
packageDbPath=~/.mcabal/system
packageDbDefaultInstall=NO_INSTALL_GIVEN
packageDbPrefix=~/.mcabal-user-dbs
...

[hasufell]

I think this sounds generally workable.

Where is the config file read from? GHCup would have to ship with this file. Are there other settings a user might want to change? Because we don't want to require the user to edit files inside ~/.ghcup. So my suggestion would be that this file is purely for package db configuration and nothing else.

I think the two main mechanisms here are:

• a wrapper script that tells mhs/mcabal via cli flag where the config file is
• the config file location is inferred from the directory the binary resides in (a bit dirty and less portable, but is actually a common method, e.g. looking for ../lib/something)

The other option for me is to just use wrapper scripts without a config file that specify those -a and -Q options directly.
Here we need to make sure that -Q can also be specified multiple times, but would be an override. Maybe you also want a cli flag for setting the % prefix.

The other implication is that we essentially now need a Makefile or configure script that creates that configuration file at install time (and/or those mhs/mcabal shell wrappers). That doesn't sound too hard, though. But we need something that creates a "MicroHs bindist". That part can be done by me though.

[hasufell]

I just noticed: wrapper scripts on windows are not really portable. So there we'd need a way to do it with just a configuration file and e.g. use getExecutablePath to infer the location of the configuration file.

[AlexandreTunstall]

This sounds good for Nixpkgs's use case too.

Besides the --install flag, it's a bit unclear what microcabal's side will look like, but the proposed flags already allow us to achieve what we need using at most a wrapper script for the mhs command.

As long as the configuration file is not hard coded to an FHS location, we can easily work with it.
Most likely, we will use it to configure mhs to use user-friendly paths (in ~) by default, while the Nixpkgs package builds will use the -i, -a, and --install flags to override the behaviour as needed.

[augustss]

The config file is currently ~/.mcabal/mhs-VERSION/packages/mhs-VERSION/data/targets.conf.
(I'll rename it.).
It is not something you normally would edit; it contains compilation commands for different targets.

The targets.conf file is generated by the Makefile so it can have ithe right flags if you want to use GMP.

The location is generated from Paths_MicroHs.getDataDir which is also generated during installation.

I'll add a flag to mhs to use a different configuration file.

[augustss]

I've not decided exactly how it will interact with mcabal, but I think mcabal also needs the -a flag so it can figure out what packages are installed and pass the right flags to mhs

[AlexandreTunstall]

The location is generated from Paths_MicroHs.getDataDir which is also generated during installation.

Yes, I believe that works well for Nixpkgs, since we try to configure package builds so that getDataDir points to the package's final resting location in the Nix store. I would expect the proposed --install flag to also influence getDataDir.

I think mcabal also needs the -a flag so it can figure out what packages are installed and pass the right flags to mhs

I think so too and I believe such an addition (or equivalent) would rid us of most, if not all, of the hacks we're currently using to make mhs and mcabal play nicely with the Nix store.

One alternative idea would be for mcabal to:

1. Allow the user to pass arbitrary flags to mhs (so we can add to -a), much like how cabal has the --ghc-options flag.
2. Ask mhs about what packages are installed so that its logic doesn't need to be kept in sync.

I don't have any strong opinions regarding which approach is best.

[hasufell]

@augustss on windows, I can't create a wrapper script to instruct mhs to use a different configuration file. And the default location doesn't work, since it's outside of ~/.ghcup. So there mhs could do something like:

• get the current location of the mhs executable (getExecutablePath)
• look for <current-executable-dir>/../data/target.conf
  • if it doesn't exist, continue to use the default

[augustss]

@hasufell to install in a non-standard place you simply say make MCABAL=$(HOME)/.ghcup/mhs, or something like that. This should adjust all the necessary paths, and you can just invoke mhs with no wrappers. I'll also make sure that there is a simple way to get the correct configuration file.