Add library field (excluded_modules)

[nojb]

This PR (which arose from a local patch that we have at LexiFi) adds a new field to the (library) stanza: (excluded_modules). This field (which is similar to (private_modules)) specifies a list of modules that:

• are compiled when the library is compiled,
• are not "seen" by non-excluded modules,
• are able to "see" all modules (excluded and non-excluded),
• are not linked into the final library, nor installed, etc

In our setting inside LexiFi, we define a library which contains the main part of our application code where the exact list of modules can vary from client to client, so we generate the list of modules dynamically using (:include). However, we want to make sure that the rest of the modules (which are not "active" for the current client) continue to compile. This feature is useful whenever you have "inactive" modules that you wish to make sure they still compile.

This feature can be simulated by introducing a second library, but doing so incurs in a substantial penalty due to the unnecessary recompilations caused by Dune's coarse-grained inter-library dependency approximation, as well as extra linking steps which can be quite slow.

Doc and tests are included.

[rgrinberg]

At first glance, this seems like a rather niche feature that's not going to be beneficial to many users. On the other hand, it does seem like there's something missing here in dune. Is it so difficult to keep this patch in your fork while we're still thinking of a way to generalize this feature?

[alainfrisch]

it does seem like there's something missing here in dune

I know little about dune, but I can propose a possible interpretation of "what's missing" : in OCaml, compilation units are not intrinsically tied to libraries/executables; but in dune (as far as I know), the only way to tell dune to compile some modules is to include them in a library (or executable); and it's impossible to link the same module in several libraries.

For the point raised in this issue, we'd like to have dune compile some modules without linking them anywhere (just to make sure they compile fine). Grouping them in a "dead" library seems like a useless hack, forcing some (slow) useless linker invocations.

And independently, we also have a frequent use of linking the same module into multiple libraries. This happen because our application compiles both to server-side native code and browser-side javascript, with some shared modules. For many components of our application, we end up having to create 3 libraries : server-only, client-only, shared. Again, one extra library (--> more linker invocations) only as a work-around for not being able to link the same modules in different libraries. (We sometimes use an even worse hack : copying the shared source code to a different directory and compile it twice.)

Would it make sense to introduce a way to specify how to compile some modules (flags, dependencies, etc), independently of included those modules in libraries/executables? And then refer to those modules in actual libraries/executables? Or, a slight variant: introduce a notion of "pseudo library", only to specify how to compile a set of modules, and allow referring to such library in libraries/executables (the semantics being to include all modules from the pseudo library). There would be no build artefact from pseudo libraries themselves. It would also allow specifying different flags for a subset of a library.

Going further, one could just stop using OCaml libraries at all... If we do that, one might want to extend OCaml to specify "optional units" when building an executable (units that are included only if it's referenced by another linked unit -- same semantics as for modules from a library).

[rgrinberg]

Thanks for the detailed response.

Indeed dune imposes the limitation that compilation units may belong to a single stanza. As you later point out, this limitation is there because it is the stanza that defines how the unit needs to be compiled (flags, wrapping, etc) and these compilation settings must be unique per compilation unit.

Having a more primitive construct (module groups?) seems like a good idea to me, but I'm not sure how this would work for wrapped libraries. Wrapped libraries want to prefix every module name with the library's name, so how would it be possible to reuse the same module group between different wrapped libraries? Maybe we'd need to limit this feature to unwrapped libraries?

I've seen the idea of getting rid of archive files floated before, and it does seem very attractive. Not only these archives require linker invocations, but they also consume precious disk space. AFAIK, it might even be possible to experiment with archive-free executables at the moment if we extract the individual cmx files we need from the entry point cmx of an executable using ocamlobjinfo.

[alainfrisch]

I haven't though of wrapped libraries (which we don't use -- so having the feature only for unwrapped libraries would be enough for us). Would it make sense to support a notion of wrapping for those "module groups" themselves?

(About getting rid of archive files: linking many object files rather than fewer archive might involve opening more files at linking time, which might have a small performance hit; also, this means we'd need to ship many object files instead of archives, which might take more space on disk. On the plus side, we avoid having to create those libraries, which can, at the end, allows for faster builds in a big monorepo with many libraries and few executables.)

[rgrinberg]

Alright, so I think this is settled for now. Let's go with some sort of first class compilation unit stanza that can be attached to unwrapped libraries.

FWIW, I had no intention of removing archives for release builds in dune. They would need to be retained as long as we need ship them to users. I would only get rid of them in dev builds to support incremental compilation.