Server design/implementation discussion points

[jonascarpay]

I have been trying to think about what the most robust server design looks like. This is an attempt to tabulate the discussion I've been having in my head. If you have any points that you feel are missing, please add them. I hope we can have a talk about this soon.

• Do we allow builds to run in parallel?
  • Con:
    • Parallelism leads to race conditions. The example I've used so far is when we get the same request twice, start 2 identical builds, and both try to insert their result into the database. I assume other similar situations will arise that we have not thought of yet. They may not be impossible to work around individually, but I suspect that in total it might not be not worth it.
    • As long as parallel builds happen on the same computer as the server, too many parallel builds may not leave enough memory for the server.
    • Parallel builds are not necessarily faster than serial builds. Individual build processes are typically already heavily parallelized, so adding extra parallelism introduces extra chances for running out of memory without being faster necessarily.
    • Parallel builds make timing guarantees harder. If we launch 5 builds simultaneously that would each occupy the entire CPU and take $t$ seconds, all 5 builds finish after $5t$, instead of finishing in order. With serial builds, we can guarantee that whoever made the first request gets their binary delivered first.
    • Strategies like pre-building during down times might be more effective at minimizing wait times than trying to optimize scheduling.
  • Pro:
    • Requests that have already been built need to be treated different from fresh builds if we don't want e.g. a slow build for a rare binary to block requests for vim.
      • Maybe Nix gives you tools to check if something has already been built, that might be an easy way around this.
      • I have a branch where upon success, store paths are cached. This means sidestepping Nix entirely for things that have already been built.
        • Pro:
          • It allows us to query the database for different queries that result in the same binary/derivation.
          • We completely eliminate Nix evaluation so responses are faster
        • Con:
          • We now need to watch out for things like GC and nixpkgs bumps invalidating our database
    • Slow builds can now hog the CPU while smaller, faster builds wait. In this case, total waiting time actually increases. Queuing theory is hard
    • If we end up using a build farm parallelization becomes free

[cdepillabout]

In our chat today, we decided that for now we'd like to use nixbuild.net, since that seems to fit out use-cases the best.

We imagine that it will be possible to start parallel builds on nixbuild.net, which should work well for us.

Since nixbuild.net costs, we'd like to try to pre-cache as many common packages as possible before actually launching binplz.dev.

[kayhide]

With nixbuild.net, I think we need to check what happens when duplicating builds are requested.

For example, while vim is being built on nixbuild.net, what would happen another vim build is requested.
I can imagine several scenarios. Depending on it, we might want to manage building status on our side.