(WIP) Allow parallel and incremental compilation for lmr (out-of-tree as well)

[bezmi]

Just putting this up so people can test. It needs a little bit of work adding the remaining targets and fixing the install target (instal now works for the most part). Filenames parallel.make are temporary and some of the abspath stuff can do with re-work

Currently only works for the lmr target.
Works for most targets except the test jacobian ones, which I haven't put in yet.

make -f parallel.make -j(nproc)

The command above will put the lmr binary (as well as all other object files, libraries and archives) in BUILD_DIR, which is ./build by default.

Disclaimer: I haven't tested with any actual sims yet. I compiled it and checked that it doesn't immediately crash when calling lmr and various sub-commands.

force_tuple_instantiation.d is there to stop the linker from complaining. No longer required with the --allinst flag.

Please test this properly for me, I'm not an eilmer user, I just enjoy the colours when all the bars in htop are maxed out.

[bezmi]

forgot to add that I've only compiled with OPT_OUT_MPI=1 so far

[bezmi]

If you want to use mold: mold: mold -run make -f parallel.make -j(nproc) worked for me. I also tried make LINKER_FLAGS="--linker=mold" -f parallel.make -j(nproc) and it works now (also with --linker=gold)

[bezmi]

Some (non-scientific) timing tests below. Times are in milliseconds.

time.sh

# time.sh from: https://unix.stackexchange.com/questions/334145/universal-non-bash-time-benchmark-alternative/334152#334152

START=$(date +%s.%N)

"$@"

END=$(date +%s.%N)
DIFF=$( echo "scale=3; (${END} - ${START})*1000/1" | bc )
echo "${DIFF}"

Standard makefile

Clean build, standard linker:

./time.sh make install
83948.345

Clean build, mold linker

./time.sh mold -run make install
83062.608

Re-compile after trivial changes to main.d, standard linker:

./time.sh make install
23553.438

Re-compile after trivial changes to main.d, mold linker:

./time.sh make install
21839.285

Parallel/Incremental build

Clean build, standard linker

./time.sh make -f parallel.make install -j12
39418.929

Clean build, mold linker

./time.sh make LINKER_FLAGS="--linker=mold" -f parallel.make install -j12
36987.783

Re-compile after trivial changes to main.d, standard linker:

./time.sh make -f parallel.make install -j12
3181.941

Re-compile after trivial changes to main.d,mold linker:

./time.sh make LINKER_FLAGS="--linker=mold" -f parallel.make install -j12
1749.701

[bezmi]

Tested 2D/convex-corner. Everything passes except two-block-mpi. 2D/sharp-cone-20-degrees. Runtimes for incremental and standard builds are similar for non-mpi. 2D/cylinder-coupled-fluid-thermal/transient works with both make run and make mpirun, but the mpi one is much slower with the incremental build than the standard build. Probably something wrong in my makefile for the mpi targets.

[bezmi]

@Alex-Muirhead, I wanted to talk about this as part of my lightning talk on Friday, so I spent some time over the weekend getting it up to date and working with MPI builds on the latest master. I also removed the majority of the extra files that I had in the previous version, and it essentially only changes the main lmr makefile.

I'll post a summary of changes, as well as benchmarks after the lightning talks!