Update OpenMPI documentation #301
Conversation
|
preview available: https://docs.tds.cscs.ch/301 |
|
preview available: https://docs.tds.cscs.ch/301 |
|
preview available: https://docs.tds.cscs.ch/301 |
|
Not ready (open TODOs) but open to feedback already. |
| !!! info "CXI provider does all communication through the network interface cards (NICs)" | ||
| When using the libfabric CXI provider, all communication goes through NICs, including intra-node communication. | ||
| This means that intra-node communication can not make use of shared memory optimizations and the maximum bandwidth will not be severely limited. | ||
| This means that intra-node communication can not make use of shared memory optimizations and the maximum bandwidth will be severely limited. |
There was a problem hiding this comment.
I'd move this note higher up (at the beginning of the section), so that the limitation is clear from the start, and add a link to the next section in this note. Something like "For an experimental workaround to this limitation, see 'Using the external LINKx provider'".
There was a problem hiding this comment.
See the other refactoring comment. I didn't explicitly move this comment further up but added a general intro that compares and explains the CXI and LINKx providers.
| The experimental [LINKx](https://ofiwg.github.io/libfabric/v2.3.1/man/fi_lnx.7.html) libfabric provider allows composing multiple libfabric providers for inter- and intra-node communication. | ||
| The CXI provider can be used for inter-node communication while the shared memory (`shm`) provider can be used to take advantage of xpmem for CPU-CPU communication and GDRCopy for GPU-GPU communication. | ||
|
|
||
| !!! warning "The LINKx provider is experimental and may contain bugs, in particular for intra-node communication" |
There was a problem hiding this comment.
| !!! warning "The LINKx provider is experimental and may contain bugs, in particular for intra-node communication" | |
| !!! danger "The LINKx provider is experimental and may contain bugs, in particular for intra-node communication" |
There was a problem hiding this comment.
I may actually go the opposite direction... The LINKx provider is experimental and it was buggy until very recently. In my latest tests, however, it's actually been very good. I may be too naive in thinking that the LINKx provider is working well for most applications though. My tests are primarily OSU...
In principle I'd like to promote LINKx as the default/primary choice and point users to CXI-only as a fallback in case they encounter issues.
In the user's shoes, how upset would you be if we recommend LINKx as the first option, only for you to find out that your application crashes with it? Or conversely, how upset would you be if you play it safe with CXI as the default choice, and then you figure out later you've been leaving a lot of performance on the table by not using LINKx even though it works just fine?
There was a problem hiding this comment.
I see your point. My suggestion was mainly triggered by " Always validate your results to ensure MPI is working correctly.", which to me also means that it could lead to wrong results (not only crash).
In principle I'd like to promote LINKx as the default/primary choice and point users to CXI-only as a fallback in case they encounter issues.
Maybe we can rephrase the whole admonition to make it clear that this is the approach we suggest to try (for performance reasons, and because it works for our tests), but that it is still experimental and therefore there might be risks involved? As it is phrased now, the whole admonition sounds quite negative (hence my "danger" suggestion).
There was a problem hiding this comment.
Given e.g. the problems @jpcoles-cscs had in his tests with lnx, I'll keep it as the potentially-better-but-secondary option and make it a danger as you proposed. I'll slightly restructure the intro though to present the two options a bit more equally.
|
preview available: https://docs.tds.cscs.ch/301 |
|
preview available: https://docs.tds.cscs.ch/301 |
| ### Versions | ||
|
|
||
| === "25.11" | ||
| TODO |
|
preview available: https://docs.tds.cscs.ch/301 |
|
I think I've somewhat successfully merged latest master containing #272. The uenv section is still first and is more detailed since it contains an explanation of both the CXI and LINKx providers. I've added a note to the containers section that the example containerfile does not install the linkx provider. Otherwise I've only very minimally touched the containers section. |
3 participants
Companion to eth-cscs/alps-uenv#263.
The list of packages in
prgenv-gnu-openmpistill needs to be updated.