Update EKAT to require C++20

[tcclevenger]

Update EKAT to require C++20

Motivation

Needed to update to Kokkos 5.0.

Testing

CI already uses C++20.

[bartgol]

Asking the obv question: are we sure that E3SM will build everywhere with C++20 required? If not, this my hold back any ekat update in e3sm.

[rljacob]

It will not build with all of our current default compilers. In particular, the "intel" compiler on Chrysalis. Possibly others.

[bartgol]

Looks good. Only possible concern is whether e3sm is ready for c++20 across all machines.

[tcclevenger]

@bartgol @rljacob Good to know. I created this PR while I'm testing E3SM on the target machines, but since we already know there will be some that are not compatible, we will need to find a different way to handle this.

Kokkos 5 is going to require C++20, what would the process be to eventually update compilers on Chrysalis (and maybe others) so that we can upgrade? Or is this going to be a bigger roadblock? I guess the first step is for me to test exactly which compilers do not work.

[bartgol]

One solution would be to mis with a cmake2.0 way of specifying the std, via something like

if (NOT CMAKE_CXX_STANDARD)
  target_compile_features(ekat_core PUBLIC cxx_std_20)
endif()

Then, in the mach files for old compilers we keep lines like

set (CMAKE_CXX_STANDARD 17)

which will force ekat to honor the required standard. Of course, for machines where we do set 17 in the mach files, we won't be able to build anything that needs kokkos 5.0 (e.g. any F case, since EAM still use kokkos for the SL transport). This means that we cannot switch to kokkos 5 until we can abandon compilers that only support c++17.

[tcclevenger]

On Chrysalis:

• gnu: default compiler is compatible
• oneapi-ifx: version is technically compatible, but we get a build error about not being able to find "concepts" file. Output indicates oneAPI is not correctly configured.
• intel: Kokkos no longer supports INTEL's classic compilers (as of Kokkos 4.6, we currently use Kokkos 4.5). Maybe they are C++20 compliant, but we will need to change anyways to update Kokkos.

[bartgol]

• oneapi-ifx: version is technically compatible, but we get a build error about not being able to find "concepts" file. Output indicates oneAPI is not correctly configured.

Ugh, that's underwhelming. I wonder if this is on the chrys IT folks' radar...

[tcclevenger]

Ugh, that's underwhelming. I wonder if this is on the chrys IT folks' radar...

Yeah, I want to try and make a small reproducer outside of E3SM. Make sure it's not something we are doing.

[tcclevenger]

Update:

• PM-CPU/GPU and Aurora: Ran some tests from nightly, C++20 builds and runs BFB with master.
• Chrysalis_oneapi-ifx: Az update the oneapi-ifx compiler and I've tested against master baselines and C++20 builds and runs BFB (yay!)
• Chrysalis_intel: No longer supported in Kokkos 5, I think current one we are using isn't c++20 compatible.
• Chrysalis_gnu: technically compliant gnu version, but internal compiler error with C++20 (somewhere in ekat_pack)

@rljacob @bartgol How should we proceed? Can we require users of Chrysalis start switching to oneapi? Do we need to have a frozen version of Kokkos as a tpl for the other compilers? Could we require anyone using ekat (running with EAMxx) use oneapi?

[bartgol]

• PM-CPU/GPU and Aurora: Ran some tests from nightly, C++20 builds and runs BFB with master.

Did you run both gnu and intel on pm-cpu? I think @ndkeen is also using nvidia as compiler on pm-gpu, so maybe we should check that too.

@rljacob @bartgol How should we proceed? Can we require users of Chrysalis start switching to oneapi? Do we need to have a frozen version of Kokkos as a tpl for the other compilers? Could we require anyone using ekat (running with EAMxx) use oneapi?

I think we can keep both intel and oneapi. But we won't be able to run anything that uses kokkos with the intel compiler. I think this includes also any EAM case though, since EAM uses hommexx's SL transport by default, so this may impact a lot of ppl.

@jgfouca wild option: can CIME update a submodule conditionally on the compiler (or any other XML var)? E.g., can we do something like

if compiler is "intel":
   update_submodule(f'{SRCROOT}/externals/ekat', 'v1.1', recursive=True)  

to switch the submodule to an older tag?

[rljacob]

We will require Chrysalis users to use oneapi-ifx. It will be made the default. ifort will still be an option for anyone who needs it and they have to use an earlier hash.

We should make a plan on when this goes to master since its a breaking change for chrysalis.

[bartgol]

We will require Chrysalis users to use oneapi-ifx. It will be made the default. ifort will still be an option for anyone who needs it and they have to use an earlier hash.

We should make a plan on when this goes to master since its a breaking change for chrysalis.

We should prob make this change first, and then update ekat/kokkos (and the required cxx standard). Maybe mid-late March? I think 2 months may be a good enough window for ppl to switch to the new compiler.

[jgfouca]

@bartgol , CIME does not support that currently. We could, in theory, add support for configuration that allows for customizing submodules, but that seems kinda hacky.

[tcclevenger]

@rljacob @bartgol I'll do some runs with EAM cases to confirm that EAM will still need to compile Kokkos, then make a PR updating the default compiler.

[tcclevenger]

Testing update

• PM-CPU
  • intel: SMS_Lh4.ne4pg2_ne4pg2.F2010-SCREAMv1.pm-cpu_intel.eamxx-output-preset-1--eamxx-prod is BFB with master baseline
  • gnu: SMS_Lh4.ne4pg2_ne4pg2.F2010-SCREAMv1.eamxx-output-preset-1--eamxx-prod is BFB with master baseline
  • nvidia: SMS_D_Ln9.ne4_ne4.F2010-SCREAMv1-noAero.eamxx-output-preset-3 runs, but no baselines exist (I'm generating my own master baseline)
• PM-GPU:
  • gnugpu: SMS_D_Ln9.ne4_ne4.F2010-SCREAMv1-noAero.eamxx-output-preset-3 is BFB with master baseline
  • nvidiagpu: No baselines, CDASH only runs e3sm_gpuacc_next_nvidiagpu, which passes

I will next go to frontier, then I think we will have all the info we need to move forward.