[Flang] Move builtin .mod generation into runtimes (Reapply #137828)

[Meinersbur]

Reapplication of #137828, changes:

• Workaround CMAKE_Fortran_PREPROCESS_SOURCE issue for CMake < 2.24: The issue is that try_compile does not forward manually-defined compiler flang variables to the test build environment; instead of just a negative test result, it aborts the configuration step itself. To be fair, manually defining these variables is deprecated since at least CMake 3.6.
• Missing flang cmd line flags for CMake < 3.28 -target=, -O2, -O3
• It is now possible to set FLANG_RT_ENABLED_STATIC=OFF and FLANG_RT_ENABLE_SHARED=OFF at the same and is the default for amdgpu and nvptx targets. In this mode, only the .mod files are compiled -- necessary for module files in lib/clang/22/finclude/flang/(nvptx64-nvidia-cuda|amdgpu-amd-amdhsa)/*.mod to be available.
• For compiling omp_lib.mod for nvptx and amdgpu, the module build functionality must be hoisted out if openmp's runtime/ directory which is only included for host targets. This PR now requires [openmp][NFCI] Move .mod code out of runtimes subdir #169909.

Move building the .mod files from openmp/flang to openmp/flang-rt using a shared mechanism. Motivations to do so are:

1. Most modules are target-dependent and need to be re-compiled for each target separately, which is something the LLVM_ENABLE_RUNTIMES system already does. Prime example is iso_c_binding.mod which encodes the target's ABI. Constants such as c_long_double also have different values. Most other modules have #ifdef-enclosed code as well. For instance this caused offload targets nvptx64-nvidia-cuda/amdgpu-amd-amdhsa to use the modules files compiled for the host which may contrain uses of the types REAL(10) or REAL(16) not available for nvptx/amdgpu.

#146876
#128015
#129742
#158790

3. CMake has support for Fortran that we should use. Among other things, it automatically determines module dependencies so there is no need to hardcode them in the CMakeLists.txt.

4. It allows using Fortran itself to implement Flang-RT. Currently, only iso_fortran_env_impl.f90 emits object files that are needed by Fortran applications ([flang] Linker for non-constant accesses to kind arrays (integer_kind, logical_kind, real_kind) #89403). The workaround of [flang][runtime] Build ISO_FORTRAN_ENV to export kind arrays as linkable symbols #95388 could be reverted (PR [Flang-RT] Remerge iso_fortran_env and iso_fortran_env_impl #169525).

If using Flang for cross-compilation or target-offloading, flang-rt must now be compiled for each target not only for the library, but also to get the target-specific module files. For instance in a bootstrapping runtime build, this can be done by adding: -DLLVM_RUNTIME_TARGETS=default;nvptx64-nvidia-cuda;amdgpu-amd-amdhsa.

Some new dependencies come into play:

• openmp depends on flang-rt for building lib_omp.mod and lib_omp_kinds.mod. Currently, if flang-rt is not found then the modules are not built.
• check-flang depends on flang-rt: If not found, the majority of tests are disabled. If not building in a bootstrpping build, the location of the module files can be pointed to using -DFLANG_INTRINSIC_MODULES_DIR=<path>, e.g. in a flang-standalone build. Alternatively, the test needing any of the intrinsic modules could be marked with REQUIRES: flangrt-modules.
• check-flang depends on openmp: Not a change; tests requiring lib_omp.mod and lib_omp_kinds.mod those are already marked with openmp_runtime.

As intrinsic are now specific to the target, their location is moved from include/flang to <resource-dir>/finclude/flang/<triple>. The mechnism to compute the location have been moved from flang-rt (previously used to compute the location of libflang_rt.*.a) to common locations in cmake/GetToolchainDirs.cmake and runtimes/CMakeLists.txt so they can be used by both, openmp and flang-rt. Potentially the mechnism could also be shared by other libraries such as compiler-rt.

finclude was chosen because gfortran uses it as well and avoids misuse such as #include <flang/iso_c_binding.mod>. The search location is now determined by ToolChain in the driver, instead of by the frontend. Another subdirectory flang avoid accidental inclusion of gfortran-modules which due to compression would result in user-unfriendly errors. Now the driver adds -fintrinsic-module-path for that location to the frontend call (Just like gfortran does). -fintrinsic-module-path had to be fixed for this because ironically it was only added to searchDirectories, but not intrinsicModuleDirectories_. Since the driver determines the location, tests invoking flang -fc1 and bbc must also be passed the location by llvm-lit. This works like llvm-lit does for finding the include dirs for Clang using -print-file-name=....

[Meinersbur]

@jhuber6 Based on the feedback in #137828, I made it easier to build the Fortran modules for nvptx and amdgpu targets. It is now possible to set both FLANG_RT_ENABLE_STATIC and FLANG_RT_ENABLE_SHARED to off, meaning no library is built, only the modules files. This is the default if the target is nvptx or amdgpu. However, this overrides the (experimental) default behavior of #131826. I could not get the gpu_sources to compile since it cannot find include files such as #include <cassert>. I could trick it to point to the libc++ versions, but I don't want other users have to do many steps. Please have a look specifically to 9041b78 (#169638)

In the spirit off adding LLVM_ENABLE_RUNTIMES=flang-rt automatically, we could also automatically add LLVM_RUNTIME_TARGETS=default;amdgcn-amd-amdhsa when the AMDGPU backend is enabled; respectively LLVM_RUNTIME_TARGETS=default;nvptx64-nvidia-cuda if LLVM_TARGETS_TO_BUILD=NVPTX is present. What do you think?

[github-actions]

🐧 Linux x64 Test Results

• 193966 tests passed
• 6243 tests skipped

✅ The build succeeded and all tests passed.

[vzakhari]

Thank you, Michael! The patchs seems to be working on my side (after some adjustments downstream). So it is good to go for the next week merge (once you merge it, I will resolve conflicts on my side).

I do not know about the experimental direct GPU build. @jhuber6 can you please comment on Michael's question above?

[Meinersbur]

After working around the CMAKE_Fortran_PREPROCESS_SOURCE failure, openmp-offload-amdgpu-clang-flang is failing with a different problem: The builder is running Fortran+OpenMP offload tests on amdgpu-amd-amdhsa. For these to compile, omp_lib.mod and the builtin modules must be compiled for amdgpu-amd-amdhsa as well.

1. The builtin modules are compiled by just adding/not removing LLVM_ENABLE_RUNTIMES=flang-rt.

2. omp_lib.mod unfortunately involves more changes. openmp skips add_subdirectory(runtime) for amdgpu/nvptx which contains the module build instructions. Some restructure was needed to compile the Fortran modules for offload triples as well, mostly moving definitions out of the runtime/ directory into a new module/ directory for building omp_lib.mod for host and device.

[jhuber6]

One problem with enabling flang-rt on the GPU is that it has a dependency on libc / libc++ on the GPU as well. Maybe there's a way to make it work there just to build the module files?

[Meinersbur]

One problem with enabling flang-rt on the GPU is that it has a dependency on libc / libc++ on the GPU as well. Maybe there's a way to make it work there just to build the module files?

This is what FLANG_RT_ENABLE_STATIC=OFF FLANG_RT_ENABLE_SHARED=OFF does. It should only build the .f90 files which have no dependency to libc or libc++ headers, and only builds object libraries, so there is no linking step which would require libc. My question specifically is whether it is OK to make this the default on nvptx/amdgpu targets which this PR currently does.

[vzakhari]

Thank you for the changes and for postponing the merge, @Meinersbur! It is good to go!

[kkwli]

On AIX, there is only iso_fortran_env_impl.mod.

$ find . -name '*.mod'
./flang-rt/lib/runtime/iso_fortran_env_impl.mod

On Linux, I can see all the modules generated.

$ find . -name '*.mod'
./lib/clang/22/finclude/flang/powerpc64le-unknown-linux-gnu/__ppc_types.mod
./lib/clang/22/finclude/flang/powerpc64le-unknown-linux-gnu/__fortran_builtins.mod
./lib/clang/22/finclude/flang/powerpc64le-unknown-linux-gnu/__cuda_builtins.mod
./lib/clang/22/finclude/flang/powerpc64le-unknown-linux-gnu/__fortran_type_info.mod
./lib/clang/22/finclude/flang/powerpc64le-unknown-linux-gnu/iso_c_binding.mod
./lib/clang/22/finclude/flang/powerpc64le-unknown-linux-gnu/iso_fortran_env_impl.mod
./lib/clang/22/finclude/flang/powerpc64le-unknown-linux-gnu/mma.mod
./lib/clang/22/finclude/flang/powerpc64le-unknown-linux-gnu/ieee_features.mod
./lib/clang/22/finclude/flang/powerpc64le-unknown-linux-gnu/__cuda_device.mod
./lib/clang/22/finclude/flang/powerpc64le-unknown-linux-gnu/__fortran_ieee_exceptions.mod
./lib/clang/22/finclude/flang/powerpc64le-unknown-linux-gnu/__ppc_intrinsics.mod
./lib/clang/22/finclude/flang/powerpc64le-unknown-linux-gnu/iso_fortran_env.mod
./lib/clang/22/finclude/flang/powerpc64le-unknown-linux-gnu/cudadevice.mod
./lib/clang/22/finclude/flang/powerpc64le-unknown-linux-gnu/ieee_exceptions.mod
./lib/clang/22/finclude/flang/powerpc64le-unknown-linux-gnu/ieee_arithmetic.mod
./lib/clang/22/finclude/flang/powerpc64le-unknown-linux-gnu/cooperative_groups.mod

[Meinersbur]

@kkwli Without access to an AIX system, how can I reproduce the issue?

I do see all the .f90 files built here:
https://lab.llvm.org/buildbot/#/builders/201/builds/7242
check-flang also succeeds successfully, so I think all module files are present.

$ find . -name '*.mod'
./flang-rt/lib/runtime/iso_fortran_env_impl.mod

This is the wrong path for the modules files. If using a bootstrapping build, it should be in lib/clang/22/finclude/flang/<triple>/*.mod of the parent build dir where flang is built.

[kkwli]

@kkwli Without access to an AIX system, how can I reproduce the issue?

I do see all the .f90 files build here: https://lab.llvm.org/buildbot/#/builders/201/builds/7242 check-flang also succeeds successfully, so I think all module files are present.

$ find . -name '*.mod'
./flang-rt/lib/runtime/iso_fortran_env_impl.mod

This is the wrong path for the modules files. If using a bootstrapping build, it should be in lib/clang/22/finclude/flang/<triple>/*.mod of the parent build dir where flang is built.

@Meinersbur My bad! I messed up the build environment. Things look good now. All .mod files are generated on AIX.

$ find . -name '*.mod'
./lib/clang/22/finclude/flang/powerpc64-ibm-aix/__cuda_builtins.mod
./lib/clang/22/finclude/flang/powerpc64-ibm-aix/__cuda_device.mod
./lib/clang/22/finclude/flang/powerpc64-ibm-aix/__fortran_builtins.mod
./lib/clang/22/finclude/flang/powerpc64-ibm-aix/__fortran_ieee_exceptions.mod
./lib/clang/22/finclude/flang/powerpc64-ibm-aix/__fortran_type_info.mod
./lib/clang/22/finclude/flang/powerpc64-ibm-aix/__ppc_intrinsics.mod
./lib/clang/22/finclude/flang/powerpc64-ibm-aix/__ppc_types.mod
./lib/clang/22/finclude/flang/powerpc64-ibm-aix/cooperative_groups.mod
./lib/clang/22/finclude/flang/powerpc64-ibm-aix/cudadevice.mod
./lib/clang/22/finclude/flang/powerpc64-ibm-aix/ieee_arithmetic.mod
./lib/clang/22/finclude/flang/powerpc64-ibm-aix/ieee_exceptions.mod
./lib/clang/22/finclude/flang/powerpc64-ibm-aix/ieee_features.mod
./lib/clang/22/finclude/flang/powerpc64-ibm-aix/iso_c_binding.mod
./lib/clang/22/finclude/flang/powerpc64-ibm-aix/iso_fortran_env.mod
./lib/clang/22/finclude/flang/powerpc64-ibm-aix/iso_fortran_env_impl.mod
./lib/clang/22/finclude/flang/powerpc64-ibm-aix/mma.mod

Sorry for the false alarm.

[DanielCChen]

LGTM.
Thanks!

[kkwli]

LG. Thanks.

[petrhosek]

This broke the runtimes build on Windows when targeting a non-Windows target with the following error:

CMake Error at C:/b/s/w/ir/x/w/llvm-llvm-project/cmake/Modules/GetToolchainDirs.cmake:76 (string):
  string sub-command REPLACE requires at least four arguments.
Call Stack (most recent call first):
  C:/b/s/w/ir/x/w/llvm-llvm-project/cmake/Modules/GetToolchainDirs.cmake:42 (get_toolchain_arch_dirname)
  CMakeLists.txt:459 (get_toolchain_library_subdir)

Would it be possible to revert the change while this is being investigated?

Since this makes significant changes to the runtimes build that affects non-Flang users, I'd also appreciate to be included as a reviewer on the reland.

[Meinersbur]

This indicates that LLVM_TARGET_TRIPLE is not defined, which seems quite unusual. How does the runtime no know what target it is building for?

You were included as reviewer in the original PR #137828 without participarting in it. LLVM's workflow explicitly mentions post-commit reviews.

Were have been updating our CI builders and are already building patches on top of it, so a revert would come with some friction to us.

[vzakhari]

Same on my side. A revert will be painful.

[Meinersbur]

Same on my side. A revert will be painful.

Too late unfortunately. I talked with our maintainers and we will keep this PR applied downstream for now.