Error: Unable to Index Using mx-index - Issue Invoking Original Compiler

[bsdb0y]

I am encountering an issue when trying to use mx-index to index the c-ares project after generating a compile_commands.json file using either bear or cmake. The command I am running is:

/root/multi/bin/mx-index --db lib.db --target c-ares/build/compile_commands.json --workspace ./mx

However, I consistently receive errors indicating that the original compiler (/usr/sbin/clang) cannot be invoked to retrieve version information, with an "Invalid argument" error. Below is a sample of the error output:

Error:

E20250331 10:37:02.274544 35908 Importer.cpp:692] Error invoking original compiler to find version information: Invalid argument; original command was: /usr/sbin/clang -DCARES_NO_DEPRECATED -DHAVE_CONFIG_H=1 -D_GNU_SOURCE -D_POSIX_C_SOURCE=200809L -D_XOPEN_SOURCE=700 -I/root/prjs/c-ares/build -I/root/prjs/c-ares -I/root/prjs/c-ares/src/lib -I/root/prjs/c-ares/src/lib/include -I/root/prjs/c-ares/include -I/root/prjs/c-ares/src/tools -Wall -Wextra -Waggregate-return -Wcast-align -Wcast-qual -Wconversion -Wdeclaration-after-statement -Wdouble-promotion -Wfloat-equal -Wformat-security -Winit-self -Wmissing-braces -Wmissing-declarations -Wmissing-format-attribute -Wmissing-include-dirs -Wmissing-prototypes -Wnested-externs -Wold-style-definition -Wpacked -Wpedantic -Wpointer-arith -Wredundant-decls -Wshadow -Wsign-conversion -Wstrict-overflow -Wstrict-prototypes -Wundef -Wunreachable-code -Wunused -Wvariadic-macros -Wvla -Wwrite-strings -Werror=implicit-int -Werror=implicit-function-declaration -Werror=partial-availability -Qunused-arguments -Wno-long-long -std=gnu90 -c -o CMakeFiles/adig.dir/adig.c.o /root/prjs/c-ares/src/tools/adig.c. Perhaps try --env /path/to/file to specify a file containing environment variables, to help me find the compiler executable
E20250331 10:37:02.275046 35906 Importer.cpp:692] Error invoking original compiler to find version information: Invalid argument; original command was: /usr/sbin/clang -DCARES_NO_DEPRECATED -DHAVE_CONFIG_H=1 -D_GNU_SOURCE -D_POSIX_C_SOURCE=200809L -D_XOPEN_SOURCE=700 -I/root/prjs/c-ares/build -I/root/prjs/c-ares -I/root/prjs/c-ares/src/lib -I/root/prjs/c-ares/src/lib/include -I/root/prjs/c-ares/include -I/root/prjs/c-ares/src/tools -Wall -Wextra -Waggregate-return -Wcast-align -Wcast-qual -Wconversion -Wdeclaration-after-statement -Wdouble-promotion -Wfloat-equal -Wformat-security -Winit-self -Wmissing-braces -Wmissing-declarations -Wmissing-format-attribute -Wmissing-include-dirs -Wmissing-prototypes -Wnested-externs -Wold-style-definition -Wpacked -Wpedantic -Wpointer-arith -Wredundant-decls -Wshadow -Wsign-conversion -Wstrict-overflow -Wstrict-prototypes -Wundef -Wunreachable-code -Wunused -Wvariadic-macros -Wvla -Wwrite-strings -Werror=implicit-int -Werror=implicit-function-declaration -Werror=partial-availability -Qunused-arguments -Wno-long-long -std=gnu90 -c -o CMakeFiles/ahost.dir/ares_getopt.c.o /root/prjs/c-ares/src/tools/ares_getopt.c. Perhaps try --env /path/to/file to specify a file containing environment variables, to help me find the compiler executable
E20250331 10:37:02.279376 35890 Importer.cpp:692] Error invoking original compiler to find version information: Invalid argument; original command was: /usr/sbin/clang -DCARES_NO_DEPRECATED -DHAVE_CONFIG_H=1 -D_GNU_SOURCE -D_POSIX_C_SOURCE=200809L -D_XOPEN_SOURCE=700 -I/root/prjs/c-ares/build -I/root/prjs/c-ares -I/root/prjs/c-ares/src/lib -I/root/prjs/c-ares/src/lib/include -I/root/prjs/c-ares/include -I/root/prjs/c-ares/src/tools -Wall -Wextra -Waggregate-return -Wcast-align -Wcast-qual -Wconversion -Wdeclaration-after-statement -Wdouble-promotion -Wfloat-equal -Wformat-security -Winit-self -Wmissing-braces -Wmissing-declarations -Wmissing-format-attribute -Wmissing-include-dirs -Wmissing-prototypes -Wnested-externs -Wold-style-definition -Wpacked -Wpedantic -Wpointer-arith -Wredundant-decls -Wshadow -Wsign-conversion -Wstrict-overflow -Wstrict-prototypes -Wundef -Wunreachable-code -Wunused -Wvariadic-macros -Wvla -Wwrite-strings -Werror=implicit-int -Werror=implicit-function-declaration -Werror=partial-availability -Qunused-arguments -Wno-long-long -std=gnu90 -c -o CMakeFiles/ahost.dir/ahost.c.o /root/prjs/c-ares/src/tools/ahost.c. Perhaps try --env /path/to/file to specify a file containing environment variables, to help me find the compiler executable
E20250331 10:37:02.378388 35928 Importer.cpp:692] Error invoking original compiler to find version information: Invalid argument; original command was: /usr/sbin/clang -DCARES_BUILDING_LIBRARY -DHAVE_CONFIG_H=1 -D_GNU_SOURCE -D_POSIX_C_SOURCE=200809L -D_XOPEN_SOURCE=700 -I/root/prjs/c-ares/build -I/root/prjs/c-ares -I/root/prjs/c-ares/include -I/root/prjs/c-ares/src/lib -I/root/prjs/c-ares/src/lib/include -Wall -Wextra -Waggregate-return -Wcast-align -Wcast-qual -Wconversion -Wdeclaration-after-statement -Wdouble-promotion -Wfloat-equal -Wformat-security -Winit-self -Wmissing-braces -Wmissing-declarations -Wmissing-format-attribute -Wmissing-include-dirs -Wmissing-prototypes -Wnested-externs -Wold-style-definition -Wpacked -Wpedantic -Wpointer-arith -Wredundant-decls -Wshadow -Wsign-conversion -Wstrict-overflow -Wstrict-prototypes -Wundef -Wunreachable-code -Wunused -Wvariadic-macros -Wvla -Wwrite-strings -Werror=implicit-int -Werror=implicit-function-declaration -Werror=partial-availability -Qunused-arguments -Wno-long-long -std=gnu90 -c -o CMakeFiles/c-ares_static.dir/util/ares_rand.c.o /root/prjs/c-ares/src/lib/util/ares_rand.c. Perhaps try --env /path/to/file to specify a file containing environment variables, to help me find the compiler executable
E20250331 10:37:02.378611 35912 Importer.cpp:692] Error invoking original compiler to find version information: Invalid argument; original command was: /usr/sbin/clang -DCARES_BUILDING_LIBRARY -DHAVE_CONFIG_H=1 -D_GNU_SOURCE -D_POSIX_C_SOURCE=200809L -D_XOPEN_SOURCE=700 -I/root/prjs/c-ares/build -I/root/prjs/c-ares -I/root/prjs/c-ares/include -I/root/prjs/c-ares/src/lib -I/root/prjs/c-ares/src/lib/include -Wall -Wextra -Waggregate-return -Wcast-align -Wcast-qual -Wconversion -Wdeclaration-after-statement -Wdouble-promotion -Wfloat-equal -Wformat-security -Winit-self -Wmissing-braces -Wmissing-declarations -Wmissing-format-attribute -Wmissing-include-dirs -Wmissing-prototypes -Wnested-externs -Wold-style-definition -Wpacked -Wpedantic -Wpointer-arith -Wredundant-decls -Wshadow -Wsign-conversion -Wstrict-overflow -Wstrict-prototypes -Wundef -Wunreachable-code -Wunused -Wvariadic-macros -Wvla -Wwrite-strings -Werror=implicit-int -Werror=implicit-function-declaration -Werror=partial-availability -Qunused-arguments -Wno-long-long -std=gnu90 -c -o CMakeFiles/c-ares_static.dir/util/ares_threads.c.o /root/prjs/c-ares/src/lib/util/ares_threads.c. Perhaps try --env /path/to/file to specify a file containing environment variables, to help me find the compiler executable
...
...
...

Environment:

• Tested on Docker images: Ubuntu 22/24/25 and Fedora 41/42
• Compiler versions: clang-18/19/20 and gcc
• Multiplier versions tested:
  • release multiplier-1705339 and compiled from source (master branch), with the installation process for Ubuntu 25.05 detailed in my Dockerfile: https://github.com/bsdb0y/OGHarn/blob/main/Dockerfile#L133

I have already tried using the --env option as suggested in the error message, but it did not resolve the issue. I provided a file with environment variables (e.g., PATH, CC, CXX), but the error persists.

Any guidance on resolving this or suggestions for additional debugging steps would be greatly appreciated!

[pgoodman]

What happens when you run one/all of those commands manually? Do they work?

[bsdb0y]

What happens when you run one/all of those commands manually? Do they work?

Yes, and to verify, I created a minimal test code with the following output:

[root@22c22dd8a395 ~]# /root/bins/bin/mx-index --target compile_commands.json --db sample.db --workspace ./mx
/root/bins/bin/mx-index: /lib64/libtinfo.so.6: no version information available (required by /root/bins/bin/mx-index)
/root/bins/bin/mx-index: /lib64/libtinfo.so.6: no version information available (required by /root/bins/bin/../lib/libLTO.so.18.1)
E20250401 18:25:28.090678   382 Importer.cpp:692] Error invoking original compiler to find version information: Invalid argument; original command was: /usr/sbin/gcc -c -o sample sample.c. Perhaps try --env /path/to/file to specify a file containing environment variables, to help me find the compiler executable
[root@22c22dd8a395 ~]# /usr/sbin/gcc -c -o sample sample.c
[root@22c22dd8a395 ~]# ls
bins  compile_commands.json  multiplier  multiplier-1705339.tar.xz  mx  sample  sample.c  sample.db
[root@22c22dd8a395 ~]# cat sample.c
void main() {}
[root@22c22dd8a395 ~]#

Just to add, the issue still persists even when there is no warning message about libtinfo

[pgoodman]

I don't have time to try to reproduce things. However, what would help me diagnose this better is the following:

Before this line, do something like:

LOG(ERROR) << "Bad command was: " << pasta::ArgumentVector(new_args).Join(" ");

So there's a function that goes and [manipulates the command a bunch, stripping out known non-portable flags, and doing some other things, and periodically its logic turns out to be bad/incomplete. I'd like to see the command is produces as a result, then have you manually run that, and then we can compare the originals (listed in our issue above) to the manipulated ones and try to spot the salient differences.

[bsdb0y]

No worries, and thanks for sharing the line to get more debug information. I recompiled the code and found that it is adding -include /trails/of/bits, which does not exist.

E20250402 20:04:28.166052   538 Importer.cpp:473] Bad command was::inside:if: /usr/bin/clang -c sample.c -w -Wno-everything -P -v -x c -include /trail/of/bits -o /dev/null
E20250402 20:04:28.169806   523 Importer.cpp:473] Bad command was::inside:if: /usr/lib/llvm-20/bin/clang -cc1 -triple x86_64-pc-linux-gnu -emit-obj -dumpdir sample- -disable-free -clear-ast-before-backend -disable-llvm-verifier -discard-value-names 2 -mframe-pointer=all -fmath-errno -ffp-contract=on -fno-rounding-math -mconstructor-aliases -funwind-tables=2 -target-cpu x86-64 -tune-cpu generic -debugger-tuning=gdb -fdebug-compilation-dir=/root/src/multiplier/build -resource-dir /usr/lib/llvm-20/lib/clang/20 -internal-isystem /usr/lib/llvm-20/lib/clang/20/include -internal-isystem /usr/local/include -internal-isystem /usr/lib/gcc/x86_64-linux-gnu/14/../../../../x86_64-linux-gnu/include -internal-externc-isystem /usr/include/x86_64-linux-gnu -internal-externc-isystem /include -internal-externc-isystem /usr/include -ferror-limit 19 -fgnuc-version=4.2.1 -fcolor-diagnostics -faddrsig -D__GCC_HAVE_DWARF2_CFI_ASM=1 sample.c -w -Wno-everything -P -v -x c -include /trail/of/bits -o /dev/null

When executing the above command:

root@07022ccd5d9c:~/src/multiplier/build# /usr/bin/clang -c sample.c -w -Wno-everything -P -v -x c -include /trail/of/bits -o /dev/null
Ubuntu clang version 20.1.1 (1~exp1ubuntu1)
Target: x86_64-pc-linux-gnu
Thread model: posix
InstalledDir: /usr/lib/llvm-20/bin
Found candidate GCC installation: /usr/lib/gcc/x86_64-linux-gnu/14
Selected GCC installation: /usr/lib/gcc/x86_64-linux-gnu/14
Candidate multilib: .;@m64
Selected multilib: .;@m64
 (in-process)
 "/usr/lib/llvm-20/bin/clang" -cc1 -triple x86_64-pc-linux-gnu -emit-obj -disable-free -clear-ast-before-backend -disable-llvm-verifier -discard-value-names -main-file-name sample.c -mrelocation-model pic -pic-level 2 -pic-is-pie -mframe-pointer=all -fmath-errno -ffp-contract=on -fno-rounding-math -mconstructor-aliases -funwind-tables=2 -target-cpu x86-64 -tune-cpu generic -debugger-tuning=gdb -fdebug-compilation-dir=/root/src/multiplier/build -v -P -fcoverage-compilation-dir=/root/src/multiplier/build -resource-dir /usr/lib/llvm-20/lib/clang/20 -include /trail/of/bits -internal-isystem /usr/lib/llvm-20/lib/clang/20/include -internal-isystem /usr/local/include -internal-isystem /usr/lib/gcc/x86_64-linux-gnu/14/../../../../x86_64-linux-gnu/include -internal-externc-isystem /usr/include/x86_64-linux-gnu -internal-externc-isystem /include -internal-externc-isystem /usr/include -Wno-everything -w -ferror-limit 19 -fgnuc-version=4.2.1 -fskip-odr-check-in-gmf -fcolor-diagnostics -faddrsig -D__GCC_HAVE_DWARF2_CFI_ASM=1 -o /dev/null -x c sample.c
clang -cc1 version 20.1.1 based upon LLVM 20.1.1 default target x86_64-pc-linux-gnu
ignoring nonexistent directory "/usr/lib/gcc/x86_64-linux-gnu/14/../../../../x86_64-linux-gnu/include"
ignoring nonexistent directory "/include"                                                                                                                                                                                                                      #include "..." search starts here:
#include <...> search starts here:                                                                                                                                                                                                                              /usr/lib/llvm-20/lib/clang/20/include                                                                                                                                                                                                                          /usr/local/include                                                                                                                                                                                                                                             /usr/include/x86_64-linux-gnu                                                                                                                                                                                                                                  /usr/include                                                                                                                                                                                                                                                  End of search list.
<built-in>:1:10: fatal error: '/trail/of/bits' file not found
    1 | #include "/trail/of/bits"
      |          ^~~~~~~~~~~~~~~~
1 error generated.

It seems like this is being added here: Importer.cpp#L449

I have quickly tried after commenting the above line which introduces this include but observed still failing with following output:

With gcc:

root@07022ccd5d9c:~/src/multiplier/build# /root/install/bin/mx-index --target compile_commands.json  --workspace ./mx --db sample.db
E20250402 22:46:42.233878 78431 Importer.cpp:471] Bad command was::inside:if: /usr/bin/gcc -c sample.c -w -Wno-everything -P -v -x c -o /dev/null
E20250402 22:46:42.234228 78431 Importer.cpp:693] Error invoking original compiler to find version information: Invalid argument; original command was: /usr/bin/gcc -c -o sample sample.c. Perhaps try --env /path/to/file to specify a file containing environment variables, to help me find the compiler executable

Executing Bad Command (for gcc):

root@07022ccd5d9c:~/src/multiplier/build# /usr/bin/gcc -c sample.c -w -Wno-everything -P -v -x c -o /dev/null
gcc: warning: '-x c' after last input file has no effect
Using built-in specs.
COLLECT_GCC=/usr/bin/gcc
OFFLOAD_TARGET_NAMES=nvptx-none:amdgcn-amdhsa
OFFLOAD_TARGET_DEFAULT=1
Target: x86_64-linux-gnu
Configured with: ../src/configure -v --with-pkgversion='Ubuntu 14.2.0-19ubuntu2' --with-bugurl=file:///usr/share/doc/gcc-14/README.Bugs --enable-languages=c,ada,c++,go,d,fortran,objc,obj-c++,m2,rust --prefix=/usr --with-gcc-major-version-only --program-suffix=-14 --program-prefix=x86_64-linux-gnu- --enable-shared --enable-linker-build-id --libexecdir=/usr/libexec --without-included-gettext --enable-threads=posix --libdir=/usr/lib --enable-nls --enable-bootstrap --enable-clocale=gnu --enable-libstdcxx-debug --enable-libstdcxx-time=yes --with-default-libstdcxx-abi=new --enable-libstdcxx-backtrace --enable-gnu-unique-object --disable-vtable-verify --enable-plugin --enable-default-pie --with-system-zlib --enable-libphobos-checking=release --with-target-system-zlib=auto --enable-objc-gc=auto --enable-multiarch --disable-werror --enable-cet --with-arch-32=i686 --with-abi=m64 --with-multilib-list=m32,m64,mx32 --enable-multilib --with-tune=generic --enable-offload-targets=nvptx-none=/build/gcc-14-C86vgL/gcc-14-14.2.0/debian/tmp-nvptx/usr,amdgcn-amdhsa=/build/gcc-14-C86vgL/gcc-14-14.2.0/debian/tmp-gcn/usr --enable-offload-defaulted --without-cuda-driver --enable-checking=release --build=x86_64-linux-gnu --host=x86_64-linux-gnu --target=x86_64-linux-gnu --with-build-config=bootstrap-lto-lean --enable-link-serialization=2
Thread model: posix
Supported LTO compression algorithms: zlib zstd
gcc version 14.2.0 (Ubuntu 14.2.0-19ubuntu2)
COLLECT_GCC_OPTIONS='-c' '-w' '-Wno-everything' '-P' '-v' '-o' '/dev/null' '-mtune=generic' '-march=x86-64'
 /usr/libexec/gcc/x86_64-linux-gnu/14/cc1 -quiet -v -P -imultiarch x86_64-linux-gnu sample.c -quiet -dumpbase sample.c -dumpbase-ext .c -mtune=generic -march=x86-64 -Wno-everything -w -version -fasynchronous-unwind-tables -fstack-protector-strong -Wformat -Wformat-security -fstack-clash-protection -fcf-protection -o /tmp/ccryodmj.s
GNU C17 (Ubuntu 14.2.0-19ubuntu2) version 14.2.0 (x86_64-linux-gnu)
        compiled by GNU C version 14.2.0, GMP version 6.3.0, MPFR version 4.2.2, MPC version 1.3.1, isl version isl-0.27-GMP

GGC heuristics: --param ggc-min-expand=100 --param ggc-min-heapsize=131072
ignoring nonexistent directory "/usr/local/include/x86_64-linux-gnu"
ignoring nonexistent directory "/usr/lib/gcc/x86_64-linux-gnu/14/include-fixed/x86_64-linux-gnu"
ignoring nonexistent directory "/usr/lib/gcc/x86_64-linux-gnu/14/include-fixed"
ignoring nonexistent directory "/usr/lib/gcc/x86_64-linux-gnu/14/../../../../x86_64-linux-gnu/include"
#include "..." search starts here:
#include <...> search starts here:
 /usr/lib/gcc/x86_64-linux-gnu/14/include
 /usr/local/include
 /usr/include/x86_64-linux-gnu
 /usr/include
End of search list.
Compiler executable checksum: 8d75a8b870b18ead058f0fb05febd3df
COLLECT_GCC_OPTIONS='-c' '-w' '-Wno-everything' '-P' '-v' '-o' '/dev/null' '-mtune=generic' '-march=x86-64'
 as -v -W --64 -o /dev/null /tmp/ccryodmj.s
GNU assembler version 2.44 (x86_64-linux-gnu) using BFD version (GNU Binutils for Ubuntu) 2.44
COMPILER_PATH=/usr/libexec/gcc/x86_64-linux-gnu/14/:/usr/libexec/gcc/x86_64-linux-gnu/14/:/usr/libexec/gcc/x86_64-linux-gnu/:/usr/lib/gcc/x86_64-linux-gnu/14/:/usr/lib/gcc/x86_64-linux-gnu/
LIBRARY_PATH=/usr/lib/gcc/x86_64-linux-gnu/14/:/usr/lib/gcc/x86_64-linux-gnu/14/../../../x86_64-linux-gnu/:/usr/lib/gcc/x86_64-linux-gnu/14/../../../../lib/:/lib/x86_64-linux-gnu/:/lib/../lib/:/usr/lib/x86_64-linux-gnu/:/usr/lib/../lib/:/usr/lib/gcc/x86_64-linux-gnu/14/../../../:/lib/:/usr/lib/
COLLECT_GCC_OPTIONS='-c' '-w' '-Wno-everything' '-P' '-v' '-o' '/dev/null' '-mtune=generic' '-march=x86-64'

With clang

root@07022ccd5d9c:~/src/multiplier/build# /root/install/bin/mx-index --target compile_commands.json  --workspace ./mx --db sample.db
E20250402 22:27:10.307521 16228 Importer.cpp:471] Bad command was::inside:if: /usr/bin/clang -c sample.c -w -Wno-everything -P -v -x c -o /dev/null
E20250402 22:27:10.307982 16228 Importer.cpp:693] Error invoking original compiler to find version information: Invalid argument; original command was: /usr/bin/clang -c -o sample sample.c. Perhaps try --env /path/to/file to specify a file containing environment variables, to help me find the compiler executable
E20250402 22:27:10.314540 16229 Importer.cpp:471] Bad command was::inside:if: /usr/lib/llvm-20/bin/clang -cc1 -triple x86_64-pc-linux-gnu -emit-obj -dumpdir sample- -disable-free -clear-ast-before-backend -disable-llvm-verifier -discard-value-names 2 -mframe-pointer=all -fmath-errno -ffp-contract=on -fno-rounding-math -mconstructor-aliases -funwind-tables=2 -target-cpu x86-64 -tune-cpu generic -debugger-tuning=gdb -fdebug-compilation-dir=/root/src/multiplier/build -resource-dir /usr/lib/llvm-20/lib/clang/20 -internal-isystem /usr/lib/llvm-20/lib/clang/20/include -internal-isystem /usr/local/include -internal-isystem /usr/lib/gcc/x86_64-linux-gnu/14/../../../../x86_64-linux-gnu/include -internal-externc-isystem /usr/include/x86_64-linux-gnu -internal-externc-isystem /include -internal-externc-isystem /usr/include -ferror-limit 19 -fgnuc-version=4.2.1 -fcolor-diagnostics -faddrsig -D__GCC_HAVE_DWARF2_CFI_ASM=1 sample.c -w -Wno-everything -P -v -x c -o /dev/null
E20250402 22:27:10.314613 16229 Importer.cpp:693] Error invoking original compiler to find version information: Invalid argument; original command was: /usr/lib/llvm-20/bin/clang -cc1 -triple x86_64-pc-linux-gnu -emit-obj -dumpdir sample- -disable-free -clear-ast-before-backend -disable-llvm-verifier -discard-value-names -main-file-name -mrelocation-model pic -pic-level 2 -pic-is-pie -mframe-pointer=all -fmath-errno -ffp-contract=on -fno-rounding-math -mconstructor-aliases -funwind-tables=2 -target-cpu x86-64 -tune-cpu generic -debugger-tuning=gdb -fdebug-compilation-dir=/root/src/multiplier/build -fcoverage-compilation-dir=/root/src/multiplier/build -resource-dir /usr/lib/llvm-20/lib/clang/20 -internal-isystem /usr/lib/llvm-20/lib/clang/20/include -internal-isystem /usr/local/include -internal-isystem /usr/lib/gcc/x86_64-linux-gnu/14/../../../../x86_64-linux-gnu/include -internal-externc-isystem /usr/include/x86_64-linux-gnu -internal-externc-isystem /include -internal-externc-isystem /usr/include -ferror-limit 19 -fgnuc-version=4.2.1 -fskip-odr-check-in-gmf -fcolor-diagnostics -faddrsig -D__GCC_HAVE_DWARF2_CFI_ASM=1 -x c -o /tmp/sample-c7ec30.o sample.c. Perhaps try --env /path/to/file to specify a file containing environment variables, to help me find the compiler executable

Executing Bad Command1:

root@07022ccd5d9c:~/src/multiplier/build# /usr/bin/clang -c sample.c -w -Wno-everything -P -v -x c -o /dev/null
Ubuntu clang version 20.1.1 (1~exp1ubuntu1)
Target: x86_64-pc-linux-gnu
Thread model: posix
InstalledDir: /usr/lib/llvm-20/bin
Found candidate GCC installation: /usr/lib/gcc/x86_64-linux-gnu/14
Selected GCC installation: /usr/lib/gcc/x86_64-linux-gnu/14
Candidate multilib: .;@m64
Selected multilib: .;@m64
 (in-process)
 "/usr/lib/llvm-20/bin/clang" -cc1 -triple x86_64-pc-linux-gnu -emit-obj -disable-free -clear-ast-before-backend -disable-llvm-verifier -discard-value-names -main-file-name sample.c -mrelocation-model pic -pic-level 2 -pic-is-pie -mframe-pointer=all -fmath-errno -ffp-contract=on -fno-rounding-math -mconstructor-aliases -funwind-tables=2 -target-cpu x86-64 -tune-cpu generic -debugger-tuning=gdb -fdebug-compilation-dir=/root/src/multiplier/build -v -P -fcoverage-compilation-dir=/root/src/multiplier/build -resource-dir /usr/lib/llvm-20/lib/clang/20 -internal-isystem /usr/lib/llvm-20/lib/clang/20/include -internal-isystem /usr/local/include -internal-isystem /usr/lib/gcc/x86_64-linux-gnu/14/../../../../x86_64-linux-gnu/include -internal-externc-isystem /usr/include/x86_64-linux-gnu -internal-externc-isystem /include -internal-externc-isystem /usr/include -Wno-everything -w -ferror-limit 19 -fgnuc-version=4.2.1 -fskip-odr-check-in-gmf -fcolor-diagnostics -faddrsig -D__GCC_HAVE_DWARF2_CFI_ASM=1 -o /dev/null -x c sample.c
clang -cc1 version 20.1.1 based upon LLVM 20.1.1 default target x86_64-pc-linux-gnu
ignoring nonexistent directory "/usr/lib/gcc/x86_64-linux-gnu/14/../../../../x86_64-linux-gnu/include"
ignoring nonexistent directory "/include"
#include "..." search starts here:
#include <...> search starts here:
 /usr/lib/llvm-20/lib/clang/20/include
 /usr/local/include
 /usr/include/x86_64-linux-gnu
 /usr/include
End of search list.

Executing Bad Command2:

root@07022ccd5d9c:~/src/multiplier/build# /usr/lib/llvm-20/bin/clang -cc1 -triple x86_64-pc-linux-gnu -emit-obj -dumpdir sample- -disable-free -clear-ast-before-backend -disable-llvm-verifier -discard-value-names 2 -mframe-pointer=all -fmath-errno -ffp-contract=on -fno-rounding-math -mconstructor-aliases -funwind-tables=2 -target-cpu x86-64 -tune-cpu generic -debugger-tuning=gdb -fdebug-compilation-dir=/root/src/multiplier/build -resource-dir /usr/lib/llvm-20/lib/clang/20 -internal-isystem /usr/lib/llvm-20/lib/clang/20/include -internal-isystem /usr/local/include -internal-isystem /usr/lib/gcc/x86_64-linux-gnu/14/../../../../x86_64-linux-gnu/include -internal-externc-isystem /usr/include/x86_64-linux-gnu -internal-externc-isystem /include -internal-externc-isystem /usr/include -ferror-limit 19 -fgnuc-version=4.2.1 -fcolor-diagnostics -faddrsig -D__GCC_HAVE_DWARF2_CFI_ASM=1 sample.c -w -Wno-everything -P -v -x c -o /dev/null
clang -cc1 version 20.1.1 based upon LLVM 20.1.1 default target x86_64-pc-linux-gnu
ignoring nonexistent directory "/usr/lib/gcc/x86_64-linux-gnu/14/../../../../x86_64-linux-gnu/include"
ignoring nonexistent directory "/include"
#include "..." search starts here:
#include <...> search starts here:
 /usr/lib/llvm-20/lib/clang/20/include
 /usr/local/include
 /usr/include/x86_64-linux-gnu
 /usr/include
End of search list.
error: error reading '2': No such file or directory
ignoring nonexistent directory "/usr/lib/gcc/x86_64-linux-gnu/14/../../../../x86_64-linux-gnu/include"
ignoring nonexistent directory "/include"
#include "..." search starts here:
#include <...> search starts here:
 /usr/lib/llvm-20/lib/clang/20/include
 /usr/local/include                                                                                                                                                                                                                                             /usr/include/x86_64-linux-gnu
 /usr/include                                                                                                                                                                                                                                                  End of search list.                                                                                                                                                                                                                                            1 error generated.

Could you please confirm if this behavior is expected, or if there’s an alternative approach to fix this?

[pgoodman]

Hrm, I'm wondering if that if statement checking the return status should be updated. Specifically, we intentionally do want the error from the -include, just in case things proceed further. The "best" solution is really to change our approach to use -###. But that is more effort.

Can you try leaving the /trail/of/bits in, then adding in std::string stdout; then pass it as &stdout as the fifth argument here, then in the error-detecting if statement, go and dump out stdout (e.g. with a std::cout and an exit). Then if it that looks "sane" to you, add some logic into the if to search for "/trail/of/bits" in the stdout value via stdout.find, and if it is not std::string::npos, then stdout.swap(output.sysroot); and keep going, otherwise bubble the error up.

[bsdb0y]

I've done some analysis and found that the current implementation isn't actually bad, it works as expected. The issue I was facing turned out to be related to the reproc library.

While debugging Importer.cpp, I analysed functions responsible for setting up the path and environment FixEnvVariablesAndPath, including BuildCommandAction::GetCompilerInfo. I noticed that things were failing before reaching the expected output stage. Specifically, it was breaking during Subprocess::Execute.

While digging into Subprocess.cpp, I suspected that the issue might be with the reproc code. To verify this, I replaced the reproc logic with a simple fork()-based model. After making this change, things started working, I was able to capture both stdout and stderr in a single variable using redirection.

To quickly confirm it was indeed a reproc issue, I tried running a simple ls -al command using reproc. That also failed with the same error I had seen during the mx-index run. Interestingly, this same test worked fine on my host machine but failed inside Docker.

#include <reproc++/run.hpp>
#include <iostream>
#include <vector>
#include <string>

int main() {
  std::vector<std::string> args = { "ls", "-la" };

  reproc::options options;
  options.redirect.parent = true;

  // Run the command and get result + error code
  auto [status, ec] = reproc::run(args, options);

  if (ec) {
    std::cerr << "Error running command: " << ec.message() << '\n';
    return 1;
  }

  std::cout << "Process exited with status: " << status << '\n';
  return 0;
}

As a temporary fix, I added a limit to the number of open files inside Docker using this command:
docker run --ulimit nofile=1024:1024 -d multi tail -f /dev/null

After applying this, I tested again, and it worked.

So that's the analysis. Sorry for the long comment!

Thank you @pgoodman for your support.

[pgoodman]

@bsdb0y great diagnosis!! Would you be willing to work in some kind of flag like, --fork_mode or --use_fork or something like that? If you do this, and contribute it as a PR, also adding some verbiage into the indexing markdown file for troubleshooting (e.g. when indexing within Docker), and mention trying the fork mode.

[bsdb0y]

@pgoodman Thanks! I’ve already started implementing it and am running some checks. I’ll update the markdown page and open a PR soon.

[bsdb0y]

@pgoodman I’ve added the changes and updated the relevant sections (#578). Could you please review and let me know if anything needs to be modified?