Load ELF symbol information from GTIRB and compare to readelf by katrinafyi · Pull Request #477 · UQ-PAC/BASIL

katrinafyi · 2025-06-23T12:07:11Z

This PR is a first pass at extracting ELF information from the GTIRB files.
To do this, we pull information from the Protobuf Symbol object and also
from the AuxData (e.g., elfSymbolInfo). The output of the GTIRB ELF loader
is designed to match the existing ReadELFLoader.visitSyms as much as possible.

Below is a diff between the old readelf and the new "gtsrelf" output.

Details

--- b-readelf.scala     2025-06-26 16:07:14.569569911 +1000
+++ b-gtsrelf.scala     2025-06-26 16:07:14.582569918 +1000
@@ -25,111 +25,40 @@
     ELFSymbol(22, BigInt("4390888"), 0, ELFSymType.SECTION, ELFBind.LOCAL, ELFVis.DEFAULT, ELFNDX.Section(22), ".got.plt"),
     ELFSymbol(23, BigInt("4391704"), 0, ELFSymType.SECTION, ELFBind.LOCAL, ELFVis.DEFAULT, ELFNDX.Section(23), ".data"),
     ELFSymbol(24, BigInt("4391728"), 0, ELFSymType.SECTION, ELFBind.LOCAL, ELFVis.DEFAULT, ELFNDX.Section(24), ".bss"),
-    ELFSymbol(25, BigInt("0"), 0, ELFSymType.SECTION, ELFBind.LOCAL, ELFVis.DEFAULT, ELFNDX.Section(25), ".comment"),
     ELFSymbol(26, BigInt("0"), 0, ELFSymType.FILE, ELFBind.LOCAL, ELFVis.DEFAULT, ELFNDX.ABS, "crt1.o"),
-    ELFSymbol(27, BigInt("4194984"), 0, ELFSymType.NOTYPE, ELFBind.LOCAL, ELFVis.DEFAULT, ELFNDX.Section(2), "$d"),
     ELFSymbol(28, BigInt("4194984"), 32, ELFSymType.OBJECT, ELFBind.LOCAL, ELFVis.DEFAULT, ELFNDX.Section(2), "__abi_tag"),
-    ELFSymbol(29, BigInt("4213248"), 0, ELFSymType.NOTYPE, ELFBind.LOCAL, ELFVis.DEFAULT, ELFNDX.Section(13), "$x"),
     ELFSymbol(30, BigInt("4213300"), 0, ELFSymType.NOTYPE, ELFBind.LOCAL, ELFVis.DEFAULT, ELFNDX.Section(13), "__wrap_main"),
-    ELFSymbol(31, BigInt("4290396"), 0, ELFSymType.NOTYPE, ELFBind.LOCAL, ELFVis.DEFAULT, ELFNDX.Section(17), "$d"),
-    ELFSymbol(32, BigInt("4270192"), 0, ELFSymType.NOTYPE, ELFBind.LOCAL, ELFVis.DEFAULT, ELFNDX.Section(15), "$d"),
-    ELFSymbol(33, BigInt("4213312"), 0, ELFSymType.NOTYPE, ELFBind.LOCAL, ELFVis.DEFAULT, ELFNDX.Section(13), "$x"),
-    ELFSymbol(34, BigInt("4290416"), 0, ELFSymType.NOTYPE, ELFBind.LOCAL, ELFVis.DEFAULT, ELFNDX.Section(17), "$d"),
     ELFSymbol(35, BigInt("0"), 0, ELFSymType.FILE, ELFBind.LOCAL, ELFVis.DEFAULT, ELFNDX.ABS, "crti.o"),
-    ELFSymbol(36, BigInt("4213316"), 0, ELFSymType.NOTYPE, ELFBind.LOCAL, ELFVis.DEFAULT, ELFNDX.Section(13), "$x"),
     ELFSymbol(37, BigInt("4213316"), 20, ELFSymType.FUNC, ELFBind.LOCAL, ELFVis.DEFAULT, ELFNDX.Section(13), "call_weak_fn"),
-    ELFSymbol(38, BigInt("4202592"), 0, ELFSymType.NOTYPE, ELFBind.LOCAL, ELFVis.DEFAULT, ELFNDX.Section(11), "$x"),
-    ELFSymbol(39, BigInt("4270160"), 0, ELFSymType.NOTYPE, ELFBind.LOCAL, ELFVis.DEFAULT, ELFNDX.Section(14), "$x"),
     ELFSymbol(40, BigInt("0"), 0, ELFSymType.FILE, ELFBind.LOCAL, ELFVis.DEFAULT, ELFNDX.ABS, "crtn.o"),
-    ELFSymbol(41, BigInt("4202608"), 0, ELFSymType.NOTYPE, ELFBind.LOCAL, ELFVis.DEFAULT, ELFNDX.Section(11), "$x"),
-    ELFSymbol(42, BigInt("4270172"), 0, ELFSymType.NOTYPE, ELFBind.LOCAL, ELFVis.DEFAULT, ELFNDX.Section(14), "$x"),
     ELFSymbol(43, BigInt("0"), 0, ELFSymType.FILE, ELFBind.LOCAL, ELFVis.DEFAULT, ELFNDX.ABS, "main.c"),
-    ELFSymbol(44, BigInt("4274744"), 0, ELFSymType.NOTYPE, ELFBind.LOCAL, ELFVis.DEFAULT, ELFNDX.Section(15), "$d"),
-    ELFSymbol(45, BigInt("4233936"), 0, ELFSymType.NOTYPE, ELFBind.LOCAL, ELFVis.DEFAULT, ELFNDX.Section(13), "$x"),
-    ELFSymbol(46, BigInt("4204288"), 0, ELFSymType.NOTYPE, ELFBind.LOCAL, ELFVis.DEFAULT, ELFNDX.Section(13), "$x"),
-    ELFSymbol(47, BigInt("4285192"), 0, ELFSymType.NOTYPE, ELFBind.LOCAL, ELFVis.DEFAULT, ELFNDX.Section(15), "$d"),
-    ELFSymbol(48, BigInt("4285024"), 0, ELFSymType.NOTYPE, ELFBind.LOCAL, ELFVis.DEFAULT, ELFNDX.Section(15), "$d"),
-    ELFSymbol(49, BigInt("4285312"), 0, ELFSymType.NOTYPE, ELFBind.LOCAL, ELFVis.DEFAULT, ELFNDX.Section(15), "$d"),
-    ELFSymbol(50, BigInt("4391792"), 0, ELFSymType.NOTYPE, ELFBind.LOCAL, ELFVis.DEFAULT, ELFNDX.Section(24), "$d"),
-    ELFSymbol(51, BigInt("4292656"), 0, ELFSymType.NOTYPE, ELFBind.LOCAL, ELFVis.DEFAULT, ELFNDX.Section(17), "$d"),
     ELFSymbol(52, BigInt("0"), 0, ELFSymType.FILE, ELFBind.LOCAL, ELFVis.DEFAULT, ELFNDX.ABS, "crtbegin.o"),
-    ELFSymbol(53, BigInt("4213344"), 0, ELFSymType.NOTYPE, ELFBind.LOCAL, ELFVis.DEFAULT, ELFNDX.Section(13), "$x"),
     ELFSymbol(54, BigInt("4213344"), 0, ELFSymType.FUNC, ELFBind.LOCAL, ELFVis.DEFAULT, ELFNDX.Section(13), "deregister_tm_clones"),
     ELFSymbol(55, BigInt("4213392"), 0, ELFSymType.FUNC, ELFBind.LOCAL, ELFVis.DEFAULT, ELFNDX.Section(13), "register_tm_clones"),
-    ELFSymbol(56, BigInt("4391712"), 0, ELFSymType.NOTYPE, ELFBind.LOCAL, ELFVis.DEFAULT, ELFNDX.Section(23), "$d"),
     ELFSymbol(57, BigInt("4213456"), 0, ELFSymType.FUNC, ELFBind.LOCAL, ELFVis.DEFAULT, ELFNDX.Section(13), "__do_global_dtors_aux"),
     ELFSymbol(58, BigInt("4391784"), 1, ELFSymType.OBJECT, ELFBind.LOCAL, ELFVis.DEFAULT, ELFNDX.Section(24), "completed.0"),
-    ELFSymbol(59, BigInt("4390352"), 0, ELFSymType.NOTYPE, ELFBind.LOCAL, ELFVis.DEFAULT, ELFNDX.Section(19), "$d"),
     ELFSymbol(60, BigInt("4390352"), 0, ELFSymType.OBJECT, ELFBind.LOCAL, ELFVis.DEFAULT, ELFNDX.Section(19), "__do_global_dtors_aux_fini_array_entry"),
     ELFSymbol(61, BigInt("4213504"), 0, ELFSymType.FUNC, ELFBind.LOCAL, ELFVis.DEFAULT, ELFNDX.Section(13), "frame_dummy"),
-    ELFSymbol(62, BigInt("4390344"), 0, ELFSymType.NOTYPE, ELFBind.LOCAL, ELFVis.DEFAULT, ELFNDX.Section(18), "$d"),
     ELFSymbol(63, BigInt("4390344"), 0, ELFSymType.OBJECT, ELFBind.LOCAL, ELFVis.DEFAULT, ELFNDX.Section(18), "__frame_dummy_init_array_entry"),
-    ELFSymbol(64, BigInt("4290440"), 0, ELFSymType.NOTYPE, ELFBind.LOCAL, ELFVis.DEFAULT, ELFNDX.Section(17), "$d"),
-    ELFSymbol(65, BigInt("4391784"), 0, ELFSymType.NOTYPE, ELFBind.LOCAL, ELFVis.DEFAULT, ELFNDX.Section(24), "$d"),
     ELFSymbol(66, BigInt("0"), 0, ELFSymType.FILE, ELFBind.LOCAL, ELFVis.DEFAULT, ELFNDX.ABS, "acl.c"),
-    ELFSymbol(67, BigInt("4270200"), 0, ELFSymType.NOTYPE, ELFBind.LOCAL, ELFVis.DEFAULT, ELFNDX.Section(15), "$d"),
-    ELFSymbol(68, BigInt("4213520"), 0, ELFSymType.NOTYPE, ELFBind.LOCAL, ELFVis.DEFAULT, ELFNDX.Section(13), "$x"),
-    ELFSymbol(69, BigInt("4290536"), 0, ELFSymType.NOTYPE, ELFBind.LOCAL, ELFVis.DEFAULT, ELFNDX.Section(17), "$d"),
     ELFSymbol(70, BigInt("0"), 0, ELFSymType.FILE, ELFBind.LOCAL, ELFVis.DEFAULT, ELFNDX.ABS, "auth.c"),
-    ELFSymbol(71, BigInt("4214336"), 0, ELFSymType.NOTYPE, ELFBind.LOCAL, ELFVis.DEFAULT, ELFNDX.Section(13), "$x"),
-    ELFSymbol(72, BigInt("4270424"), 0, ELFSymType.NOTYPE, ELFBind.LOCAL, ELFVis.DEFAULT, ELFNDX.Section(15), "$d"),
-    ELFSymbol(73, BigInt("4285016"), 0, ELFSymType.NOTYPE, ELFBind.LOCAL, ELFVis.DEFAULT, ELFNDX.Section(15), "$d"),
-    ELFSymbol(74, BigInt("4290656"), 0, ELFSymType.NOTYPE, ELFBind.LOCAL, ELFVis.DEFAULT, ELFNDX.Section(17), "$d"),
     ELFSymbol(75, BigInt("0"), 0, ELFSymType.FILE, ELFBind.LOCAL, ELFVis.DEFAULT, ELFNDX.ABS, "config.c"),
-    ELFSymbol(76, BigInt("4270656"), 0, ELFSymType.NOTYPE, ELFBind.LOCAL, ELFVis.DEFAULT, ELFNDX.Section(15), "$d"),
-    ELFSymbol(77, BigInt("4215536"), 0, ELFSymType.NOTYPE, ELFBind.LOCAL, ELFVis.DEFAULT, ELFNDX.Section(13), "$x"),
-    ELFSymbol(78, BigInt("4290848"), 0, ELFSymType.NOTYPE, ELFBind.LOCAL, ELFVis.DEFAULT, ELFNDX.Section(17), "$d"),
     ELFSymbol(79, BigInt("0"), 0, ELFSymType.FILE, ELFBind.LOCAL, ELFVis.DEFAULT, ELFNDX.ABS, "direct.c"),
-    ELFSymbol(80, BigInt("4216688"), 0, ELFSymType.NOTYPE, ELFBind.LOCAL, ELFVis.DEFAULT, ELFNDX.Section(13), "$x"),
-    ELFSymbol(81, BigInt("4270696"), 0, ELFSymType.NOTYPE, ELFBind.LOCAL, ELFVis.DEFAULT, ELFNDX.Section(15), "$d"),
-    ELFSymbol(82, BigInt("4291072"), 0, ELFSymType.NOTYPE, ELFBind.LOCAL, ELFVis.DEFAULT, ELFNDX.Section(17), "$d"),
     ELFSymbol(83, BigInt("0"), 0, ELFSymType.FILE, ELFBind.LOCAL, ELFVis.DEFAULT, ELFNDX.ABS, "forward.c"),
-    ELFSymbol(84, BigInt("4271856"), 0, ELFSymType.NOTYPE, ELFBind.LOCAL, ELFVis.DEFAULT, ELFNDX.Section(15), "$d"),
-    ELFSymbol(85, BigInt("4220992"), 0, ELFSymType.NOTYPE, ELFBind.LOCAL, ELFVis.DEFAULT, ELFNDX.Section(13), "$x"),
-    ELFSymbol(86, BigInt("4285048"), 0, ELFSymType.NOTYPE, ELFBind.LOCAL, ELFVis.DEFAULT, ELFNDX.Section(15), "$d"),
-    ELFSymbol(87, BigInt("4291400"), 0, ELFSymType.NOTYPE, ELFBind.LOCAL, ELFVis.DEFAULT, ELFNDX.Section(17), "$d"),
     ELFSymbol(88, BigInt("0"), 0, ELFSymType.FILE, ELFBind.LOCAL, ELFVis.DEFAULT, ELFNDX.ABS, "http.c"),
-    ELFSymbol(89, BigInt("4273728"), 0, ELFSymType.NOTYPE, ELFBind.LOCAL, ELFVis.DEFAULT, ELFNDX.Section(15), "$d"),
-    ELFSymbol(90, BigInt("4227024"), 0, ELFSymType.NOTYPE, ELFBind.LOCAL, ELFVis.DEFAULT, ELFNDX.Section(13), "$x"),
-    ELFSymbol(91, BigInt("4291768"), 0, ELFSymType.NOTYPE, ELFBind.LOCAL, ELFVis.DEFAULT, ELFNDX.Section(17), "$d"),
     ELFSymbol(92, BigInt("0"), 0, ELFSymType.FILE, ELFBind.LOCAL, ELFVis.DEFAULT, ELFNDX.ABS, "ntlm.c"),
-    ELFSymbol(93, BigInt("4237952"), 0, ELFSymType.NOTYPE, ELFBind.LOCAL, ELFVis.DEFAULT, ELFNDX.Section(13), "$x"),
     ELFSymbol(94, BigInt("4237952"), 148, ELFSymType.FUNC, ELFBind.LOCAL, ELFVis.DEFAULT, ELFNDX.Section(13), "ntlm_set_key"),
-    ELFSymbol(95, BigInt("4280912"), 0, ELFSymType.NOTYPE, ELFBind.LOCAL, ELFVis.DEFAULT, ELFNDX.Section(15), "$d"),
-    ELFSymbol(96, BigInt("4285040"), 0, ELFSymType.NOTYPE, ELFBind.LOCAL, ELFVis.DEFAULT, ELFNDX.Section(15), "$d"),
-    ELFSymbol(97, BigInt("4293248"), 0, ELFSymType.NOTYPE, ELFBind.LOCAL, ELFVis.DEFAULT, ELFNDX.Section(17), "$d"),
     ELFSymbol(98, BigInt("0"), 0, ELFSymType.FILE, ELFBind.LOCAL, ELFVis.DEFAULT, ELFNDX.ABS, "pages.c"),
-    ELFSymbol(99, BigInt("4281368"), 0, ELFSymType.NOTYPE, ELFBind.LOCAL, ELFVis.DEFAULT, ELFNDX.Section(15), "$d"),
-    ELFSymbol(100, BigInt("4241920"), 0, ELFSymType.NOTYPE, ELFBind.LOCAL, ELFVis.DEFAULT, ELFNDX.Section(13), "$x"),
-    ELFSymbol(101, BigInt("4293536"), 0, ELFSymType.NOTYPE, ELFBind.LOCAL, ELFVis.DEFAULT, ELFNDX.Section(17), "$d"),
     ELFSymbol(102, BigInt("0"), 0, ELFSymType.FILE, ELFBind.LOCAL, ELFVis.DEFAULT, ELFNDX.ABS, "proxy.c"),
-    ELFSymbol(103, BigInt("4242304"), 0, ELFSymType.NOTYPE, ELFBind.LOCAL, ELFVis.DEFAULT, ELFNDX.Section(13), "$x"),
-    ELFSymbol(104, BigInt("4282232"), 0, ELFSymType.NOTYPE, ELFBind.LOCAL, ELFVis.DEFAULT, ELFNDX.Section(15), "$d"),
-    ELFSymbol(105, BigInt("4391984"), 0, ELFSymType.NOTYPE, ELFBind.LOCAL, ELFVis.DEFAULT, ELFNDX.Section(24), "$d"),
-    ELFSymbol(106, BigInt("4293688"), 0, ELFSymType.NOTYPE, ELFBind.LOCAL, ELFVis.DEFAULT, ELFNDX.Section(17), "$d"),
     ELFSymbol(107, BigInt("0"), 0, ELFSymType.FILE, ELFBind.LOCAL, ELFVis.DEFAULT, ELFNDX.ABS, "scanner.c"),
-    ELFSymbol(108, BigInt("4274480"), 0, ELFSymType.NOTYPE, ELFBind.LOCAL, ELFVis.DEFAULT, ELFNDX.Section(15), "$d"),
-    ELFSymbol(109, BigInt("4246752"), 0, ELFSymType.NOTYPE, ELFBind.LOCAL, ELFVis.DEFAULT, ELFNDX.Section(13), "$x"),
-    ELFSymbol(110, BigInt("4294432"), 0, ELFSymType.NOTYPE, ELFBind.LOCAL, ELFVis.DEFAULT, ELFNDX.Section(17), "$d"),
     ELFSymbol(111, BigInt("0"), 0, ELFSymType.FILE, ELFBind.LOCAL, ELFVis.DEFAULT, ELFNDX.ABS, "socket.c"),
-    ELFSymbol(112, BigInt("4275384"), 0, ELFSymType.NOTYPE, ELFBind.LOCAL, ELFVis.DEFAULT, ELFNDX.Section(15), "$d"),
-    ELFSymbol(113, BigInt("4250528"), 0, ELFSymType.NOTYPE, ELFBind.LOCAL, ELFVis.DEFAULT, ELFNDX.Section(13), "$x"),
-    ELFSymbol(114, BigInt("4294536"), 0, ELFSymType.NOTYPE, ELFBind.LOCAL, ELFVis.DEFAULT, ELFNDX.Section(17), "$d"),
     ELFSymbol(115, BigInt("0"), 0, ELFSymType.FILE, ELFBind.LOCAL, ELFVis.DEFAULT, ELFNDX.ABS, "utils.c"),
-    ELFSymbol(116, BigInt("4284408"), 0, ELFSymType.NOTYPE, ELFBind.LOCAL, ELFVis.DEFAULT, ELFNDX.Section(15), "$d"),
-    ELFSymbol(117, BigInt("4252464"), 0, ELFSymType.NOTYPE, ELFBind.LOCAL, ELFVis.DEFAULT, ELFNDX.Section(13), "$x"),
-    ELFSymbol(118, BigInt("4285328"), 0, ELFSymType.NOTYPE, ELFBind.LOCAL, ELFVis.DEFAULT, ELFNDX.Section(15), "$d"),
-    ELFSymbol(119, BigInt("4285360"), 0, ELFSymType.NOTYPE, ELFBind.LOCAL, ELFVis.DEFAULT, ELFNDX.Section(15), "$d"),
     ELFSymbol(120, BigInt("4285360"), 17, ELFSymType.OBJECT, ELFBind.LOCAL, ELFVis.DEFAULT, ELFNDX.Section(15), "hextab"),
     ELFSymbol(121, BigInt("4285392"), 512, ELFSymType.OBJECT, ELFBind.LOCAL, ELFVis.DEFAULT, ELFNDX.Section(15), "hexindex"),
     ELFSymbol(122, BigInt("4285904"), 64, ELFSymType.OBJECT, ELFBind.LOCAL, ELFVis.DEFAULT, ELFNDX.Section(15), "base64"),
     ELFSymbol(123, BigInt("4285968"), 512, ELFSymType.OBJECT, ELFBind.LOCAL, ELFVis.DEFAULT, ELFNDX.Section(15), "index64"),
-    ELFSymbol(124, BigInt("4294984"), 0, ELFSymType.NOTYPE, ELFBind.LOCAL, ELFVis.DEFAULT, ELFNDX.Section(17), "$d"),
     ELFSymbol(125, BigInt("0"), 0, ELFSymType.FILE, ELFBind.LOCAL, ELFVis.DEFAULT, ELFNDX.ABS, "xcrypt.c"),
-    ELFSymbol(126, BigInt("4258400"), 0, ELFSymType.NOTYPE, ELFBind.LOCAL, ELFVis.DEFAULT, ELFNDX.Section(13), "$x"),
     ELFSymbol(127, BigInt("4258400"), 812, ELFSymType.FUNC, ELFBind.LOCAL, ELFVis.DEFAULT, ELFNDX.Section(13), "des_key_schedule"),
-    ELFSymbol(128, BigInt("4285344"), 0, ELFSymType.NOTYPE, ELFBind.LOCAL, ELFVis.DEFAULT, ELFNDX.Section(15), "$d"),
-    ELFSymbol(129, BigInt("4286480"), 0, ELFSymType.NOTYPE, ELFBind.LOCAL, ELFVis.DEFAULT, ELFNDX.Section(15), "$d"),
     ELFSymbol(130, BigInt("4286480"), 64, ELFSymType.OBJECT, ELFBind.LOCAL, ELFVis.DEFAULT, ELFNDX.Section(15), "leftkey_swap"),
     ELFSymbol(131, BigInt("4286544"), 64, ELFSymType.OBJECT, ELFBind.LOCAL, ELFVis.DEFAULT, ELFNDX.Section(15), "rightkey_swap"),
     ELFSymbol(132, BigInt("4286608"), 16, ELFSymType.OBJECT, ELFBind.LOCAL, ELFVis.DEFAULT, ELFNDX.Section(15), "encrypt_rotate_tab"),
@@ -143,15 +72,12 @@
     ELFSymbol(140, BigInt("4288672"), 256, ELFSymType.OBJECT, ELFBind.LOCAL, ELFVis.DEFAULT, ELFNDX.Section(15), "sbox3"),
     ELFSymbol(141, BigInt("4288928"), 256, ELFSymType.OBJECT, ELFBind.LOCAL, ELFVis.DEFAULT, ELFNDX.Section(15), "sbox1"),
     ELFSymbol(142, BigInt("4289184"), 64, ELFSymType.OBJECT, ELFBind.LOCAL, ELFVis.DEFAULT, ELFNDX.Section(15), "fillbuf"),
-    ELFSymbol(143, BigInt("4296880"), 0, ELFSymType.NOTYPE, ELFBind.LOCAL, ELFVis.DEFAULT, ELFNDX.Section(17), "$d"),
     ELFSymbol(144, BigInt("0"), 0, ELFSymType.FILE, ELFBind.LOCAL, ELFVis.DEFAULT, ELFNDX.ABS, "crtend.o"),
-    ELFSymbol(145, BigInt("4297916"), 0, ELFSymType.NOTYPE, ELFBind.LOCAL, ELFVis.DEFAULT, ELFNDX.Section(17), "$d"),
     ELFSymbol(146, BigInt("4297916"), 0, ELFSymType.OBJECT, ELFBind.LOCAL, ELFVis.DEFAULT, ELFNDX.Section(17), "__FRAME_END__"),
     ELFSymbol(147, BigInt("0"), 0, ELFSymType.FILE, ELFBind.LOCAL, ELFVis.DEFAULT, ELFNDX.ABS, ""),
     ELFSymbol(148, BigInt("4390360"), 0, ELFSymType.OBJECT, ELFBind.LOCAL, ELFVis.DEFAULT, ELFNDX.Section(20), "_DYNAMIC"),
     ELFSymbol(149, BigInt("4289248"), 0, ELFSymType.NOTYPE, ELFBind.LOCAL, ELFVis.DEFAULT, ELFNDX.Section(16), "__GNU_EH_FRAME_HDR"),
     ELFSymbol(150, BigInt("4390856"), 0, ELFSymType.OBJECT, ELFBind.LOCAL, ELFVis.DEFAULT, ELFNDX.Section(21), "_GLOBAL_OFFSET_TABLE_"),
-    ELFSymbol(151, BigInt("4202624"), 0, ELFSymType.NOTYPE, ELFBind.LOCAL, ELFVis.DEFAULT, ELFNDX.Section(12), "$x"),
     ELFSymbol(152, BigInt("0"), 0, ELFSymType.FUNC, ELFBind.GLOBAL, ELFVis.DEFAULT, ELFNDX.UND, "memcpy"),
     ELFSymbol(153, BigInt("4268448"), 1268, ELFSymType.FUNC, ELFBind.GLOBAL, ELFVis.DEFAULT, ELFNDX.Section(13), "hmac_md5"),
     ELFSymbol(154, BigInt("4242112"), 80, ELFSymType.FUNC, ELFBind.GLOBAL, ELFVis.DEFAULT, ELFNDX.Section(13), "gen_denied_page"),
@@ -479,8 +405,6 @@
     ExternalFunction("memset", BigInt("4391176")),
     ExternalFunction("open", BigInt("4391160")),
     ExternalFunction("openlog", BigInt("4391320")),
-    ExternalFunction("optarg", BigInt("4391744")),
-    ExternalFunction("optind", BigInt("4391752")),
     ExternalFunction("perror", BigInt("4390984")),
     ExternalFunction("printf", BigInt("4391616")),
     ExternalFunction("pthread_attr_destroy", BigInt("4391168")),
@@ -508,9 +432,6 @@
     ExternalFunction("snprintf", BigInt("4391080")),
     ExternalFunction("socket", BigInt("4391504")),
     ExternalFunction("srandom", BigInt("4391568")),
-    ExternalFunction("stderr", BigInt("4391728")),
-    ExternalFunction("stdin", BigInt("4391776")),
-    ExternalFunction("stdout", BigInt("4391760")),
     ExternalFunction("strcasecmp", BigInt("4391232")),
     ExternalFunction("strchr", BigInt("4391464")),
     ExternalFunction("strcmp", BigInt("4391376")),

We can see that most of the rows are the same.
One remaining discrepancy is that we remove external global variables
from the external functions set. This is probably the more correct behaviour.
The GTIRB relf is also missing the SECTION entries and the NOTYPE LOCAL entries
with names starting with $. Aside from this, the only differences are in the
@GLIBC suffix (omitted in the diff above).

So, the only substantive difference is in the full symbol list. It looks like this
is only used in the interpreter, which looks for __end__, __bss_start__, and
__bss_end__. These symbols are correct in the GTIRB output, so hopefully
this is fine.

In this PR, the Gtirb ELF loading is implemented in a "shadow" mode.
When using any GTIRB input, the new GTIRB ELF loader is run alongside
the existing readelf loader. We always return the original readelf result,
but we compare two results and print a warning if there's a discrepancy.
(Note: this comparison does some normalisation to discount the acceptable
differences, as described above).

This shadowing means that we can test the new loader in practice and make
sure it behaves the same with all the examples we use. After some period
of time, we can cutover to the gtirb relf loader if it's stable.

testing

We reiterate that the gtsrelf functionality is tested in the background
whenever a GTIRB input is used.

To more clearly inspect the two relf outputs, you can use the new --dump-relf
option:

./mill run --load-directory-gtirb src/test/correct/function/clang --dump-relf=function

This will write function-readelf.scala and function-gtsrelf.scala, then exit Basil.

To only run the GTIRB/readelf check without writing file, you can pass an
empty string with --dump-relf=.

$ ./mill -w run --load-directory-gtirb src/test/correct/jumptable2/clang --dump-relf= 
[INFO]   Found ./src/test/correct/jumptable2/clang/jumptable2.gts ./src/test/correct/jumptable2/clang/jumptable2.relf 
[WARN]   PLEASE REPORT THIS ISSUE! include the gts and relf files. gtirb relf discrepancy, symbol tables differ:
         gtirb - relf = HashSet()
         relf - gtirb = HashSet(ELFSymbol(59,0,0,FILE,LOCAL,DEFAULT,ABS,crtstuff.c)) [checkReadELFCompatibility@GTIRBReadELF.scala:224]
[DEBUG]   gtirb relf and readelf relf compatible: false [checkReadELFCompatibility@GTIRBReadELF.scala:226]

This demonstrates the error message when there is a mismatch between the two ELF loaders.
Keep in mind that this is not a crash. If --dump-relf is not passed, Basil will
continue with its normal functions.

bonus things

Most of the ELF parsing is done by ddisasm (yay!), but we do manually parse the
relocation sections .rela.dyn and .rela.plt. This is surely the weakest link in this PR.
Address long-standing todo by making a case class for the parsed readelf data.
GTIRBResolver with GTIRBRef types to abstract away GTIRB querying operations.
This pattern could be extended to clean up the GTIRB frontend, if desired.
Changes the derived ToScala methods to produce multi-line output if any of their
elements are multi-line.
Slaps derives ToScala on all the classes and enums that make up ELFSymbol.

Fixes #175

the previous MapDecoder is hard-coded to decode only certain structures in the aux_data. this makes it more compositional, so we can decode more complex types across more auxdata fields.

maybe we just vendor the java file lol

companion objects are awful. i can't access the java public static inner class ;-;

This reverts commit 31e0cfe.

This reverts commit cebe023.

the static class problem in 31e0cfe was me being silly

This reverts commit 3c45043. Conflicts: src/main/scala/translating/GTIRBReadELF.scala

Conflicts: src/main/scala/gtirb/AuxDecoder.scala src/main/scala/translating/GTIRBToIR.scala

ailrst · 2025-06-25T05:42:17Z

More important than the symbols list matching is that the externalFunctions, globals, funcEntries and globalOffsets are the same with the gtirb-based relf loader, as there are actually used by basil.

I don't think the LOCAL rows are used so shouldn't matter, and sections are already loaded into Program.initialMemory.

ailrst

Probably a good idea to fix the issues with the cntlm before merging, since they look straightforward enough, but I think this is good so far.

ailrst · 2025-06-25T05:42:52Z

src/main/scala/Main.scala

+
+        // skip writing files if the given path is an empty string
+        if (relfOut.trim.isEmpty)
+          return


Significant indentation 👎

ailrst · 2025-06-25T05:46:39Z

src/main/scala/gtirb/GTIRBReadELF.scala

+    gtirb.symbolEntriesByUuid.view.collect { case (symid, (size, "OBJECT", "GLOBAL", "DEFAULT", idx)) =>
+      val blk = symid.getReferentUuid.get.getOption
+      // val sec = blk.section
+      // assert(gtirb.mod.sections(idx.toInt - 1) == sec)


ailrst · 2025-06-25T05:46:47Z

src/main/scala/gtirb/GTIRBReadELF.scala

+
+  // Full ELF32 specification: https://refspecs.linuxfoundation.org/elf/elf.pdf
+
+  // Full ELF64 specification: https://irix7.com/techpubs/007-4658-001.pdf


Doc comment

ailrst · 2025-06-25T05:48:05Z

src/main/scala/gtirb/GTIRBReadELF.scala

+      case 1024 => Left((BigInt(0), BigInt(0)))
+    }
+
+  def getAllSymbols() = {


katrinafyi · 2025-06-26T06:09:29Z

new updates to fix cntlm's things. also added SECTION entries to more closely match readelf.

the only remaining discrepancy is that we remove external global variables from the external functions. this is probably the more correct behaviour.

diff, with glibc versions removed:

Details

--- b-readelf.scala     2025-06-26 16:07:14.569569911 +1000
+++ b-gtsrelf.scala     2025-06-26 16:07:14.582569918 +1000
@@ -25,111 +25,40 @@
     ELFSymbol(22, BigInt("4390888"), 0, ELFSymType.SECTION, ELFBind.LOCAL, ELFVis.DEFAULT, ELFNDX.Section(22), ".got.plt"),
     ELFSymbol(23, BigInt("4391704"), 0, ELFSymType.SECTION, ELFBind.LOCAL, ELFVis.DEFAULT, ELFNDX.Section(23), ".data"),
     ELFSymbol(24, BigInt("4391728"), 0, ELFSymType.SECTION, ELFBind.LOCAL, ELFVis.DEFAULT, ELFNDX.Section(24), ".bss"),
-    ELFSymbol(25, BigInt("0"), 0, ELFSymType.SECTION, ELFBind.LOCAL, ELFVis.DEFAULT, ELFNDX.Section(25), ".comment"),
     ELFSymbol(26, BigInt("0"), 0, ELFSymType.FILE, ELFBind.LOCAL, ELFVis.DEFAULT, ELFNDX.ABS, "crt1.o"),
-    ELFSymbol(27, BigInt("4194984"), 0, ELFSymType.NOTYPE, ELFBind.LOCAL, ELFVis.DEFAULT, ELFNDX.Section(2), "$d"),
     ELFSymbol(28, BigInt("4194984"), 32, ELFSymType.OBJECT, ELFBind.LOCAL, ELFVis.DEFAULT, ELFNDX.Section(2), "__abi_tag"),
-    ELFSymbol(29, BigInt("4213248"), 0, ELFSymType.NOTYPE, ELFBind.LOCAL, ELFVis.DEFAULT, ELFNDX.Section(13), "$x"),
     ELFSymbol(30, BigInt("4213300"), 0, ELFSymType.NOTYPE, ELFBind.LOCAL, ELFVis.DEFAULT, ELFNDX.Section(13), "__wrap_main"),
-    ELFSymbol(31, BigInt("4290396"), 0, ELFSymType.NOTYPE, ELFBind.LOCAL, ELFVis.DEFAULT, ELFNDX.Section(17), "$d"),
-    ELFSymbol(32, BigInt("4270192"), 0, ELFSymType.NOTYPE, ELFBind.LOCAL, ELFVis.DEFAULT, ELFNDX.Section(15), "$d"),
-    ELFSymbol(33, BigInt("4213312"), 0, ELFSymType.NOTYPE, ELFBind.LOCAL, ELFVis.DEFAULT, ELFNDX.Section(13), "$x"),
-    ELFSymbol(34, BigInt("4290416"), 0, ELFSymType.NOTYPE, ELFBind.LOCAL, ELFVis.DEFAULT, ELFNDX.Section(17), "$d"),
     ELFSymbol(35, BigInt("0"), 0, ELFSymType.FILE, ELFBind.LOCAL, ELFVis.DEFAULT, ELFNDX.ABS, "crti.o"),
-    ELFSymbol(36, BigInt("4213316"), 0, ELFSymType.NOTYPE, ELFBind.LOCAL, ELFVis.DEFAULT, ELFNDX.Section(13), "$x"),
     ELFSymbol(37, BigInt("4213316"), 20, ELFSymType.FUNC, ELFBind.LOCAL, ELFVis.DEFAULT, ELFNDX.Section(13), "call_weak_fn"),
-    ELFSymbol(38, BigInt("4202592"), 0, ELFSymType.NOTYPE, ELFBind.LOCAL, ELFVis.DEFAULT, ELFNDX.Section(11), "$x"),
-    ELFSymbol(39, BigInt("4270160"), 0, ELFSymType.NOTYPE, ELFBind.LOCAL, ELFVis.DEFAULT, ELFNDX.Section(14), "$x"),
     ELFSymbol(40, BigInt("0"), 0, ELFSymType.FILE, ELFBind.LOCAL, ELFVis.DEFAULT, ELFNDX.ABS, "crtn.o"),
-    ELFSymbol(41, BigInt("4202608"), 0, ELFSymType.NOTYPE, ELFBind.LOCAL, ELFVis.DEFAULT, ELFNDX.Section(11), "$x"),
-    ELFSymbol(42, BigInt("4270172"), 0, ELFSymType.NOTYPE, ELFBind.LOCAL, ELFVis.DEFAULT, ELFNDX.Section(14), "$x"),
     ELFSymbol(43, BigInt("0"), 0, ELFSymType.FILE, ELFBind.LOCAL, ELFVis.DEFAULT, ELFNDX.ABS, "main.c"),
-    ELFSymbol(44, BigInt("4274744"), 0, ELFSymType.NOTYPE, ELFBind.LOCAL, ELFVis.DEFAULT, ELFNDX.Section(15), "$d"),
-    ELFSymbol(45, BigInt("4233936"), 0, ELFSymType.NOTYPE, ELFBind.LOCAL, ELFVis.DEFAULT, ELFNDX.Section(13), "$x"),
-    ELFSymbol(46, BigInt("4204288"), 0, ELFSymType.NOTYPE, ELFBind.LOCAL, ELFVis.DEFAULT, ELFNDX.Section(13), "$x"),
-    ELFSymbol(47, BigInt("4285192"), 0, ELFSymType.NOTYPE, ELFBind.LOCAL, ELFVis.DEFAULT, ELFNDX.Section(15), "$d"),
-    ELFSymbol(48, BigInt("4285024"), 0, ELFSymType.NOTYPE, ELFBind.LOCAL, ELFVis.DEFAULT, ELFNDX.Section(15), "$d"),
-    ELFSymbol(49, BigInt("4285312"), 0, ELFSymType.NOTYPE, ELFBind.LOCAL, ELFVis.DEFAULT, ELFNDX.Section(15), "$d"),
-    ELFSymbol(50, BigInt("4391792"), 0, ELFSymType.NOTYPE, ELFBind.LOCAL, ELFVis.DEFAULT, ELFNDX.Section(24), "$d"),
-    ELFSymbol(51, BigInt("4292656"), 0, ELFSymType.NOTYPE, ELFBind.LOCAL, ELFVis.DEFAULT, ELFNDX.Section(17), "$d"),
     ELFSymbol(52, BigInt("0"), 0, ELFSymType.FILE, ELFBind.LOCAL, ELFVis.DEFAULT, ELFNDX.ABS, "crtbegin.o"),
-    ELFSymbol(53, BigInt("4213344"), 0, ELFSymType.NOTYPE, ELFBind.LOCAL, ELFVis.DEFAULT, ELFNDX.Section(13), "$x"),
     ELFSymbol(54, BigInt("4213344"), 0, ELFSymType.FUNC, ELFBind.LOCAL, ELFVis.DEFAULT, ELFNDX.Section(13), "deregister_tm_clones"),
     ELFSymbol(55, BigInt("4213392"), 0, ELFSymType.FUNC, ELFBind.LOCAL, ELFVis.DEFAULT, ELFNDX.Section(13), "register_tm_clones"),
-    ELFSymbol(56, BigInt("4391712"), 0, ELFSymType.NOTYPE, ELFBind.LOCAL, ELFVis.DEFAULT, ELFNDX.Section(23), "$d"),
     ELFSymbol(57, BigInt("4213456"), 0, ELFSymType.FUNC, ELFBind.LOCAL, ELFVis.DEFAULT, ELFNDX.Section(13), "__do_global_dtors_aux"),
     ELFSymbol(58, BigInt("4391784"), 1, ELFSymType.OBJECT, ELFBind.LOCAL, ELFVis.DEFAULT, ELFNDX.Section(24), "completed.0"),
-    ELFSymbol(59, BigInt("4390352"), 0, ELFSymType.NOTYPE, ELFBind.LOCAL, ELFVis.DEFAULT, ELFNDX.Section(19), "$d"),
     ELFSymbol(60, BigInt("4390352"), 0, ELFSymType.OBJECT, ELFBind.LOCAL, ELFVis.DEFAULT, ELFNDX.Section(19), "__do_global_dtors_aux_fini_array_entry"),
     ELFSymbol(61, BigInt("4213504"), 0, ELFSymType.FUNC, ELFBind.LOCAL, ELFVis.DEFAULT, ELFNDX.Section(13), "frame_dummy"),
-    ELFSymbol(62, BigInt("4390344"), 0, ELFSymType.NOTYPE, ELFBind.LOCAL, ELFVis.DEFAULT, ELFNDX.Section(18), "$d"),
     ELFSymbol(63, BigInt("4390344"), 0, ELFSymType.OBJECT, ELFBind.LOCAL, ELFVis.DEFAULT, ELFNDX.Section(18), "__frame_dummy_init_array_entry"),
-    ELFSymbol(64, BigInt("4290440"), 0, ELFSymType.NOTYPE, ELFBind.LOCAL, ELFVis.DEFAULT, ELFNDX.Section(17), "$d"),
-    ELFSymbol(65, BigInt("4391784"), 0, ELFSymType.NOTYPE, ELFBind.LOCAL, ELFVis.DEFAULT, ELFNDX.Section(24), "$d"),
     ELFSymbol(66, BigInt("0"), 0, ELFSymType.FILE, ELFBind.LOCAL, ELFVis.DEFAULT, ELFNDX.ABS, "acl.c"),
-    ELFSymbol(67, BigInt("4270200"), 0, ELFSymType.NOTYPE, ELFBind.LOCAL, ELFVis.DEFAULT, ELFNDX.Section(15), "$d"),
-    ELFSymbol(68, BigInt("4213520"), 0, ELFSymType.NOTYPE, ELFBind.LOCAL, ELFVis.DEFAULT, ELFNDX.Section(13), "$x"),
-    ELFSymbol(69, BigInt("4290536"), 0, ELFSymType.NOTYPE, ELFBind.LOCAL, ELFVis.DEFAULT, ELFNDX.Section(17), "$d"),
     ELFSymbol(70, BigInt("0"), 0, ELFSymType.FILE, ELFBind.LOCAL, ELFVis.DEFAULT, ELFNDX.ABS, "auth.c"),
-    ELFSymbol(71, BigInt("4214336"), 0, ELFSymType.NOTYPE, ELFBind.LOCAL, ELFVis.DEFAULT, ELFNDX.Section(13), "$x"),
-    ELFSymbol(72, BigInt("4270424"), 0, ELFSymType.NOTYPE, ELFBind.LOCAL, ELFVis.DEFAULT, ELFNDX.Section(15), "$d"),
-    ELFSymbol(73, BigInt("4285016"), 0, ELFSymType.NOTYPE, ELFBind.LOCAL, ELFVis.DEFAULT, ELFNDX.Section(15), "$d"),
-    ELFSymbol(74, BigInt("4290656"), 0, ELFSymType.NOTYPE, ELFBind.LOCAL, ELFVis.DEFAULT, ELFNDX.Section(17), "$d"),
     ELFSymbol(75, BigInt("0"), 0, ELFSymType.FILE, ELFBind.LOCAL, ELFVis.DEFAULT, ELFNDX.ABS, "config.c"),
-    ELFSymbol(76, BigInt("4270656"), 0, ELFSymType.NOTYPE, ELFBind.LOCAL, ELFVis.DEFAULT, ELFNDX.Section(15), "$d"),
-    ELFSymbol(77, BigInt("4215536"), 0, ELFSymType.NOTYPE, ELFBind.LOCAL, ELFVis.DEFAULT, ELFNDX.Section(13), "$x"),
-    ELFSymbol(78, BigInt("4290848"), 0, ELFSymType.NOTYPE, ELFBind.LOCAL, ELFVis.DEFAULT, ELFNDX.Section(17), "$d"),
     ELFSymbol(79, BigInt("0"), 0, ELFSymType.FILE, ELFBind.LOCAL, ELFVis.DEFAULT, ELFNDX.ABS, "direct.c"),
-    ELFSymbol(80, BigInt("4216688"), 0, ELFSymType.NOTYPE, ELFBind.LOCAL, ELFVis.DEFAULT, ELFNDX.Section(13), "$x"),
-    ELFSymbol(81, BigInt("4270696"), 0, ELFSymType.NOTYPE, ELFBind.LOCAL, ELFVis.DEFAULT, ELFNDX.Section(15), "$d"),
-    ELFSymbol(82, BigInt("4291072"), 0, ELFSymType.NOTYPE, ELFBind.LOCAL, ELFVis.DEFAULT, ELFNDX.Section(17), "$d"),
     ELFSymbol(83, BigInt("0"), 0, ELFSymType.FILE, ELFBind.LOCAL, ELFVis.DEFAULT, ELFNDX.ABS, "forward.c"),
-    ELFSymbol(84, BigInt("4271856"), 0, ELFSymType.NOTYPE, ELFBind.LOCAL, ELFVis.DEFAULT, ELFNDX.Section(15), "$d"),
-    ELFSymbol(85, BigInt("4220992"), 0, ELFSymType.NOTYPE, ELFBind.LOCAL, ELFVis.DEFAULT, ELFNDX.Section(13), "$x"),
-    ELFSymbol(86, BigInt("4285048"), 0, ELFSymType.NOTYPE, ELFBind.LOCAL, ELFVis.DEFAULT, ELFNDX.Section(15), "$d"),
-    ELFSymbol(87, BigInt("4291400"), 0, ELFSymType.NOTYPE, ELFBind.LOCAL, ELFVis.DEFAULT, ELFNDX.Section(17), "$d"),
     ELFSymbol(88, BigInt("0"), 0, ELFSymType.FILE, ELFBind.LOCAL, ELFVis.DEFAULT, ELFNDX.ABS, "http.c"),
-    ELFSymbol(89, BigInt("4273728"), 0, ELFSymType.NOTYPE, ELFBind.LOCAL, ELFVis.DEFAULT, ELFNDX.Section(15), "$d"),
-    ELFSymbol(90, BigInt("4227024"), 0, ELFSymType.NOTYPE, ELFBind.LOCAL, ELFVis.DEFAULT, ELFNDX.Section(13), "$x"),
-    ELFSymbol(91, BigInt("4291768"), 0, ELFSymType.NOTYPE, ELFBind.LOCAL, ELFVis.DEFAULT, ELFNDX.Section(17), "$d"),
     ELFSymbol(92, BigInt("0"), 0, ELFSymType.FILE, ELFBind.LOCAL, ELFVis.DEFAULT, ELFNDX.ABS, "ntlm.c"),
-    ELFSymbol(93, BigInt("4237952"), 0, ELFSymType.NOTYPE, ELFBind.LOCAL, ELFVis.DEFAULT, ELFNDX.Section(13), "$x"),
     ELFSymbol(94, BigInt("4237952"), 148, ELFSymType.FUNC, ELFBind.LOCAL, ELFVis.DEFAULT, ELFNDX.Section(13), "ntlm_set_key"),
-    ELFSymbol(95, BigInt("4280912"), 0, ELFSymType.NOTYPE, ELFBind.LOCAL, ELFVis.DEFAULT, ELFNDX.Section(15), "$d"),
-    ELFSymbol(96, BigInt("4285040"), 0, ELFSymType.NOTYPE, ELFBind.LOCAL, ELFVis.DEFAULT, ELFNDX.Section(15), "$d"),
-    ELFSymbol(97, BigInt("4293248"), 0, ELFSymType.NOTYPE, ELFBind.LOCAL, ELFVis.DEFAULT, ELFNDX.Section(17), "$d"),
     ELFSymbol(98, BigInt("0"), 0, ELFSymType.FILE, ELFBind.LOCAL, ELFVis.DEFAULT, ELFNDX.ABS, "pages.c"),
-    ELFSymbol(99, BigInt("4281368"), 0, ELFSymType.NOTYPE, ELFBind.LOCAL, ELFVis.DEFAULT, ELFNDX.Section(15), "$d"),
-    ELFSymbol(100, BigInt("4241920"), 0, ELFSymType.NOTYPE, ELFBind.LOCAL, ELFVis.DEFAULT, ELFNDX.Section(13), "$x"),
-    ELFSymbol(101, BigInt("4293536"), 0, ELFSymType.NOTYPE, ELFBind.LOCAL, ELFVis.DEFAULT, ELFNDX.Section(17), "$d"),
     ELFSymbol(102, BigInt("0"), 0, ELFSymType.FILE, ELFBind.LOCAL, ELFVis.DEFAULT, ELFNDX.ABS, "proxy.c"),
-    ELFSymbol(103, BigInt("4242304"), 0, ELFSymType.NOTYPE, ELFBind.LOCAL, ELFVis.DEFAULT, ELFNDX.Section(13), "$x"),
-    ELFSymbol(104, BigInt("4282232"), 0, ELFSymType.NOTYPE, ELFBind.LOCAL, ELFVis.DEFAULT, ELFNDX.Section(15), "$d"),
-    ELFSymbol(105, BigInt("4391984"), 0, ELFSymType.NOTYPE, ELFBind.LOCAL, ELFVis.DEFAULT, ELFNDX.Section(24), "$d"),
-    ELFSymbol(106, BigInt("4293688"), 0, ELFSymType.NOTYPE, ELFBind.LOCAL, ELFVis.DEFAULT, ELFNDX.Section(17), "$d"),
     ELFSymbol(107, BigInt("0"), 0, ELFSymType.FILE, ELFBind.LOCAL, ELFVis.DEFAULT, ELFNDX.ABS, "scanner.c"),
-    ELFSymbol(108, BigInt("4274480"), 0, ELFSymType.NOTYPE, ELFBind.LOCAL, ELFVis.DEFAULT, ELFNDX.Section(15), "$d"),
-    ELFSymbol(109, BigInt("4246752"), 0, ELFSymType.NOTYPE, ELFBind.LOCAL, ELFVis.DEFAULT, ELFNDX.Section(13), "$x"),
-    ELFSymbol(110, BigInt("4294432"), 0, ELFSymType.NOTYPE, ELFBind.LOCAL, ELFVis.DEFAULT, ELFNDX.Section(17), "$d"),
     ELFSymbol(111, BigInt("0"), 0, ELFSymType.FILE, ELFBind.LOCAL, ELFVis.DEFAULT, ELFNDX.ABS, "socket.c"),
-    ELFSymbol(112, BigInt("4275384"), 0, ELFSymType.NOTYPE, ELFBind.LOCAL, ELFVis.DEFAULT, ELFNDX.Section(15), "$d"),
-    ELFSymbol(113, BigInt("4250528"), 0, ELFSymType.NOTYPE, ELFBind.LOCAL, ELFVis.DEFAULT, ELFNDX.Section(13), "$x"),
-    ELFSymbol(114, BigInt("4294536"), 0, ELFSymType.NOTYPE, ELFBind.LOCAL, ELFVis.DEFAULT, ELFNDX.Section(17), "$d"),
     ELFSymbol(115, BigInt("0"), 0, ELFSymType.FILE, ELFBind.LOCAL, ELFVis.DEFAULT, ELFNDX.ABS, "utils.c"),
-    ELFSymbol(116, BigInt("4284408"), 0, ELFSymType.NOTYPE, ELFBind.LOCAL, ELFVis.DEFAULT, ELFNDX.Section(15), "$d"),
-    ELFSymbol(117, BigInt("4252464"), 0, ELFSymType.NOTYPE, ELFBind.LOCAL, ELFVis.DEFAULT, ELFNDX.Section(13), "$x"),
-    ELFSymbol(118, BigInt("4285328"), 0, ELFSymType.NOTYPE, ELFBind.LOCAL, ELFVis.DEFAULT, ELFNDX.Section(15), "$d"),
-    ELFSymbol(119, BigInt("4285360"), 0, ELFSymType.NOTYPE, ELFBind.LOCAL, ELFVis.DEFAULT, ELFNDX.Section(15), "$d"),
     ELFSymbol(120, BigInt("4285360"), 17, ELFSymType.OBJECT, ELFBind.LOCAL, ELFVis.DEFAULT, ELFNDX.Section(15), "hextab"),
     ELFSymbol(121, BigInt("4285392"), 512, ELFSymType.OBJECT, ELFBind.LOCAL, ELFVis.DEFAULT, ELFNDX.Section(15), "hexindex"),
     ELFSymbol(122, BigInt("4285904"), 64, ELFSymType.OBJECT, ELFBind.LOCAL, ELFVis.DEFAULT, ELFNDX.Section(15), "base64"),
     ELFSymbol(123, BigInt("4285968"), 512, ELFSymType.OBJECT, ELFBind.LOCAL, ELFVis.DEFAULT, ELFNDX.Section(15), "index64"),
-    ELFSymbol(124, BigInt("4294984"), 0, ELFSymType.NOTYPE, ELFBind.LOCAL, ELFVis.DEFAULT, ELFNDX.Section(17), "$d"),
     ELFSymbol(125, BigInt("0"), 0, ELFSymType.FILE, ELFBind.LOCAL, ELFVis.DEFAULT, ELFNDX.ABS, "xcrypt.c"),
-    ELFSymbol(126, BigInt("4258400"), 0, ELFSymType.NOTYPE, ELFBind.LOCAL, ELFVis.DEFAULT, ELFNDX.Section(13), "$x"),
     ELFSymbol(127, BigInt("4258400"), 812, ELFSymType.FUNC, ELFBind.LOCAL, ELFVis.DEFAULT, ELFNDX.Section(13), "des_key_schedule"),
-    ELFSymbol(128, BigInt("4285344"), 0, ELFSymType.NOTYPE, ELFBind.LOCAL, ELFVis.DEFAULT, ELFNDX.Section(15), "$d"),
-    ELFSymbol(129, BigInt("4286480"), 0, ELFSymType.NOTYPE, ELFBind.LOCAL, ELFVis.DEFAULT, ELFNDX.Section(15), "$d"),
     ELFSymbol(130, BigInt("4286480"), 64, ELFSymType.OBJECT, ELFBind.LOCAL, ELFVis.DEFAULT, ELFNDX.Section(15), "leftkey_swap"),
     ELFSymbol(131, BigInt("4286544"), 64, ELFSymType.OBJECT, ELFBind.LOCAL, ELFVis.DEFAULT, ELFNDX.Section(15), "rightkey_swap"),
     ELFSymbol(132, BigInt("4286608"), 16, ELFSymType.OBJECT, ELFBind.LOCAL, ELFVis.DEFAULT, ELFNDX.Section(15), "encrypt_rotate_tab"),
@@ -143,15 +72,12 @@
     ELFSymbol(140, BigInt("4288672"), 256, ELFSymType.OBJECT, ELFBind.LOCAL, ELFVis.DEFAULT, ELFNDX.Section(15), "sbox3"),
     ELFSymbol(141, BigInt("4288928"), 256, ELFSymType.OBJECT, ELFBind.LOCAL, ELFVis.DEFAULT, ELFNDX.Section(15), "sbox1"),
     ELFSymbol(142, BigInt("4289184"), 64, ELFSymType.OBJECT, ELFBind.LOCAL, ELFVis.DEFAULT, ELFNDX.Section(15), "fillbuf"),
-    ELFSymbol(143, BigInt("4296880"), 0, ELFSymType.NOTYPE, ELFBind.LOCAL, ELFVis.DEFAULT, ELFNDX.Section(17), "$d"),
     ELFSymbol(144, BigInt("0"), 0, ELFSymType.FILE, ELFBind.LOCAL, ELFVis.DEFAULT, ELFNDX.ABS, "crtend.o"),
-    ELFSymbol(145, BigInt("4297916"), 0, ELFSymType.NOTYPE, ELFBind.LOCAL, ELFVis.DEFAULT, ELFNDX.Section(17), "$d"),
     ELFSymbol(146, BigInt("4297916"), 0, ELFSymType.OBJECT, ELFBind.LOCAL, ELFVis.DEFAULT, ELFNDX.Section(17), "__FRAME_END__"),
     ELFSymbol(147, BigInt("0"), 0, ELFSymType.FILE, ELFBind.LOCAL, ELFVis.DEFAULT, ELFNDX.ABS, ""),
     ELFSymbol(148, BigInt("4390360"), 0, ELFSymType.OBJECT, ELFBind.LOCAL, ELFVis.DEFAULT, ELFNDX.Section(20), "_DYNAMIC"),
     ELFSymbol(149, BigInt("4289248"), 0, ELFSymType.NOTYPE, ELFBind.LOCAL, ELFVis.DEFAULT, ELFNDX.Section(16), "__GNU_EH_FRAME_HDR"),
     ELFSymbol(150, BigInt("4390856"), 0, ELFSymType.OBJECT, ELFBind.LOCAL, ELFVis.DEFAULT, ELFNDX.Section(21), "_GLOBAL_OFFSET_TABLE_"),
-    ELFSymbol(151, BigInt("4202624"), 0, ELFSymType.NOTYPE, ELFBind.LOCAL, ELFVis.DEFAULT, ELFNDX.Section(12), "$x"),
     ELFSymbol(152, BigInt("0"), 0, ELFSymType.FUNC, ELFBind.GLOBAL, ELFVis.DEFAULT, ELFNDX.UND, "memcpy"),
     ELFSymbol(153, BigInt("4268448"), 1268, ELFSymType.FUNC, ELFBind.GLOBAL, ELFVis.DEFAULT, ELFNDX.Section(13), "hmac_md5"),
     ELFSymbol(154, BigInt("4242112"), 80, ELFSymType.FUNC, ELFBind.GLOBAL, ELFVis.DEFAULT, ELFNDX.Section(13), "gen_denied_page"),
@@ -479,8 +405,6 @@
     ExternalFunction("memset", BigInt("4391176")),
     ExternalFunction("open", BigInt("4391160")),
     ExternalFunction("openlog", BigInt("4391320")),
-    ExternalFunction("optarg", BigInt("4391744")),
-    ExternalFunction("optind", BigInt("4391752")),
     ExternalFunction("perror", BigInt("4390984")),
     ExternalFunction("printf", BigInt("4391616")),
     ExternalFunction("pthread_attr_destroy", BigInt("4391168")),
@@ -508,9 +432,6 @@
     ExternalFunction("snprintf", BigInt("4391080")),
     ExternalFunction("socket", BigInt("4391504")),
     ExternalFunction("srandom", BigInt("4391568")),
-    ExternalFunction("stderr", BigInt("4391728")),
-    ExternalFunction("stdin", BigInt("4391776")),
-    ExternalFunction("stdout", BigInt("4391760")),
     ExternalFunction("strcasecmp", BigInt("4391232")),
     ExternalFunction("strchr", BigInt("4391464")),
     ExternalFunction("strcmp", BigInt("4391376")),

Conflicts: src/main/scala/specification/Specification.scala

Conflicts: src/test/scala/IrreducibleLoop.scala

ailrst · 2025-07-04T01:24:54Z

Can we enable this when no relf is provided using gtirb (probably keep a warning saying its experimental)

katrinafyi · 2025-07-04T01:28:51Z

Yeah sure. You can make this change if you want. the system tests are still turning up some mismatch warnings, but we can file bugs and fix those later.

katrinafyi · 2025-07-08T04:31:59Z

We're still seeing mismatches between gtirb's ELF information and readelf, but only in some test cases. for example:

- incorrect/nestedifglobal/gcc:GTIRB (1 second, 152 milliseconds)
  + [WARN]   PLEASE REPORT THIS ISSUE! include the gts and relf files. gtirb relf discrepancy, function entries differ:
         gtirb - relf = Set(FuncEntry(main,1056,1812))
         relf - gtirb = Set(FuncEntry(main,992,1812)) [checkReadELFCompatibility@GTIRBReadELF.scala:261] 
  + [WARN]   PLEASE REPORT THIS ISSUE! include the gts and relf files. gtirb relf discrepancy, global variables differ:
         gtirb - relf = Set(SpecGlobal(_IO_stdin_used,32,None,1964))
         relf - gtirb = Set(SpecGlobal(_IO_stdin_used,32,None,1956)) [checkReadELFCompatibility@GTIRBReadELF.scala:263] 
  + [WARN]   PLEASE REPORT THIS ISSUE! include the gts and relf files. gtirb relf discrepancy, symbol tables differ:
         gtirb - relf = HashSet(ELFSymbol(59,2192,0,OBJECT,LOCAL,DEFAULT,Section(17),__FRAME_END__), ELFSymbol(80,1964,4,OBJECT,GLOBAL,DEFAULT,Section(15),_IO_stdin_used), ELFSymbol(62,1968,0,NOTYPE,LOCAL,DEFAULT,Section(16),__GNU_EH_FRAME_HDR), ELFSymbol(74,1944,0,FUNC,GLOBAL,HIDDEN,Section(14),_fini), ELFSymbol(85,1812,132,FUNC,GLOBAL,DEFAULT,Section(13),main))
         relf - gtirb = HashSet(ELFSymbol(85,1812,124,FUNC,GLOBAL,DEFAULT,Section(13),main), ELFSymbol(62,1960,0,NOTYPE,LOCAL,DEFAULT,Section(16),__GNU_EH_FRAME_HDR), ELFSymbol(80,1956,4,OBJECT,GLOBAL,DEFAULT,Section(15),_IO_stdin_used), ELFSymbol(59,2184,0,OBJECT,LOCAL,DEFAULT,Section(17),__FRAME_END__), ELFSymbol(74,1936,0,FUNC,GLOBAL,HIDDEN,Section(14),_fini)) [checkReadELFCompatibility@GTIRBReadELF.scala:265] 

 - incorrect/nestedifglobal/clang_pic:GTIRB (1 second, 481 milliseconds)
  + [WARN]   PLEASE REPORT THIS ISSUE! include the gts and relf files. gtirb relf discrepancy, function entries differ:
         gtirb - relf = Set(FuncEntry(main,1312,1876))
         relf - gtirb = Set(FuncEntry(main,1248,1876)) [checkReadELFCompatibility@GTIRBReadELF.scala:261] 
  + [WARN]   PLEASE REPORT THIS ISSUE! include the gts and relf files. gtirb relf discrepancy, global variables differ:
         gtirb - relf = Set(SpecGlobal(_IO_stdin_used,32,None,2060))
         relf - gtirb = Set(SpecGlobal(_IO_stdin_used,32,None,2052)) [checkReadELFCompatibility@GTIRBReadELF.scala:263] 
  + [WARN]   PLEASE REPORT THIS ISSUE! include the gts and relf files. gtirb relf discrepancy, symbol tables differ:
         gtirb - relf = HashSet(ELFSymbol(87,1876,164,FUNC,GLOBAL,DEFAULT,Section(13),main), ELFSymbol(61,2308,0,OBJECT,LOCAL,DEFAULT,Section(17),__FRAME_END__), ELFSymbol(76,2040,0,FUNC,GLOBAL,HIDDEN,Section(14),_fini), ELFSymbol(82,2060,4,OBJECT,GLOBAL,DEFAULT,Section(15),_IO_stdin_used), ELFSymbol(64,2064,0,NOTYPE,LOCAL,DEFAULT,Section(16),__GNU_EH_FRAME_HDR))
         relf - gtirb = HashSet(ELFSymbol(87,1876,156,FUNC,GLOBAL,DEFAULT,Section(13),main), ELFSymbol(64,2056,0,NOTYPE,LOCAL,DEFAULT,Section(16),__GNU_EH_FRAME_HDR), ELFSymbol(82,2052,4,OBJECT,GLOBAL,DEFAULT,Section(15),_IO_stdin_used), ELFSymbol(76,2032,0,FUNC,GLOBAL,HIDDEN,Section(14),_fini), ELFSymbol(61,2300,0,OBJECT,LOCAL,DEFAULT,Section(17),__FRAME_END__)) [checkReadELFCompatibility@GTIRBReadELF.scala:265] 

- incorrect/nestedifglobal/gcc_pic:GTIRB (1 second, 246 milliseconds)
  + [WARN]   PLEASE REPORT THIS ISSUE! include the gts and relf files. gtirb relf discrepancy, function entries differ:
         gtirb - relf = Set(FuncEntry(main,1056,1876))
         relf - gtirb = Set(FuncEntry(main,992,1876)) [checkReadELFCompatibility@GTIRBReadELF.scala:261] 
  + [WARN]   PLEASE REPORT THIS ISSUE! include the gts and relf files. gtirb relf discrepancy, global variables differ:
         gtirb - relf = Set(SpecGlobal(_IO_stdin_used,32,None,2028))
         relf - gtirb = Set(SpecGlobal(_IO_stdin_used,32,None,2020)) [checkReadELFCompatibility@GTIRBReadELF.scala:263] 
  + [WARN]   PLEASE REPORT THIS ISSUE! include the gts and relf files. gtirb relf discrepancy, symbol tables differ:
         gtirb - relf = HashSet(ELFSymbol(80,2028,4,OBJECT,GLOBAL,DEFAULT,Section(15),_IO_stdin_used), ELFSymbol(74,2008,0,FUNC,GLOBAL,HIDDEN,Section(14),_fini), ELFSymbol(62,2032,0,NOTYPE,LOCAL,DEFAULT,Section(16),__GNU_EH_FRAME_HDR), ELFSymbol(85,1876,132,FUNC,GLOBAL,DEFAULT,Section(13),main), ELFSymbol(59,2256,0,OBJECT,LOCAL,DEFAULT,Section(17),__FRAME_END__))
         relf - gtirb = HashSet(ELFSymbol(74,2000,0,FUNC,GLOBAL,HIDDEN,Section(14),_fini), ELFSymbol(80,2020,4,OBJECT,GLOBAL,DEFAULT,Section(15),_IO_stdin_used), ELFSymbol(62,2024,0,NOTYPE,LOCAL,DEFAULT,Section(16),__GNU_EH_FRAME_HDR), ELFSymbol(59,2248,0,OBJECT,LOCAL,DEFAULT,Section(17),__FRAME_END__), ELFSymbol(85,1876,124,FUNC,GLOBAL,DEFAULT,Section(13),main)) [checkReadELFCompatibility@GTIRBReadELF.scala:265]

I feel like this is possibly an issue of the checked-in gts and relf files diverging. Point Basil to the test files from #288, these test cases do not exhibit the mismatch. The Git history also shows the gts files were updated separately from the other files and with large gaps in time. This seems suspicious, and I'd really like to have something like #288 to give us some assurances here.

Conflicts: build.mill src/main/scala/gtirb/AuxDecoder.scala

the argument works in this way: ``` # --load-directory-* defaults to using .relf file if present ./mill -w run --load-directory-gtirb src/test/incorrect/nestedifglobal/clang # ... if relf and gts is given, --gts-relf directs Basil to use gtirb-as-relf ./mill -w run --load-directory-gtirb src/test/incorrect/nestedifglobal/clang --gts-relf # if only gtirb input is given, Basil will also use that for elf. ./mill -w run -i src/test/incorrect/nestedifglobal/clang/nestedifglobal.gts ```

ailrst

lgtm

katrinafyi added 30 commits June 20, 2025 14:14

case class readelf, toscala readelf, --dump-relf

595756a

we can get external function symbols. TODO: dig into auxdata

ba5dec4

new AuxDecoder for compositional decoding of AuxData

167325f

the previous MapDecoder is hard-coded to decode only certain structures in the aux_data. this makes it more compositional, so we can decode more complex types across more auxdata fields.

use auxdecoder in gtirb loader

7397b88

scalafmt

efdf667

i cant tell if this is shorter

19df41e

foldLeft1 is just reduce - allie

191730d

Merge remote-tracking branch 'origin/mapdecoder2' into gtirb-relf

fe91ff6

what the fkkkkkkkkkkkkkkkkkk

b94a1a4

kaitai https://formats.kaitai.io/elf/java.html

cebe023

maybe we just vendor the java file lol

kaitai integration. ASS BECAUSE SCALA [no ci]

31e0cfe

companion objects are awful. i can't access the java public static inner class ;-;

Revert "kaitai integration. ASS BECAUSE SCALA [no ci]"

f67df70

This reverts commit 31e0cfe.

Revert "kaitai https://formats.kaitai.io/elf/java.html"

6adf92d

This reverts commit cebe023.

groupMapReduce

677bea2

change r_type to long, add docs. BTW, ...

1f80736

the static class problem in 31e0cfe was me being silly

unfold

856616c

a lot more work. all parts from readelfloader work now

1d280c8

ToScala: deriving creates multi-line output sometimes

805342f

scalafmt

8f9911f

tehe revert me

3c45043

macros lol

91d84c0

auto tupling??

5f86644

ReadTuple

e6e38a5

move

1446863

resolver

018da5f

Revert "tehe revert me"

0e38d1b

This reverts commit 3c45043. Conflicts: src/main/scala/translating/GTIRBReadELF.scala

Merge remote-tracking branch 'origin/main' into gtirb-relf

a2e49bd

scalafmt

c7e98f6

Merge remote-tracking branch 'origin/main' into gtirb-relf

615f9bb

Conflicts: src/main/scala/gtirb/AuxDecoder.scala src/main/scala/translating/GTIRBToIR.scala

allow string in Uuid inputs

07fc539

katrinafyi added 2 commits June 23, 2025 23:31

no println

10f68c2

scalafmt

bedc54d

.sorted method for ReadELFData

28cb1a8

ailrst approved these changes Jun 25, 2025

View reviewed changes

katrinafyi added 3 commits June 26, 2025 15:26

working on R_AARCH64_COPY. add SECTION entries too

f0de62e

fix relocated global objects

bb019a1

checkset, remove -100 symbols

9aa41b4

katrinafyi added 7 commits June 26, 2025 16:10

scalafmt

1052400

urls in doc comments

53275d3

type annotations

62cb830

Merge remote-tracking branch 'origin/main' into gtirb-relf

d6b688a

Conflicts: src/main/scala/specification/Specification.scala

format

3b99169

Merge remote-tracking branch 'origin/main' into gtirb-relf

698d42d

Conflicts: src/test/scala/IrreducibleLoop.scala

reformat

e4afa50

ailrst approved these changes Jul 3, 2025

View reviewed changes

katrinafyi added 2 commits July 8, 2025 14:33

Merge remote-tracking branch 'origin/main' into gtirb-relf

fd8500b

Conflicts: build.mill src/main/scala/gtirb/AuxDecoder.scala

touch scaladoc mappings

b7a5165

katrinafyi requested a review from ailrst July 8, 2025 05:52

katrinafyi force-pushed the gtirb-relf branch from 2b31754 to 71e1cf5 Compare July 8, 2025 05:53

ailrst approved these changes Jul 9, 2025

View reviewed changes

katrinafyi merged commit a5ac9be into main Jul 9, 2025
12 checks passed

katrinafyi deleted the gtirb-relf branch July 9, 2025 01:47

katrinafyi linked an issue Jul 16, 2025 that may be closed by this pull request

ReadELF/gts-relf discrepancies megathread #509

Closed

j-tobler mentioned this pull request Aug 14, 2025

Runner cleanup #449

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Load ELF symbol information from GTIRB and compare to readelf#477

Load ELF symbol information from GTIRB and compare to readelf#477
katrinafyi merged 57 commits intomainfrom
gtirb-relf

katrinafyi commented Jun 23, 2025 •

edited

Loading

Uh oh!

ailrst commented Jun 25, 2025

Uh oh!

ailrst left a comment •

edited

Loading

Uh oh!

ailrst Jun 25, 2025

Uh oh!

ailrst Jun 25, 2025

Uh oh!

ailrst Jun 25, 2025

Uh oh!

ailrst Jun 25, 2025

Uh oh!

katrinafyi commented Jun 26, 2025

Uh oh!

ailrst commented Jul 4, 2025

Uh oh!

katrinafyi commented Jul 4, 2025 •

edited

Loading

Uh oh!

katrinafyi commented Jul 8, 2025

Uh oh!

ailrst left a comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants


		// Full ELF32 specification: https://refspecs.linuxfoundation.org/elf/elf.pdf

		// Full ELF64 specification: https://irix7.com/techpubs/007-4658-001.pdf

Conversation

katrinafyi commented Jun 23, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

testing

bonus things

Uh oh!

ailrst commented Jun 25, 2025

Uh oh!

ailrst left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

ailrst Jun 25, 2025

Choose a reason for hiding this comment

Uh oh!

ailrst Jun 25, 2025

Choose a reason for hiding this comment

Uh oh!

ailrst Jun 25, 2025

Choose a reason for hiding this comment

Uh oh!

ailrst Jun 25, 2025

Choose a reason for hiding this comment

Uh oh!

katrinafyi commented Jun 26, 2025

Uh oh!

ailrst commented Jul 4, 2025

Uh oh!

katrinafyi commented Jul 4, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

katrinafyi commented Jul 8, 2025

Uh oh!

ailrst left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

katrinafyi commented Jun 23, 2025 •

edited

Loading

ailrst left a comment •

edited

Loading

katrinafyi commented Jul 4, 2025 •

edited

Loading