914 Fix stacktrace symbols when dladdr fails

[lifflander]

Fixes #914

[codecov]

Codecov Report

Merging #915 into develop will not change coverage.
The diff coverage is n/a.

@@           Coverage Diff            @@
##           develop     #915   +/-   ##
========================================
  Coverage    82.79%   82.79%           
========================================
  Files          356      356           
  Lines        11232    11232           
========================================
  Hits          9300     9300           
  Misses        1932     1932           

[PhilMiller]

Before going down this path, why not try out libbacktrace? It's part of gcc, but it's BSD licensed

[lifflander]

Before going down this path, why not try out libbacktrace? It's part of gcc, but it's BSD licensed

I considered that, but it's another dependency (which, by the way, doesn't have cmake). Our current stack traces work correctly except for extracting the symbols for some gcc versions. I remedy that by calling addr2line or atos depending on the platform. I don't see a strong advantage to using an external library compared to the existing simpler solution.

[PhilMiller]

In a process failure setting, especially, calling popen seems super sketchy. That's a fork() and exec(), per frame, when we may have already blown up memory.

Given the license, libbacktrace could be vendored into our source tree as a stand-alone piece.

There's also boost stacktrace, which again, could be imported into our tree if we didn't want an external dependency.

[lifflander]

In a process failure setting, especially, calling popen seems super sketchy. That's a fork() and exec(), per frame, when we may have already blown up memory.

Given the license, libbacktrace could be vendored into our source tree as a stand-alone piece.

There's also boost stacktrace, which again, could be imported into our tree if we didn't want an external dependency.

Yes, I'm familiar. There's a proposal to put stack traces in the standard that follows the boost implementation:

http://www.open-std.org/jtc1/sc22/wg21/docs/papers/2019/p0881r5.html

[PhilMiller]

Plus, with an external binary, there's no guarantee that its presence in the build environment means it's available in the execution environment.

[lifflander]

Plus, with an external binary, there's no guarantee that its presence in the build environment means it's available in the execution environment.

Another option would be to put addr2line in lib. It's a simple utility with about 400 lines of code. Do you think that libbacktrace compiles everywhere we need it?

[PhilMiller]

addr2line is GPL licensed as part of GNU binutils, and depends on libbfd, which is also part of the GPL licensed GNU binutils

[PhilMiller]

Do you think that libbacktrace compiles everywhere we need it?

'Everywhere we need it' is relatively modern Linux and macOS? It has ELF and Mach-O support, so we should be good.

[PhilMiller]

For a separate concern with anything using popen of an external binary, how do MPI, Kokkos (including OpenMP usage as well), and CUDA feel about fork() calls?

[lifflander]

Here is a project that adds cmake support for libbacktrace with an older version. We should use it's cmake with the latest libbacktrace code:
https://github.com/ErwanLegrand/libbacktrace

There is also a PR open in the base libbacktrace for adding cmake support, but it is incomplete and it refers to this other fork:

ianlancetaylor/libbacktrace#19

edit: actually someone tried to use it with the newer libbacktrace code and was more trouble than it was worth

[lifflander]

The boost stacktrace library relies on a lot of boostisms and macros and might be hard to extract. It also doesn't have cmake:

https://github.com/boostorg/stacktrace

[lifflander]

The other thing to note is that most these libraries depend on a low-level system function: _Unwind_Backtrace (and _Unwind_GetIP in some cases). Boost stack trace is really just a very pretty wrapper.

[lifflander]

Rust supports backtraces and has recently moved away from libbacktrace due to several portability and maintainability issues outlined here: rust-lang/rust#73441

Their new solution is rolled from the ground up and it written in rust (not super helpful): https://github.com/gimli-rs

[lifflander]

@PhilMiller Actually, I might have just found a really good alternative with cmake support: https://github.com/bombela/backward-cpp

[PhilMiller]

Here is a project that adds cmake support for libbacktrace with an older version. We should use it's cmake with the latest libbacktrace code:
https://github.com/ErwanLegrand/libbacktrace

There is also a PR open in the base libbacktrace for adding cmake support, but it is incomplete and it refers to this other fork:

ianlancetaylor/libbacktrace#19

edit: actually someone tried to use it with the newer libbacktrace code and was more trouble than it was worth

Ian Lance Taylor is the upstream maintainer of libbacktrace. I suspect that if you gave that PR a nudge, possibly with suitable testing report, that it would move ahead.

[lifflander]

Here is a project that adds cmake support for libbacktrace with an older version. We should use it's cmake with the latest libbacktrace code:
https://github.com/ErwanLegrand/libbacktrace
There is also a PR open in the base libbacktrace for adding cmake support, but it is incomplete and it refers to this other fork:
ianlancetaylor/libbacktrace#19
edit: actually someone tried to use it with the newer libbacktrace code and was more trouble than it was worth

Ian Lance Taylor is the upstream maintainer of libbacktrace. I suspect that if you gave that PR a nudge, possibly with suitable testing report, that it would move ahead.

I'm now thinking that https://github.com/bombela/backward-cpp might be the best library to embed. It's more modern, handles reading dwarf, has wide support across many platforms, and an MIT license.

[PhilMiller]

Backward depends on either libbfd or libdw from elfutils. The former is GPL, the latter is dual GPL/LGPL. Either way, it's doing the pretty-printing, not the actual DWARF-reading.

[lifflander]

Backward depends on either libbfd or libdw from elfutils. The former is GPL, the latter is dual GPL/LGPL. Either way, it's doing the pretty-printing, not the actual DWARF-reading.

We don't need to include those in the source, right? It makes the call to those libraries to read the symbols (that's what I meant): load_object_with_dwarf.

I'm thinking we embed backward; we will only need those libs for the pretty backtrace.

You will need to install some dependencies to get the ultimate stack trace. Two libraries are currently supported, the only difference is which one is the easiest for you to install, so pick your poison:

[PhilMiller]

Re -rdynamic: https://stackoverflow.com/a/8624223/90002 See point number 4 at the bottom about static linking

[PhilMiller]

How about libunwind:
https://www.nongnu.org/libunwind/man/libunwind(3).html
https://github.com/libunwind/libunwind
MIT licensed, seemingly standalone

[PhilMiller]

How about libunwind:
https://www.nongnu.org/libunwind/man/libunwind(3).html
https://github.com/libunwind/libunwind
MIT licensed, seemingly standalone

Looks like it will similarly be limited in symbol naming by lack of .dynsym as the general backtrace()

[lifflander]

@PhilMiller I have -rdynamic working now and the stack traces are printing correctly on gcc.

[PhilMiller]

Even with fully static builds? That's great news.

[PhilMiller]

I'm +1 on passing the -rdynamic flag as described, but can't at all knowledgeably comment on the CMake aspects of how you're doing this. Probably good to get someone who knows it better to look at that.

[lifflander]

@nlslatt Can you look at the cmake here?

[PhilMiller]

OK, CMake's policy and variable scoping rules are really weird, but they make good sense for the use case.

For anyone else following along:

The include(CheckLinkerFlag) pushes a fresh frame on the 'policy stack', and the end of the file pops that frame. Thus, the cmake_policy command only applies inside that file.

The function command captures the top of the policy stack, and all invocations of the function are evaluated with that policy set in place.

Files input and calls to functions each create their own variable scope. So, the set only applies within the function.