bmcweb: Log throw-site stack on std::terminate Uncaught exceptions in gbmcweb reach std::terminate only after asio's io_context and the sdbusplus posted-rethrow loop have already unwound the stack, so the default terminate handler prints a backtrace that no longer contains the original throw site. This makes fleet crashes from stray exceptions very hard to triage. Interpose __cxa_throw (the folly exception_tracer technique, reduced to a single self-contained translation unit) to record the throw-site stack of every exception, keyed by the exception object address, and install a std::terminate handler that looks the record up via the in-flight exception_ptr and logs it. The terminate path performs no allocation, takes no locks and cannot re-enter, since it may run under suspected heap corruption. For the interposer to record throws from libstdc++ and other shared objects -- not just the executable's own code -- it must win dynamic symbol resolution against libstdc++'s __cxa_throw. Export it with default visibility plus a linker --dynamic-list, and compile the tracer translation unit directly into the executable so -Wl,--exclude-libs,ALL cannot force it back to a local, non-interposing symbol. Without this the interposer only captures throws whose __cxa_throw call site is compiled into the executable, and misses standard-library and cross-library throws. A runtime kill switch (--exception_stack_tracer=false) disables recording and keeps the default terminate handler. The tracer depends on glibc backtrace(), libgcc unwinding and the Itanium __cxa_throw ABI, so it is built only in the meson/yocto (gcc) image. Every reference to it from webserver_main_setup is guarded by BMCWEB_ENABLE_EXCEPTION_STACK_TRACER, which is defined only by meson, so the google3/blaze build carries no dependency on the tracer. The tracer source, header and test are GoB-only. Tested: - Built gbmcweb and ran the unit tests inside the gbmcweb Gerrit presubmit gcc image (openbmc-ubuntu-unit-test) with the presubmit meson flags; exception_stack_tracer_test passes 4/4. - Verified the guarded references compile out when BMCWEB_ENABLE_EXCEPTION_STACK_TRACER is undefined. - On an emulated BMC image in Mimik (QEMU), using an out-of-tree debug-only endpoint that throws from a separate shared library through asio's inline executor (catch + async rethrow), confirmed the terminate handler logged the throw-site stack and that it symbolized across both modules. Before exporting the interposer the same cross-module throw was not recorded; after, it is. Verified with readelf that __cxa_throw is GLOBAL DEFAULT in the executable's dynamic symbol table. Google-Bug-Id: 553672958 Signed-off-by: Yuli Fiterman <fiterman@google.com> Change-Id: Ib461c25ffd44188707b628fad9d85f08ef516549
diff --git a/exception_stack_tracer.dynlist b/exception_stack_tracer.dynlist new file mode 100644 index 0000000..c7a6f59 --- /dev/null +++ b/exception_stack_tracer.dynlist
@@ -0,0 +1,8 @@ +# Force-export just the exception stack tracer's __cxa_throw interposer into the +# executable's dynamic symbol table so it interposes throws from ALL modules +# (libstdc++ and other shared objects), overriding the default -fvisibility=hidden +# / --exclude-libs,ALL that would otherwise leave it a LOCAL symbol. Nothing else +# is exported. +{ + __cxa_throw; +};
diff --git a/include/exception_stack_tracer.hpp b/include/exception_stack_tracer.hpp new file mode 100644 index 0000000..95fd998 --- /dev/null +++ b/include/exception_stack_tracer.hpp
@@ -0,0 +1,23 @@ +#ifndef THIRD_PARTY_GBMCWEB_INCLUDE_EXCEPTION_STACK_TRACER_HPP_ +#define THIRD_PARTY_GBMCWEB_INCLUDE_EXCEPTION_STACK_TRACER_HPP_ + +// Installs a std::terminate handler that prints the ORIGINAL throw-site stack +// of the uncaught exception, recorded at throw time by a __cxa_throw +// interposer (folly exception_tracer mechanism). Exceptions captured into a +// std::exception_ptr and rethrown later (asio scheduler +// rethrow_pending_exception, sdbusplus do_unpack posted rethrow) otherwise +// terminate with a stack that only shows the rethrow site. +// +// The terminate path never allocates, takes no locks, and guards against +// re-entry; see src/exception_stack_tracer.cpp for the full constraints. +// +// The interposer is active as soon as its TU is linked in; this call installs +// the terminate handler and snapshots the module table used to symbolize +// frames. Call it first thing in run() so nothing on the crash path +// allocates, dlopens, or takes the loader lock. +// +// With enable == false (--exception_stack_tracer=false kill switch), +// recording becomes a no-op and the default terminate handler is kept. +void installExceptionStackTracer(bool enable = true); + +#endif // THIRD_PARTY_GBMCWEB_INCLUDE_EXCEPTION_STACK_TRACER_HPP_
diff --git a/include/exception_stack_tracer_internal.hpp b/include/exception_stack_tracer_internal.hpp new file mode 100644 index 0000000..f4a2bbf --- /dev/null +++ b/include/exception_stack_tracer_internal.hpp
@@ -0,0 +1,25 @@ +#ifndef THIRD_PARTY_GBMCWEB_INCLUDE_EXCEPTION_STACK_TRACER_INTERNAL_HPP_ +#define THIRD_PARTY_GBMCWEB_INCLUDE_EXCEPTION_STACK_TRACER_INTERNAL_HPP_ + +// Internal surface of the exception stack tracer, exposed only so unit tests +// can exercise the record lookup and terminate-formatting logic in-process. +// The production terminate handler aborts the process, so exercising that logic +// solely through death tests keeps it out of coverage (a forked child that +// abort()s never flushes gcov counters) and makes it awkward to assert on. +// This is not part of the public API; do not use outside the tracer. +namespace exception_stack_tracer_internal { + +// Logs "terminate: <type>", the exception's what(), and the recorded +// throw-site stack (or an "unavailable" line) for the exception currently being +// handled. Safe to call with no active exception (logs that and returns). +// Shared by the real terminate handler and the unit tests. +void logThrowSiteStackOfCurrentException(); + +// Restores recording state for hermetic tests: re-enables tracing, empties the +// throw ring, and forces the module table to be re-snapshotted on the next +// installExceptionStackTracer(true) call. +void resetForTest(); + +} // namespace exception_stack_tracer_internal + +#endif // THIRD_PARTY_GBMCWEB_INCLUDE_EXCEPTION_STACK_TRACER_INTERNAL_HPP_
diff --git a/include/webserver_main_setup.hpp b/include/webserver_main_setup.hpp index ade7c4e..1e2bc32 100644 --- a/include/webserver_main_setup.hpp +++ b/include/webserver_main_setup.hpp
@@ -22,6 +22,9 @@ #include "bmcweb_config.h" #include "app.hpp" #include "app_singleton.hpp" +#ifdef BMCWEB_ENABLE_EXCEPTION_STACK_TRACER +#include "exception_stack_tracer.hpp" +#endif #include "logging.hpp" #include "cors_preflight.hpp" #include "persistent_data.hpp" @@ -72,6 +75,7 @@ ABSL_DECLARE_FLAG(absl::optional<int>, rde_errors_max); ABSL_DECLARE_FLAG(absl::optional<int>, rde_skips_max); ABSL_DECLARE_FLAG(bool, multi_thread_get); +ABSL_DECLARE_FLAG(bool, exception_stack_tracer); ABSL_DECLARE_FLAG(bool, enable_tlbmc); ABSL_DECLARE_FLAG(bool, enable_tlbmc_trace); ABSL_DECLARE_FLAG(bool, enable_tlbmc_thermal_control); @@ -195,6 +199,12 @@ } inline int run() { + // First so even startup-path exceptions terminate with a throw-site stack. + // Also the explicit reference that pulls the tracer TU (and its __cxa_throw + // interposer) out of the bmcweblib archive despite --gc-sections. +#ifdef BMCWEB_ENABLE_EXCEPTION_STACK_TRACER + installExceptionStackTracer(absl::GetFlag(FLAGS_exception_stack_tracer)); +#endif // BMCWEB_ENABLE_EXCEPTION_STACK_TRACER std::shared_ptr<milotic_tlbmc::CredentialManager> credential_manager = nullptr; #ifdef BMCWEB_ENABLE_GRPC
diff --git a/meson.build b/meson.build index 8f78352..37c914c 100644 --- a/meson.build +++ b/meson.build
@@ -448,6 +448,14 @@ bmcweb_dependencies += abseil_deps + # The exception stack tracer installs a __cxa_throw interposer (the "folly + # trick") plus a std::terminate handler. It is meson/yocto-only: the g3/blaze + # build does not compile src/exception_stack_tracer.cpp, so every reference to + # it in webserver_main_setup is guarded by this macro to avoid a hard + # dependency in g3. Defined here because srcfiles_bmcweb below always builds + # the tracer translation unit for the meson/yocto build. + add_project_arguments('-DBMCWEB_ENABLE_EXCEPTION_STACK_TRACER', language : 'cpp') + # Source files fs = import('fs') @@ -474,6 +482,7 @@ 'redfish-core/src/utils/json_utils.cpp', 'redfish-core/src/utils/subprocess_utils.cpp', 'src/webserver_main_setup.cpp', + 'src/exception_stack_tracer.cpp', 'src/boost_asio_ssl.cpp', 'src/boost_asio.cpp', 'src/boost_beast.cpp', @@ -550,16 +559,30 @@ executable( 'bmcweb', 'src/webserver_main.cpp', + # Compiled directly into the executable (in addition to bmcweblib, which the + # unit tests link) so the __cxa_throw interposer is a direct object, not an + # archive member: --exclude-libs,ALL force-hides archive symbols and would + # otherwise defeat the --dynamic-list export below. The duplicate archive + # member is simply not pulled, since the direct object already defines it. + 'src/exception_stack_tracer.cpp', include_directories : incdir, dependencies: bmcweb_dependencies, link_with: libs_link_with, - link_args: '-Wl,--gc-sections', + # --gc-sections trims unused code; --dynamic-list force-exports only the + # exception stack tracer's __cxa_throw interposer so it interposes throws + # from every module (libstdc++ and other .so's), not just the executable. + link_args: [ + '-Wl,--gc-sections', + '-Wl,--dynamic-list=@0@/exception_stack_tracer.dynlist'.format( + meson.current_source_dir()), + ], install: true, install_dir: bindir, ) srcfiles_unittest = [ 'src/managed_store_http_test.cpp', + 'test/src/exception_stack_tracer_test.cpp', 'test/g3/mock_managed_store_test.cpp', 'test/http/crow_getroutes_test.cpp', 'test/http/http_response_test.cpp',
diff --git a/src/exception_stack_tracer.cpp b/src/exception_stack_tracer.cpp new file mode 100644 index 0000000..7c5ad5a --- /dev/null +++ b/src/exception_stack_tracer.cpp
@@ -0,0 +1,568 @@ +// =========================================================================== +// Overview (plain language) +// =========================================================================== +// Problem: when gbmcweb hits an uncaught C++ exception it crashes, but by the +// time the crash handler runs the stack has already unwound, so the usual +// backtrace points at framework code (asio / sdbusplus re-throwing a stashed +// exception) instead of the line that actually threw. That makes fleet crashes +// very hard to debug. +// +// Trick: every C++ `throw` funnels through one libstdc++ function, __cxa_throw. +// We define our own __cxa_throw in the main binary, so the linker makes the +// whole process (libstdc++ included) call ours. On each throw we snapshot the +// call stack *before* it unwinds, tag it with the exception object's address, +// and stash it in a small fixed ring. If the program later dies from an +// uncaught exception, our std::terminate handler finds that exception's +// snapshot and logs it -- the real throw site. +// +// How the pieces fit: +// * Capture: backtrace() returns a list of raw code addresses ("here, called +// from here, ..."), with no names. +// * Store: a 64-slot circular buffer (the newest throw overwrites the oldest) +// with no memory allocation. Each slot is tagged with the exception's +// address. A lock-free seqlock lets the crash handler read a slot while +// other threads keep throwing, without locks and without ever trusting a +// half-written entry. +// * Report at crash: look up the dying exception's snapshot and print each +// address as "module + hex offset". +// * Decode later: a developer runs the standard `addr2line` tool on the +// matching debug binary to turn "module + offset" into file:line. We do not +// resolve names at crash time -- that needs locks/allocation and the +// process may be corrupt -- so at startup we snapshot once which address +// ranges belong to which loaded binary. +// +// Why so defensive (no allocation, no locks, no re-entry): the handler runs +// when the process may already be corrupt; anything that could block, allocate, +// or crash again would lose the log entirely. +// +// Scope: built only in the gcc/yocto firmware image, since it relies on GNU +// libstdc++ exception internals; other builds compile it to "unavailable". It +// works on x86-64, ARM64 and ARM32 (it uses the standard stack unwinder, not a +// CPU-specific hack, so -fomit-frame-pointer and optimization levels do not +// affect it). +// +// The links below point to the specs and mechanisms this file relies on; the +// rationale now lives as comments next to the code each point describes. +// =========================================================================== + +// References (formats and mechanisms this file depends on): +// - Itanium C++ ABI, Exception Handling — __cxa_throw, the __cxa_exception +// header layout, __cxa_current_exception_type, and the exception_ptr / +// primary-object model this code keys on: +// https://itanium-cxx-abi.github.io/cxx-abi/abi-eh.html +// - folly exception_tracer — the __cxa_throw interposer this file reduces to a +// single self-contained TU: +// https://github.com/facebook/folly/blob/main/folly/debugging/exception_tracer/ExceptionTracerLib.cpp +// - glibc backtrace(3) — the throw-site stack capture primitive: +// https://man7.org/linux/man-pages/man3/backtrace.3.html +// - dl_iterate_phdr(3) — source of the ELF program-header module table +// snapshotted at install time: +// https://man7.org/linux/man-pages/man3/dl_iterate_phdr.3.html +// - Seqlock — the lock-free single-writer scheme each ring slot uses: +// https://www.kernel.org/doc/html/latest/locking/seqlock.html +// - Unwind tables consumed by backtrace(): +// DWARF .eh_frame (ARM64/x86-64) — LSB Core, Exception Frames: +// https://refspecs.linuxfoundation.org/LSB_5.0.0/LSB-Core-generic/LSB-Core-generic/ehframechpt.html +// ARM32 EHABI .ARM.exidx — Exception Handling ABI for the Arm Arch: +// https://github.com/ARM-software/abi-aa/blob/main/ehabi32/ehabi32.rst +// ARM64 frame-record chain (x29/x30), AAPCS64: +// https://github.com/ARM-software/abi-aa/blob/main/aapcs64/aapcs64.rst + +#include "exception_stack_tracer.hpp" + +#include "exception_stack_tracer_internal.hpp" + +#include <cxxabi.h> +#include <dlfcn.h> +#include <execinfo.h> +#include <link.h> + +#include <algorithm> +#include <array> +#include <atomic> +#include <cstddef> +#include <cstdint> +#include <cstdlib> +#include <cstring> +#include <exception> +#include <typeinfo> + +#include "absl/base/internal/raw_logging.h" + +namespace { + +// Frame 0 is the interposer itself, so ~9 caller frames survive; enough to +// clear the __throw_* helper and land in application code. +constexpr int kMaxFrames = 10; +constexpr uint32_t kRingSize = 64; +constexpr size_t kMaxModules = 64; +constexpr size_t kModuleNameSize = 40; + +// A captured call stack as written by backtrace(): an array of up to +// kMaxFrames return addresses (code pointers / program counters), innermost +// frame first. Each entry is an opaque instruction address, later resolved to +// module + offset for offline addr2line; it is never dereferenced. +using FrameArray = std::array<void*, kMaxFrames>; + +// A throw-site stack copied out of the ring for the terminate handler to print. +struct ThrowRecordSnapshot { + const std::type_info* type = nullptr; + int depth = 0; + FrameArray frames{}; +}; + +// --------------------------------------------------------------------------- +// ThrowSiteRing: a lock-free ring of throw-site stacks. +// +// Writers are the throwing threads (via __cxa_throw); the single reader is the +// terminate handler. Each slot is published as a seqlock: `object` is the +// sequence word, stored last with release order and cleared first by a writer, +// so a reader trusts a copy only if `object` matched before and after. +// --------------------------------------------------------------------------- +class ThrowSiteRing { + public: + void setEnabled(bool enable) { + enabled_.store(enable, std::memory_order_relaxed); + } + + // Writer, called from __cxa_throw before the unwinder runs: at catch time the + // throw-site frames no longer exist. + void record(void* object, const std::type_info* type) { + if (!enabled_.load(std::memory_order_relaxed)) { + return; + } + // backtrace() writes up to kMaxFrames return addresses into `frames` and + // returns how many it wrote (`depth`), innermost frame first. + FrameArray frames{}; + int depth = backtrace(frames.data(), kMaxFrames); + nextSlot().publish(object, type, frames, depth); + } + + // Reader: scan newest-first so that after wraparound a reused object address + // resolves to the live exception's record, not an older evicted throw. + bool lookup(void* object, ThrowRecordSnapshot& out) const { + if (object == nullptr) { + return false; + } + uint32_t end = next_.load(std::memory_order_acquire); + for (uint32_t i = 0; i < kRingSize; ++i) { + const Slot& slot = slots_[(end - 1 - i) % kRingSize]; + switch (slot.tryRead(object, out)) { + case Slot::ReadResult::kNoMatch: + continue; + case Slot::ReadResult::kCopied: + return true; + case Slot::ReadResult::kEvicted: + // The address matched but the slot was rewritten during the copy. + // The caller's exception_ptr pins `object` alive, so no concurrent + // throw can record this address; a rewrite therefore means this slot + // was reused by an unrelated throw and the record is gone. Report it + // evicted rather than matching a stale older same-address slot. + return false; + } + } + return false; + } + + // Test-only: re-enable recording and drop all records. + void reset() { + enabled_.store(true, std::memory_order_relaxed); + next_.store(0, std::memory_order_relaxed); + for (Slot& slot : slots_) { + slot.invalidate(); + } + } + + private: + class Slot { + public: + enum class ReadResult { kNoMatch, kCopied, kEvicted }; + + // Writer half of the seqlock: invalidate, fill the payload, then publish + // `object` last so a concurrent reader sees either the old value, a + // no-match sentinel, or the fully written record — never a torn mix. + void publish(void* object, const std::type_info* type, + const FrameArray& frames, int depth) { + object_.store(nullptr, std::memory_order_relaxed); + // A release *store* only keeps preceding writes above it; it does not stop + // the relaxed payload writes below from being hoisted above the + // invalidation. This release fence is the store-store barrier that keeps + // the object_=nullptr invalidation ordered before the payload writes, so a + // reader still seeing the old object during an eviction can never observe + // a torn payload and revalidate it as kCopied. + std::atomic_thread_fence(std::memory_order_release); + type_.store(type, std::memory_order_relaxed); + depth_.store(depth, std::memory_order_relaxed); + for (size_t i = 0; i < kMaxFrames; ++i) { + frames_[i].store(frames[i], std::memory_order_relaxed); + } + object_.store(object, std::memory_order_release); + } + + // Reader half: copy the payload iff the slot holds `object`, then report + // whether the sequence word still matches (kCopied) or was rewritten + // mid-copy (kEvicted). Payload fields are atomic so the mid-copy read that + // kEvicted discards is well defined rather than a data race. + ReadResult tryRead(void* object, ThrowRecordSnapshot& out) const { + if (object_.load(std::memory_order_acquire) != object) { + return ReadResult::kNoMatch; + } + out.type = type_.load(std::memory_order_relaxed); + out.depth = sanitizedDepth(depth_.load(std::memory_order_relaxed)); + for (size_t i = 0; i < kMaxFrames; ++i) { + out.frames[i] = frames_[i].load(std::memory_order_relaxed); + } + // Symmetric to publish()'s release fence. An acquire *load* on object_ + // only orders operations that follow it; it would not stop the relaxed + // payload loads above from being reordered *after* the revalidation, so a + // slot rewritten mid-copy could be revalidated as kCopied over a torn + // payload. This acquire fence is the load-load barrier that keeps the + // payload reads ordered before the revalidation load, so a mid-copy + // rewrite is reliably detected as kEvicted. With the fence in place the + // revalidation load itself only needs to be relaxed. + std::atomic_thread_fence(std::memory_order_acquire); + return object_.load(std::memory_order_relaxed) == object + ? ReadResult::kCopied + : ReadResult::kEvicted; + } + + void invalidate() { object_.store(nullptr, std::memory_order_relaxed); } + + private: + // A valid frame count is 0..kMaxFrames; anything else can only come from a + // torn read (later rejected by revalidation), so map it to 0 -- show no + // frames rather than a bogus count. + static int sanitizedDepth(int depth) { + return (depth < 0 || depth > kMaxFrames) ? 0 : depth; + } + + std::atomic<void*> object_{nullptr}; + std::atomic<const std::type_info*> type_{nullptr}; + std::atomic<int> depth_{0}; + std::array<std::atomic<void*>, kMaxFrames> frames_{}; + }; + + Slot& nextSlot() { + return slots_[next_.fetch_add(1, std::memory_order_relaxed) % kRingSize]; + } + + // Kill switch (--exception_stack_tracer=false): recording is a no-op and the + // default terminate handler stays installed. + std::atomic<bool> enabled_{true}; + std::atomic<uint32_t> next_{0}; + std::array<Slot, kRingSize> slots_{}; +}; + +// --------------------------------------------------------------------------- +// ModuleTable: maps a runtime code address back to (module, in-file offset). +// +// backtrace() gives us raw runtime addresses. To symbolize them offline we need +// two things per address: which binary it came from, and its offset within that +// binary's file (that is what `addr2line -e <file> <offset>` consumes). +// +// Why we keep our own table instead of resolving at crash time: +// The normal way to map an address to its module is dladdr(), but dladdr +// takes the dynamic loader lock and may allocate -- both forbidden in the +// terminate handler, which runs under suspected corruption and must never +// block or allocate. So we walk the loaded modules once at install time, +// while the process is healthy, and cache exactly what we need in a +// fixed-size table the handler can scan with plain array reads: no locks, no +// allocation, no loader calls. This stays correct because bmcweb does not +// dlopen anything after startup, so the module map never changes. +// +// What the table holds: +// One ModuleInfo row per loaded module (the executable and each .so). Each +// row records that module's runtime address range (to test which module a +// frame falls in), its load bias (to convert the frame to an in-file offset), +// and its short name (for the log line) -- everything needed to turn a raw +// address into "module + offset". +// +// ELF background (how a program is laid out in memory, i.e. where those numbers +// come from): +// * A "module" is one ELF file mapped into the process: the main executable +// plus every shared library (.so) it loaded. +// * Each ELF file carries a "program header" table describing how the loader +// should map it. Each entry is a segment; the ones with type PT_LOAD are +// the chunks of code/data actually copied into memory (other types hold +// dynamic-linking metadata, notes, etc., and take up no code space). +// * Addresses inside the file are recorded relative to a link-time base of 0 +// (for position-independent code -- PIEs and all .so files). At load time +// ASLR places the module at a random base address; the loader then shifts +// every address by a fixed amount called the "load bias". +// runtime address = load bias + in-file address +// * So a PT_LOAD segment lives at runtime in +// [ bias + p_vaddr , bias + p_vaddr + p_memsz ) +// where p_vaddr is the segment's in-file address and p_memsz its size in +// memory. Unioning all PT_LOAD segments gives the module's whole runtime +// address range (ModuleInfo::begin/end). +// * To symbolize a frame we reverse the shift: in-file offset = addr - bias. +// +// The snapshot is taken with dl_iterate_phdr (see loadedRange below). +// --------------------------------------------------------------------------- +// One row of the table: a single loaded module (the executable or one .so), and +// everything the terminate handler needs to place and symbolize an address that +// falls inside it. +struct ModuleInfo { + // Load bias (dl_phdr_info::dlpi_addr): the address a position-independent + // object was mapped at. `frame - bias` is the in-file offset addr2line wants. + uintptr_t bias = 0; + // Half-open runtime address range [begin, end) the module occupies, used to + // match a captured frame to its module. + uintptr_t begin = 0; + uintptr_t end = 0; + // Module basename ("exe" for the main program), NUL-terminated. + std::array<char, kModuleNameSize> name{}; + + // Copy `n` into the fixed buffer, always leaving it NUL-terminated. + void setName(const char* n) { + std::strncpy(name.data(), n, kModuleNameSize - 1); + name[kModuleNameSize - 1] = '\0'; + } +}; + +class ModuleTable { + public: + // Snapshot the currently loaded modules. Call once, at install time. + void snapshot() { dl_iterate_phdr(&phdrCallback, this); } + + // Find the module containing `addr`, or nullptr if none does. + const ModuleInfo* resolve(uintptr_t addr) const { + size_t modules = count_.load(std::memory_order_acquire); + for (size_t m = 0; m < modules; ++m) { + if (addr >= modules_[m].begin && addr < modules_[m].end) { + return &modules_[m]; + } + } + return nullptr; + } + + void reset() { count_.store(0, std::memory_order_relaxed); } + + private: + static int phdrCallback(dl_phdr_info* info, size_t /*size*/, void* self) { + return static_cast<ModuleTable*>(self)->addModule(*info); + } + + // A half-open [begin, end) address range, accumulated across PT_LOAD + // segments. Starts empty (begin > end) so min/max fold correctly. + struct AddressRange { + uintptr_t begin = UINTPTR_MAX; + uintptr_t end = 0; + bool empty() const { return end == 0; } + }; + + // dl_iterate_phdr hands us one dl_phdr_info per loaded ELF object (the main + // program and each shared library). The fields we use: + // dlpi_addr - load bias: the base address the object was mapped at. + // dlpi_name - the object's file path ("" for the main program). + // dlpi_phdr - array of the object's ELF program headers (ElfW(Phdr), + // dlpi_phnum which is Elf32/Elf64 chosen for the target). + // Each PT_LOAD program header describes a segment actually mapped into + // memory; its runtime span is [dlpi_addr + p_vaddr, + p_memsz). Unioning the + // PT_LOAD segments yields the whole object's occupied address range. + static AddressRange loadedRange(const dl_phdr_info& info) { + AddressRange range; + for (int i = 0; i < info.dlpi_phnum; ++i) { + // NOLINTNEXTLINE(cppcoreguidelines-pro-bounds-pointer-arithmetic) + const ElfW(Phdr)& phdr = info.dlpi_phdr[i]; + if (phdr.p_type != PT_LOAD) { + continue; // only PT_LOAD segments are mapped into memory + } + uintptr_t segmentBegin = info.dlpi_addr + phdr.p_vaddr; + uintptr_t segmentEnd = segmentBegin + phdr.p_memsz; + range.begin = std::min(range.begin, segmentBegin); + range.end = std::max(range.end, segmentEnd); + } + return range; + } + + // Reduce a module path to its basename for compact logging; the main + // program's dlpi_name is empty, so report it as "exe". + static const char* moduleBasename(const char* path) { + if (path == nullptr || path[0] == '\0') { + return "exe"; + } + const char* slash = std::strrchr(path, '/'); + // NOLINTNEXTLINE(cppcoreguidelines-pro-bounds-pointer-arithmetic) + return slash != nullptr ? slash + 1 : path; + } + + // dl_iterate_phdr callback. Returns 1 to stop iteration once the table is + // full, 0 to continue. + int addModule(const dl_phdr_info& info) { + size_t count = count_.load(std::memory_order_relaxed); + if (count >= kMaxModules) { + return 1; + } + AddressRange range = loadedRange(info); + if (range.empty()) { + return 0; // no PT_LOAD segments (e.g. a linker placeholder); skip + } + ModuleInfo& m = modules_[count]; + m.bias = info.dlpi_addr; + m.begin = range.begin; + m.end = range.end; + m.setName(moduleBasename(info.dlpi_name)); + count_.store(count + 1, std::memory_order_release); + return 0; + } + + // The snapshot: modules_[0, count_) are the loaded modules captured at + // install time. Fixed capacity so no allocation is ever needed; count_ is + // published with release so resolve() sees fully-written rows. + std::array<ModuleInfo, kMaxModules> modules_{}; + std::atomic<size_t> count_{0}; +}; + +// The two process-wide tables. constinit guarantees constant initialization, +// so they are ready before any static constructor (or early throw) runs. +// NOLINTNEXTLINE(cppcoreguidelines-avoid-non-const-global-variables) +constinit ThrowSiteRing gThrowRing{}; +// NOLINTNEXTLINE(cppcoreguidelines-avoid-non-const-global-variables) +constinit ModuleTable gModuleTable{}; + +// Returns the address that identifies the in-flight exception -- the same +// pointer __cxa_throw recorded -- which the handler uses to look up its stack. +// On libstdc++ (GNU), a std::exception_ptr is exactly one pointer to that +// object, so we read it out directly (layout checked by static_assert). On any +// other C++ runtime (e.g. clang/libc++) we don't know the layout, so we return +// nullptr; the handler then just reports "stack unavailable" and everything +// else keeps working. +#if defined(__GLIBCXX__) +void* primaryExceptionObject(const std::exception_ptr& p) { + static_assert(sizeof(std::exception_ptr) == sizeof(void*), + "libstdc++ exception_ptr layout assumption violated"); + void* object = nullptr; + std::memcpy(&object, &p, sizeof(object)); + return object; +} +#else +void* primaryExceptionObject(const std::exception_ptr& /*p*/) { + return nullptr; // unknown exception_ptr layout: lookups report unavailable +} +#endif + +void logResolvedFrames(const FrameArray& frames, int depth) { + for (int i = 0; i < depth; ++i) { + // NOLINTNEXTLINE(cppcoreguidelines-pro-type-reinterpret-cast) + const auto addr = reinterpret_cast<uintptr_t>(frames[static_cast<size_t>(i)]); + const ModuleInfo* module = gModuleTable.resolve(addr); + if (module != nullptr) { + // Module-relative offset: symbolizable offline against the matching + // .debug file (addr2line -e) even for a stripped PIE. + ABSL_RAW_LOG(ERROR, " #%02d %s+0x%lx", i, module->name.data(), + static_cast<unsigned long>(addr - module->bias)); + } else { + ABSL_RAW_LOG(ERROR, " #%02d [0x%lx]", i, + static_cast<unsigned long>(addr)); + } + } +} + +[[noreturn]] void terminateWithThrowStack() { + // If anything below terminates again (a throwing what(), a second uncaught + // exception on another thread), die immediately instead of recursing. + // NOLINTNEXTLINE(cppcoreguidelines-avoid-non-const-global-variables) + static std::atomic<bool> entered{false}; + if (entered.exchange(true, std::memory_order_relaxed)) { + std::abort(); + } + exception_stack_tracer_internal::logThrowSiteStackOfCurrentException(); + std::abort(); +} + +} // namespace + +namespace exception_stack_tracer_internal { + +void logThrowSiteStackOfCurrentException() { + const std::type_info* type = abi::__cxa_current_exception_type(); + if (type == nullptr) { + ABSL_RAW_LOG(ERROR, "terminate: called without an active exception"); + return; + } + ABSL_RAW_LOG(ERROR, "terminate: uncaught exception of type %s", type->name()); + + // __cxa_call_terminate (or a test's active catch) began the catch, so the + // in-flight exception is rethrowable in place. Unlike std::rethrow_exception + // this allocates nothing (same technique as libstdc++'s + // __verbose_terminate_handler). + try { + throw; + } catch (const std::exception& e) { + // A user-defined what() override may return nullptr; passing that to %s is + // undefined behaviour, so fall back to a literal (mirrors the type-name + // guard below). + const char* what = e.what(); + ABSL_RAW_LOG(ERROR, " what(): %s", what != nullptr ? what : "(null)"); + } catch (...) { + } + + // For a native exception this only bumps a refcount on the existing object + // — no allocation. It also pins the object for the ring's revalidation. + if (std::exception_ptr eptr = std::current_exception()) { + ThrowRecordSnapshot snapshot; + if (gThrowRing.lookup(primaryExceptionObject(eptr), snapshot)) { + ABSL_RAW_LOG(ERROR, "throw-site stack (recorded as %s):", + snapshot.type != nullptr ? snapshot.type->name() : "?"); + logResolvedFrames(snapshot.frames, snapshot.depth); + } else { + ABSL_RAW_LOG(ERROR, + "throw-site stack unavailable (ring evicted, foreign " + "exception, or non-libstdc++ runtime)"); + } + } +} + +void resetForTest() { + gThrowRing.reset(); + gModuleTable.reset(); +} + +} // namespace exception_stack_tracer_internal + +// Called once from run() at startup. Besides the one-time setup below, this +// explicit call is also what keeps this file in the binary: with no reference to +// it, the linker's --gc-sections would discard the whole translation unit -- and +// our __cxa_throw override along with it. enable=false (the +// --exception_stack_tracer kill switch) leaves the default terminate handler in +// place and turns recording into a no-op. +void installExceptionStackTracer(bool enable) { + if (!enable) { + gThrowRing.setEnabled(false); + return; + } + // Snapshot the module table and force glibc's lazy libgcc unwinder load now + // so the throw path and the terminate path never allocate, dlopen, or take + // the loader lock. + gModuleTable.snapshot(); + std::array<void*, 2> warm{}; + backtrace(warm.data(), 2); + std::set_terminate(terminateWithThrowStack); +} + +// Our replacement for the C++ runtime's throw entry point. Every `throw` in the +// process -- including libstdc++'s own __throw_* helpers and throws originating +// in other shared objects -- calls this function, because our definition is +// exported from the main executable (default visibility + a linker +// --dynamic-list entry for __cxa_throw; see meson.build) and the executable is +// searched first in the global symbol scope, so it interposes the library's. +// We record the throw site, then forward to the real __cxa_throw (found via +// dlsym(RTLD_NEXT, ...)) so throwing behaves exactly as normal. +// +// NOTE: exporting this symbol is what makes cross-module capture work. Without +// it (e.g. built LOCAL under -fvisibility=hidden), only throws whose __cxa_throw +// call site is compiled into the executable are recorded; throws executing +// inside libstdc++ or any other .so bind to the library's own __cxa_throw and +// are missed. +extern "C" [[gnu::visibility("default")]] void __cxa_throw( + void* thrown_object, std::type_info* tinfo, void (*dest)(void*)) { + gThrowRing.record(thrown_object, tinfo); + using ThrowFn = void (*)(void*, std::type_info*, void (*)(void*)); + // NOLINTBEGIN(cppcoreguidelines-pro-type-reinterpret-cast) + static const ThrowFn realCxaThrow = + reinterpret_cast<ThrowFn>(dlsym(RTLD_NEXT, "__cxa_throw")); + // NOLINTEND(cppcoreguidelines-pro-type-reinterpret-cast) + realCxaThrow(thrown_object, tinfo, dest); + __builtin_unreachable(); +}
diff --git a/src/webserver_main_setup.cpp b/src/webserver_main_setup.cpp index 1933481..d57565e 100644 --- a/src/webserver_main_setup.cpp +++ b/src/webserver_main_setup.cpp
@@ -54,6 +54,10 @@ "Number of requests skipped when rate limiting enabled"); ABSL_FLAG(bool, multi_thread_get, enableMultiThreadGet, "Enable multi-thread for Redfish GET"); +ABSL_FLAG(bool, exception_stack_tracer, true, + "Record throw-site stacks and print them from the terminate " + "handler. Kill switch: set to false to disable recording and keep " + "the default terminate handler."); ABSL_FLAG(bool, enable_tlbmc, enableTlbmc, "Enable tlBMC features"); ABSL_FLAG(bool, enable_tlbmc_trace, enableTlbmcTrace, "Enable tlBMC tracer"); ABSL_FLAG(bool, enable_tlbmc_thermal_control, enableTlbmcThermalControl,
diff --git a/test/src/exception_stack_tracer_test.cpp b/test/src/exception_stack_tracer_test.cpp new file mode 100644 index 0000000..2f3575e --- /dev/null +++ b/test/src/exception_stack_tracer_test.cpp
@@ -0,0 +1,204 @@ +#include "exception_stack_tracer.hpp" + +#include "exception_stack_tracer_internal.hpp" + +#include <exception> +#include <stdexcept> +#include <string> +#include <thread> // NOLINT +#include <utility> + +#include "gtest/gtest.h" + +namespace { + +// Depth of the synthetic throw-site call chain. Comfortably exceeds the five +// frames the capture test asserts, leaving margin under kMaxFrames. +constexpr int kThrowDepth = 6; + +// Recurse to build a deep, real call stack and throw at the bottom, so the +// capture test can assert the tracer recorded several throw-site frames rather +// than just one. [[gnu::noinline]] keeps each level a distinct frame and the +// post-call barrier defeats the tail call that would otherwise elide it. +[[gnu::noinline]] void throwFromDepth(int depth) { + if (depth <= 0) { + throw std::runtime_error("boom-from-throw-site"); + } + throwFromDepth(depth - 1); + asm volatile("" ::: "memory"); +} + +// A distinct noinline entry so the throw site is not the test body. +[[gnu::noinline]] void throwRuntimeError() { throwFromDepth(kThrowDepth); } + +// Throws and captures on a separate thread, so a later rethrow exercises the +// cross-thread exception_ptr transport that asio/sdbusplus laundering uses. +std::exception_ptr captureOnOtherThread() { + std::exception_ptr eptr; + std::thread thrower([&eptr] { + try { + throwRuntimeError(); + } catch (...) { + eptr = std::current_exception(); + } + }); + thrower.join(); + return eptr; +} + +// Runs `fn` on a separate thread and lets any exception escape the thread's +// entry function. An exception escaping a std::thread invokes std::terminate +// directly -- this is the real uncaught path (asio io_context /sdbusplus posted +// rethrow) and the ONLY way to reach our terminate handler from inside +// EXPECT_DEATH: gtest wraps the death-test statement in a try/catch that would +// otherwise intercept a main-thread throw before std::terminate ever runs. +template <typename Fn> +void terminateViaEscapingThread(Fn fn) { + std::thread t(std::move(fn)); + t.join(); // Not reached: the escaping exception terminates the process. +} + +// All tests fork via EXPECT_DEATH and match the child's stderr, since the +// behavior under test IS the abort. threadsafe style re-executes the test +// binary for the child, which is required once threads are involved. + +TEST(ExceptionStackTracerDeathTest, UncaughtThrowPrintsThrowSiteStack) { + testing::FLAGS_gtest_death_test_style = "threadsafe"; + EXPECT_DEATH( + { + installExceptionStackTracer(); + terminateViaEscapingThread(throwRuntimeError); + }, + "uncaught exception of type St13runtime_error" + ".*what\\(\\): boom-from-throw-site" + ".*throw-site stack \\(recorded as St13runtime_error\\)" + // At least five frames, each resolved to a module+offset (not the + // unresolved "[0x..]" fallback): proves a real multi-frame stack walk. + ".*#00 [^ ]+\\+0x[0-9a-f]+" + ".*#01 [^ ]+\\+0x[0-9a-f]+" + ".*#02 [^ ]+\\+0x[0-9a-f]+" + ".*#03 [^ ]+\\+0x[0-9a-f]+" + ".*#04 [^ ]+\\+0x[0-9a-f]+"); +} + +TEST(ExceptionStackTracerDeathTest, RethrownExceptionPtrKeepsThrowSiteStack) { + testing::FLAGS_gtest_death_test_style = "threadsafe"; + EXPECT_DEATH( + { + installExceptionStackTracer(); + std::exception_ptr eptr = captureOnOtherThread(); + // Uncaught rethrow on a third thread: the terminate happens on a + // different thread than the throw, via a dependent exception -- the + // record must still resolve through the primary object. + terminateViaEscapingThread([eptr] { std::rethrow_exception(eptr); }); + }, + "what\\(\\): boom-from-throw-site" + ".*throw-site stack \\(recorded as St13runtime_error\\)" + ".*#00 "); +} + +TEST(ExceptionStackTracerDeathTest, EvictedRecordDegradesGracefully) { + testing::FLAGS_gtest_death_test_style = "threadsafe"; + EXPECT_DEATH( + { + installExceptionStackTracer(); + std::exception_ptr eptr = captureOnOtherThread(); + // Wrap the 64-slot ring so the captured exception's record is gone. + for (int i = 0; i < 100; ++i) { + try { + throw std::logic_error("filler"); + } catch (const std::logic_error&) { + } + } + terminateViaEscapingThread([eptr] { std::rethrow_exception(eptr); }); + }, + "what\\(\\): boom-from-throw-site" + ".*throw-site stack unavailable"); +} + +TEST(ExceptionStackTracerDeathTest, DisabledTracerKeepsDefaultTerminate) { + testing::FLAGS_gtest_death_test_style = "threadsafe"; + // libstdc++'s verbose terminate handler output, not ours. + EXPECT_DEATH( + { + installExceptionStackTracer(false); + terminateViaEscapingThread(throwRuntimeError); + }, + "terminate called after throwing an instance of"); +} + +// In-process (non-death) tests. Unlike the death tests above -- whose forked +// children abort() and so never flush coverage counters -- these run the +// record, lookup, seqlock-validation and frame-printing logic in the parent +// process via the extracted logThrowSiteStackOfCurrentException() helper. That +// makes the terminate-path code visible to coverage and lets us assert on the +// captured output directly instead of only matching a dying child's stderr. + +using exception_stack_tracer_internal::logThrowSiteStackOfCurrentException; +using exception_stack_tracer_internal::resetForTest; + +TEST(ExceptionStackTracerTest, RecordsAndResolvesThrowSiteInProcess) { + resetForTest(); + installExceptionStackTracer(true); + testing::internal::CaptureStderr(); + try { + throwRuntimeError(); + } catch (...) { + logThrowSiteStackOfCurrentException(); + } + const std::string err = testing::internal::GetCapturedStderr(); + EXPECT_NE(err.find("uncaught exception of type St13runtime_error"), + std::string::npos); + EXPECT_NE(err.find("what(): boom-from-throw-site"), std::string::npos); + EXPECT_NE(err.find("throw-site stack (recorded as St13runtime_error)"), + std::string::npos); + // At least five frames captured, proving a real multi-frame stack walk. + for (const char* frame : {"#00 ", "#01 ", "#02 ", "#03 ", "#04 "}) { + EXPECT_NE(err.find(frame), std::string::npos) << "missing frame " << frame; + } +} + +TEST(ExceptionStackTracerTest, DisabledTracerRecordsNothingInProcess) { + resetForTest(); + installExceptionStackTracer(false); + testing::internal::CaptureStderr(); + try { + throwRuntimeError(); + } catch (...) { + logThrowSiteStackOfCurrentException(); + } + const std::string err = testing::internal::GetCapturedStderr(); + EXPECT_NE(err.find("throw-site stack unavailable"), std::string::npos); +} + +TEST(ExceptionStackTracerTest, EvictedRecordReportsUnavailableInProcess) { + resetForTest(); + installExceptionStackTracer(true); + std::exception_ptr eptr = captureOnOtherThread(); + // Wrap the 64-slot ring so the captured exception's record is overwritten. + for (int i = 0; i < 100; ++i) { + try { + throw std::logic_error("filler"); + } catch (const std::logic_error&) { + } + } + testing::internal::CaptureStderr(); + try { + std::rethrow_exception(eptr); + } catch (...) { + logThrowSiteStackOfCurrentException(); + } + const std::string err = testing::internal::GetCapturedStderr(); + EXPECT_NE(err.find("throw-site stack unavailable"), std::string::npos); +} + +TEST(ExceptionStackTracerTest, NoActiveExceptionIsHandledInProcess) { + resetForTest(); + installExceptionStackTracer(true); + testing::internal::CaptureStderr(); + logThrowSiteStackOfCurrentException(); // no exception in flight + const std::string err = testing::internal::GetCapturedStderr(); + EXPECT_NE(err.find("called without an active exception"), std::string::npos); +} + +} // namespace