Cockatrice/cmake/CrashReporting.cmake
BruebachL 9f59f5c495
[CrashReporting] Breakpad capture, session detection and symbol upload (#7523)
* [CrashReporting] Breakpad everywhere, session detection, symbol CI (inc 3)

Breakpad on all platforms (no crashpad packaging); remove the Windows
MiniDumpWithFullMemory filter and the installer WER LocalDumps full-dump
registration. QLockFile per-process session tracking replaces the
Windows-only .dmp scan, placed after parser.process and the instance
hand-off so help/version exits and handed-off secondaries never leak a
lock. CI: Release-only DSN bake from SENTRY_DSN secret, tag-only
sentry-cli symbol upload (org cockatrice, project native).

Tests: 533/533 ctest green, format clean, code gate PASS.

* [CrashReporting] Enable http_retry, harden symbol upload, fix NSIS context

Sets sentry_options_set_http_retry so the cache_keep-held crash
envelope in Ask mode is written with a retry filename and actually
flushes on giveConsent() instead of sitting in <db>/cache until
pruning.

Adds continue-on-error to both sentry-cli install steps so a Sentry
registry outage or broken Git Bash install cannot block release
assets.

Cleans the legacy WER CrashDumps folder under SetShellVarContext
current (under all it resolves to ProgramData, not the user's
AppData) in installer and uninstaller; registry delete works in
either context.

* [CrashReporting] Extract the startup crash-report flow from main() per review

* [CrashReporting] Break up the CrashReporter class doc per review

* [CrashReporting] Drop the invalid --release from the symbol upload

`debug-files upload` never accepted `--release` (checked against the pinned
sentry-cli 3.8.0: the subcommand declares --org/--project and a set of
type/id/scan options, and execute() reads no release; the only global flags are
--url/--header/--auth-token/--log-level/--quiet/--allow-failure). Clap rejects
the unknown flag before any upload happens, so every tag build has been shipping
without symbols, with continue-on-error keeping the release pipeline green.

Release association is also the wrong mechanism: debug information files are
matched to events by debug ID via the event's debug_meta. The Sentry release of
a native event comes from the event itself (COCKATRICE_SENTRY_RELEASE), not from
the DIF upload. Remove the flag and the now-unused SENTRY_RELEASE env from both
invocations (Linux, and Windows/macOS).

The symbol upload stays continue-on-error (a Sentry outage must not block the
release assets); revisit once it has proven itself on a tag.

* [CrashReporting] Split debug information out of packaged release binaries

-g is required so the CI symbol upload has DWARF to send, but it was left in
the shipped binaries too: the Linux Release client is 563 MiB, 552 MiB of it
debug info, and every distribution channel pays for that.

Keep the build tree unchanged (CI uploads symbols from build/) and strip only
the packaged copy. A CPack pre-build script extracts each ELF's debug info to
COCKATRICE_DEBUG_SYMBOL_DIR and then runs `strip --strip-debug --strip-unneeded`.
.eh_frame is allocatable and survives, so unwinding is unaffected, and the build
ID is identical in the stripped binary and the sidecar.

Measured on a local Release build (GCC -O2 -g, WITH_CRASH_REPORTING=ON),
bytes:

  binary      build tree  packaged before  packaged after  .debug sidecar
  cockatrice 590,722,560      590,722,560      28,086,480     579,623,576
  oracle      53,432,608       53,432,608       1,649,888      52,075,344
  tar.gz of both trees: 289,964,521 -> 19,018,058 (~15x smaller).

Only ELF (Linux/BSD) is handled: Windows already links /DEBUG with separate
PDBs, and macOS extracts a dSYM from the bundle. Opt-out via
COCKATRICE_SPLIT_DEBUG_SYMBOLS; skipped with a warning when objcopy/strip are
missing. CPACK_PRE_BUILD_SCRIPTS (not CPACK_INSTALL_SCRIPTS, which runs before
staging) invokes it, and the script refuses to run unless CPACK_TEMPORARY_DIRECTORY
is a real CPack staging tree so a missing variable can never turn its glob into
a filesystem-wide walk.

* [CrashReporting] Upload Windows PDBs from where Ninja actually puts them

The Windows branch of the symbol upload looked in build/<target>/Release, which
is the Visual Studio generator's layout. The Windows leg uses Ninja, which is
single-config: linker PDBs land in build/<target>/, next to the binaries — the
same paths the "[Windows] Upload PDBs" artifact step below globs with
if-no-files-found: error.

So paths was always empty on Windows, the step printed "No symbol paths found,
skipping symbol upload" and exited 0: a silent no-op behind continue-on-error.
Every Windows build would then report "A required debug information file was
missing" for cockatrice.exe with a PDB-style debug id (GUID-age), exactly as if
no upload step existed at all.

Use the artifact step's globs. Also turn the empty-path case into a failure in
both symbol steps: a silent skip is indistinguishable from a successful upload
(which is how the wrong path went unnoticed), and continue-on-error already
keeps a broken upload from failing the release.

Verified by extracting both steps' run blocks and executing the Windows branch
against a Ninja-style tree: it now passes the three linker PDBs to sentry-cli
(and not the CMakeFiles/*.pdb intermediates), and exits non-zero with an
::error:: annotation when there is nothing to upload.

* [CrashReporting] Archive PDBs on tag builds so a failed upload is recoverable

The "[Windows] Upload PDBs" artifact step skipped tag builds
(github.ref_type != 'tag'), so for a release the symbols existed only inside
the Sentry upload — which is continue-on-error. A failed upload therefore lost
the ability to symbolicate that release permanently: no rebuild produces the
same PDB, since the debug id is a per-build GUID+age. That is how every
Windows crash reported so far became unreadable.

Archive the PDBs for tag builds as well, and order the step after
"Upload to release" so a missing PDB can never block a release from being
published. Retention stays at the repository default; nothing consumes the
artifact automatically, it is the manual recovery path
(gh run download <run> -n <name>-PDBs, then sentry-cli debug-files upload).

Verified by re-parsing the workflow and asserting the step order
(release -> PDBs -> attest) and the broadened condition.

* [CrashReporting] Upload the Windows executables with their PDBs

Sentry needs two things per image on Windows: symbolication and stack
unwinding. The PDB covers symbolication, but for a 64-bit PE the Call Frame
Information (.pdata/.xdata) lives in the executable and is not copied into the
PDB, so uploading PDBs alone leaves the image at "Stack Unwinding: Missing".
That is exactly what the event's Images panel showed for cockatrice.exe, while
Sentry's own Microsoft symbol server resolved KERNELBASE.dll and ucrtbase.dll
from the same dump.

sentry-cli also needs the pair in the same invocation: only then does it
associate the PE's code identifier with the PDB's debug identifier, otherwise
"the identifiers diverge and Sentry might not be able to resolve the correct
file for symbolication" (docs: Debug Identifiers).

Do the same for macOS, where the dSYM carries the debug information but the
Mach-O carries the unwind information, and include the executables in the
symbol artifact so a re-upload still has the pair to offer.

Verified by extracting the step and running both branches against stub
harnesses: Windows passes the three exe/pdb pairs in a single sentry-cli
invocation and still fails loudly on an empty tree; macOS passes the dSYM and
the Mach-O together.

* [CrashReporting] Upload the shipped Qt runtime DLLs too

Qt ships no PDBs with its MSVC SDK, so every Qt frame in a crash report
resolved to nothing: an image with neither PDB nor PE has no unwind
information (on 64-bit Windows the Call Frame Information lives in the PE
and is not copied into the PDB) and no symbol source to fall back on.

The list is required-runtime.txt, the manifest .ci/compile.sh already
asserts the installer against, so what gets uploaded is exactly what an
install carries. Qt frames now get unwinding plus export-table names; line
numbers still need Qt's debug information files, which the SDK omits.

The guards are not cosmetic: the step runs under `bash -eo pipefail`, where
a sed over a missing file or an unmatched ls glob aborts the whole step, and
continue-on-error makes that look identical to a successful upload.

* [CrashReporting] Upload only the binaries, archive symbols only on tags

The Linux leg handed sentry-cli whole build directories: 471 candidate files,
of which 7 were object files that can never match a stack frame. Only the
three linked binaries carry the build-ids Sentry looks up; the packaged copy
is split and stripped but keeps that same build-id.

The Windows symbol archive ran on every packaging build even though only tag
builds upload to Sentry, so a PR build paid for an artifact nothing could
ever consume. Tags keep it as the durable copy for a failed upload.

---------

Co-authored-by: Lukas Brübach <Bruebach.Lukas@bdosecurity.de>
2026-10-19 14:57:04 +02:00

181 lines
8.8 KiB
CMake

# Crash reporting via sentry-native (Sentry protocol, MIT).
# Included once from the root CMakeLists.txt, right after the sanitizer/coverage helpers.
#
# Inputs:
# WITH_CRASH_REPORTING -- build the client with the sentry-native SDK (default OFF, our release
# CI passes ON; packagers and from-source builds stay SDK-free until this
# ships something real, so offline package builds never fetch the SDK)
# COCKATRICE_SENTRY_DSN -- Sentry DSN baked into the build (default empty = SDK init skipped at runtime)
# COCKATRICE_SENTRY_TAG -- sentry-native release tag to vendor via FetchContent
# ENABLE_SANITIZERS -- forces the sentry backend to "none" (crashpad vs ASan conflict)
# Defines: cockatrice_target_enable_crash_reporting(<target>)
option(WITH_CRASH_REPORTING "Build with crash reporting via sentry-native" OFF)
set(COCKATRICE_SENTRY_DSN
""
CACHE STRING "Sentry DSN baked into the client; empty disables crash-report upload at runtime"
)
# Pinned stable release; tag verified at https://github.com/getsentry/sentry-native/tags.
set(COCKATRICE_SENTRY_TAG
"0.17.3"
CACHE STRING "sentry-native release tag to vendor via FetchContent"
)
# Wire the SDK (or the no-op fallback) into a client target. Always callable: with the
# option OFF the compile definitions below make crash_reporter.cpp compile to no-ops.
function(cockatrice_target_enable_crash_reporting _target)
if(WITH_CRASH_REPORTING AND TARGET sentry::sentry)
target_link_libraries(${_target} PRIVATE sentry::sentry)
target_compile_definitions(${_target} PRIVATE COCKATRICE_WITH_CRASH_REPORTING=1)
else()
target_compile_definitions(${_target} PRIVATE COCKATRICE_WITH_CRASH_REPORTING=0)
endif()
# Always baked in (possibly empty): main() prefers the COCKATRICE_SENTRY_DSN env var at runtime.
target_compile_definitions(${_target} PRIVATE "COCKATRICE_SENTRY_DSN=\"${COCKATRICE_SENTRY_DSN}\"")
# Sentry release for events from this target: PROJECT_VERSION (getversion.cmake ran before this
# file was included). Deliberately not PROJECT_VERSION_FRIENDLY / VERSION_STRING — that carries
# a build date, so every build of the same version would land in a different Sentry release.
# PROJECT_VERSION matches what the installer reports in its own log and what CPack derives the
# package version from, so client and installer events group under one release.
target_compile_definitions(${_target} PRIVATE "COCKATRICE_SENTRY_RELEASE=\"${PROJECT_VERSION}\"")
endfunction()
if(WITH_CRASH_REPORTING AND WITH_CLIENT)
include(FetchContent)
if(ENABLE_SANITIZERS)
# The crash handler cannot run under ASan/UBSan, so there is no backend to capture with.
set(_sentry_backend "none")
else()
# Breakpad on every OS (increment 3 probe, 2026-10-14): the crashpad sources
# do vendor friction-free (FetchContent inits all submodules recursively by
# default, verified populated in a local build tree), but the crashpad backend
# would need its out-of-process crashpad_handler binary installed next to the
# exe, into the macOS bundle, and through NSIS, plus handler-path wiring and
# notarization care. Breakpad is in-process, needs no extra binary and no
# packaging on any OS sentry-native builds, so simplicity wins.
set(_sentry_backend "breakpad")
endif()
# FORCE so toggling ENABLE_SANITIZERS on a reused build dir cannot leave a stale backend behind.
set(SENTRY_BACKEND
"${_sentry_backend}"
CACHE STRING "Crash handler backend for sentry-native" FORCE
)
unset(_sentry_backend)
if(NOT MSVC)
# Symbol upload needs debug info in Release binaries: the top-level flags build GCC
# Release as -s -O2 (stripped, no DWARF) and Clang/Apple Release as plain -O2 (no DWARF),
# both of which upload nothing useful. Keep -g and drop the strip flag while the SDK is
# on; the OFF default leaves local, distro and Debug builds untouched. MSVC already
# links /DEBUG with PDBs in release (see top-level CMakeLists.txt), so it needs nothing.
string(REGEX REPLACE "(^| )-s( |$)" "\\1\\2" CMAKE_CXX_FLAGS_RELEASE "${CMAKE_CXX_FLAGS_RELEASE}")
if(NOT CMAKE_CXX_FLAGS_RELEASE MATCHES "(^| )-g( |$)")
set(CMAKE_CXX_FLAGS_RELEASE "${CMAKE_CXX_FLAGS_RELEASE} -g")
endif()
endif()
# No Qt integration: it runs find_package(Qt6) inside the vendored build, which
# cannot see our thin/aqt Qt layouts (macOS CI) under foreign toolchains, and its
# breadcrumbs could carry unredacted log lines into native crash events that bypass
# before_send. Our attached redacted log preview already carries the log context.
set(SENTRY_INTEGRATION_QT
OFF
CACHE BOOL "Qt log-message breadcrumbs for sentry-native (off: discovery + secret surface)" FORCE
)
# The transport stays at the SDK default (curl on Apple/Linux, winhttp on Windows).
set(SENTRY_BUILD_TESTS
OFF
CACHE BOOL "Build sentry-native tests" FORCE
)
set(SENTRY_BUILD_EXAMPLES
OFF
CACHE BOOL "Build sentry-native examples" FORCE
)
fetchcontent_declare(
sentry
GIT_REPOSITORY https://github.com/getsentry/sentry-native.git
GIT_TAG ${COCKATRICE_SENTRY_TAG}
GIT_SHALLOW TRUE
)
fetchcontent_makeavailable(sentry)
# Vendored sentry-native (and its breakpad/crashpad trees) warn under our -Wall -Wextra
# (e.g. breakpad's -Wmissing-field-initializers), and WARNING_AS_ERROR would fail the build
# on code we cannot fix. Those findings are not actionable for us, so opt the whole vendored
# tree out of warnings-as-errors while keeping the warnings themselves visible; our own
# targets keep -Werror. Interface libraries compile nothing, so they are skipped outright:
# an INTERFACE -Wno-error would propagate to our consumers and weaken their build.
function(_cockatrice_sentry_opt_out_of_werror _dir)
get_property(
_sentry_targets
DIRECTORY "${_dir}"
PROPERTY BUILDSYSTEM_TARGETS
)
foreach(_sentry_target IN LISTS _sentry_targets)
get_target_property(_sentry_type ${_sentry_target} TYPE)
if(_sentry_type STREQUAL "INTERFACE_LIBRARY")
continue()
elseif(_sentry_type MATCHES "LIBRARY|EXECUTABLE")
get_target_property(_sentry_imported ${_sentry_target} IMPORTED)
if(NOT _sentry_imported)
target_compile_options(${_sentry_target} PRIVATE -Wno-error)
endif()
endif()
endforeach()
get_property(
_sentry_subdirs
DIRECTORY "${_dir}"
PROPERTY SUBDIRECTORIES
)
foreach(_sentry_subdir IN LISTS _sentry_subdirs)
_cockatrice_sentry_opt_out_of_werror("${_sentry_subdir}")
endforeach()
endfunction()
if(NOT MSVC)
_cockatrice_sentry_opt_out_of_werror("${sentry_SOURCE_DIR}")
endif()
message(STATUS "Crash reporting: sentry-native ${COCKATRICE_SENTRY_TAG} (backend: ${SENTRY_BACKEND})")
else()
message(STATUS "Crash reporting: SDK not vendored "
"(WITH_CRASH_REPORTING=${WITH_CRASH_REPORTING}, WITH_CLIENT=${WITH_CLIENT})"
)
endif()
# Packaged release binaries must not ship the DWARF that the symbol upload needs: it is
# 80-90% of an unstripped binary and nobody downloads a 500 MB installer to get it. The
# build tree keeps -g (CI uploads symbols from there); the *packaged copy* is split at
# install time by a CPack install script, which extracts the debug info next to the staging
# dir and strips the binary. Only ELF (Linux/BSD) needs this: Windows links /DEBUG with
# separate PDBs, and macOS keeps its dSYM extracted from the bundle.
option(COCKATRICE_SPLIT_DEBUG_SYMBOLS "Strip DWARF out of packaged release binaries" ON)
set(COCKATRICE_DEBUG_SYMBOL_DIR
"${CMAKE_BINARY_DIR}/debug-symbols"
CACHE PATH "Where the packaged binaries' split debug information is written (kept out of the package)"
)
if(WITH_CRASH_REPORTING
AND WITH_CLIENT
AND COCKATRICE_SPLIT_DEBUG_SYMBOLS
AND CMAKE_BUILD_TYPE STREQUAL "Release"
AND UNIX
AND NOT APPLE
)
find_program(COCKATRICE_OBJCOPY NAMES objcopy)
find_program(COCKATRICE_STRIP_TOOL NAMES strip)
if(COCKATRICE_OBJCOPY AND COCKATRICE_STRIP_TOOL)
configure_file(
"${CMAKE_CURRENT_LIST_DIR}/SplitDebugSymbols.cmake.in" "${CMAKE_BINARY_DIR}/cmake/SplitDebugSymbols.cmake" @ONLY
)
# PRE_BUILD runs after CPack stages the install tree and before the package is produced.
# CPACK_INSTALL_SCRIPTS would run too early (staging is still empty); the script walks
# CPACK_TEMPORARY_DIRECTORY, never CMAKE_INSTALL_PREFIX, and refuses to run otherwise.
list(APPEND CPACK_PRE_BUILD_SCRIPTS "${CMAKE_BINARY_DIR}/cmake/SplitDebugSymbols.cmake")
message(STATUS "Crash reporting: packaged binaries stripped, debug info split to ${COCKATRICE_DEBUG_SYMBOL_DIR}")
else()
message(WARNING "COCKATRICE_SPLIT_DEBUG_SYMBOLS is ON but objcopy/strip were not found; "
"packaged release binaries will keep their debug information"
)
endif()
endif()