[CrashReporting] Breakpad capture, session detection and symbol upload (#7523)

* [CrashReporting] Breakpad everywhere, session detection, symbol CI (inc 3)

Breakpad on all platforms (no crashpad packaging); remove the Windows
MiniDumpWithFullMemory filter and the installer WER LocalDumps full-dump
registration. QLockFile per-process session tracking replaces the
Windows-only .dmp scan, placed after parser.process and the instance
hand-off so help/version exits and handed-off secondaries never leak a
lock. CI: Release-only DSN bake from SENTRY_DSN secret, tag-only
sentry-cli symbol upload (org cockatrice, project native).

Tests: 533/533 ctest green, format clean, code gate PASS.

* [CrashReporting] Enable http_retry, harden symbol upload, fix NSIS context

Sets sentry_options_set_http_retry so the cache_keep-held crash
envelope in Ask mode is written with a retry filename and actually
flushes on giveConsent() instead of sitting in <db>/cache until
pruning.

Adds continue-on-error to both sentry-cli install steps so a Sentry
registry outage or broken Git Bash install cannot block release
assets.

Cleans the legacy WER CrashDumps folder under SetShellVarContext
current (under all it resolves to ProgramData, not the user's
AppData) in installer and uninstaller; registry delete works in
either context.

* [CrashReporting] Extract the startup crash-report flow from main() per review

* [CrashReporting] Break up the CrashReporter class doc per review

* [CrashReporting] Drop the invalid --release from the symbol upload

`debug-files upload` never accepted `--release` (checked against the pinned
sentry-cli 3.8.0: the subcommand declares --org/--project and a set of
type/id/scan options, and execute() reads no release; the only global flags are
--url/--header/--auth-token/--log-level/--quiet/--allow-failure). Clap rejects
the unknown flag before any upload happens, so every tag build has been shipping
without symbols, with continue-on-error keeping the release pipeline green.

Release association is also the wrong mechanism: debug information files are
matched to events by debug ID via the event's debug_meta. The Sentry release of
a native event comes from the event itself (COCKATRICE_SENTRY_RELEASE), not from
the DIF upload. Remove the flag and the now-unused SENTRY_RELEASE env from both
invocations (Linux, and Windows/macOS).

The symbol upload stays continue-on-error (a Sentry outage must not block the
release assets); revisit once it has proven itself on a tag.

* [CrashReporting] Split debug information out of packaged release binaries

-g is required so the CI symbol upload has DWARF to send, but it was left in
the shipped binaries too: the Linux Release client is 563 MiB, 552 MiB of it
debug info, and every distribution channel pays for that.

Keep the build tree unchanged (CI uploads symbols from build/) and strip only
the packaged copy. A CPack pre-build script extracts each ELF's debug info to
COCKATRICE_DEBUG_SYMBOL_DIR and then runs `strip --strip-debug --strip-unneeded`.
.eh_frame is allocatable and survives, so unwinding is unaffected, and the build
ID is identical in the stripped binary and the sidecar.

Measured on a local Release build (GCC -O2 -g, WITH_CRASH_REPORTING=ON),
bytes:

  binary      build tree  packaged before  packaged after  .debug sidecar
  cockatrice 590,722,560      590,722,560      28,086,480     579,623,576
  oracle      53,432,608       53,432,608       1,649,888      52,075,344
  tar.gz of both trees: 289,964,521 -> 19,018,058 (~15x smaller).

Only ELF (Linux/BSD) is handled: Windows already links /DEBUG with separate
PDBs, and macOS extracts a dSYM from the bundle. Opt-out via
COCKATRICE_SPLIT_DEBUG_SYMBOLS; skipped with a warning when objcopy/strip are
missing. CPACK_PRE_BUILD_SCRIPTS (not CPACK_INSTALL_SCRIPTS, which runs before
staging) invokes it, and the script refuses to run unless CPACK_TEMPORARY_DIRECTORY
is a real CPack staging tree so a missing variable can never turn its glob into
a filesystem-wide walk.

* [CrashReporting] Upload Windows PDBs from where Ninja actually puts them

The Windows branch of the symbol upload looked in build/<target>/Release, which
is the Visual Studio generator's layout. The Windows leg uses Ninja, which is
single-config: linker PDBs land in build/<target>/, next to the binaries — the
same paths the "[Windows] Upload PDBs" artifact step below globs with
if-no-files-found: error.

So paths was always empty on Windows, the step printed "No symbol paths found,
skipping symbol upload" and exited 0: a silent no-op behind continue-on-error.
Every Windows build would then report "A required debug information file was
missing" for cockatrice.exe with a PDB-style debug id (GUID-age), exactly as if
no upload step existed at all.

Use the artifact step's globs. Also turn the empty-path case into a failure in
both symbol steps: a silent skip is indistinguishable from a successful upload
(which is how the wrong path went unnoticed), and continue-on-error already
keeps a broken upload from failing the release.

Verified by extracting both steps' run blocks and executing the Windows branch
against a Ninja-style tree: it now passes the three linker PDBs to sentry-cli
(and not the CMakeFiles/*.pdb intermediates), and exits non-zero with an
::error:: annotation when there is nothing to upload.

* [CrashReporting] Archive PDBs on tag builds so a failed upload is recoverable

The "[Windows] Upload PDBs" artifact step skipped tag builds
(github.ref_type != 'tag'), so for a release the symbols existed only inside
the Sentry upload — which is continue-on-error. A failed upload therefore lost
the ability to symbolicate that release permanently: no rebuild produces the
same PDB, since the debug id is a per-build GUID+age. That is how every
Windows crash reported so far became unreadable.

Archive the PDBs for tag builds as well, and order the step after
"Upload to release" so a missing PDB can never block a release from being
published. Retention stays at the repository default; nothing consumes the
artifact automatically, it is the manual recovery path
(gh run download <run> -n <name>-PDBs, then sentry-cli debug-files upload).

Verified by re-parsing the workflow and asserting the step order
(release -> PDBs -> attest) and the broadened condition.

* [CrashReporting] Upload the Windows executables with their PDBs

Sentry needs two things per image on Windows: symbolication and stack
unwinding. The PDB covers symbolication, but for a 64-bit PE the Call Frame
Information (.pdata/.xdata) lives in the executable and is not copied into the
PDB, so uploading PDBs alone leaves the image at "Stack Unwinding: Missing".
That is exactly what the event's Images panel showed for cockatrice.exe, while
Sentry's own Microsoft symbol server resolved KERNELBASE.dll and ucrtbase.dll
from the same dump.

sentry-cli also needs the pair in the same invocation: only then does it
associate the PE's code identifier with the PDB's debug identifier, otherwise
"the identifiers diverge and Sentry might not be able to resolve the correct
file for symbolication" (docs: Debug Identifiers).

Do the same for macOS, where the dSYM carries the debug information but the
Mach-O carries the unwind information, and include the executables in the
symbol artifact so a re-upload still has the pair to offer.

Verified by extracting the step and running both branches against stub
harnesses: Windows passes the three exe/pdb pairs in a single sentry-cli
invocation and still fails loudly on an empty tree; macOS passes the dSYM and
the Mach-O together.

* [CrashReporting] Upload the shipped Qt runtime DLLs too

Qt ships no PDBs with its MSVC SDK, so every Qt frame in a crash report
resolved to nothing: an image with neither PDB nor PE has no unwind
information (on 64-bit Windows the Call Frame Information lives in the PE
and is not copied into the PDB) and no symbol source to fall back on.

The list is required-runtime.txt, the manifest .ci/compile.sh already
asserts the installer against, so what gets uploaded is exactly what an
install carries. Qt frames now get unwinding plus export-table names; line
numbers still need Qt's debug information files, which the SDK omits.

The guards are not cosmetic: the step runs under `bash -eo pipefail`, where
a sed over a missing file or an unmatched ls glob aborts the whole step, and
continue-on-error makes that look identical to a successful upload.

* [CrashReporting] Upload only the binaries, archive symbols only on tags

The Linux leg handed sentry-cli whole build directories: 471 candidate files,
of which 7 were object files that can never match a stack frame. Only the
three linked binaries carry the build-ids Sentry looks up; the packaged copy
is split and stripped but keeps that same build-id.

The Windows symbol archive ran on every packaging build even though only tag
builds upload to Sentry, so a PR build paid for an artifact nothing could
ever consume. Tags keep it as the durable copy for a failed upload.

---------

Co-authored-by: Lukas Brübach <Bruebach.Lukas@bdosecurity.de>

cmake/CrashReporting.cmake

@@ -46,12 +46,14 @@ if(WITH_CRASH_REPORTING AND WITH_CLIENT)
   if(ENABLE_SANITIZERS)
     # The crash handler cannot run under ASan/UBSan, so there is no backend to capture with.
     set(_sentry_backend "none")
-  elseif(WIN32 OR APPLE)
-    # Out-of-process handler; shipping the handler binary is a later increment, which unifies
-    # every OS on breakpad (in-process, no extra binary to ship).
-    set(_sentry_backend "crashpad")
   else()
-    # In-process, no extra binary to ship.
+    # Breakpad on every OS (increment 3 probe, 2026-10-14): the crashpad sources
+    # do vendor friction-free (FetchContent inits all submodules recursively by
+    # default, verified populated in a local build tree), but the crashpad backend
+    # would need its out-of-process crashpad_handler binary installed next to the
+    # exe, into the macOS bundle, and through NSIS, plus handler-path wiring and
+    # notarization care. Breakpad is in-process, needs no extra binary and no
+    # packaging on any OS sentry-native builds, so simplicity wins.
     set(_sentry_backend "breakpad")
   endif()
   # FORCE so toggling ENABLE_SANITIZERS on a reused build dir cannot leave a stale backend behind.
@@ -61,6 +63,18 @@ if(WITH_CRASH_REPORTING AND WITH_CLIENT)
   )
   unset(_sentry_backend)
 
+  if(NOT MSVC)
+    # Symbol upload needs debug info in Release binaries: the top-level flags build GCC
+    # Release as -s -O2 (stripped, no DWARF) and Clang/Apple Release as plain -O2 (no DWARF),
+    # both of which upload nothing useful. Keep -g and drop the strip flag while the SDK is
+    # on; the OFF default leaves local, distro and Debug builds untouched. MSVC already
+    # links /DEBUG with PDBs in release (see top-level CMakeLists.txt), so it needs nothing.
+    string(REGEX REPLACE "(^| )-s( |$)" "\\1\\2" CMAKE_CXX_FLAGS_RELEASE "${CMAKE_CXX_FLAGS_RELEASE}")
+    if(NOT CMAKE_CXX_FLAGS_RELEASE MATCHES "(^| )-g( |$)")
+      set(CMAKE_CXX_FLAGS_RELEASE "${CMAKE_CXX_FLAGS_RELEASE} -g")
+    endif()
+  endif()
+
   # No Qt integration: it runs find_package(Qt6) inside the vendored build, which
   # cannot see our thin/aqt Qt layouts (macOS CI) under foreign toolchains, and its
   # breadcrumbs could carry unredacted log lines into native crash events that bypass
@@ -129,3 +143,39 @@ else()
                  "(WITH_CRASH_REPORTING=${WITH_CRASH_REPORTING}, WITH_CLIENT=${WITH_CLIENT})"
   )
 endif()
+
+# Packaged release binaries must not ship the DWARF that the symbol upload needs: it is
+# 80-90% of an unstripped binary and nobody downloads a 500 MB installer to get it. The
+# build tree keeps -g (CI uploads symbols from there); the *packaged copy* is split at
+# install time by a CPack install script, which extracts the debug info next to the staging
+# dir and strips the binary. Only ELF (Linux/BSD) needs this: Windows links /DEBUG with
+# separate PDBs, and macOS keeps its dSYM extracted from the bundle.
+option(COCKATRICE_SPLIT_DEBUG_SYMBOLS "Strip DWARF out of packaged release binaries" ON)
+set(COCKATRICE_DEBUG_SYMBOL_DIR
+    "${CMAKE_BINARY_DIR}/debug-symbols"
+    CACHE PATH "Where the packaged binaries' split debug information is written (kept out of the package)"
+)
+if(WITH_CRASH_REPORTING
+   AND WITH_CLIENT
+   AND COCKATRICE_SPLIT_DEBUG_SYMBOLS
+   AND CMAKE_BUILD_TYPE STREQUAL "Release"
+   AND UNIX
+   AND NOT APPLE
+)
+  find_program(COCKATRICE_OBJCOPY NAMES objcopy)
+  find_program(COCKATRICE_STRIP_TOOL NAMES strip)
+  if(COCKATRICE_OBJCOPY AND COCKATRICE_STRIP_TOOL)
+    configure_file(
+      "${CMAKE_CURRENT_LIST_DIR}/SplitDebugSymbols.cmake.in" "${CMAKE_BINARY_DIR}/cmake/SplitDebugSymbols.cmake" @ONLY
+    )
+    # PRE_BUILD runs after CPack stages the install tree and before the package is produced.
+    # CPACK_INSTALL_SCRIPTS would run too early (staging is still empty); the script walks
+    # CPACK_TEMPORARY_DIRECTORY, never CMAKE_INSTALL_PREFIX, and refuses to run otherwise.
+    list(APPEND CPACK_PRE_BUILD_SCRIPTS "${CMAKE_BINARY_DIR}/cmake/SplitDebugSymbols.cmake")
+    message(STATUS "Crash reporting: packaged binaries stripped, debug info split to ${COCKATRICE_DEBUG_SYMBOL_DIR}")
+  else()
+    message(WARNING "COCKATRICE_SPLIT_DEBUG_SYMBOLS is ON but objcopy/strip were not found; "
+                    "packaged release binaries will keep their debug information"
+    )
+  endif()
+endif()

cmake/NSIS.template.in

@@ -1530,11 +1530,22 @@ ${If} $PortableMode = 0
 	${GetSize} "$INSTDIR" "/S=0K" $0 $1 $2
 	IntFmt $0 "0x%08X" $0
 
-	; Enable Windows User-Mode Dumps
+	; No Windows Error Reporting LocalDumps registration: crash capture belongs to
+	; the in-app reporter (Sentry/breakpad, MiniDumpNormal-class data only). Opting
+	; machines into OS-level full dumps (DumpType 2) would leave unredacted memory
+	; dumps on user machines that nothing consumes.
 	; https://learn.microsoft.com/en-us/windows/win32/wer/collecting-user-mode-dumps
-	WriteRegExpandStr HKLM "Software\Microsoft\Windows\Windows Error Reporting\LocalDumps\cockatrice.exe" "DumpFolder" "%LOCALAPPDATA%\CrashDumps\Cockatrice"
-	WriteRegDWORD HKLM "Software\Microsoft\Windows\Windows Error Reporting\LocalDumps\cockatrice.exe" "DumpCount" "5"
-	WriteRegDWORD HKLM "Software\Microsoft\Windows\Windows Error Reporting\LocalDumps\cockatrice.exe" "DumpType" "2"
+
+	; Older installers did register cockatrice.exe above (and the removed
+	; MiniDumpWithFullMemory writer filled that folder), and that key survives this
+	; install, so WER would keep writing full dumps. Remove it, and clear the
+	; legacy dumps nothing consumes anymore. $LOCALAPPDATA resolves per shell context:
+	; under "all" it points at ProgramData, so switch to "current" for the per-user
+	; dumps folder and back afterwards; the registry delete works in either context.
+	DeleteRegKey HKLM "Software\Microsoft\Windows\Windows Error Reporting\LocalDumps\cockatrice.exe"
+	SetShellVarContext current
+	RMDir /r "$LOCALAPPDATA\CrashDumps\Cockatrice"
+	SetShellVarContext all
 
 	WriteRegStr HKLM "Software\Microsoft\Windows\CurrentVersion\Uninstall\Cockatrice" "DisplayIcon" "$INSTDIR\cockatrice.exe"
 	WriteRegStr HKLM "Software\Microsoft\Windows\CurrentVersion\Uninstall\Cockatrice" "DisplayName" "Cockatrice"
@@ -1635,6 +1646,15 @@ Section "un.Application" UnSecApplication
 
 	DeleteRegKey HKLM "Software\Microsoft\Windows\CurrentVersion\Uninstall\Cockatrice"
 
+	; Leftover WER LocalDumps key from pre-Sentry installers (see the installer note
+	; above): removing it here covers machines that uninstall without reinstalling.
+	; The legacy full-memory dumps go with it; nothing consumes them anymore. Same shell
+	; context note as the installer: the per-user dumps folder needs "current".
+	DeleteRegKey HKLM "Software\Microsoft\Windows\Windows Error Reporting\LocalDumps\cockatrice.exe"
+	SetShellVarContext current
+	RMDir /r "$LOCALAPPDATA\CrashDumps\Cockatrice"
+	SetShellVarContext all
+
 	; Only remove the file/protocol associations if we registered them (i.e. the
 	; install was not portable) and .cod is still owned by Cockatrice, so we don't
 	; clobber a .cod association installed by another application.

cmake/SplitDebugSymbols.cmake.in

@@ -0,0 +1,80 @@
+# Splits DWARF out of the packaged ELF binaries, then strips them.
+#
+# This is a CPack pre-build script (CPACK_PRE_BUILD_SCRIPTS, wired in CrashReporting.cmake):
+# CPack runs it after staging the install tree and before producing the package, so it only
+# ever touches the copy that becomes the release artifact. The build tree is left alone: -g
+# stays on, and CI uploads symbols to Sentry straight from build/.
+#
+# For every ELF file under CPACK_TEMPORARY_DIRECTORY the script extracts the debug information
+# to COCKATRICE_DEBUG_SYMBOL_DIR and removes the debug sections from the packaged binary.
+# COCKATRICE_DEBUG_SYMBOL_DIR is deliberately outside the staging tree, so sidecars never end
+# up in the package; CI can archive/upload them if wanted (it normally uploads straight from
+# the build tree). The only thing that must survive stripping is .eh_frame (allocatable, used
+# for unwinding), which neither --only-keep-debug nor --strip-debug removes.
+#
+# Tool paths, the symbol directory and the tree to walk are baked in / provided at configure /
+# run time; the guards below refuse to run on anything that is not a CPack staging directory,
+# so a missing variable can never turn the glob into a filesystem-wide walk.
+
+set(_objcopy "@COCKATRICE_OBJCOPY@")
+set(_strip "@COCKATRICE_STRIP_TOOL@")
+set(_symbolDir "@COCKATRICE_DEBUG_SYMBOL_DIR@")
+
+if(_objcopy STREQUAL "" OR _strip STREQUAL "")
+  message(WARNING "split debug symbols: objcopy/strip not found at configure time, packaging unstripped binaries")
+  return()
+endif()
+if(_symbolDir STREQUAL "")
+  message(FATAL_ERROR "split debug symbols: COCKATRICE_DEBUG_SYMBOL_DIR is empty")
+endif()
+
+# CPack sets CPACK_TEMPORARY_DIRECTORY to the staging tree; CMAKE_INSTALL_PREFIX is still the
+# configured /usr prefix here, so globbing it would walk system directories. Refuse to run
+# unless the path exists and really is a CPack staging tree.
+if(NOT CPACK_TEMPORARY_DIRECTORY OR NOT IS_DIRECTORY "${CPACK_TEMPORARY_DIRECTORY}")
+  message(STATUS "split debug symbols: no CPack staging directory, skipping")
+  return()
+endif()
+if(NOT CPACK_TEMPORARY_DIRECTORY MATCHES "_CPack_Packages")
+  message(WARNING "split debug symbols: ${CPACK_TEMPORARY_DIRECTORY} is not a CPack staging tree, skipping")
+  return()
+endif()
+
+file(MAKE_DIRECTORY "${_symbolDir}")
+
+file(
+  GLOB_RECURSE _entries
+  LIST_DIRECTORIES true
+  "${CPACK_TEMPORARY_DIRECTORY}/*"
+)
+set(_processed 0)
+foreach(_file IN LISTS _entries)
+  if(IS_DIRECTORY "${_file}" OR IS_SYMLINK "${_file}")
+    continue()
+  endif()
+  # Only ELF: objcopy would fail on the icons, translations and Qt resources next to it.
+  file(
+    READ "${_file}" _magic
+    OFFSET 0
+    LIMIT 4
+    HEX
+  )
+  if(NOT _magic STREQUAL "7f454c46")
+    continue()
+  endif()
+  get_filename_component(_name "${_file}" NAME)
+  set(_debug "${_symbolDir}/${_name}.debug")
+  execute_process(COMMAND "${_objcopy}" --only-keep-debug "${_file}" "${_debug}" RESULT_VARIABLE _rc)
+  if(NOT _rc EQUAL 0)
+    message(WARNING "split debug symbols: objcopy --only-keep-debug failed for ${_file}")
+    continue()
+  endif()
+  execute_process(COMMAND "${_strip}" --strip-debug --strip-unneeded "${_file}" RESULT_VARIABLE _rc)
+  if(NOT _rc EQUAL 0)
+    message(WARNING "split debug symbols: strip failed for ${_file}")
+    continue()
+  endif()
+  math(EXPR _processed "${_processed} + 1")
+endforeach()
+
+message(STATUS "Split debug symbols for ${_processed} packaged ELF file(s); sidecars in ${_symbolDir}")