jeirmeister/Cockatrice
1
0
Fork 0
mirror of https://github.com/Cockatrice/Cockatrice.git synced 2026-11-05 09:33:25 -08:00
Code Issues Projects Releases Packages Wiki Activity Actions 5

[CrashReporting] Breakpad capture, session detection and symbol upload (#7523)

Browse source 
* [CrashReporting] Breakpad everywhere, session detection, symbol CI (inc 3)

Breakpad on all platforms (no crashpad packaging); remove the Windows
MiniDumpWithFullMemory filter and the installer WER LocalDumps full-dump
registration. QLockFile per-process session tracking replaces the
Windows-only .dmp scan, placed after parser.process and the instance
hand-off so help/version exits and handed-off secondaries never leak a
lock. CI: Release-only DSN bake from SENTRY_DSN secret, tag-only
sentry-cli symbol upload (org cockatrice, project native).

Tests: 533/533 ctest green, format clean, code gate PASS.

* [CrashReporting] Enable http_retry, harden symbol upload, fix NSIS context

Sets sentry_options_set_http_retry so the cache_keep-held crash
envelope in Ask mode is written with a retry filename and actually
flushes on giveConsent() instead of sitting in <db>/cache until
pruning.

Adds continue-on-error to both sentry-cli install steps so a Sentry
registry outage or broken Git Bash install cannot block release
assets.

Cleans the legacy WER CrashDumps folder under SetShellVarContext
current (under all it resolves to ProgramData, not the user's
AppData) in installer and uninstaller; registry delete works in
either context.

* [CrashReporting] Extract the startup crash-report flow from main() per review

* [CrashReporting] Break up the CrashReporter class doc per review

* [CrashReporting] Drop the invalid --release from the symbol upload

`debug-files upload` never accepted `--release` (checked against the pinned
sentry-cli 3.8.0: the subcommand declares --org/--project and a set of
type/id/scan options, and execute() reads no release; the only global flags are
--url/--header/--auth-token/--log-level/--quiet/--allow-failure). Clap rejects
the unknown flag before any upload happens, so every tag build has been shipping
without symbols, with continue-on-error keeping the release pipeline green.

Release association is also the wrong mechanism: debug information files are
matched to events by debug ID via the event's debug_meta. The Sentry release of
a native event comes from the event itself (COCKATRICE_SENTRY_RELEASE), not from
the DIF upload. Remove the flag and the now-unused SENTRY_RELEASE env from both
invocations (Linux, and Windows/macOS).

The symbol upload stays continue-on-error (a Sentry outage must not block the
release assets); revisit once it has proven itself on a tag.

* [CrashReporting] Split debug information out of packaged release binaries

-g is required so the CI symbol upload has DWARF to send, but it was left in
the shipped binaries too: the Linux Release client is 563 MiB, 552 MiB of it
debug info, and every distribution channel pays for that.

Keep the build tree unchanged (CI uploads symbols from build/) and strip only
the packaged copy. A CPack pre-build script extracts each ELF's debug info to
COCKATRICE_DEBUG_SYMBOL_DIR and then runs `strip --strip-debug --strip-unneeded`.
.eh_frame is allocatable and survives, so unwinding is unaffected, and the build
ID is identical in the stripped binary and the sidecar.

Measured on a local Release build (GCC -O2 -g, WITH_CRASH_REPORTING=ON),
bytes:

  binary      build tree  packaged before  packaged after  .debug sidecar
  cockatrice 590,722,560      590,722,560      28,086,480     579,623,576
  oracle      53,432,608       53,432,608       1,649,888      52,075,344
  tar.gz of both trees: 289,964,521 -> 19,018,058 (~15x smaller).

Only ELF (Linux/BSD) is handled: Windows already links /DEBUG with separate
PDBs, and macOS extracts a dSYM from the bundle. Opt-out via
COCKATRICE_SPLIT_DEBUG_SYMBOLS; skipped with a warning when objcopy/strip are
missing. CPACK_PRE_BUILD_SCRIPTS (not CPACK_INSTALL_SCRIPTS, which runs before
staging) invokes it, and the script refuses to run unless CPACK_TEMPORARY_DIRECTORY
is a real CPack staging tree so a missing variable can never turn its glob into
a filesystem-wide walk.

* [CrashReporting] Upload Windows PDBs from where Ninja actually puts them

The Windows branch of the symbol upload looked in build/<target>/Release, which
is the Visual Studio generator's layout. The Windows leg uses Ninja, which is
single-config: linker PDBs land in build/<target>/, next to the binaries — the
same paths the "[Windows] Upload PDBs" artifact step below globs with
if-no-files-found: error.

So paths was always empty on Windows, the step printed "No symbol paths found,
skipping symbol upload" and exited 0: a silent no-op behind continue-on-error.
Every Windows build would then report "A required debug information file was
missing" for cockatrice.exe with a PDB-style debug id (GUID-age), exactly as if
no upload step existed at all.

Use the artifact step's globs. Also turn the empty-path case into a failure in
both symbol steps: a silent skip is indistinguishable from a successful upload
(which is how the wrong path went unnoticed), and continue-on-error already
keeps a broken upload from failing the release.

Verified by extracting both steps' run blocks and executing the Windows branch
against a Ninja-style tree: it now passes the three linker PDBs to sentry-cli
(and not the CMakeFiles/*.pdb intermediates), and exits non-zero with an
::error:: annotation when there is nothing to upload.

* [CrashReporting] Archive PDBs on tag builds so a failed upload is recoverable

The "[Windows] Upload PDBs" artifact step skipped tag builds
(github.ref_type != 'tag'), so for a release the symbols existed only inside
the Sentry upload — which is continue-on-error. A failed upload therefore lost
the ability to symbolicate that release permanently: no rebuild produces the
same PDB, since the debug id is a per-build GUID+age. That is how every
Windows crash reported so far became unreadable.

Archive the PDBs for tag builds as well, and order the step after
"Upload to release" so a missing PDB can never block a release from being
published. Retention stays at the repository default; nothing consumes the
artifact automatically, it is the manual recovery path
(gh run download <run> -n <name>-PDBs, then sentry-cli debug-files upload).

Verified by re-parsing the workflow and asserting the step order
(release -> PDBs -> attest) and the broadened condition.

* [CrashReporting] Upload the Windows executables with their PDBs

Sentry needs two things per image on Windows: symbolication and stack
unwinding. The PDB covers symbolication, but for a 64-bit PE the Call Frame
Information (.pdata/.xdata) lives in the executable and is not copied into the
PDB, so uploading PDBs alone leaves the image at "Stack Unwinding: Missing".
That is exactly what the event's Images panel showed for cockatrice.exe, while
Sentry's own Microsoft symbol server resolved KERNELBASE.dll and ucrtbase.dll
from the same dump.

sentry-cli also needs the pair in the same invocation: only then does it
associate the PE's code identifier with the PDB's debug identifier, otherwise
"the identifiers diverge and Sentry might not be able to resolve the correct
file for symbolication" (docs: Debug Identifiers).

Do the same for macOS, where the dSYM carries the debug information but the
Mach-O carries the unwind information, and include the executables in the
symbol artifact so a re-upload still has the pair to offer.

Verified by extracting the step and running both branches against stub
harnesses: Windows passes the three exe/pdb pairs in a single sentry-cli
invocation and still fails loudly on an empty tree; macOS passes the dSYM and
the Mach-O together.

* [CrashReporting] Upload the shipped Qt runtime DLLs too

Qt ships no PDBs with its MSVC SDK, so every Qt frame in a crash report
resolved to nothing: an image with neither PDB nor PE has no unwind
information (on 64-bit Windows the Call Frame Information lives in the PE
and is not copied into the PDB) and no symbol source to fall back on.

The list is required-runtime.txt, the manifest .ci/compile.sh already
asserts the installer against, so what gets uploaded is exactly what an
install carries. Qt frames now get unwinding plus export-table names; line
numbers still need Qt's debug information files, which the SDK omits.

The guards are not cosmetic: the step runs under `bash -eo pipefail`, where
a sed over a missing file or an unmatched ls glob aborts the whole step, and
continue-on-error makes that look identical to a successful upload.

* [CrashReporting] Upload only the binaries, archive symbols only on tags

The Linux leg handed sentry-cli whole build directories: 471 candidate files,
of which 7 were object files that can never match a stack frame. Only the
three linked binaries carry the build-ids Sentry looks up; the packaged copy
is split and stripped but keeps that same build-id.

The Windows symbol archive ran on every packaging build even though only tag
builds upload to Sentry, so a PR build paid for an artifact nothing could
ever consume. Tags keep it as the durable copy for a failed upload.

---------

Co-authored-by: Lukas Brübach <Bruebach.Lukas@bdosecurity.de>
This commit is contained in:
BruebachL 2026-10-19 14:57:04 +02:00 • committed by GitHub
parent ad0fd2d4b0
commit 9f59f5c495
No known key found for this signature in database
GPG key ID: B5690EEEBB952194
9 changed files with 648 additions and 156 deletions
Download patch file Download diff file Expand all files Collapse all files

222
.github/workflows/desktop-build.yml vendored
View file

@ -46,6 +46,11 @@ jobs:
outputs:
tag: ${{ steps.configure.outputs.tag }}
sha: ${{ steps.configure.outputs.sha }}
# MAJ.MIN.PATCH[-beta[.N]] of a tag build; empty otherwise. Mirrors what
# cmake/getversion.cmake derives as PROJECT_VERSION, which the client bakes as its
# Sentry release (cmake/CrashReporting.cmake) — symbol uploads must use the same
# string so stack traces de-symbolicate against this build's binaries.
sentry_release: ${{ steps.configure.outputs.sentry_release }}
steps:
- name: "Configure"
@ -56,6 +61,9 @@ jobs:
run: |
if [[ "$GITHUB_REF_TYPE" == 'tag' ]]; then # release
echo "tag=$GITHUB_REF_NAME" >> "$GITHUB_OUTPUT"
# Tag format is YYYY-MM-DD-{Release|Development}-MAJ.MIN.PATCH[-beta[.N]];
# PROJECT_VERSION is the tail after the type (see cmake/getversion.cmake).
echo "sentry_release=$(sed -E 's/^[0-9]{4}-[0-9]{2}-[0-9]{2}-(Release|Development)-//' <<< "$GITHUB_REF_NAME")" >> "$GITHUB_OUTPUT"
fi
echo "sha=$RESOLVED_SHA" >> "$GITHUB_OUTPUT"
@ -240,6 +248,11 @@ jobs:
package: '${{ matrix.package }}'
server_only: '${{ matrix.server_only }}'
WITH_CRASH_REPORTING: 'ON'
# Tag builds only: artifacts from master pushes and PR branches must not report to
# Sentry. Forks get no secrets on pull_request, so they keep the mock fallback either way.
# configure.outputs.tag is set for ANY tag, so beta (Development) tags get the DSN
# and the symbol upload below just like full releases.
COCKATRICE_SENTRY_DSN: ${{ needs.configure.outputs.tag != null && secrets.SENTRY_DSN || '' }}
run: |
source .ci/docker.sh
args=()
@ -248,9 +261,58 @@ jobs:
args+=(--ccache "$CCACHE_SIZE")
args+=(--cmake-generator "$CMAKE_GENERATOR")
args+=(--suffix "$SUFFIX")
# The container inherits no environment, so the DSN rides in explicitly. -e without
# =value passes it through (possibly empty); compile.sh only consumes it for Release
# configures, and empty keeps the mock fallback.
export RUN_ARGS="${RUN_ARGS:-} -e COCKATRICE_SENTRY_DSN"
RUN --server --release --package "$package" "${args[@]}"
- name: "Install sentry-cli"
if: matrix.package != 'skip' && needs.configure.outputs.tag != null
# Downloads from Sentry's release registry: a Sentry outage or a broken install under
# Git Bash must not take down the release assets below.
continue-on-error: true
shell: bash
env:
SENTRY_CLI_VERSION: '3.8.0'
run: |
export INSTALL_DIR="$GITHUB_WORKSPACE/.sentry-cli"
curl -sL https://sentry.io/get-cli/ | bash
echo "$GITHUB_WORKSPACE/.sentry-cli" >> "$GITHUB_PATH"
- name: "Upload debug symbols to Sentry"
# Like the DSN above, this runs for every tag — beta (Development) tags included.
if: matrix.package != 'skip' && needs.configure.outputs.tag != null
# A Sentry outage or rotated token must not take down the release assets below.
continue-on-error: true
shell: bash
env:
SENTRY_AUTH_TOKEN: ${{ secrets.SENTRY_AUTH_TOKEN }}
run: |
paths=()
# The binaries only, not the directories: a directory makes sentry-cli walk the
# whole build tree, where it finds (and uploads) object files and other fragments
# that can never match a stack frame. The packaged copy is split and stripped by
# cpack, but keeps this binary's GNU build-id, which is how Sentry matches it.
for app in cockatrice oracle servatrice; do
binary="build/$app/$app"
[[ -f $binary ]] && paths+=("$binary")
done
if [[ ${#paths[@]} == 0 ]]; then
# Fail loudly instead of skipping: see the Windows/macOS leg below — this step
# is the only way symbols reach Sentry and `continue-on-error` keeps the job
# green either way, so a silent skip looks exactly like a successful upload.
echo "::error::No debug symbol files found; nothing would be uploaded"
exit 1
fi
# Project slug on Sentry; must match the project the baked DSN points at.
# No --release here: `debug-files upload` has no such option (sentry-cli errors on
# the unknown flag and uploads nothing), and debug information files are matched to
# events by debug ID via the event's debug_meta, not by release. The release grouping
# comes from the client event itself (COCKATRICE_SENTRY_RELEASE).
sentry-cli debug-files upload --org cockatrice --project cockatrice-desktop "${paths[@]}"
# Delete used cache to emulate a ccache update. See https://github.com/actions/cache/issues/342
- name: "Delete remote compiler cache (ccache)"
if: github.ref == 'refs/heads/master' && steps.restore_ccache.outputs.cache-hit
@ -536,6 +598,11 @@ jobs:
BUILDTYPE: '${{ matrix.type }}'
CCACHE_EVICTION_AGE: ${{ matrix.ccache_eviction_age }}
CMAKE_GENERATOR: ${{ matrix.cmake_generator }}
# Tag builds only: artifacts from master pushes and PR branches must not report to
# Sentry. Forks get no secrets on pull_request, so they keep the mock fallback either way.
# configure.outputs.tag is set for ANY tag, so beta (Development) tags get the DSN
# and the symbol upload below just like full releases.
COCKATRICE_SENTRY_DSN: ${{ needs.configure.outputs.tag != null && secrets.SENTRY_DSN || '' }}
DEVELOPER_DIR: '/Applications/Xcode_${{ matrix.xcode }}.app/Contents/Developer'
GITHUB_TOKEN: ${{ github.token }} # needed for vcpkg dependency graph updates, see VCPKG_FEATURE_FLAGS
MACOS_CERTIFICATE: ${{ secrets.PROD_MACOS_CERTIFICATE }}
@ -563,6 +630,121 @@ jobs:
WITH_CRASH_REPORTING: ${{ matrix.type == 'Release' && 'ON' || '' }}
run: .ci/compile.sh --server --test --vcpkg
- name: "Install sentry-cli"
if: needs.configure.outputs.tag != null && matrix.make_package == '1'
# Same as the Linux leg above: the install downloads from Sentry's registry, so a
# failure here must not block the release upload below.
continue-on-error: true
shell: bash
env:
SENTRY_CLI_VERSION: '3.8.0'
run: |
export INSTALL_DIR="$GITHUB_WORKSPACE/.sentry-cli"
curl -sL https://sentry.io/get-cli/ | bash
echo "$GITHUB_WORKSPACE/.sentry-cli" >> "$GITHUB_PATH"
- name: "Upload debug symbols to Sentry"
# Like the DSN above, this runs for every tag — beta (Development) tags included.
if: needs.configure.outputs.tag != null && matrix.make_package == '1'
# A Sentry outage or rotated token must not take down the release assets below.
continue-on-error: true
shell: bash
env:
SENTRY_AUTH_TOKEN: ${{ secrets.SENTRY_AUTH_TOKEN }}
run: |
paths=()
if [[ '${{ matrix.os }}' == 'Windows' ]]; then
# Top-level CMakeLists.txt links /DEBUG even in release, so the linker PDBs
# land next to the binaries. Ninja is single-config, so that is
# build/<target>/ and NOT build/<target>/Release/ — the same globs the symbol
# artifact step below uses.
#
# The executables are uploaded as well, in the same invocation. On 64-bit
# Windows the Call Frame Information (.pdata/.xdata) lives in the PE and is
# not copied into the PDB, so a PDB on its own leaves Sentry reporting
# "Stack Unwinding: Missing" for the image. sentry-cli also needs to see the
# PE and its PDB together: only then does it associate the PE's code
# identifier with the PDB's debug identifier.
shopt -s nullglob
for target in cockatrice oracle servatrice; do
paths+=("build/$target"/*.exe "build/$target"/*.pdb)
done
shopt -u nullglob
# Qt's own runtime DLLs, taken from required-runtime.txt — the manifest
# .ci/compile.sh asserts the installer against — so the set uploaded is
# the set an install actually carries. The PE is the only carrier of
# unwind information for a 64-bit image, and sending it in the same
# invocation as a PDB is what lets sentry-cli associate the PE's code
# identifier with the PDB's debug identifier. Qt ships no PDBs with the
# SDK, so Qt frames resolve through the export table: names, no lines.
qt_dir=
qt_core_dir=
# Existence guards matter: the step runs under `bash -eo pipefail`, so a
# redirection-less sed over a missing file would abort the step outright
# and continue-on-error would hide it.
if [[ -f build/CMakeCache.txt ]]; then
qt_core_dir=$(sed -n 's/^Qt6Core_DIR:PATH=//p' build/CMakeCache.txt | head -1)
fi
if [[ -n $qt_core_dir ]]; then
qt_dir=$(dirname "$(dirname "$(dirname "$qt_core_dir")")")
fi
if [[ -z $qt_dir || ! -d $qt_dir ]]; then
# Warnings, not failures: the upload of our own binaries is what this
# step is for, and continue-on-error already means an upload failure
# cannot be told apart from a successful one.
echo "::warning::Qt runtime directory not resolved from build/CMakeCache.txt; Qt images get no unwind information"
elif [[ ! -f build/required-runtime.txt ]]; then
echo "::warning::build/required-runtime.txt is missing; cannot tell which Qt DLLs ship"
else
while IFS= read -r runtime; do
name=${runtime##*/}
if [[ $runtime == */* ]]; then
# A Qt plugin: the install root keeps it as plugins/<type>/<name>.
# `|| true`: an unmatched glob makes ls exit non-zero, and pipefail
# turns that into a step abort.
runtime_path=$(ls "$qt_dir/plugins"/*/"$name" 2>/dev/null | head -1) || true
else
runtime_path="$qt_dir/bin/$name"
fi
if [[ -f ${runtime_path:-} ]]; then
paths+=("$runtime_path")
else
echo "::warning::shipped Qt runtime $runtime not found under $qt_dir"
fi
done < <(grep -E '^(Qt6[^/]*\.dll|Plugins/)' build/required-runtime.txt)
fi
else
# Release builds carry DWARF (-g, kept by CrashReporting.cmake); macOS keeps its
# debug info in dSYM bundles, so materialize them before uploading.
for app in cockatrice oracle servatrice; do
binary="build/$app/$app.app/Contents/MacOS/$app"
if [[ -f $binary ]]; then
if [[ '${{ matrix.os }}' == 'macOS' ]]; then
dsymutil "$binary" -o "$binary.dSYM"
paths+=("$binary.dSYM")
# The dSYM holds the debug information, but the unwind information
# Sentry needs to walk a minidump stack lives in the Mach-O itself.
paths+=("$binary")
else
paths+=("$binary")
fi
fi
done
fi
if [[ ${#paths[@]} == 0 ]]; then
# Fail loudly instead of skipping: this step is the only way symbols reach
# Sentry, and `continue-on-error` keeps the job green either way, so a
# silent skip is indistinguishable from a successful upload. It also hid a
# wrong Windows path (build/<target>/Release, which Ninja never creates).
echo "::error::No debug symbol files found; nothing would be uploaded"
exit 1
fi
# Project slug on Sentry; must match the project the baked DSN points at.
# No --release: see the Linux leg above — the flag does not exist on
# `debug-files upload` and DIFs are matched by debug ID, not release.
sentry-cli debug-files upload --org cockatrice --project cockatrice-desktop "${paths[@]}"
# CPack reports a failed makensis run as "Problem running NSIS command"
# plus a path to NSISOutput.log, and that log lives only on the runner, so
# the actual compile error is otherwise nowhere in the workflow output.
@ -706,19 +888,6 @@ jobs:
if-no-files-found: error
path: ${{ steps.build.outputs.path }}
# Ninja is single-config: binaries and their PDBs land directly in the
# target directories instead of a Release subdirectory.
- name: "[Windows] Upload PDBs (Program Databases)"
if: matrix.os == 'Windows' && github.ref_type != 'tag'
uses: actions/upload-artifact@v7
with:
if-no-files-found: error
name: ${{ steps.build.outputs.name }}-PDBs
path: |
build/cockatrice/*.pdb
build/oracle/*.pdb
build/servatrice/*.pdb
- name: "Upload to release"
if: needs.configure.outputs.tag != null && matrix.make_package == '1'
id: upload_release
@ -730,6 +899,33 @@ jobs:
tag_name: ${{ needs.configure.outputs.tag }}
run: gh release upload "$tag_name" "$asset_path#$asset_name"
# Ninja is single-config: binaries and their PDBs land directly in the
# target directories instead of a Release subdirectory.
#
# Archived for tag builds only, and ordered after the release upload so
# a missing PDB can never block a release from being published. On a tag
# build this artifact is the only durable copy of the symbols: the Sentry
# upload is continue-on-error, so without it a failed upload would lose
# the ability to symbolicate that release permanently. Other builds upload
# no symbols at all, so archiving them there would only cost storage.
#
# The executables travel with their PDBs: re-uploading the pair together is
# what lets sentry-cli associate the PE's code identifier with the PDB's
# debug identifier, and the PE is what carries the unwind information.
- name: "[Windows] Upload debug symbols (exe + pdb)"
if: matrix.os == 'Windows' && needs.configure.outputs.tag != null
uses: actions/upload-artifact@v7
with:
if-no-files-found: error
name: ${{ steps.build.outputs.name }}-Symbols
path: |
build/cockatrice/*.exe
build/cockatrice/*.pdb
build/oracle/*.exe
build/oracle/*.pdb
build/servatrice/*.exe
build/servatrice/*.pdb
- name: "Attest binary provenance"
if: steps.upload_release.outcome == 'success'
id: attestation