mirror of
https://github.com/Cockatrice/Cockatrice.git
synced 2026-11-05 01:23:24 -08:00
[Tests] Replay XML test files through a headless runner (#7694)
Co-authored-by: Lukas Brübach <Bruebach.Lukas@bdosecurity.de>
This commit is contained in:
BruebachL
GitHub
parent
2e0a718cd7
commit
ee57d740c2
9 changed files with 469 additions and 4 deletions
52
Testing/README.md
Normal file
52
Testing/README.md
Normal file
|
|
@ -0,0 +1,52 @@
|
|||
# GUI tests
|
||||
|
||||
Widget-level playback tests for the Cockatrice client, driven by
|
||||
[QtTesting](https://github.com/Kitware/QtTesting) (`external/qttesting`, a submodule).
|
||||
Each `*_test.xml` file is a list of events that QtTesting replays against the running
|
||||
application; a test passes when every event is handled without error.
|
||||
|
||||
## Running
|
||||
|
||||
```bash
|
||||
git submodule update --init external/qttesting # once
|
||||
cmake -S . -B build/debug -DCMAKE_BUILD_TYPE=Debug -DWITH_SERVER=0 -DWITH_GUI_TEST=ON
|
||||
cmake --build build/debug -j 24
|
||||
|
||||
./Testing/run_test.sh --all # headless, offscreen
|
||||
./Testing/run_test.sh Testing/smoke_test.xml # one file
|
||||
./Testing/run_test.sh --visual Testing/smoke_test.xml # shows the window
|
||||
```
|
||||
|
||||
`WITH_GUI_TEST` is on by default; a checkout without the submodule configures with a
|
||||
warning and no harness. `COCKATRICE_BIN` points the runner at another build, and
|
||||
`TEST_TIMEOUT` (seconds, default 180) bounds a single test — a failing event inside a
|
||||
modal dialog can otherwise leave the process alive. The client also hard-exits a few
|
||||
seconds after playback returns, whatever the result.
|
||||
|
||||
The tests are **not** part of `ctest`: they need a card database and a developer's
|
||||
settings, so they stay a local loop for now.
|
||||
|
||||
## Caveats
|
||||
|
||||
- **Not hermetic.** The client reads the local settings and card database, so a test can
|
||||
pass on one machine and fail on another. Avoid asserting things that restored
|
||||
workspace state or user preferences decide (for example the number of open tabs).
|
||||
- **English labels only.** Menu entries are addressed by their (translated) text, so the
|
||||
tests expect the source language. Widgets are addressed by objectName paths, which are
|
||||
stable across translations.
|
||||
- Actions that open a dialog are followed by a `pause` event: playback continues in the
|
||||
dialog's own event loop, and the pause gives the UI time to build itself.
|
||||
- Widget paths include Qt-internal names where a QTabWidget is involved
|
||||
(`qt_tabwidget_stackedwidget`), and the deck views in a game are indexed
|
||||
(`tabbedDeckView_0`, `tabbedDeckView_1`).
|
||||
|
||||
## Writing a test
|
||||
|
||||
A test is a `<events>` list. Widgets are named by their path from the main window, e.g.
|
||||
`MainWindow/tabSupervisor/stackedWidget/TabGame/phasesMenu`. Useful commands:
|
||||
|
||||
| Command | Arguments | Effect |
|
||||
|---------|-----------|--------|
|
||||
| `activate` | action or menu name | triggers a QAction (menus are opened first) |
|
||||
| `key` | Qt key code | presses that key on the focused widget |
|
||||
| `pause` | milliseconds | waits, letting the UI settle |
|
||||
118
Testing/run_test.sh
Executable file
118
Testing/run_test.sh
Executable file
|
|
@ -0,0 +1,118 @@
|
|||
#!/bin/bash
|
||||
# Run QtTesting XML test files against the Cockatrice build.
|
||||
#
|
||||
# Usage:
|
||||
# ./Testing/run_test.sh [--visual] [--all | <test-file.xml> ...]
|
||||
#
|
||||
# Options:
|
||||
# --visual Run with native display (shows the GUI window).
|
||||
# Default is headless (QT_QPA_PLATFORM=offscreen).
|
||||
# --all Run every *.xml file in Testing/.
|
||||
#
|
||||
# Environment:
|
||||
# COCKATRICE_BIN Path to the client binary (default: build/debug/cockatrice/cockatrice,
|
||||
# or the .app bundle binary on macOS).
|
||||
# TEST_TIMEOUT Seconds a single test may run before it is killed (default: 180).
|
||||
# Uses timeout, or gtimeout from coreutils on macOS.
|
||||
#
|
||||
# Requires the debug build to exist at build/debug/cockatrice/cockatrice.
|
||||
|
||||
set -euo pipefail
|
||||
shopt -s nullglob
|
||||
|
||||
REPO_ROOT="$(cd "$(dirname "$0")/.." && pwd -P)"
|
||||
if [ -n "${COCKATRICE_BIN:-}" ]; then
|
||||
COCKATRICE="$COCKATRICE_BIN"
|
||||
elif [ "$(uname -s)" = "Darwin" ]; then
|
||||
# A macOS build produces an app bundle, not a bare executable.
|
||||
COCKATRICE="$REPO_ROOT/build/debug/cockatrice/cockatrice.app/Contents/MacOS/cockatrice"
|
||||
else
|
||||
COCKATRICE="$REPO_ROOT/build/debug/cockatrice/cockatrice"
|
||||
fi
|
||||
TEST_DIR="$REPO_ROOT/Testing"
|
||||
TEST_TIMEOUT="${TEST_TIMEOUT:-180}"
|
||||
|
||||
# Stock macOS has no `timeout`; coreutils provides `gtimeout`. Without either the
|
||||
# runner still works, but a hung test is not killed.
|
||||
TIMEOUT=()
|
||||
if command -v timeout >/dev/null 2>&1; then
|
||||
TIMEOUT=(timeout --foreground "$TEST_TIMEOUT")
|
||||
elif command -v gtimeout >/dev/null 2>&1; then
|
||||
TIMEOUT=(gtimeout --foreground "$TEST_TIMEOUT")
|
||||
else
|
||||
echo "Warning: neither timeout nor gtimeout found; tests are not time-limited." >&2
|
||||
fi
|
||||
|
||||
if [ ! -f "$COCKATRICE" ]; then
|
||||
echo "Error: $COCKATRICE not found. Build first:" >&2
|
||||
echo " cmake --build build/debug" >&2
|
||||
exit 1
|
||||
fi
|
||||
|
||||
VISUAL=0
|
||||
if [ "${1:-}" = "--visual" ]; then
|
||||
VISUAL=1
|
||||
shift
|
||||
fi
|
||||
|
||||
if [ "${1:-}" = "--all" ]; then
|
||||
shift
|
||||
TEST_FILES=("$TEST_DIR"/*.xml)
|
||||
if [ ${#TEST_FILES[@]} -eq 0 ]; then
|
||||
echo "No *.xml files found in $TEST_DIR" >&2
|
||||
exit 1
|
||||
fi
|
||||
elif [ $# -ge 1 ]; then
|
||||
TEST_FILES=("$@")
|
||||
else
|
||||
echo "Usage: $0 [--visual] [--all | <test-file.xml> ...]" >&2
|
||||
echo " Run a single test: $0 Testing/smoke_test.xml" >&2
|
||||
echo " Run all tests: $0 --all" >&2
|
||||
echo " Run visually: $0 --visual Testing/smoke_test.xml" >&2
|
||||
echo " Run all visually: $0 --visual --all" >&2
|
||||
exit 1
|
||||
fi
|
||||
|
||||
PASS=0
|
||||
FAIL=0
|
||||
FAILED_TESTS=()
|
||||
|
||||
run_one() {
|
||||
local file="$1"
|
||||
|
||||
local -a cmd=()
|
||||
if [ "${#TIMEOUT[@]}" -gt 0 ]; then
|
||||
cmd=("${TIMEOUT[@]}")
|
||||
fi
|
||||
cmd+=("$COCKATRICE" --no-connect --test "$file")
|
||||
|
||||
if [ "$VISUAL" -eq 1 ]; then
|
||||
"${cmd[@]}"
|
||||
else
|
||||
env QT_QPA_PLATFORM=offscreen "${cmd[@]}"
|
||||
fi
|
||||
}
|
||||
|
||||
for f in "${TEST_FILES[@]}"; do
|
||||
name="$(basename "$f")"
|
||||
printf "%-40s " "$name"
|
||||
if run_one "$f"; then
|
||||
echo "PASS"
|
||||
PASS=$((PASS + 1))
|
||||
else
|
||||
echo "FAIL"
|
||||
FAIL=$((FAIL + 1))
|
||||
FAILED_TESTS+=("$name")
|
||||
fi
|
||||
done
|
||||
|
||||
TOTAL=$((PASS + FAIL))
|
||||
echo ""
|
||||
echo "Results: $PASS/$TOTAL passed, $FAIL failed"
|
||||
if [ "$FAIL" -gt 0 ]; then
|
||||
echo "Failed:"
|
||||
for t in "${FAILED_TESTS[@]}"; do
|
||||
echo " $t"
|
||||
done
|
||||
exit 1
|
||||
fi
|
||||
8
Testing/smoke_test.xml
Normal file
8
Testing/smoke_test.xml
Normal file
|
|
@ -0,0 +1,8 @@
|
|||
<events>
|
||||
<!-- Open Help menu by activating it on the menu bar -->
|
||||
<event widget="MainWindow/menubar" command="activate" arguments="helpMenu" />
|
||||
<!-- Click About Cockatrice (opens modal QMessageBox) -->
|
||||
<event widget="MainWindow/menubar/helpMenu" command="activate" arguments="&About Cockatrice" />
|
||||
<!-- Close the modal dialog by pressing Escape (Qt::Key_Escape = 16777216) -->
|
||||
<event widget="MainWindow/aboutDialog" command="key" arguments="16777216" />
|
||||
</events>
|
||||
Loading…
Add table
Add a link
Reference in a new issue