diff --git a/Testing/README.md b/Testing/README.md index cb990ec29..d052fb3c5 100644 --- a/Testing/README.md +++ b/Testing/README.md @@ -2,8 +2,8 @@ Widget-level playback tests for the Cockatrice client, driven by [QtTesting](https://github.com/Kitware/QtTesting) (`external/qttesting`, a submodule). -Each `*_test.xml` file is a list of events that QtTesting replays against the running -application; a test passes when every event is handled without error. +Each `*_test.xml` file lists events that QtTesting replays against the running +application. A test passes when every event is handled without error. ## Running @@ -17,39 +17,43 @@ cmake --build build/debug -j 24 ./Testing/run_test.sh --visual Testing/smoke_test.xml # shows the window ``` -`WITH_GUI_TEST` is on by default; a checkout without the submodule configures with a +`WITH_GUI_TEST` is on by default. Without the submodule the build configures with a warning and no harness. `COCKATRICE_BIN` points the runner at another build, and -`TEST_TIMEOUT` (seconds, default 180) bounds a single test — a failing event inside a -modal dialog can otherwise leave the process alive. The client also hard-exits a few +`TEST_TIMEOUT` (seconds, default 180) bounds a single test, because a failing event +inside a modal dialog can leave the process alive. The client also hard-exits a few seconds after playback returns, whatever the result. -The tests are **not** part of `ctest`: they need a card database and a developer's +The tests are **not** part of `ctest`. They need a card database and a developer's settings, so they stay a local loop for now. ## Caveats - **Not hermetic.** The client reads the local settings and card database, so a test can pass on one machine and fail on another. Avoid asserting things that restored - workspace state or user preferences decide (for example the number of open tabs). -- **English labels only.** Menu entries are addressed by their (translated) text, so the + workspace state or user preferences decide, such as the number of open tabs. +- **English labels only.** Menu entries are addressed by their translated text, so the tests expect the source language. Widgets are addressed by objectName paths, which are stable across translations. -- Actions that open a dialog are followed by a `pause` event: playback continues in the +- Actions that open a dialog are followed by a `pause` event. Playback continues in the dialog's own event loop, and the pause gives the UI time to build itself. +- `pause` sleeps the thread, so nothing is delivered while it runs. A reply from a server + never arrives during a pause. Waiting for the client to talk to a server needs + `process_events`, which runs the event loop (see *Screenshots* below). - Widget paths include Qt-internal names where a QTabWidget is involved (`qt_tabwidget_stackedwidget`), and the deck views in a game are indexed (`tabbedDeckView_0`, `tabbedDeckView_1`). ## Writing a test -A test is a `` list. Widgets are named by their path from the main window, e.g. +A test is an `` list. Widgets are named by their path from the main window, e.g. `MainWindow/tabSupervisor/stackedWidget/TabGame/phasesMenu`. Useful commands: | Command | Arguments | Effect | |---------|-----------|--------| | `activate` | action or menu name | triggers a QAction (menus are opened first) | | `key` | Qt key code | presses that key on the focused widget | -| `pause` | milliseconds | waits, letting the UI settle | +| `pause` | milliseconds | sleeps, letting the UI settle (no events are processed) | +| `process_events` | milliseconds | runs the event loop, so timers and network replies are handled | | `clickCard` | `card,zone` | double click the card found in that zone (plays it from hand) | | `flipCard` | `card,zone` | right-click → *Turn Over* | | `cardAction` | `card,zone,label` | right-click → first menu entry whose label contains `label` (submenus included) | @@ -59,8 +63,61 @@ A test is a `` list. Widgets are named by their path from the main windo | `selectItem` | `item label` | selects the tree/list entry with that label | | `typeText` | `text` | types text one key event at a time | | `expectProperty` | `property,value` | fails playback unless the widget property matches | +| `screenshot` | `path` | saves an image of the addressed widget; `$VAR` expands from the environment | +| `dumpTree` | `path` | writes the addressed widget's subtree with the paths tests address | +| `resize` | `WIDTHxHEIGHT` | resizes the addressed widget's window | The card, item and property commands are `pqCockatriceEventPlayer` (`cockatrice/src/game_graphics/pq_cockatrice_event_player.h`). They resolve cards by name and zone through the `QGraphicsScene` and send real mouse events at the card's current viewport position, so a test does not depend on the window size or screen resolution. + +The capture commands are `pqWidgetTestEventPlayer` +(`cockatrice/src/interface/pq_widget_test_event_player.h`). + +## Screenshots + +`run_test.sh` sets `COCKATRICE_SCREENSHOT_DIR` to `build/screenshots`, and `screenshot` +and `dumpTree` write there. Override it in the environment to send captures elsewhere. +That directory is not part of the source tree, so a test run does not leave untracked +files in the repository: + +```xml + + + +``` + +The addressed widget is what gets grabbed, so `MainWindow` is the whole window and a +panel's path is that panel. The format comes from the file extension. A path with no +known extension is a test failure. + +Capture at a fixed, generous size. `1920x1080` is a safe default, and narrower windows +clip columns of wide tables. A headless run renders through the offscreen platform, so +the palette and layout are the real ones, but `raise()`, keyboard-grab and +`propagateSizeHints()` warnings on stderr are normal. + +### Waiting for a server + +`pause` blocks, so a reply that has to arrive over the network never does. Use +`process_events` with a budget that covers the round trip: + +```xml + + +``` + +### Paths drift between runs + +`pqObjectNaming` names an unnamed widget ``, where the index counts +same-class unnamed siblings and a leading digit records visibility (`1` visible, `0` +hidden). The numbering depends on what the previous events left on screen, so a path that +worked in one playback can miss in the next. The player retries a leading `0` as `1`, +which resolves some hidden-page paths but not all of them. + +Record the tree in the same playback that clicks, and copy paths from that dump: + +```bash +./Testing/run_test.sh Testing/screenshot_test.xml +less build/screenshots/widget-tree.txt +``` diff --git a/Testing/run_test.sh b/Testing/run_test.sh index 8ff3fe028..42a806f45 100755 --- a/Testing/run_test.sh +++ b/Testing/run_test.sh @@ -12,6 +12,10 @@ # Environment: # COCKATRICE_BIN Path to the client binary (default: build/debug/cockatrice/cockatrice, # or the .app bundle binary on macOS). +# COCKATRICE_SCREENSHOT_DIR +# Directory the tests write screenshots and widget dumps to +# (default: build/screenshots). Tests address it as +# $COCKATRICE_SCREENSHOT_DIR/name.png. # TEST_TIMEOUT Seconds a single test may run before it is killed (default: 180). # Uses timeout, or gtimeout from coreutils on macOS. # @@ -32,6 +36,12 @@ fi TEST_DIR="$REPO_ROOT/Testing" TEST_TIMEOUT="${TEST_TIMEOUT:-180}" +# Tests that capture the UI address this as $COCKATRICE_SCREENSHOT_DIR/name.png. +# It lives under the ignored build/ tree, so a run does not leave untracked files. +SCREENSHOT_DIR="${COCKATRICE_SCREENSHOT_DIR:-$REPO_ROOT/build/screenshots}" +mkdir -p "$SCREENSHOT_DIR" +export COCKATRICE_SCREENSHOT_DIR="$SCREENSHOT_DIR" + # Stock macOS has no `timeout`; coreutils provides `gtimeout`. Without either the # runner still works, but a hung test is not killed. TIMEOUT=() diff --git a/Testing/screenshot_test.xml b/Testing/screenshot_test.xml new file mode 100644 index 000000000..f69ecfd44 --- /dev/null +++ b/Testing/screenshot_test.xml @@ -0,0 +1,23 @@ + + + + + + + + + + + + + + + + + diff --git a/cockatrice/CMakeLists.txt b/cockatrice/CMakeLists.txt index e9949551a..46643b5e5 100644 --- a/cockatrice/CMakeLists.txt +++ b/cockatrice/CMakeLists.txt @@ -599,9 +599,12 @@ target_link_libraries( cockatrice_target_enable_crash_reporting(cockatrice) if(WITH_GUI_TEST) - # These two translate between XML test files and the QtTesting event pipeline; - # they need the QtTesting headers, so they only exist with the harness. - target_sources(cockatrice PRIVATE src/game_graphics/pq_cockatrice_event_player.cpp src/interface/recording.cpp) + # QtTesting integration and the Cockatrice-specific playback commands. They + # need the QtTesting headers, so they only exist with the harness. + target_sources( + cockatrice PRIVATE src/game_graphics/pq_cockatrice_event_player.cpp src/interface/pq_widget_test_event_player.cpp + src/interface/recording.cpp + ) target_link_libraries(cockatrice PRIVATE QtTesting) target_compile_definitions(cockatrice PRIVATE WITH_GUI_TEST) endif() diff --git a/cockatrice/src/interface/pq_widget_test_event_player.cpp b/cockatrice/src/interface/pq_widget_test_event_player.cpp new file mode 100644 index 000000000..60e19541a --- /dev/null +++ b/cockatrice/src/interface/pq_widget_test_event_player.cpp @@ -0,0 +1,223 @@ +#include "pq_widget_test_event_player.h" + +#include +#include +#include +#include +#include +#include +#include +#include +#include + +Q_LOGGING_CATEGORY(PQ_WIDGET_TEST_EVENT_PLAYER, "cockatrice.guiTest.widgetEventPlayer") + +namespace +{ +void dumpSubtree(QObject *object, QTextStream &out, int depth) +{ + if (auto *widget = qobject_cast(object)) { + const QString path = pqObjectNaming::GetName(*widget); + if (!path.isEmpty()) { + out << QString(depth * 2, ' ') << path << " [" << widget->metaObject()->className() << "]"; + if (!widget->objectName().isEmpty()) { + out << " objectName=" << widget->objectName(); + } + out << '\n'; + } + } + + // children() returns a snapshot, so the recursion below cannot invalidate it. + const QObjectList children = object->children(); + for (QObject *child : children) { + dumpSubtree(child, out, depth + 1); + } +} + +/** + * Expands $NAME and ${NAME} from the environment. + * + * A test file is static, so this lets the runner send screenshots to a scratch + * directory instead of hardcoding a path that only exists on one machine. An + * unset variable is an error, not an empty string: writing to the working + * directory would look like a pass. + */ +bool expandEnvironment(const QString &command, const QString &path, QString &expanded) +{ + expanded.clear(); + for (int i = 0; i < path.size(); ++i) { + if (path.at(i) != QLatin1Char('$')) { + expanded += path.at(i); + continue; + } + + int nameStart = i + 1; + int nameEnd = nameStart; + if (nameStart < path.size() && path.at(nameStart) == QLatin1Char('{')) { + nameStart++; + nameEnd = path.indexOf(QLatin1Char('}'), nameStart); + if (nameEnd < 0) { + qCCritical(PQ_WIDGET_TEST_EVENT_PLAYER) << command << ": unterminated ${ in path" << path; + return false; + } + i = nameEnd; + } else { + while (nameEnd < path.size() && + (path.at(nameEnd).isLetterOrNumber() || path.at(nameEnd) == QLatin1Char('_'))) { + nameEnd++; + } + i = nameEnd - 1; + } + + const QString name = path.mid(nameStart, nameEnd - nameStart); + if (name.isEmpty()) { + expanded += QLatin1Char('$'); + continue; + } + + const QString value = qEnvironmentVariable(name.toUtf8().constData()); + if (value.isEmpty()) { + qCCritical(PQ_WIDGET_TEST_EVENT_PLAYER) << command << ": environment variable" << name << "is not set"; + return false; + } + expanded += value; + } + return true; +} +} // namespace + +pqWidgetTestEventPlayer::pqWidgetTestEventPlayer(QObject *parent) : pqWidgetEventPlayer(parent) +{ +} + +bool pqWidgetTestEventPlayer::playEvent(QObject *object, const QString &command, const QString &arguments, bool &error) +{ + // A command we know that fails is still a handled event: QtTesting logs + // "Unhandled event" when playEvent returns false, so set `error` and return + // true, as pqCockatriceEventPlayer does. + const auto run = [&error](bool ok) { + if (!ok) { + error = true; + } + return true; + }; + + if (command == "screenshot") { + return run(playScreenshot(object, arguments)); + } + if (command == "dumpTree") { + return run(playDumpTree(object, arguments)); + } + if (command == "resize") { + return run(playResize(object, arguments)); + } + return false; +} + +bool pqWidgetTestEventPlayer::playScreenshot(QObject *object, const QString &arguments) +{ + auto *widget = qobject_cast(object); + if (!widget) { + qCCritical(PQ_WIDGET_TEST_EVENT_PLAYER) << "screenshot: target is not a QWidget"; + return false; + } + + QString path; + if (!resolveOutputPath(QStringLiteral("screenshot"), arguments.trimmed(), path)) { + return false; + } + + // repaint() is synchronous, even on the offscreen platform, so the grab + // contains what the previous events drew. + widget->repaint(); + const QPixmap pixmap = widget->grab(); + if (pixmap.isNull()) { + qCCritical(PQ_WIDGET_TEST_EVENT_PLAYER) << "screenshot: grab returned a null pixmap for" << widget; + return false; + } + if (!pixmap.save(path)) { + qCCritical(PQ_WIDGET_TEST_EVENT_PLAYER) + << "screenshot: could not save" << path << "- an image format is chosen from the file extension"; + return false; + } + + qCInfo(PQ_WIDGET_TEST_EVENT_PLAYER) << "screenshot saved to" << path; + return true; +} + +bool pqWidgetTestEventPlayer::playDumpTree(QObject *object, const QString &arguments) +{ + if (!qobject_cast(object)) { + qCCritical(PQ_WIDGET_TEST_EVENT_PLAYER) << "dumpTree: target is not a QWidget"; + return false; + } + + QString path; + if (!resolveOutputPath(QStringLiteral("dumpTree"), arguments.trimmed(), path)) { + return false; + } + + QFile file(path); + if (!file.open(QIODevice::WriteOnly | QIODevice::Text | QIODevice::Truncate)) { + qCCritical(PQ_WIDGET_TEST_EVENT_PLAYER) << "dumpTree: could not open" << path << file.errorString(); + return false; + } + + QTextStream out(&file); + dumpSubtree(object, out, 0); + + qCInfo(PQ_WIDGET_TEST_EVENT_PLAYER) << "widget tree dumped to" << path; + return true; +} + +bool pqWidgetTestEventPlayer::playResize(QObject *object, const QString &arguments) +{ + auto *widget = qobject_cast(object); + if (!widget) { + qCCritical(PQ_WIDGET_TEST_EVENT_PLAYER) << "resize: target is not a QWidget"; + return false; + } + + const QStringList parts = arguments.split(QLatin1Char('x')); + if (parts.size() != 2) { + qCCritical(PQ_WIDGET_TEST_EVENT_PLAYER) << "resize: arguments must be 'WIDTHxHEIGHT', got:" << arguments; + return false; + } + + bool okWidth = false; + bool okHeight = false; + const int width = parts.at(0).trimmed().toInt(&okWidth); + const int height = parts.at(1).trimmed().toInt(&okHeight); + if (!okWidth || !okHeight || width <= 0 || height <= 0) { + qCCritical(PQ_WIDGET_TEST_EVENT_PLAYER) << "resize: invalid size" << arguments; + return false; + } + + // Resize the top-level window: the layout owns the children. + widget->window()->resize(width, height); + QApplication::processEvents(); + return true; +} + +bool pqWidgetTestEventPlayer::resolveOutputPath(const QString &command, const QString &path, QString &resolved) +{ + if (path.isEmpty()) { + qCCritical(PQ_WIDGET_TEST_EVENT_PLAYER) << command << ": output path is empty"; + return false; + } + + QString expanded; + if (!expandEnvironment(command, path, expanded)) { + return false; + } + + const QFileInfo info(expanded); + const QString directory = info.absolutePath(); + if (!QDir().mkpath(directory)) { + qCCritical(PQ_WIDGET_TEST_EVENT_PLAYER) << command << ": could not create directory" << directory; + return false; + } + + resolved = info.absoluteFilePath(); + return true; +} diff --git a/cockatrice/src/interface/pq_widget_test_event_player.h b/cockatrice/src/interface/pq_widget_test_event_player.h new file mode 100644 index 000000000..758c2df6b --- /dev/null +++ b/cockatrice/src/interface/pq_widget_test_event_player.h @@ -0,0 +1,45 @@ +#ifndef PQ_WIDGET_TEST_EVENT_PLAYER_H +#define PQ_WIDGET_TEST_EVENT_PLAYER_H + +#include "pqWidgetEventPlayer.h" + +/** + * Widget-level commands for the UI tests. + * + * pqCockatriceEventPlayer handles cards and zones inside the game scene. This + * player covers the rest of the client: capturing a widget as an image, + * dumping the paths QtTesting addresses widgets by, and sizing the window. + * + * Commands: + * screenshot "" + * grabs the addressed widget and saves it as an image. Parent directories + * are created, and $NAME / ${NAME} in the path expand from the + * environment, so the runner decides where captures land + * dumpTree "" + * writes the addressed widget's subtree with the paths QtTesting + * addresses it by, for writing a test + * resize "WIDTHxHEIGHT" + * resizes the addressed widget's top-level window + */ +class pqWidgetTestEventPlayer : public pqWidgetEventPlayer +{ + Q_OBJECT + typedef pqWidgetEventPlayer Superclass; + +public: + explicit pqWidgetTestEventPlayer(QObject *parent = nullptr); + + using Superclass::playEvent; + + bool playEvent(QObject *object, const QString &command, const QString &arguments, bool &error) override; + +private: + bool playScreenshot(QObject *object, const QString &arguments); + bool playDumpTree(QObject *object, const QString &arguments); + bool playResize(QObject *object, const QString &arguments); + + /** @brief Expands $NAME / ${NAME} and creates the file's parent directory. */ + static bool resolveOutputPath(const QString &command, const QString &path, QString &resolved); +}; + +#endif // PQ_WIDGET_TEST_EVENT_PLAYER_H diff --git a/cockatrice/src/interface/window_main.cpp b/cockatrice/src/interface/window_main.cpp index 715505926..8657a251e 100644 --- a/cockatrice/src/interface/window_main.cpp +++ b/cockatrice/src/interface/window_main.cpp @@ -62,6 +62,7 @@ #include "pqEventSource.h" #include "pqEventTypes.h" #include "pqTestUtility.h" +#include "pq_widget_test_event_player.h" #include "recording.h" #endif @@ -692,6 +693,7 @@ MainWindow::MainWindow(QWidget *parent) testUtility->addEventObserver("xml", new XMLEventObserver(this)); testUtility->addEventSource("xml", new XMLEventSource(this)); testUtility->eventPlayer()->addWidgetEventPlayer(new pqCockatriceEventPlayer(this)); + testUtility->eventPlayer()->addWidgetEventPlayer(new pqWidgetTestEventPlayer(this)); #endif }