[Updater] Close a running Cockatrice when the installer asks (#7672)

* [Updater] Refuse a close that arrives while one is already asking

MainWindow::closeEvent() had a re-entrancy guard that returned without touching
the event. A QCloseEvent is accepted unless something ignores it, so a second
close request - an installer replacing our files posts one every 30 s - accepted
the close while the first one was still blocked on its own confirmation. close()
then hid this window, hid the confirmation dialog with it, and left the process
running inside the nested loop that dialog had opened: no window, no way to
answer it, and every later close request accepted into the same dead end. That
is what an update looked like when it could not close a running client, and it
is why the installer could name the process, bring it forward and ask it to
close for minutes without anything happening.

Track how far the shutdown has got instead. While a confirmation is up, a close
from anywhere else is refused, which leaves the question in front of the user;
once the shutdown has been accepted, a duplicate close (Qt delivers the event
twice on some platforms) is accepted again so it cannot cancel the quit.

* [Installer] Leave an instance that is asking a question alone

An instance that answered a close by hiding rather than quitting - the state the
client used to get into - has no visible window left, and the confirmation it is
waiting on is hidden with the window that owns it. The close probe only ever
looked at visible windows, so it restored the main window, posted WM_CLOSE into
the same dead end, and reported "still running" without saying why.

Show every window the instance left hidden before choosing the one to close, and
skip the close when the window it would reach is disabled: a modal dialog of the
instance's own is up, and posting the close then only hides the question again.
Name what is left of the window in the report as well - gone, still open, hidden
again, and whether the thread answers at all - so the log tells a refused close
from one that was never delivered.

* [Installer] Offer to end a Cockatrice that will not close

A close is a request, and an instance that does not answer it - waiting on a
question nobody is looking at, or stuck where a close cannot reach - leaves the
user with no way forward: the files cannot be replaced while it runs, and
clicking Yes again only repeats the request.

After a second close that changed nothing, offer the only thing that can free
the files: end it. The prompt says what it costs - an unsaved deck, a running
game or a setting it has not written yet - the default button is the safe one,
and it is a separate answer rather than a relabelled "Yes", so it is not
reached by clicking through the close prompt. Only cockatrice.exe is ever
ended: Oracle writes the card database in place and Servatrice is a service, so
both are named and left to the user, exactly as they are for the close.

The probe gains an --end-holders mode for it, and the uninstaller's guard gets
the same offer.

* [Installer] Never name the process that is doing the asking

The old uninstaller is started with _?= so it runs in place, from the very
directory it is emptying, and an installer can be downloaded into the folder it
installs to. Both are then processes running from the directory, so the probe
named them - and the guard asked the user to close the program that was asking.
That question cannot be answered: a close and the end offer both skip anything
that is not cockatrice.exe, so the guard repeated itself until the update or the
uninstall was cancelled.

The probe and the processes that started it are now never named, by either
lookup. The walk up the parent chain stops at the first program the payload
installs, because in an in-app update the installer's own parent is the
Cockatrice instance being replaced - the one the guard exists to name.

The Windows check held its locked file in the script that starts the probe, so
that holder is excluded by design now and the check read it as "nothing found".
It holds the file from a process of its own instead, and asserts the exclusion
itself: a file that only the asking process holds comes back as nothing.

* [Installer] Restore only the windows a user could answer

Showing every hidden window an instance owns put Qt's own bookkeeping windows on
screen: the tray icon keeps one to receive the TaskbarCreated message, styled
WS_OVERLAPPED and titled QTrayIconMessageWindow. Restored, it is an empty white
rectangle in the taskbar with no title bar to close it - it has no window
procedure that would hide it again, and the process is busy in the very loop the
restore is meant to break, so it stayed until the process was force-ended.

Only framed, non-empty windows are restored and considered for the close now.
That also keeps the close off Qt's tray window, which answers WM_CLOSE by asking
the application to terminate - a quit that skips the confirmation the close is
supposed to reach.

* [Installer] Uninstall the previous version before installing over it

A regular install over an existing one asked "a previous version must be
uninstalled", then ran the old uninstaller interactively. That uninstaller
put its confirmation page behind the installer window, where it could not be
answered, and it only removed the installation after the installer had
already written the new one over it.

Run the previous version's uninstaller silently instead. The installer has
already asked the question the user answered, so a second, hidden confirmation
is pure risk: with /S there is no window to be behind, and ExecWait is then
waiting for something that cannot outlive it. QuietUninstallString is used
when the old installation has one, otherwise /S is appended.

ExecWait only waits while the uninstaller runs in place: one that did not get
a usable _?= copies itself to %TEMP% and returns at once, while the copy keeps
deleting - which is how an uninstall can finish after the install it was
supposed to precede. Log the exit code and wait for the old cockatrice.exe to
disappear before restoring user data or writing anything, and abort with a
message instead of installing into a directory that is still being emptied.

Cancelling the "must be uninstalled" prompt now quits the installer; it used
to skip the page and install over the old version anyway.

* [Installer] Do not show windows the application has hidden

The guard restored every hidden window an instance owned before asking it to
close, so that a confirmation hidden together with its window would come back.
Windows does as it is told and shows them; the application does not, because Qt
still believes they are hidden and never paints them. What the user gets is a
blank white rectangle, on screen and on the taskbar, that answers nothing - and
the question it was meant to bring back is not answerable in it either.

An instance with no window a user could answer is named and left to the user
instead, and the prompt says what to do about it: open it from the notification
area. Ending it stays the last resort, offered once a close has been asked for
twice.

The abort after a failed uninstall now says what it saw - the old version is
still there after its uninstaller ran - instead of promising a removal that is
not happening, which is the state a cancelled old uninstaller leaves behind.

---------

Co-authored-by: Lukas Brübach <Bruebach.Lukas@bdosecurity.de>

tools/installer-holder-probe/close_holders.cpp

@@ -44,39 +44,123 @@ std::wstring processLabel(const Holder &holder)
     return label;
 }
 
+/// Joins the per-holder lines into the one line the installer logs and cuts it to what one
+/// NSIS string holds.
+std::wstring joinReport(const std::vector<std::wstring> &lines)
+{
+    std::wstring report;
+    for (const std::wstring &line : lines) {
+        if (!report.empty()) {
+            report += L"; ";
+        }
+        report += line;
+    }
+    if (report.size() > REPORT_CHAR_MAX) {
+        report.erase(REPORT_CHAR_MAX);
+        report += L"...";
+    }
+    return report;
+}
+
 struct WindowSearch
 {
     DWORD processId = 0;
-    HWND topmost = nullptr; ///< first visible window in Z-order: a dialog when one is up
-    HWND main = nullptr;    ///< first visible window without an owner: the one to close
-    HWND any = nullptr;     ///< first window at all, used when nothing is visible
+    std::vector<HWND> windows; ///< every top-level window of the process, in Z-order
 };
 
-BOOL CALLBACK findProcessWindow(HWND window, LPARAM parameter)
+BOOL CALLBACK collectProcessWindow(HWND window, LPARAM parameter)
 {
     auto &search = *reinterpret_cast<WindowSearch *>(parameter);
     DWORD processId = 0;
     GetWindowThreadProcessId(window, &processId);
-    if (processId != search.processId) {
-        return TRUE;
-    }
-    if (search.any == nullptr) {
-        search.any = window;
-    }
-    // A tray icon leaves the main window hidden, so a window that is not visible is still
-    // kept as a fallback. EnumWindows walks the Z-order top down, so the first visible
-    // window is the one the user would see in front.
-    if (IsWindowVisible(window) != FALSE) {
-        if (search.topmost == nullptr) {
-            search.topmost = window;
-        }
-        if (search.main == nullptr && GetWindow(window, GW_OWNER) == nullptr) {
-            search.main = window;
-        }
+    if (processId == search.processId) {
+        search.windows.push_back(window);
     }
     return TRUE;
 }
 
+/// Whether a window is one the user could be asked to answer.
+///
+/// Only a framed, non-empty window qualifies. Qt keeps hidden top-level windows of its own
+/// next to the application's: the tray icon creates one to receive the TaskbarCreated
+/// message, styled WS_OVERLAPPED - which is no frame at all - and titled
+/// QTrayIconMessageWindow. Showing one of those puts an empty white rectangle on the
+/// taskbar that nothing ever closes again: it has no window procedure of its own to hide it,
+/// and the process is busy in the loop this is meant to break, so it stays until the process
+/// dies. Posting a close to one is worse than useless too: Qt's tray window answers WM_CLOSE
+/// by asking the application to terminate, without going through the confirmation the close
+/// is supposed to reach.
+bool isAnswerableWindow(HWND window)
+{
+    if ((GetWindowLongPtr(window, GWL_EXSTYLE) & WS_EX_TOOLWINDOW) != 0) {
+        return false;
+    }
+    if ((GetWindowLongPtr(window, GWL_STYLE) & WS_CAPTION) == 0) {
+        return false;
+    }
+    RECT bounds{};
+    if (GetWindowRect(window, &bounds) == FALSE) {
+        return false;
+    }
+    return bounds.right > bounds.left && bounds.bottom > bounds.top;
+}
+
+/// Picks the window to post the close to and the one the user should end up looking at.
+struct CloseTarget
+{
+    HWND main = nullptr;    ///< first visible window without an owner: the application's own
+    HWND toRaise = nullptr; ///< topmost visible window: a dialog when one is up
+    bool modalOpen = false; ///< the window a close would reach is disabled, so a dialog of its own is up
+};
+
+/// Only the windows the user could answer are candidates - see isAnswerableWindow. An
+/// instance left with nothing but Qt's own bookkeeping windows visible has no window to ask:
+/// the caller reports that instead of posting a close the application would answer with a
+/// terminate that skips its own confirmation.
+CloseTarget chooseCloseTarget(const std::vector<HWND> &windows)
+{
+    CloseTarget target;
+    HWND topmost = nullptr;
+    for (const HWND window : windows) {
+        if (IsWindowVisible(window) == FALSE || !isAnswerableWindow(window)) {
+            continue;
+        }
+        if (topmost == nullptr) {
+            topmost = window;
+        }
+        if (target.main == nullptr && GetWindow(window, GW_OWNER) == nullptr) {
+            target.main = window;
+        }
+    }
+    target.toRaise = topmost != nullptr ? topmost : target.main;
+    // A modal dialog disables the windows behind it, so a disabled main window is the
+    // instance asking a question of its own.
+    target.modalOpen =
+        target.toRaise != nullptr && IsWindowEnabled(target.main != nullptr ? target.main : target.toRaise) == FALSE;
+    return target;
+}
+
+/// What is left of the window the close was posted to, so a log line tells an instance that
+/// answered by hiding from one that never answered at all. A process that hid its window and
+/// stayed alive is the one that needs the next close to bring it back into view.
+std::wstring describeWindowState(HWND window)
+{
+    if (window == nullptr || IsWindow(window) == FALSE) {
+        return L", its window is gone";
+    }
+    // WM_NULL is the cheapest way to ask whether the thread behind the window still dispatches.
+    // A hung instance does not answer at all, which is a different problem from one that
+    // answered the close and stayed open, and the two want different words in the prompt.
+    DWORD_PTR ignored = 0;
+    const bool responding =
+        SendMessageTimeoutW(window, WM_NULL, 0, 0, SMTO_ABORTIFHUNG | SMTO_BLOCK, 1000, &ignored) != 0;
+    std::wstring state = IsWindowVisible(window) != FALSE ? L", window still open" : L", window hidden again";
+    if (!responding) {
+        state += L" and not responding";
+    }
+    return state;
+}
+
 /// Restores a minimized window and takes the foreground, so a confirmation Cockatrice
 /// answers the close with is in front of the installer instead of behind it.
 ///
@@ -133,6 +217,7 @@ CloseHoldersResult closeCockatriceHolders(const std::vector<Holder> &holders, un
     std::vector<std::wstring> lines;
     std::vector<HANDLE> asked;
     std::vector<std::size_t> askedLines;
+    std::vector<HWND> askedWindows;
 
     for (const Holder &holder : holders) {
         if (holder.isService || !isCockatriceImage(holder.imageName)) {
@@ -144,23 +229,41 @@ CloseHoldersResult closeCockatriceHolders(const std::vector<Holder> &holders, un
 
         WindowSearch search;
         search.processId = holder.processId;
-        EnumWindows(findProcessWindow, reinterpret_cast<LPARAM>(&search));
-        if (search.any == nullptr) {
+        EnumWindows(collectProcessWindow, reinterpret_cast<LPARAM>(&search));
+        if (search.windows.empty()) {
             lines.push_back(processLabel(holder) + L" has no window to close, left to you");
             continue;
         }
 
-        const HWND target =
-            search.main != nullptr ? search.main : (search.topmost != nullptr ? search.topmost : search.any);
-        const HWND toRaise = search.topmost != nullptr ? search.topmost : target;
+        const CloseTarget target = chooseCloseTarget(search.windows);
+        if (target.main == nullptr) {
+            // Nothing visible that a user could answer: the instance is hidden - in the
+            // notification area, say - or has nothing but Qt's own bookkeeping windows on
+            // screen. Neither is a window a close could reach, and showing one from here does
+            // not help: Windows would display it while the application still believes it is
+            // hidden, so it stays blank and answers nothing. The instance is named and left to
+            // the user, who can open it from the notification area.
+            lines.push_back(processLabel(holder) + L" has no window to ask, left to you");
+            continue;
+        }
         std::wstring line = processLabel(holder);
-        line += bringForward(toRaise) ? L" brought forward" : L" could not be brought forward";
+        line += bringForward(target.toRaise) ? L" brought forward" : L" could not be brought forward";
+
+        // Posting the close to an instance that is asking a question of its own is what made
+        // an earlier attempt useless: the client accepts it, hides this window and the dialog
+        // with it, and stays alive inside the nested loop the dialog opened, where nothing can
+        // reach it any more. The question is the request now, so it is brought forward and
+        // left alone.
+        if (target.modalOpen) {
+            lines.push_back(line + L", and is waiting for an answer to a question of its own");
+            continue;
+        }
 
         // Kept open across the wait so a pid reused after the process exits cannot be
         // mistaken for the instance that was asked to close.
         const HANDLE process = OpenProcess(SYNCHRONIZE, FALSE, holder.processId);
         const DWORD openError = process == nullptr ? GetLastError() : ERROR_SUCCESS;
-        if (PostMessageW(target, WM_CLOSE, 0, 0) == FALSE) {
+        if (PostMessageW(target.main, WM_CLOSE, 0, 0) == FALSE) {
             // Read the error before CloseHandle, which is free to reset it.
             const DWORD postError = GetLastError();
             if (process != nullptr) {
@@ -177,6 +280,7 @@ CloseHoldersResult closeCockatriceHolders(const std::vector<Holder> &holders, un
         } else {
             askedLines.push_back(lines.size());
             asked.push_back(process);
+            askedWindows.push_back(target.main);
         }
         lines.push_back(std::move(line));
     }
@@ -186,8 +290,12 @@ CloseHoldersResult closeCockatriceHolders(const std::vector<Holder> &holders, un
         const bool allExited = waitForExit(asked, waitMilliseconds, elapsed);
         for (std::size_t index = 0; index < asked.size(); ++index) {
             const bool exited = allExited || WaitForSingleObject(asked[index], 0) == WAIT_OBJECT_0;
-            lines[askedLines[index]] += exited ? L", exited after " + std::to_wstring(elapsed) + L" ms"
-                                               : L", still running after " + std::to_wstring(elapsed) + L" ms";
+            if (exited) {
+                lines[askedLines[index]] += L", exited after " + std::to_wstring(elapsed) + L" ms";
+            } else {
+                lines[askedLines[index]] += L", still running after " + std::to_wstring(elapsed) + L" ms" +
+                                            describeWindowState(askedWindows[index]);
+            }
         }
     }
     for (const HANDLE process : asked) {
@@ -198,15 +306,61 @@ CloseHoldersResult closeCockatriceHolders(const std::vector<Holder> &holders, un
         result.report = L"nothing is holding the files";
         return result;
     }
-    for (const std::wstring &line : lines) {
-        if (!result.report.empty()) {
-            result.report += L"; ";
-        }
-        result.report += line;
-    }
-    if (result.report.size() > REPORT_CHAR_MAX) {
-        result.report.erase(REPORT_CHAR_MAX);
-        result.report += L"...";
-    }
+    result.report = joinReport(lines);
+    return result;
+}
+
+EndHoldersResult endCockatriceHolders(const std::vector<Holder> &holders, unsigned long waitMilliseconds)
+{
+    EndHoldersResult result;
+    std::vector<std::wstring> lines;
+    std::vector<HANDLE> ended;
+    std::vector<std::size_t> endedLines;
+
+    for (const Holder &holder : holders) {
+        if (holder.isService || !isCockatriceImage(holder.imageName)) {
+            // The same split as the close: Oracle writes the card database in place and a
+            // service has nothing to ask, so neither is ever ended here.
+            lines.push_back(processLabel(holder) + L" left to you");
+            continue;
+        }
+
+        const HANDLE process = OpenProcess(PROCESS_TERMINATE | SYNCHRONIZE, FALSE, holder.processId);
+        if (process == nullptr) {
+            lines.push_back(processLabel(holder) + L" could not be ended (win32 " + std::to_wstring(GetLastError()) +
+                            L")");
+            continue;
+        }
+        if (TerminateProcess(process, 1) == FALSE) {
+            // Read the error before CloseHandle, which is free to reset it.
+            const DWORD error = GetLastError();
+            CloseHandle(process);
+            lines.push_back(processLabel(holder) + L" could not be ended (win32 " + std::to_wstring(error) + L")");
+            continue;
+        }
+        ++result.ended;
+        endedLines.push_back(lines.size());
+        lines.push_back(processLabel(holder) + L" ended");
+        ended.push_back(process);
+    }
+
+    if (!ended.empty() && waitMilliseconds > 0) {
+        unsigned long long elapsed = 0;
+        const bool allExited = waitForExit(ended, waitMilliseconds, elapsed);
+        for (std::size_t index = 0; index < ended.size(); ++index) {
+            const bool exited = allExited || WaitForSingleObject(ended[index], 0) == WAIT_OBJECT_0;
+            lines[endedLines[index]] += exited ? L", exited after " + std::to_wstring(elapsed) + L" ms"
+                                               : L", still running after " + std::to_wstring(elapsed) + L" ms";
+        }
+    }
+    for (const HANDLE process : ended) {
+        CloseHandle(process);
+    }
+
+    if (lines.empty()) {
+        result.report = L"nothing is holding the files";
+        return result;
+    }
+    result.report = joinReport(lines);
     return result;
 }