[Installer] End a launcher that has no window to ask instead of prompting (#7681)

* [Installer] End a launcher that has no window to ask instead of prompting

The in-app updater starts this installer detached and then closes itself. A
client that accepts that close hides its window before it exits, and if it
then hangs in shutdown there is nothing left to ask: the holder probe
deliberately never posts WM_CLOSE to an instance it cannot see a window for,
so the "ask it to close again" answer only repeated itself until the user
picked Cancel and the installation aborted (Sentry COCKATRICE-DESKTOP-J).

The launcher wait now asks the launcher itself before asking the user. A new
probe mode, --close-launcher, answers whether that one process has a window a
close could reach at all; it is limited to a pid that still runs from the
directory being replaced, so a reused process id cannot be hit. When there is
no window, the installer gives the shutdown it already started one full grace
and then ends it through the handle it waited on, which is what the dialog's
Yes answer did. The prompt remains for the case it exists for: a window is up,
the instance may be waiting on its own question, and No now means only "keep
waiting".

The client is left alone: an already-installed version runs the installer
during its own update, so a change there cannot reach the run that aborts.

* [libcockatrice_rng] Include immintrin.h for AVX2 builds in SFMT

---------

Co-authored-by: sentry[bot] <39604003+sentry[bot]@users.noreply.github.com>

tools/installer-holder-probe/close_holders.cpp

@@ -15,8 +15,6 @@
 
 namespace
 {
-constexpr wchar_t COCKATRICE_IMAGE[] = L"cockatrice.exe";
-
 /// How long one poll of the asked processes takes. One WaitForSingleObject per process
 /// and no enumeration, so polling costs nothing next to the seconds this is called on.
 constexpr unsigned long WAIT_POLL_MILLISECONDS = 200;
@@ -26,11 +24,6 @@ constexpr unsigned long WAIT_POLL_MILLISECONDS = 200;
 /// the line stays far below it.
 constexpr std::size_t REPORT_CHAR_MAX = 900;
 
-bool isCockatriceImage(const std::wstring &imageName)
-{
-    return CompareStringOrdinal(imageName.c_str(), -1, COCKATRICE_IMAGE, -1, TRUE) == CSTR_EQUAL;
-}
-
 /// The holder as the log has to name it to be actionable. Deliberately not
 /// formatHolder(): that one is worded for the dialog, where ", running" says something
 /// the line here continues after.
@@ -112,7 +105,7 @@ struct CloseTarget
 {
     HWND main = nullptr;    ///< first visible window without an owner: the application's own
     HWND toRaise = nullptr; ///< topmost visible window: a dialog when one is up
-    bool modalOpen = false; ///< the window a close would reach is disabled, so a dialog of its own is up
+    bool modalOpen = false; ///< the only window a user could answer is the instance's own question
 };
 
 /// Only the windows the user could answer are candidates - see isAnswerableWindow. An
@@ -136,9 +129,11 @@ CloseTarget chooseCloseTarget(const std::vector<HWND> &windows)
     }
     target.toRaise = topmost != nullptr ? topmost : target.main;
     // A modal dialog disables the windows behind it, so a disabled main window is the
-    // instance asking a question of its own.
-    target.modalOpen =
-        target.toRaise != nullptr && IsWindowEnabled(target.main != nullptr ? target.main : target.toRaise) == FALSE;
+    // instance asking a question of its own. A visible window with no unowned main window
+    // next to it is one too: what is on screen is then the question itself - the window
+    // behind it is hidden, not gone - and there is no window a close could be posted to
+    // without hiding the question with it.
+    target.modalOpen = target.toRaise != nullptr && (target.main == nullptr || IsWindowEnabled(target.main) == FALSE);
     return target;
 }
 
@@ -213,6 +208,12 @@ bool waitForExit(const std::vector<HANDLE> &processes, unsigned long waitMillise
 }
 } // namespace
 
+bool isCockatriceImage(const std::wstring &imageName)
+{
+    constexpr wchar_t COCKATRICE_IMAGE[] = L"cockatrice.exe";
+    return CompareStringOrdinal(imageName.c_str(), -1, COCKATRICE_IMAGE, -1, TRUE) == CSTR_EQUAL;
+}
+
 CloseHoldersResult closeCockatriceHolders(const std::vector<Holder> &holders, unsigned long waitMilliseconds)
 {
     CloseHoldersResult result;
@@ -233,18 +234,20 @@ CloseHoldersResult closeCockatriceHolders(const std::vector<Holder> &holders, un
         search.processId = holder.processId;
         EnumWindows(collectProcessWindow, reinterpret_cast<LPARAM>(&search));
         if (search.windows.empty()) {
+            ++result.withoutWindow;
             lines.push_back(processLabel(holder) + L" has no window to close, left to you");
             continue;
         }
 
         const CloseTarget target = chooseCloseTarget(search.windows);
-        if (target.main == nullptr) {
+        if (target.toRaise == nullptr) {
             // Nothing visible that a user could answer: the instance is hidden - in the
             // notification area, say - or has nothing but Qt's own bookkeeping windows on
             // screen. Neither is a window a close could reach, and showing one from here does
             // not help: Windows would display it while the application still believes it is
             // hidden, so it stays blank and answers nothing. The instance is named and left to
             // the user, who can open it from the notification area.
+            ++result.withoutWindow;
             lines.push_back(processLabel(holder) + L" has no window to ask, left to you");
             continue;
         }
@@ -255,7 +258,8 @@ CloseHoldersResult closeCockatriceHolders(const std::vector<Holder> &holders, un
         // an earlier attempt useless: the client accepts it, hides this window and the dialog
         // with it, and stays alive inside the nested loop the dialog opened, where nothing can
         // reach it any more. The question is the request now, so it is brought forward and
-        // left alone.
+        // left alone - including a question whose owner window is hidden, which is the only
+        // thing left on screen and has no unowned window a close could be posted to.
         if (target.modalOpen) {
             lines.push_back(line + L", and is waiting for an answer to a question of its own");
             continue;

tools/installer-holder-probe/close_holders.h

@@ -11,13 +11,27 @@
  *
  * @c report is one line naming every holder, including the ones this deliberately
  * leaves alone, because the installer repeats it in its log and in the second prompt.
+ * @c withoutWindow counts the holders that had no visible window a user could answer: they
+ * were named and left alone, and the launcher mode reads that count as "nothing to ask"
+ * rather than as a window the user still has to answer.
  */
 struct CloseHoldersResult
 {
-    unsigned int askedToClose = 0; ///< holders a WM_CLOSE was posted to
-    std::wstring report;           ///< one line, narrowed to ASCII when written
+    unsigned int askedToClose = 0;  ///< holders a WM_CLOSE was posted to
+    unsigned int withoutWindow = 0; ///< holders with no visible window a user could answer
+    std::wstring report;            ///< one line, narrowed to ASCII when written
 };
 
+/**
+ * @brief Whether @p imageName is the client the update belongs to, matched without case.
+ *
+ * The close mode asks only this image to close, and the launcher mode is only allowed to
+ * report it as safe to end, so both read the answer from here.
+ *
+ * @param imageName the file name of a process image, empty when it could not be read
+ */
+bool isCockatriceImage(const std::wstring &imageName);
+
 /**
  * @brief Brings every running Cockatrice holding an installer file forward and asks it
  *        to close.

tools/installer-holder-probe/main.cpp

@@ -3,6 +3,7 @@
 #include "restart_manager_holders.h"
 
 #include <algorithm>
+#include <cstdlib>
 #include <string>
 #include <vector>
 #include <windows.h>
@@ -14,6 +15,10 @@ namespace
 /// trailing backslash then never has to survive NSIS quoting or a command line.
 constexpr wchar_t INSTALL_DIR_VARIABLE[] = L"COCKATRICE_GUARD_DIR";
 
+/// The installer sets this to the process it waits on (/PID=), so the launcher mode reads
+/// the same process id the wait does instead of one that has to survive NSIS quoting.
+constexpr wchar_t LAUNCHER_PID_VARIABLE[] = L"COCKATRICE_LAUNCHER_PID";
+
 /// How long the close mode waits for the instance it asked to exit before it reports the
 /// process as still running. The installer measures the purge right after this call, so
 /// the wait belongs here: it is the only place that holds a handle on the process whose
@@ -30,16 +35,19 @@ constexpr unsigned long END_WAIT_MILLISECONDS = 5000;
 /// In the close mode 1 means a close was actually sent and 0 that every holder was left
 /// to the user, which is what the report line next to the code spells out. 3 is only for
 /// a lookup that failed and named nobody: one that failed but still named a process
-/// running from the directory is answered with that process.
+/// running from the directory is answered with that process. 4 is only the launcher mode:
+/// the process the installer waits on has no window a user could answer, which is the one
+/// answer the installer acts on by itself instead of asking the user.
 enum ExitCode
 {
     EXIT_NO_HOLDERS = 0,
     EXIT_HOLDERS_FOUND = 1,
     EXIT_NOT_CONFIGURED = 2,
     EXIT_PROBE_FAILED = 3,
+    EXIT_NO_WINDOW = 4,
 };
 
-/// The probe has three modes, selected by the first argument:
+/// The probe has four modes, selected by the first argument:
 ///
 ///   (none)           name what holds the payload, one line, for the guard prompt and the
 ///                     abort dialog/log
@@ -48,18 +56,22 @@ enum ExitCode
 ///   --end-holders    end the running Cockatrice, Oracle and Servatrice and wait for them to
 ///                     be gone, for the last resort the guard prompt offers when a close did
 ///                     not free the files
+///   --close-launcher ask the one process the installer waits on (/PID=) to close and say
+///                     whether there was a window to ask at all, because an instance with
+///                     no window has answered its own close and is stuck in shutdown
 ///
-/// All three answer from two lookups that cover each other's blind spot: the Restart Manager
-/// names whoever has a payload file open, and every process running from the directory is
-/// named as well, because a running instance holds its own image and the modules it
-/// loaded as mappings rather than open files and the Restart Manager does not always
+/// The guard's three modes answer from two lookups that cover each other's blind spot: the
+/// Restart Manager names whoever has a payload file open, and every process running from the
+/// directory is named as well, because a running instance holds its own image and the modules
+/// it loaded as mappings rather than open files and the Restart Manager does not always
 /// report those. Neither lookup alone is the guard's authority: an instance the file
 /// handles do not name is exactly the case that left a half-replaced install behind.
 enum class Mode
 {
-    Name,  ///< name the holders
-    Close, ///< ask the holders to close
-    End,   ///< end the holders
+    Name,          ///< name the holders
+    Close,         ///< ask the holders to close
+    End,           ///< end the holders
+    CloseLauncher, ///< ask the one process the installer waits on to close
 };
 
 Mode requestedMode(int argumentCount, char **arguments)
@@ -72,6 +84,9 @@ Mode requestedMode(int argumentCount, char **arguments)
         if (mode == "--end-holders") {
             return Mode::End;
         }
+        if (mode == "--close-launcher") {
+            return Mode::CloseLauncher;
+        }
     }
     return Mode::Name;
 }
@@ -92,11 +107,11 @@ void writeLine(const std::wstring &text)
     WriteFile(GetStdHandle(STD_OUTPUT_HANDLE), narrow.data(), static_cast<DWORD>(narrow.size()), &written, nullptr);
 }
 
-std::wstring installDirFromEnvironment()
+std::wstring environmentVariable(const wchar_t *name)
 {
     for (DWORD size = MAX_PATH; size <= 32768U; size *= 2) {
         std::vector<wchar_t> buffer(size);
-        const DWORD length = GetEnvironmentVariableW(INSTALL_DIR_VARIABLE, buffer.data(), size);
+        const DWORD length = GetEnvironmentVariableW(name, buffer.data(), size);
         if (length == 0) {
             return {};
         }
@@ -106,6 +121,68 @@ std::wstring installDirFromEnvironment()
     }
     return {};
 }
+
+std::wstring installDirFromEnvironment()
+{
+    return environmentVariable(INSTALL_DIR_VARIABLE);
+}
+
+/// The process id the installer waits on, or false when it was not passed or is not a
+/// number. A pid of 0 is refused because the installer never passes one.
+bool launcherProcessIdFromEnvironment(unsigned long &processId)
+{
+    const std::wstring text = environmentVariable(LAUNCHER_PID_VARIABLE);
+    if (text.empty()) {
+        return false;
+    }
+    wchar_t *end = nullptr;
+    const unsigned long parsed = std::wcstoul(text.c_str(), &end, 10);
+    if (parsed == 0 || end == nullptr || *end != L'\0') {
+        return false;
+    }
+    processId = parsed;
+    return true;
+}
+
+/// Asks the one process the installer waits on to close and answers what it could do.
+///
+/// The guard closes a set and reports every holder; here there is exactly one process, and
+/// the installer needs an answer it can act on rather than a sentence: a launcher with no
+/// window to ask is past its own prompts and stuck in shutdown, so no answer the installer
+/// offers would ever reach it. Everything else - a window that was asked, or one that is
+/// waiting on a question of its own - is still the user's decision.
+int closeLauncher(const std::wstring &installDir)
+{
+    unsigned long processId = 0;
+    if (!launcherProcessIdFromEnvironment(processId)) {
+        writeLine(L"the probe was not given the launcher process id");
+        return EXIT_NOT_CONFIGURED;
+    }
+
+    // Only an instance that still runs from the directory being replaced is asked: a bare
+    // process id names anything once it is reused, while a process running from the
+    // directory is the one the installer started this update from.
+    for (const Holder &holder : findProcessesRunningFrom(installDir)) {
+        if (holder.processId != processId) {
+            continue;
+        }
+        if (!isCockatriceImage(holder.imageName)) {
+            writeLine(L"the launcher is not cockatrice.exe, left to you");
+            return EXIT_PROBE_FAILED;
+        }
+        const CloseHoldersResult result = closeCockatriceHolders({holder}, CLOSE_WAIT_MILLISECONDS);
+        writeLine(result.report);
+        if (result.askedToClose > 0) {
+            return EXIT_HOLDERS_FOUND;
+        }
+        return result.withoutWindow > 0 ? EXIT_NO_WINDOW : EXIT_HOLDERS_FOUND;
+    }
+    // Not an answer: the lookup may have failed or refused the image, and the installer
+    // holds a live handle on that process, so "not found" must not read as "gone". The
+    // installer shows its dialog for this code instead of acting on it.
+    writeLine(L"the launcher is not running from the directory being replaced, left to you");
+    return EXIT_PROBE_FAILED;
+}
 } // namespace
 
 int main(int argumentCount, char **arguments)
@@ -116,6 +193,14 @@ int main(int argumentCount, char **arguments)
         return EXIT_NOT_CONFIGURED;
     }
 
+    const Mode mode = requestedMode(argumentCount, arguments);
+    // The launcher mode answers about exactly one process, so the holder set below is what it
+    // does not need - and the Restart Manager call behind it is the slowest thing this tool
+    // does, which must not delay a wait that started with a ten-second grace.
+    if (mode == Mode::CloseLauncher) {
+        return closeLauncher(installDir);
+    }
+
     // Processes running from the directory first: they are the instance the user is asked
     // to close, and the Restart Manager may not name any of them (see
     // findProcessesRunningFrom). The file holders follow, deduplicated by process, so an
@@ -147,7 +232,6 @@ int main(int argumentCount, char **arguments)
         return EXIT_PROBE_FAILED;
     }
 
-    const Mode mode = requestedMode(argumentCount, arguments);
     if (holders.empty()) {
         // No output in the naming mode: the installer words the "nothing is holding them"
         // case, and it owns the message either way. The acting modes say it themselves,