[Installer] Offer to close blocking processes and retry purge on upgrade (#7664)

* [Installer] Offer to close blocking processes and retry purge on upgrade

* [Installer] Retry the purge after closing only our own applications

The first version of this fix closed every Restart Manager holder through
RmShutdown, forcing what refused to close. RmShutdown also stops services, and
the only API that brings them back is RmRestart, which would restart Cockatrice
itself into the files that are about to be replaced; the forced pass also
contradicts the running-instance guard, which never force-closes Cockatrice
because it may be holding an unsaved deck. The guard ran first anyway, so for
the case the fix exists for - a Cockatrice restarted during the update - the
probe never got to run: the guard waited 60 s and aborted.

Keep the prompt and the retry, but ask only Cockatrice, Oracle and Servatrice to
close, with WM_CLOSE and nothing else, and leave every other holder named by the
probe to the user. Repeat the prompt while the user keeps saying Yes, refresh
the names after a failed retry, and keep the /R abort message pointing at the
downloaded installer.

Also clear the read-only attribute before every purge attempt. Delete and RMDir
leave such a file in place, so the retry could never make progress, and the
diagnostics branch that named the cause had no action behind it.

The holder probe goes back to being diagnostic-only, so its Restart Manager
code and the --close argument are dropped.

* [Installer] Make an accepted close of the running client actually land

WM_CLOSE is a request, and Cockatrice answers it: while a card database
update is running, a game is open or a deck is unsaved it puts up its own
"are you sure?" confirmation. That dialog opened behind the installer's
prompt, so the user saw nothing happen; the next Yes was posted to a main
window the app's own modal dialog had disabled, so it was ignored as well,
and the identical yes/no prompt came back with no hint of why.

- Activate the matching window before posting WM_CLOSE, so the app's own
  confirmation is in front where it can be answered.
- Wait up to 30 s for cockatrice.exe to exit instead of purging after a
  fixed 2 s, so the purge measures the app gone rather than its dialog.
- On the second and later rounds ask with Retry/Cancel and say what is
  needed, instead of repeating the same offer that just did nothing.
- Stop promising Oracle and Servatrice: they are console applications, so
  CloseMainWindow() has nothing to send WM_CLOSE to. They are not
  force-stopped either - Oracle writes the card database in place.

* [Installer] Close the client from the holder probe, not PowerShell

Accepting the offer to close did nothing on a real machine: the instance stayed
up, never came forward, and the same prompt came back.

The prompt names its holders from the Restart Manager probe, but the close itself
went through powershell.exe: Get-Process -Name, a strict "path starts with
$INSTDIR" filter, then Process.CloseMainWindow() and a WScript.Shell AppActivate.
Every step of that is silent. A process whose path cannot be read is filtered
out, a process .NET finds no main window for makes CloseMainWindow() return
false without posting anything, and AppActivate is refused for a background
caller - so an attempted close, a missed close and a close request the client
ignored all looked the same in the log: nothing.

Ask the probe instead. installer-holder-probe --close-holders closes exactly the
processes Restart Manager just named as holding a file of the version being
replaced, so no executable path has to be guessed and a third party application
that merely shares the image name is never in the set. Per instance it restores
and brings the top-level window forward (AttachThreadInput, so the client's own
"really quit?" confirmation is on screen rather than behind the installer), posts
WM_CLOSE to the main window, and waits up to 30 s for the process to exit. Its
report names every holder with what happened to it, including the ones left to
the user, and goes into the installer log as one line.

That also removes the separate PowerShell polling loop that used to wait for the
close: the probe holds a handle on the process it asked, so the purge that
follows measures a settled directory.

Oracle and Servatrice are still never closed, and nothing is force-stopped.

---------

Co-authored-by: sentry[bot] <39604003+sentry[bot]@users.noreply.github.com>
Co-authored-by: Lukas Brübach <Bruebach.Lukas@bdosecurity.de>

cmake/NSIS.template.in

@@ -38,6 +38,9 @@ Var AbortReason
 Var HolderExit
 Var HolderOutput
 Var HolderFound
+; OfferToCloseAndRetryPurge counts the accepted close-and-retry rounds, so the
+; prompt can stop repeating an offer that provably did not help (see there).
+Var HolderCloseRounds
 Var WaitPid
 Var LauncherExited
 Var InstallLogPath
@@ -766,8 +769,16 @@ Function CloseMatchingApps
     ; or one owned by another account running as SYSTEM) as a match would make
     ; the installer close a third party application, and it runs elevated, so
     ; CloseMainWindow() would succeed on it.
+    ;
+    ; The window is activated first, because WM_CLOSE is answered, not obeyed:
+    ; Cockatrice replies with a confirmation of its own while a card database
+    ; update is running, a game is open or a deck is unsaved. Without the
+    ; activation that dialog opens behind the installer's own prompt, where it
+    ; looks like the close did nothing and the second WM_CLOSE is delivered to a
+    ; main window its own modal dialog has disabled, which is ignored. Failing to
+    ; activate is harmless and swallowed: the close request still goes out.
     System::Call 'kernel32::SetEnvironmentVariable(t, t) i ("COCKATRICE_GUARD_DIR", "$GuardDir")'
-    nsExec::ExecToLog 'powershell -NoProfile -Command "Get-Process -Name $\'$R2$\' -ErrorAction SilentlyContinue | Where-Object { try { $$p = $$_.Path; if ($$p) { $$p.StartsWith($$env:COCKATRICE_GUARD_DIR, [StringComparison]::OrdinalIgnoreCase) } else { $$false } } catch { $$false } } | ForEach-Object { $$null = $$_.CloseMainWindow() }"'
+    nsExec::ExecToLog 'powershell -NoProfile -Command "Get-Process -Name $\'$R2$\' -ErrorAction SilentlyContinue | Where-Object { try { $$p = $$_.Path; if ($$p) { $$p.StartsWith($$env:COCKATRICE_GUARD_DIR, [StringComparison]::OrdinalIgnoreCase) } else { $$false } } catch { $$false } } | ForEach-Object { try { $$null = (New-Object -ComObject WScript.Shell).AppActivate($$_.Id) } catch { }; $$null = $$_.CloseMainWindow() }"'
     Pop $R3
 FunctionEnd
 
@@ -1185,15 +1196,74 @@ Function CheckStaleRuntimeMask
     ${EndIf}
 FunctionEnd
 
+; Clears the read-only attribute of everything the purge is about to delete.
+;
+; Delete and RMDir only set the error flag: a read-only file is left on disk
+; untouched, and the retry loop then cannot make progress however long it waits,
+; because the file is not held - it is protected. The abort diagnostics have a
+; branch for exactly that ("read-only, which stops Delete"), so a survivor that
+; is both free and read-only was a dead end until now. This runs on every
+; attempt, so an attribute set again between attempts is cleared as well.
+; SetFileAttributes has no wildcard support, hence the walks.
+Function ClearReadOnlyPurgeTargets
+    FindFirst $StaleFindHandle $StaleFindName "$INSTDIR\*.dll"
+    ${If} $StaleFindHandle != ""
+    crp_dll_loop:
+        SetFileAttributes "$INSTDIR\$StaleFindName" NORMAL
+        ClearErrors
+        FindNext $StaleFindHandle $StaleFindName
+        IfErrors 0 crp_dll_loop
+        FindClose $StaleFindHandle
+    ${EndIf}
+    StrCpy $R6 "$INSTDIR\Plugins"
+    Call ClearReadOnlyTree
+FunctionEnd
+
+; usage: set $R6 to a directory. Clears the read-only attribute of every entry
+; below it, so RMDir /r is not stopped by a single file. Saves and restores the
+; registers it works in, because it recurses into itself.
+Function ClearReadOnlyTree
+    Push $R6
+    Push $R7
+    Push $R8
+    FindFirst $R7 $R8 "$R6\*.*"
+    ${If} $R7 != ""
+    crt_loop:
+        ${If} $R8 != "."
+        ${AndIf} $R8 != ".."
+            SetFileAttributes "$R6\$R8" NORMAL
+            ; Only an entry with contents has anything below it that the purge
+            ; would fail on; an empty directory's own attribute is already
+            ; cleared above and there is nothing left to descend into.
+            ${If} ${FileExists} "$R6\$R8\*.*"
+                Push $R6
+                StrCpy $R6 "$R6\$R8"
+                Call ClearReadOnlyTree
+                Pop $R6
+            ${EndIf}
+        ${EndIf}
+        ClearErrors
+        FindNext $R7 $R8
+        IfErrors 0 crt_loop
+        FindClose $R7
+    ${EndIf}
+    Pop $R8
+    Pop $R7
+    Pop $R6
+FunctionEnd
+
 ; Removes the Plugins directory and all top-level DLLs from $INSTDIR, then
 ; retries up to two more times (2 s apart) when files remain locked, so that
 ; transient holders such as antivirus scanners have a chance to release their
-; handles before the installer gives up. Three attempts total.
+; handles before the installer gives up. Three attempts total. The read-only
+; attribute is cleared before each attempt: a protected file is not a held one
+; and would never clear on its own.
 ; Sets $R0 to 0 when the purge is clean, or to 1 when stale DLLs remain
 ; after all attempts.
 Function RetryDllPurge
     StrCpy $R5 0        ; counts retries after the initial attempt
     rdp_loop:
+    Call ClearReadOnlyPurgeTargets
     RMDir /r "$INSTDIR\Plugins"
     Delete "$INSTDIR\*.dll"
     Call HasStaleRuntimeFiles
@@ -1253,12 +1323,14 @@ FunctionEnd
 ;
 ; The probe opens each survivor for DELETE access and then closes it again
 ; without deleting anything. The code is read by the plugin right after that call
-; (?e), so a failure reports what Windows refused; clearing LastError first is
-; what makes a zero code mean "this open worked", i.e. the purge-time failure was
-; transient (an antivirus scan that has finished, an instance still shutting
-; down) rather than a standing condition. The plugin parses the call's arguments
-; between the clear and the call, so a non-zero code is reported as what Windows
-; said rather than as proof of a cause - the wording below never claims one.
+; (?e), so a failure reports what Windows refused; clearing LastError first makes
+; a zero code mean "the open worked". That is all it means: Windows lets a DLL a
+; running process has mapped be opened for DELETE, and DeleteFile still refuses
+; it until the image is unloaded, so a zero code is not evidence that the
+; failure was transient - the holders line is what names a running process. The
+; plugin parses the call's arguments between the clear and the call, so a
+; non-zero code is reported as what Windows said rather than as proof of a
+; cause - the wording below never claims one.
 Function LogStaleRuntimeDiagnostics
     FindFirst $StaleFindHandle $StaleFindName "$INSTDIR\*.dll"
     ${If} $StaleFindHandle != ""
@@ -1275,11 +1347,12 @@ Function LogStaleRuntimeDiagnostics
         ${Else}
             Pop $R2
             ${If} $R2 = 0
-                ; A valid handle: close it. The open proves the file is not held
-                ; open, but not that Delete would succeed: a read-only file opens
-                ; for DELETE and still fails DeleteFile until the attribute is
-                ; cleared, which the purge does not do. Report that standing
-                ; cause instead of calling the failure transient.
+                ; A valid handle: close it. The open says nothing about Delete:
+                ; a DLL a running process has mapped opens for DELETE and still
+                ; refuses DeleteFile, and a read-only file does the same until its
+                ; attribute is cleared (the purge clears it before every attempt
+                ; now, so a survivor that is read-only had it set again). Report
+                ; that standing cause instead of calling the failure transient.
                 System::Call 'kernel32::CloseHandle(p R1)'
                 System::Call 'kernel32::GetFileAttributesW(w "$INSTDIR\$StaleFindName") i.R3'
                 ${If} $R3 = -1
@@ -1287,9 +1360,9 @@ Function LogStaleRuntimeDiagnostics
                 ${Else}
                     IntOp $R3 $R3 & 1
                     ${If} $R3 = 1
-                        StrCpy $R0 "a delete-access open now succeeds, but it is still read-only, which Delete does not clear"
+                        StrCpy $R0 "a delete-access open now succeeds, but it is read-only, which stops Delete"
                     ${Else}
-                        StrCpy $R0 "a delete-access open now succeeds"
+                        StrCpy $R0 "a delete-access open now succeeds, but a DLL a running process has mapped still refuses Delete - see holders"
                     ${EndIf}
                 ${EndIf}
             ${Else}
@@ -1360,6 +1433,125 @@ Function LogProcessesUsingInstallDir
     !insertmacro CockLog "holders: $HolderOutput"
 FunctionEnd
 
+; Called after a purge left files behind and LogStaleRuntimeDiagnostics named who
+; is holding them. Instead of aborting straight away - which on /R strands the
+; user, since the old version is already uninstalled - ask, close what this
+; installer owns, purge again, and retry while the user keeps asking. Sets
+; $StaleRuntimeAfterPurge to the last purge's verdict, so the caller still aborts
+; when the files survive.
+;
+; Cockatrice is asked to close (WM_CLOSE, the same request as clicking the
+; window's close button) and keeps the last word: it may be waiting for an answer
+; to its own "really quit?" confirmation (a running card database update, an open
+; game, an unsaved deck), and that stays its decision, exactly as the
+; running-instance guard treats it. The instance is brought forward first, so that
+; confirmation is on screen rather than behind this installer's prompt. Oracle and
+; Servatrice have no window to send a close to at all, and nothing here is
+; force-stopped: Oracle writes the card database in place and killing it mid-write
+; would corrupt it. Everything the probe named that this installer cannot close -
+; those two included - is left to the user, whom the prompt names it to.
+;
+; Restart Manager's RmShutdown was the alternative and was rejected: it shuts down
+; services as well as applications, and the only way to bring them back is
+; RmRestart, which would restart this very application into files that are about
+; to be replaced.
+;
+; No offer when nobody is at the keyboard (a /S install without /R, e.g. a package
+; manager): a dialog there would block forever. /R is the in-app updater, whose
+; client has already quit, so the user is there - the same reasoning AbortInstall
+; uses for its message box. No offer either when the probe named nobody: there is
+; nothing to close, and the abort's per-file diagnostics already record what
+; Windows said.
+;
+; The first round offers the close; every later one stops offering it and says
+; what to do instead. Repeating the same question is what made an accepted close
+; look like a Yes that was ignored: when Cockatrice was blocked on its own
+; confirmation, the second Yes was delivered to a window that dialog had disabled,
+; so nothing closed, the holders were unchanged, and the identical prompt came
+; back with no hint of what had happened.
+Function OfferToCloseAndRetryPurge
+    ${If} $ReinstallMode = 0
+    ${AndIf} ${Silent}
+        !insertmacro CockLog "close-holders: unattended install, not offering to close holders"
+        Return
+    ${EndIf}
+    ${If} $HolderFound != 1
+        !insertmacro CockLog "close-holders: no holder was named, nothing to offer"
+        Return
+    ${EndIf}
+    StrCpy $HolderCloseRounds 0
+    ocrp_prompt:
+    ${If} $HolderCloseRounds = 0
+        MessageBox MB_YESNO|MB_ICONEXCLAMATION|MB_DEFBUTTON2 \
+            "Files of the previous version of Cockatrice are still in use by:$\r$\n$\r$\n$HolderOutput$\r$\n$\r$\nThe installer can close Cockatrice for you. It may first ask you to confirm quitting - it does while a card database update is running, a game is open or a deck is unsaved - so answer that window. Close Oracle, Servatrice and any other program listed above by hand.$\r$\n$\r$\nChoose Yes to close Cockatrice and try again, or No to cancel the installation." \
+            IDYES ocrp_close
+        !insertmacro CockLog "close-holders: declined by user"
+        Return
+    ${EndIf}
+    MessageBox MB_RETRYCANCEL|MB_ICONEXCLAMATION|MB_DEFBUTTON1 \
+        "Files of the previous version of Cockatrice are still in use by:$\r$\n$\r$\n$HolderOutput$\r$\n$\r$\nCockatrice may be waiting for an answer in its own window - it asks before quitting while a card database update is running, a game is open or a deck is unsaved. Switch to it and confirm, then close Oracle, Servatrice and any other program listed above.$\r$\n$\r$\nChoose Retry once they are closed, or Cancel to cancel the installation." \
+        IDRETRY ocrp_close
+    !insertmacro CockLog "close-holders: declined by user after $HolderCloseRounds round(s)"
+    Return
+    ocrp_close:
+    IntOp $HolderCloseRounds $HolderCloseRounds + 1
+    !insertmacro CockLog "close-holders: accepted (round $HolderCloseRounds), holders were [$HolderOutput]"
+    ; The close is a request, not a fact: AskCockatriceAppsToClose waits for the
+    ; instances it asked to exit, so a confirmation of Cockatrice's that is still up is
+    ; reported as still running instead of being measured as a successful close.
+    Call AskCockatriceAppsToClose
+    Call RetryDllPurge
+    StrCpy $StaleRuntimeAfterPurge $R0
+    !insertmacro CockLogListing "after-close-purge" "*.dll"
+    !insertmacro CockLogListing "after-close-plugins" "Plugins\*.*"
+    !insertmacro CockLog "close-holders: stale-runtime after retry=$StaleRuntimeAfterPurge"
+    ${If} $StaleRuntimeAfterPurge = 0
+        DetailPrint "Files of the previous version were removed; continuing the installation."
+        Return
+    ${EndIf}
+    ; Still stuck: refresh the names, so the next prompt and the log describe what
+    ; survived the close and not what was there before it.
+    Call LogStaleRuntimeDiagnostics
+    ${If} $HolderFound != 1
+        Return
+    ${EndIf}
+    Goto ocrp_prompt
+FunctionEnd
+
+; Asks the running Cockatrice to close, and waits for it to exit.
+;
+; The set is the Restart Manager answer from LogProcessesUsingInstallDir: exactly the
+; processes holding a file of the version being replaced. tools\installer-holder-probe
+; closes that set by pid, which is why nothing here has to match an executable path or
+; depend on powershell.exe. Only cockatrice.exe is asked: Oracle writes the card database
+; in place and Servatrice has no window, so both stay the user's job, as the prompt says.
+;
+; Each instance is brought to the foreground before WM_CLOSE is posted to its main
+; window. WM_CLOSE is answered, not obeyed - Cockatrice asks before it quits while a card
+; database update is running, a game is open or a deck is unsaved - and that question is
+; only useful if it is in front of the installer instead of behind it.
+;
+; The probe waits up to 30 s for the processes it asked to exit, so the purge that follows
+; measures a settled directory rather than sleeping for a fixed time. Its line goes into
+; the log as it stands: it names every holder, including the ones left to the user, so a
+; close that did nothing can be told apart from one that was never sent.
+Function AskCockatriceAppsToClose
+    InitPluginsDir
+    ; LogProcessesUsingInstallDir extracted it already; the check keeps a single copy of
+    ; the executable in the installer and still works if that order ever changes.
+    IfFileExists "$PLUGINSDIR\installer-holder-probe.exe" ck_close_probe_ready
+        File "/oname=$PLUGINSDIR\installer-holder-probe.exe" "..\..\..\installer-holder-probe.exe"
+    ck_close_probe_ready:
+    System::Call 'kernel32::SetEnvironmentVariable(t, t) i ("COCKATRICE_GUARD_DIR", "$INSTDIR")'
+    ; The probe waits 30 s itself. The timeout here only bounds a probe that never
+    ; returns; the Restart Manager call inside it is the same one the holder listing makes.
+    nsExec::ExecToStack /TIMEOUT=40000 '"$PLUGINSDIR\installer-holder-probe.exe" --close-holders'
+    Pop $R3
+    Pop $R4
+    !insertmacro CockLog "close-holders: probe exit=$R3 [$R4]"
+    DetailPrint "Closing Cockatrice: $R4"
+FunctionEnd
+
 ; Uninstaller copies of the same routines (the uninstaller gets its own
 ; function set compiled in, it cannot call the installer functions).
 Function un.IsAppRunning
@@ -1786,13 +1978,21 @@ ${If} $IsUpgrade = 1
 ${Else}
     StrCpy $StaleRuntimeAfterPurge 0
 ${EndIf}
+${If} $StaleRuntimeAfterPurge = 1
+    !insertmacro CockLog "section: files of a previous version survived the purge"
+    ; Who is holding the payload is recorded first, so the log that reaches us
+    ; answers the question the message can only guess at, and so the prompt can
+    ; name them and offer to close the ones this installer owns.
+    Call LogStaleRuntimeDiagnostics
+    ; On /R the previous version is already uninstalled at this point, so an abort
+    ; leaves the user with no working Cockatrice. Offer to close and purge again,
+    ; repeating while the user keeps saying Yes; this updates
+    ; $StaleRuntimeAfterPurge with the last retry's verdict.
+    Call OfferToCloseAndRetryPurge
+${EndIf}
 ${If} $StaleRuntimeAfterPurge = 1
     !insertmacro CockLog "section: aborting, files of a previous version survived the purge"
     DetailPrint "Files of a previous version could not be removed; aborting the installation."
-    ; Who is holding the payload is recorded before the abort, so the log that
-    ; reaches us answers the question the message can only guess at, and so the
-    ; message can name the processes to close whenever any were found.
-    Call LogStaleRuntimeDiagnostics
     ; The in-app updater runs this installer silent and detached and never looks
     ; at its exit code, so the user has to be told what happened. A /S install
     ; without /R is unattended (a package manager, for instance) and a message
@@ -1803,6 +2003,12 @@ ${If} $StaleRuntimeAfterPurge = 1
     ${If} $HolderFound = 1
         StrCpy $AbortReason "$AbortReason$\r$\n$\r$\nStill using the install directory:$\r$\n$HolderOutput"
     ${EndIf}
+    ; "Run the installer again" is not enough on an in-app update: the old version
+    ; was uninstalled before the purge, so point at the installer that was downloaded.
+    ${If} $ReinstallMode = 1
+    ${AndIf} $PortableMode = 0
+        StrCpy $AbortReason "$AbortReason$\r$\n$\r$\nThe previous version has already been removed. To finish the update, run this installer again:$\r$\n$EXEPATH"
+    ${EndIf}
     Call AbortInstall
 ${EndIf}