[CI] Measure Ninja against the Visual Studio generator on Windows

Ninja scales its job pool to the machine it runs on, and it lets a
compiler launcher through, which MSBuild does not; those two are the
reasons to expect it to beat the Visual Studio generator on a 4-core
runner. That is an expectation, not a measurement, so run it as an A/B
next to the Visual Studio job for a week: the extra matrix entry is
allowed to fail, keeps its own " (Ninja)" suffix in the job name, is
never uploaded to a release and never contributes PDBs, so it can
neither block a pull request nor ship anything.

Ninja needs cl.exe and the MSVC environment on PATH, which the Visual
Studio generator gets for free by locating the compiler itself, hence
the setup-msvc-dev step for the experimental job alone. Its build gets
neither the MSBuild switches nor a job count: Ninja rejects -p: options
and sizes its own pool, while the Visual Studio job keeps both. cmake
takes the generator platform from the environment, so it is unset for
everything that is not a Visual Studio generator, where it would
otherwise follow the generator around.

Delete this entry once the numbers are in: if Ninja wins, switch the
Windows matrix to it and fix the PDB paths with it.
This commit is contained in:
Lukas Brübach 2026-10-02 14:03:51 +02:00 • committed by BruebachL
parent 285e6d6825
commit d80c77bf6f
2 changed files with 62 additions and 22 deletions

View file

@ -401,9 +401,26 @@ jobs:
vcpkg_scope: 'vcpkg/windows10-x64-vs2026/'
vcpkg_upload: 1 # only one leg per vcpkg_scope hands its archives over
name: ${{ matrix.os }} ${{ matrix.target }}${{ matrix.soc == 'Intel' && ' Intel' || '' }}${{ matrix.type == 'Debug' && ' Debug' || '' }}
# A/B run: is Ninja faster than the Visual Studio generator on this runner?
# It is allowed to fail and is never published, see the steps that build
# a release below. Delete this entry once the answer is known.
- os: Windows
target: 10
runner: windows-2025
cmake_generator: Ninja
experiment: yes
make_package: 1
package_suffix: "-Win10-ninja"
qt_version: 6.11.*
qt_modules: qtimageformats qtmultimedia qtshadertools qtwebsockets
type: Release
variant: ' (Ninja)'
name: ${{ matrix.os }} ${{ matrix.target }}${{ matrix.soc == 'Intel' && ' Intel' || '' }}${{ matrix.type == 'Debug' && ' Debug' || '' }}${{ matrix.variant }}
needs: configure
runs-on: ${{ matrix.runner }}
continue-on-error: ${{ matrix.experiment == 'yes' }}
timeout-minutes: 100
env:
BUILD_PROFILE: '${{ matrix.build_profile }}'
@ -489,6 +506,14 @@ jobs:
modules: ${{ matrix.qt_modules }}
version: ${{ matrix.qt_version }}
# Ninja invokes cl.exe directly and needs the MSVC environment on PATH; the
# Visual Studio generator locates the compiler on its own and skips this.
- name: "[Windows/Ninja] Set up MSVC environment"
if: matrix.os == 'Windows' && matrix.cmake_generator == 'Ninja'
uses: TheMrMilchmann/setup-msvc-dev@v4.1.0
with:
arch: x64
- name: "[Windows] Install NSIS"
if: matrix.os == 'Windows'
shell: bash
@ -697,8 +722,11 @@ jobs:
if-no-files-found: error
path: ${{ steps.build.outputs.path }}
# Ninja writes its PDBs next to the binaries instead of into a Release
# subdirectory, and the experimental job is not a source of release debug
# symbols, so it keeps out of both.
- name: "[Windows] Upload PDBs (Program Databases)"
if: matrix.os == 'Windows' && github.ref_type != 'tag'
if: matrix.os == 'Windows' && matrix.experiment != 'yes' && github.ref_type != 'tag'
uses: actions/upload-artifact@v7
with:
if-no-files-found: error
@ -720,7 +748,7 @@ jobs:
retention-days: 7
- name: "Upload to release"
if: needs.configure.outputs.tag != null && matrix.make_package == '1'
if: needs.configure.outputs.tag != null && matrix.make_package == '1' && matrix.experiment != 'yes'
id: upload_release
shell: bash
env: