[CI] Measure Ninja against the Visual Studio generator on Windows

Ninja scales its job pool to the machine it runs on, and it lets a
compiler launcher through, which MSBuild does not; those two are the
reasons to expect it to beat the Visual Studio generator on a 4-core
runner. That is an expectation, not a measurement, so run it as an A/B
next to the Visual Studio job for a week: the extra matrix entry is
allowed to fail, keeps its own " (Ninja)" suffix in the job name, is
never uploaded to a release and never contributes PDBs, so it can
neither block a pull request nor ship anything.

Ninja needs cl.exe and the MSVC environment on PATH, which the Visual
Studio generator gets for free by locating the compiler itself, hence
the setup-msvc-dev step for the experimental job alone. Its build gets
neither the MSBuild switches nor a job count: Ninja rejects -p: options
and sizes its own pool, while the Visual Studio job keeps both. cmake
takes the generator platform from the environment, so it is unset for
everything that is not a Visual Studio generator, where it would
otherwise follow the generator around.

Delete this entry once the numbers are in: if Ninja wins, switch the
Windows matrix to it and fix the PDB paths with it.
This commit is contained in:
Lukas Brübach 2026-10-02 14:03:51 +02:00 • committed by BruebachL
parent 285e6d6825
commit d80c77bf6f
2 changed files with 62 additions and 22 deletions

View file

@ -138,6 +138,12 @@ done
set -e
# cmake reads the generator platform from the environment, so a leftover platform
# follows the generator around; Ninja has no platform to select.
if [[ $CMAKE_GENERATOR != "Visual Studio"* ]]; then
unset CMAKE_GENERATOR_PLATFORM
fi
# Setup
./servatrice/check_schema_version.sh
if [[ ! $BUILDTYPE ]]; then
@ -314,26 +320,32 @@ if [[ $RUNNER_OS == macOS ]]; then
fi
elif [[ $RUNNER_OS == Windows ]]; then
# cmake only hands MSBuild a job count when one is asked for, and without it
# MSBuild builds a single node: one project at a time, whatever MTT does inside
# it. Ask for a node per core.
if [[ ! $BUILD_PARALLEL_LEVEL ]]; then
BUILD_PARALLEL_LEVEL="${NUMBER_OF_PROCESSORS:-1}"
fi
buildflags+=(--parallel "$BUILD_PARALLEL_LEVEL")
if [[ $CMAKE_GENERATOR == "Visual Studio"* ]]; then
# cmake only hands MSBuild a job count when it is asked for, and without it
# MSBuild builds a single node: one project at a time, whatever MTT does inside
# it. Ask for a node per core.
if [[ ! $BUILD_PARALLEL_LEVEL ]]; then
BUILD_PARALLEL_LEVEL="${NUMBER_OF_PROCESSORS:-1}"
fi
buildflags+=(--parallel "$BUILD_PARALLEL_LEVEL")
# Enable MTT, see https://devblogs.microsoft.com/cppblog/improved-parallelism-in-msbuild/
# and https://devblogs.microsoft.com/cppblog/cpp-build-throughput-investigation-and-tune-up/#multitooltask-mtt
# EnforceProcessCountAcrossBuilds makes the CL_MPCount ceiling apply across all
# the projects the nodes are building, instead of each node running its own
# cl.exe processes and /m and MTT multiplying into an oversubscribed machine.
buildflags+=(-- -p:UseMultiToolTask=true -p:EnableClServerMode=true -p:EnforceProcessCountAcrossBuilds=true -p:CL_MPCount="$BUILD_PARALLEL_LEVEL")
if [[ $BUILD_PROFILE ]]; then
# A performance summary says which projects and tasks cost what, the binary log
# keeps the full timeline; both are far too noisy to always produce. The files
# land next to the solution, i.e. in the build dir the workflow uploads. These
# join the MTT options above, which already carried the "--" separator.
buildflags+=("-flp:PerformanceSummary;v=q;LogFile=msbuild-perf.log" -bl:msbuild.binlog)
# Enable MTT, see https://devblogs.microsoft.com/cppblog/improved-parallelism-in-msbuild/
# and https://devblogs.microsoft.com/cppblog/cpp-build-throughput-investigation-and-tune-up/#multitooltask-mtt
# EnforceProcessCountAcrossBuilds makes the CL_MPCount ceiling apply across all
# the projects the nodes are building, instead of each node running its own
# cl.exe processes and /m and MTT multiplying into an oversubscribed machine.
buildflags+=(-- -p:UseMultiToolTask=true -p:EnableClServerMode=true -p:EnforceProcessCountAcrossBuilds=true -p:CL_MPCount="$BUILD_PARALLEL_LEVEL")
if [[ $BUILD_PROFILE ]]; then
# A performance summary says which projects and tasks cost what, the binary log
# keeps the full timeline; both are far too noisy to always produce. The files
# land next to the solution, i.e. in the build dir the workflow uploads. These
# join the MTT options above, which already carried the "--" separator.
buildflags+=("-flp:PerformanceSummary;v=q;LogFile=msbuild-perf.log" -bl:msbuild.binlog)
fi
else
# Ninja runs the compiler itself, so it neither needs the MSVC switches nor
# wants a job count: it sizes its own pool for the cores it finds.
echo "ninja $(ninja --version)"
fi
fi