Compare commits

...

5 commits

Author SHA1 Message Date
Lukas Brübach
1f8d252c68 [CI] Give the Ninja leg the same vcpkg scope as the Visual Studio leg 2026-10-14 08:14:20 +02:00
Lukas Brübach
28a5bd44e7 [CI] Measure Ninja against the Visual Studio generator on Windows
Ninja scales its job pool to the machine it runs on, and it lets a
compiler launcher through, which MSBuild does not; those two are the
reasons to expect it to beat the Visual Studio generator on a 4-core
runner. That is an expectation, not a measurement, so run it as an A/B
next to the Visual Studio job for a week: the extra matrix entry is
allowed to fail, keeps its own " (Ninja)" suffix in the job name, is
never uploaded to a release and never contributes PDBs, so it can
neither block a pull request nor ship anything.

Ninja needs cl.exe and the MSVC environment on PATH, which the Visual
Studio generator gets for free by locating the compiler itself, hence
the setup-msvc-dev step for the experimental job alone. Its build gets
neither the MSBuild switches nor a job count: Ninja rejects -p: options
and sizes its own pool, while the Visual Studio job keeps both. cmake
takes the generator platform from the environment, so it is unset for
everything that is not a Visual Studio generator, where it would
otherwise follow the generator around.

Delete this entry once the numbers are in: if Ninja wins, switch the
Windows matrix to it and fix the PDB paths with it.
2026-10-14 08:14:20 +02:00
Lukas Brübach
255c713ce4 [CI] Profile the MSBuild run this pull request measures 2026-10-14 08:14:19 +02:00
Lukas Brübach
af05b85f29 [CI] Build the Windows solution on more than one MSBuild node
cmake --build only passes a job count to MSBuild when it is asked for
one, and .ci/compile.sh never asked, so every Visual Studio generator
build ran single-node: projects were built one after another and MTT
only ever parallelized the files of whichever project happened to be
building. Ask for one node per core, overridable with
BUILD_PARALLEL_LEVEL, and fall back to a single node when the core
count is not exposed.

Alongside it set EnforceProcessCountAcrossBuilds and CL_MPCount, the
knobs Microsoft's build throughput write-up pairs with MTT: without
them the node count and MTT multiply into more cl.exe processes than
the runner has cores for.
2026-10-14 08:14:19 +02:00
Lukas Brübach
8ed6354595 [CI] Collect an MSBuild performance summary and binary log on request
The Windows job spends about 28 minutes inside one "Build project"
group, with nothing in it that says where the time goes: MSBuild runs
single-node because no --parallel is passed, so projects build one
after another and it is not visible which projects, tasks or phases are
responsible.

Add a --profile flag that makes the Windows build write
-flp:PerformanceSummary and -bl, repeat the summary in the job log so
the numbers need no download, and report the core count, which decides
how much parallelism is worth asking for. The binary log is uploaded as
an artifact only on the runs that asked for it.

The workflow drives it from a build_profile matrix key that is empty by
default, so no build changes behaviour until someone sets it.
2026-10-14 08:14:19 +02:00
2 changed files with 121 additions and 8 deletions

View file

@ -17,8 +17,9 @@
# --dir <dir> sets the name of the build dir, default is "build"
# --cmake-generator <generator> sets CMAKE_GENERATOR as used by cmake
# --target-macos-version <version> sets the min os version - only used for macOS builds
# uses env: BUILDTYPE MAKE_INSTALL MAKE_PACKAGE PACKAGE_TYPE PACKAGE_SUFFIX MAKE_SERVER MAKE_NO_CLIENT MAKE_TEST USE_CCACHE CCACHE_SIZE CCACHE_EVICTION_AGE BUILD_DIR CMAKE_GENERATOR TARGET_MACOS_VERSION MAKE_SANITIZE MAKE_COVERAGE VCPKG_TARGET_MACOS_VERSION
# (correspond to args: --debug/--release --install --package <package type> --suffix <suffix> --server --test --sanitize --coverage --ccache <ccache_size> --dir <dir>)
# --profile writes an MSBuild performance summary and binary log, Windows only
# uses env: BUILDTYPE MAKE_INSTALL MAKE_PACKAGE PACKAGE_TYPE PACKAGE_SUFFIX MAKE_SERVER MAKE_NO_CLIENT MAKE_TEST USE_CCACHE CCACHE_SIZE CCACHE_EVICTION_AGE BUILD_DIR CMAKE_GENERATOR TARGET_MACOS_VERSION MAKE_SANITIZE MAKE_COVERAGE VCPKG_TARGET_MACOS_VERSION BUILD_PARALLEL_LEVEL BUILD_PROFILE
# (correspond to args: --debug/--release --install --package <package type> --suffix <suffix> --server --test --sanitize --coverage --ccache <ccache_size> --dir <dir> --profile)
# exitcode: 1 for failure, 3 for invalid arguments
# Read arguments
@ -115,6 +116,10 @@ while [[ $# != 0 ]]; do
export CMAKE_GENERATOR=$1
shift
;;
'--profile')
BUILD_PROFILE=1
shift
;;
'--target-macos-version')
shift
if [[ $# == 0 ]]; then
@ -133,6 +138,12 @@ done
set -e
# cmake reads the generator platform from the environment, so a leftover platform
# follows the generator around; Ninja has no platform to select.
if [[ $CMAKE_GENERATOR != "Visual Studio"* ]]; then
unset CMAKE_GENERATOR_PLATFORM
fi
# Setup
./servatrice/check_schema_version.sh
if [[ ! $BUILDTYPE ]]; then
@ -209,6 +220,22 @@ function ccachestatsverbose() {
fi
}
function cpuinfo() {
# The number of cores decides how much build parallelism is useful; the hosted
# runners are not all sized alike, so read it off the machine instead of assuming.
case "$RUNNER_OS" in
Windows)
echo "cores: ${NUMBER_OF_PROCESSORS:-unknown}"
;;
macOS)
echo "cores: $(sysctl -n hw.ncpu)"
;;
*)
echo "cores: $(nproc 2>/dev/null || echo unknown)"
;;
esac
}
# Compile
if [[ $RUNNER_OS == macOS ]]; then
# QTDIR is needed for macOS since we actually only use the cached thin Qt binaries instead of the install-qt-action,
@ -293,9 +320,33 @@ if [[ $RUNNER_OS == macOS ]]; then
fi
elif [[ $RUNNER_OS == Windows ]]; then
# Enable MTT, see https://devblogs.microsoft.com/cppblog/improved-parallelism-in-msbuild/
# and https://devblogs.microsoft.com/cppblog/cpp-build-throughput-investigation-and-tune-up/#multitooltask-mtt
buildflags+=(-- -p:UseMultiToolTask=true -p:EnableClServerMode=true)
if [[ $CMAKE_GENERATOR == "Visual Studio"* ]]; then
# cmake only hands MSBuild a job count when it is asked for, and without it
# MSBuild builds a single node: one project at a time, whatever MTT does inside
# it. Ask for a node per core.
if [[ ! $BUILD_PARALLEL_LEVEL ]]; then
BUILD_PARALLEL_LEVEL="${NUMBER_OF_PROCESSORS:-1}"
fi
buildflags+=(--parallel "$BUILD_PARALLEL_LEVEL")
# Enable MTT, see https://devblogs.microsoft.com/cppblog/improved-parallelism-in-msbuild/
# and https://devblogs.microsoft.com/cppblog/cpp-build-throughput-investigation-and-tune-up/#multitooltask-mtt
# EnforceProcessCountAcrossBuilds makes the CL_MPCount ceiling apply across all
# the projects the nodes are building, instead of each node running its own
# cl.exe processes and /m and MTT multiplying into an oversubscribed machine.
buildflags+=(-- -p:UseMultiToolTask=true -p:EnableClServerMode=true -p:EnforceProcessCountAcrossBuilds=true -p:CL_MPCount="$BUILD_PARALLEL_LEVEL")
if [[ $BUILD_PROFILE ]]; then
# A performance summary says which projects and tasks cost what, the binary log
# keeps the full timeline; both are far too noisy to always produce. The files
# land next to the solution, i.e. in the build dir the workflow uploads. These
# join the MTT options above, which already carried the "--" separator.
buildflags+=("-flp:PerformanceSummary;v=q;LogFile=msbuild-perf.log" -bl:msbuild.binlog)
fi
else
# Ninja runs the compiler itself, so it neither needs the MSVC switches nor
# wants a job count: it sizes its own pool for the cores it finds.
echo "ninja $(ninja --version)"
fi
fi
if [[ $USE_CCACHE ]]; then
@ -311,10 +362,19 @@ cmake .. "${flags[@]}"
echo "::endgroup::"
echo "::group::Build project"
cpuinfo
echo "Running cmake --build with flags: ${buildflags[*]}"
cmake --build . "${buildflags[@]}"
echo "::endgroup::"
if [[ $BUILD_PROFILE && $RUNNER_OS == Windows && -f msbuild-perf.log ]]; then
# Repeat the summary in the job log so the numbers are readable without
# downloading the binary log, which is hundreds of megabytes.
echo "::group::MSBuild performance summary"
cat msbuild-perf.log
echo "::endgroup::"
fi
if [[ $USE_CCACHE ]]; then
if [[ $CCACHE_EVICTION_AGE ]]; then
echo "::group::evict ccache files older than $CCACHE_EVICTION_AGE"

View file

@ -385,6 +385,12 @@ jobs:
target: 10
runner: windows-2025 # https://github.com/actions/runner-images/blob/main/images/windows/Windows2025-VS2026-Readme.md
# Set build_profile: 1 to have the build write an MSBuild performance
# summary and a binary log; empty on every entry that runs by default.
# Enabled here on purpose: this pull request measures MSBuild
# parallelism, so its run is the one worth profiling.
build_profile: 1
cmake_generator: "Visual Studio 18 2026"
cmake_generator_platform: x64
make_package: 1
@ -395,11 +401,36 @@ jobs:
vcpkg_scope: 'vcpkg/windows10-x64-vs2026/'
vcpkg_upload: 1 # only one leg per vcpkg_scope hands its archives over
name: ${{ matrix.os }} ${{ matrix.target }}${{ matrix.soc == 'Intel' && ' Intel' || '' }}${{ matrix.type == 'Debug' && ' Debug' || '' }}
# A/B run: is Ninja faster than the Visual Studio generator on this runner?
# It is allowed to fail and is never published, see the steps that build
# a release below. Delete this entry once the answer is known.
- os: Windows
target: 10
runner: windows-2025
cmake_generator: Ninja
experiment: yes
make_package: 1
package_suffix: "-Win10-ninja"
qt_version: 6.11.*
qt_modules: qtimageformats qtmultimedia qtshadertools qtwebsockets
type: Release
variant: ' (Ninja)'
# The same scope as the Visual Studio leg, so both halves of the A/B
# start from the same vcpkg cache and the comparison is about the
# generator alone. Without it this leg's prefix is empty, which makes
# the cache action restore the whole vcpkg/ namespace and write entries
# that no scope claims. vcpkg_upload is deliberately left unset, so only
# the Visual Studio leg hands its archives over.
vcpkg_scope: 'vcpkg/windows10-x64-vs2026/'
name: ${{ matrix.os }} ${{ matrix.target }}${{ matrix.soc == 'Intel' && ' Intel' || '' }}${{ matrix.type == 'Debug' && ' Debug' || '' }}${{ matrix.variant }}
needs: configure
runs-on: ${{ matrix.runner }}
continue-on-error: ${{ matrix.experiment == 'yes' }}
timeout-minutes: 100
env:
BUILD_PROFILE: '${{ matrix.build_profile }}'
CCACHE_DIR: ${{ github.workspace }}/.cache/
CCACHE_SIZE: 600M # space of all repo is 10Gi: https://docs.github.com/en/actions/using-workflows/caching-dependencies-to-speed-up-workflows#usage-limits-and-eviction-policy
@ -482,6 +513,14 @@ jobs:
modules: ${{ matrix.qt_modules }}
version: ${{ matrix.qt_version }}
# Ninja invokes cl.exe directly and needs the MSVC environment on PATH; the
# Visual Studio generator locates the compiler on its own and skips this.
- name: "[Windows/Ninja] Set up MSVC environment"
if: matrix.os == 'Windows' && matrix.cmake_generator == 'Ninja'
uses: TheMrMilchmann/setup-msvc-dev@v4.1.0
with:
arch: x64
- name: "[Windows] Install NSIS"
if: matrix.os == 'Windows'
shell: bash
@ -690,8 +729,11 @@ jobs:
if-no-files-found: error
path: ${{ steps.build.outputs.path }}
# Ninja writes its PDBs next to the binaries instead of into a Release
# subdirectory, and the experimental job is not a source of release debug
# symbols, so it keeps out of both.
- name: "[Windows] Upload PDBs (Program Databases)"
if: matrix.os == 'Windows' && github.ref_type != 'tag'
if: matrix.os == 'Windows' && matrix.experiment != 'yes' && github.ref_type != 'tag'
uses: actions/upload-artifact@v7
with:
if-no-files-found: error
@ -701,8 +743,19 @@ jobs:
build/oracle/Release/*.pdb
build/servatrice/Release/*.pdb
# Written only when the matrix entry sets build_profile, so the artifact
# exists exactly on the runs that were asked to produce one.
- name: "[Windows] Upload MSBuild binary log"
if: always() && matrix.os == 'Windows' && env.BUILD_PROFILE == '1'
uses: actions/upload-artifact@v7
with:
if-no-files-found: error
name: ${{ steps.build.outputs.name }}-msbuild
path: build/msbuild.binlog
retention-days: 7
- name: "Upload to release"
if: needs.configure.outputs.tag != null && matrix.make_package == '1'
if: needs.configure.outputs.tag != null && matrix.make_package == '1' && matrix.experiment != 'yes'
id: upload_release
shell: bash
env: