[Client/Server/Protocol] Surface live metrics in the Developer tab (#7212)

* [Server] Instrument command processing, game starts, and event loops

Add a lock-free MetricsRegistry that accumulates per-command processing
times in preallocated histogram slots (one per protobuf command type,
bucketed at 1/5/10/25/50/100/250/500/1000/2500/5000 ms +Inf). The
hot-path observeCommand() uses only relaxed atomic adds — no locks,
no allocations, no cache-line ping-pong beyond the unavoidable counter
updates.

Wire the registry into AbstractServerSocketInterface::processCommandContainer()
so every processed command is attributed with its container's wall-clock
time. When a container exceeds metrics/slow_command_ms (default 500),
a warning is logged including the connected username.

Add an EventLoopWatchdog heartbeat that runs on every socket pool thread.
If a heartbeat overshoots metrics/stall_warn_ms (default 2000 ms), the
overshoot is recorded in atomic counters and a warning is logged. Both
thresholds are configurable in servatrice.ini; setting stall_warn_ms to 0
disables the watchdogs entirely.

Track game-start durations via a separate histogram in MetricsRegistry.
Server_Game::startGameNow() measures the time from zone creation through
player materialization and reports it via Server::observeGameStartDurationMs().

Add a live card-count gauge: Server_Game exposes getCardsInGame() and
Servatrice::getCardsInGamesTotal() sums across all running games under
the appropriate read locks.

Include a standalone metrics_registry_test (Google Test) that validates
empty registries, single/multi-sample histograms, kind encoding,
overflow-slot collapse, negative-duration clamping, gauge rendering,
and the game-start histogram separation.

Took 10 minutes

* [Client/Server/Protocol] Surface live metrics in the Developer tab

Extend Response_GetServerStats with live counters from the in-process
MetricsRegistry: cards in games, event loop stall totals/worst,
total commands processed, average command time, active command types,
and game-start count/duration. Add a repeated CommandStats message
carrying per-command breakdowns (kind, extension number, resolved
protobuf name, count, total ms) for every type that has seen at
least one sample.

Server-side cmdGetServerStats() populates all new fields after the
existing DB uptime snapshot query, resolving protobuf extension names
via the descriptor pool for human-readable labels like
session/Command_Ping.

Expand TabDeveloper with two tables: an overview section (existing
DB stats plus the new live metrics) and a per-command breakdown table
(Command / Count / Total ms / Avg ms) sorted by total_ms descending
so the hottest commands surface first.

Took 55 minutes

Took 47 seconds

* [Server] Drop dead Prometheus histogram, add developer command metrics, fix watchdog init order

- metrics_registry: remove toPrometheusText/appendCumulativeBuckets and the time-bucket histogram that nothing in production ever emitted (the future /metrics exporter can bring it back); keep counts/totals read by the Developer tab
- Fix +Inf bucket routing that never incremented, and its test that locked the bug in
- Instrument developer_command container (kind 6) in processCommandContainer and stats label resolution
- Read metrics/{slow_command_ms,stall_warn_ms} at the top of initServer() so stall_warn_ms=0 disables the watchdogs before pool threads start
- Shrink KindStride to 1280 (largest extension in use is 1206) with a static_assert; document scrape cost of getCardsInGamesTotal; note slow_command logging has no rate limit in servatrice.ini.example

* [Tests] Give metrics_registry_test an explicit main

* [Server] Record only the dispatched command family; drop unused totals

processCommandContainer recorded every family in a container even though
the base if/else-if dispatch processes at most one. An unauthenticated
client could batch a session command (login) with fabricated developer,
moderator, and admin entries and forge genuine-looking samples that were
never executed or authorized. Mirror the base's selection, skip when the
handler was already deleted, and skip entries whose extension number is
-1 (which would otherwise wrap into the previous kind's id range).

[Server] Drop dead process-lifetime byte/uptime counters

txBytesTotal/rxBytesTotal added an atomic RMW to every socket write and
read for counters nothing consumes (cmdGetServerStats fills tx_bytes,
rx_bytes, and uptime_secs from the DB snapshot). Remove the two atomics
and the getTxBytesTotal/getRxBytesTotal/getUptimeSeconds getters; the
incTxBytes/incRxBytes slots and mutexes remain for the ISL legacy
counters.

[Protocol] Document kind 5 as developer in CommandStats

NumKinds is 6 and the server emits kind_index = 5 for developer
commands; the comment stopped at 4.

* [Client] Togglable auto-refresh for Developer stats tab

* [Oracle] Fix clang-format alignment of card type priority list

---------

Co-authored-by: Lukas Brübach <Bruebach.Lukas@bdosecurity.de>
This commit is contained in:
BruebachL 2026-09-11 17:18:56 +02:00 committed by GitHub
parent d5d99e4dfb
commit 202a5ac958
No known key found for this signature in database
GPG key ID: B5690EEEBB952194
22 changed files with 818 additions and 7 deletions

View file

@ -15,6 +15,7 @@ add_test(NAME server_developer_role_test COMMAND server_developer_role_test)
add_test(NAME warning_categories_test COMMAND warning_categories_test)
add_test(NAME lag_monitor_test COMMAND lag_monitor_test)
add_test(NAME latency_tracker_test COMMAND latency_tracker_test)
add_test(NAME metrics_registry_test COMMAND metrics_registry_test)
add_test(NAME deck_hash_performance_test COMMAND deck_hash_performance_test)
set_tests_properties(deck_hash_performance_test PROPERTIES TIMEOUT 15)
@ -36,6 +37,7 @@ add_executable(warning_categories_test warning_categories_test.cpp)
add_executable(lag_monitor_test ${CMAKE_SOURCE_DIR}/cockatrice/src/client/lag_monitor.cpp lag_monitor_test.cpp)
target_include_directories(lag_monitor_test PRIVATE ${CMAKE_SOURCE_DIR}/cockatrice/src)
add_executable(latency_tracker_test latency_tracker_test.cpp)
add_executable(metrics_registry_test ../servatrice/src/metrics_registry.cpp metrics_registry_test.cpp)
find_package(GTest)
@ -76,6 +78,7 @@ if(NOT GTEST_FOUND)
add_dependencies(warning_categories_test gtest)
add_dependencies(lag_monitor_test gtest)
add_dependencies(latency_tracker_test gtest)
add_dependencies(metrics_registry_test gtest)
endif()
include_directories(${GTEST_INCLUDE_DIRS})
@ -118,6 +121,8 @@ target_link_libraries(lag_monitor_test Threads::Threads ${GTEST_BOTH_LIBRARIES}
target_link_libraries(
latency_tracker_test libcockatrice_network Threads::Threads ${GTEST_BOTH_LIBRARIES} ${TEST_QT_MODULES}
)
target_include_directories(metrics_registry_test PRIVATE ${CMAKE_SOURCE_DIR}/servatrice/src)
target_link_libraries(metrics_registry_test ${TEST_QT_MODULES} Threads::Threads ${GTEST_BOTH_LIBRARIES})
add_subdirectory(card_zone_algorithms)
add_subdirectory(carddatabase)

View file

@ -0,0 +1,83 @@
#include <QCoreApplication>
#include <QList>
#include <gtest/gtest.h>
#include <metrics_registry.h>
TEST(MetricsRegistryTest, EmptyRegistryHasZeroedCounters)
{
MetricsRegistry registry;
EXPECT_EQ(0, registry.totalCommands());
EXPECT_EQ(0, registry.totalTimeMs());
EXPECT_EQ(0, registry.activeTypeCount());
EXPECT_EQ(0, registry.getGameStartSnapshot().count);
}
TEST(MetricsRegistryTest, SampleIsRecordedInTotalsAndSlot)
{
MetricsRegistry registry;
registry.observeCommand(MetricsRegistry::typeIdFor(0, 1000), 7);
EXPECT_EQ(1, registry.totalCommands());
EXPECT_EQ(7, registry.totalTimeMs());
EXPECT_EQ(1, registry.activeTypeCount());
const auto stats = registry.collectActiveStats();
ASSERT_EQ(1, stats.size());
EXPECT_EQ(MetricsRegistry::typeIdFor(0, 1000), stats[0].typeId);
EXPECT_EQ(1, stats[0].count);
EXPECT_EQ(7, stats[0].totalMs);
}
TEST(MetricsRegistryTest, KindEncodingSeparatesSameExtensionNumber)
{
MetricsRegistry registry;
const int sessionPing = MetricsRegistry::typeIdFor(0, 1000);
const int roomLeaveRoom = MetricsRegistry::typeIdFor(1, 1000);
ASSERT_NE(sessionPing, roomLeaveRoom);
registry.observeCommand(sessionPing, 1);
registry.observeCommand(roomLeaveRoom, 5000);
EXPECT_EQ(2, registry.activeTypeCount());
}
TEST(MetricsRegistryTest, OutOfRangeIdsLandInOverflowSlot)
{
MetricsRegistry registry;
registry.observeCommand(-1, 4);
registry.observeCommand(MetricsRegistry::MaxTypes + 12345, 4);
EXPECT_EQ(2, registry.totalCommands());
EXPECT_EQ(1, registry.activeTypeCount()); // both collapsed into one slot
EXPECT_EQ(8, registry.totalTimeMs());
}
TEST(MetricsRegistryTest, NegativeDurationsAreClamped)
{
MetricsRegistry registry;
registry.observeCommand(MetricsRegistry::typeIdFor(0, 1000), -50);
EXPECT_EQ(0, registry.totalTimeMs());
}
TEST(MetricsRegistryTest, GameStartTrackedSeparatelyFromCommands)
{
MetricsRegistry registry;
registry.observeGameStartDurationMs(120);
EXPECT_EQ(0, registry.totalCommands());
EXPECT_EQ(0, registry.totalTimeMs());
EXPECT_EQ(0, registry.activeTypeCount());
const auto snapshot = registry.getGameStartSnapshot();
EXPECT_EQ(1, snapshot.count);
EXPECT_EQ(120, snapshot.totalMs);
}
int main(int argc, char **argv)
{
QCoreApplication app(argc, argv);
::testing::InitGoogleTest(&argc, argv);
return RUN_ALL_TESTS();
}