jeirmeister/Cockatrice
1
0
Fork 0
mirror of https://github.com/Cockatrice/Cockatrice.git synced 2026-09-21 09:05:10 -07:00
Code Issues Projects Releases Packages Wiki Activity Actions 3

[Client/Server/Protocol] Surface live metrics in the Developer tab (#7212)

Browse source 
* [Server] Instrument command processing, game starts, and event loops

Add a lock-free MetricsRegistry that accumulates per-command processing
times in preallocated histogram slots (one per protobuf command type,
bucketed at 1/5/10/25/50/100/250/500/1000/2500/5000 ms +Inf). The
hot-path observeCommand() uses only relaxed atomic adds — no locks,
no allocations, no cache-line ping-pong beyond the unavoidable counter
updates.

Wire the registry into AbstractServerSocketInterface::processCommandContainer()
so every processed command is attributed with its container's wall-clock
time. When a container exceeds metrics/slow_command_ms (default 500),
a warning is logged including the connected username.

Add an EventLoopWatchdog heartbeat that runs on every socket pool thread.
If a heartbeat overshoots metrics/stall_warn_ms (default 2000 ms), the
overshoot is recorded in atomic counters and a warning is logged. Both
thresholds are configurable in servatrice.ini; setting stall_warn_ms to 0
disables the watchdogs entirely.

Track game-start durations via a separate histogram in MetricsRegistry.
Server_Game::startGameNow() measures the time from zone creation through
player materialization and reports it via Server::observeGameStartDurationMs().

Add a live card-count gauge: Server_Game exposes getCardsInGame() and
Servatrice::getCardsInGamesTotal() sums across all running games under
the appropriate read locks.

Include a standalone metrics_registry_test (Google Test) that validates
empty registries, single/multi-sample histograms, kind encoding,
overflow-slot collapse, negative-duration clamping, gauge rendering,
and the game-start histogram separation.

Took 10 minutes

* [Client/Server/Protocol] Surface live metrics in the Developer tab

Extend Response_GetServerStats with live counters from the in-process
MetricsRegistry: cards in games, event loop stall totals/worst,
total commands processed, average command time, active command types,
and game-start count/duration. Add a repeated CommandStats message
carrying per-command breakdowns (kind, extension number, resolved
protobuf name, count, total ms) for every type that has seen at
least one sample.

Server-side cmdGetServerStats() populates all new fields after the
existing DB uptime snapshot query, resolving protobuf extension names
via the descriptor pool for human-readable labels like
session/Command_Ping.

Expand TabDeveloper with two tables: an overview section (existing
DB stats plus the new live metrics) and a per-command breakdown table
(Command / Count / Total ms / Avg ms) sorted by total_ms descending
so the hottest commands surface first.

Took 55 minutes

Took 47 seconds

* [Server] Drop dead Prometheus histogram, add developer command metrics, fix watchdog init order

- metrics_registry: remove toPrometheusText/appendCumulativeBuckets and the time-bucket histogram that nothing in production ever emitted (the future /metrics exporter can bring it back); keep counts/totals read by the Developer tab
- Fix +Inf bucket routing that never incremented, and its test that locked the bug in
- Instrument developer_command container (kind 6) in processCommandContainer and stats label resolution
- Read metrics/{slow_command_ms,stall_warn_ms} at the top of initServer() so stall_warn_ms=0 disables the watchdogs before pool threads start
- Shrink KindStride to 1280 (largest extension in use is 1206) with a static_assert; document scrape cost of getCardsInGamesTotal; note slow_command logging has no rate limit in servatrice.ini.example

* [Tests] Give metrics_registry_test an explicit main

* [Server] Record only the dispatched command family; drop unused totals

processCommandContainer recorded every family in a container even though
the base if/else-if dispatch processes at most one. An unauthenticated
client could batch a session command (login) with fabricated developer,
moderator, and admin entries and forge genuine-looking samples that were
never executed or authorized. Mirror the base's selection, skip when the
handler was already deleted, and skip entries whose extension number is
-1 (which would otherwise wrap into the previous kind's id range).

[Server] Drop dead process-lifetime byte/uptime counters

txBytesTotal/rxBytesTotal added an atomic RMW to every socket write and
read for counters nothing consumes (cmdGetServerStats fills tx_bytes,
rx_bytes, and uptime_secs from the DB snapshot). Remove the two atomics
and the getTxBytesTotal/getRxBytesTotal/getUptimeSeconds getters; the
incTxBytes/incRxBytes slots and mutexes remain for the ISL legacy
counters.

[Protocol] Document kind 5 as developer in CommandStats

NumKinds is 6 and the server emits kind_index = 5 for developer
commands; the comment stopped at 4.

* [Client] Togglable auto-refresh for Developer stats tab

* [Oracle] Fix clang-format alignment of card type priority list

---------

Co-authored-by: Lukas Brübach <Bruebach.Lukas@bdosecurity.de>
This commit is contained in:
BruebachL 2026-09-11 17:18:56 +02:00 • committed by GitHub
parent d5d99e4dfb
commit 202a5ac958
No known key found for this signature in database
GPG key ID: B5690EEEBB952194
22 changed files with 818 additions and 7 deletions
Download patch file Download diff file Expand all files Collapse all files

119
servatrice/src/serversocketinterface.cpp
View file

@ -30,6 +30,7 @@
#include <QDateTime>
#include <QDebug>
#include <QElapsedTimer>
#include <QHostAddress>
#include <QJsonDocument>
#include <QJsonObject>
@ -39,8 +40,10 @@
#include <QSqlQuery>
#include <QString>
#include <game/server_player.h>
#include <google/protobuf/descriptor.h>
#include <iostream>
#include <libcockatrice/deck_list/deck_list.h>
#include <libcockatrice/protocol/get_pb_extension.h>
#include <libcockatrice/protocol/pb/command_deck_del.pb.h>
#include <libcockatrice/protocol/pb/command_deck_del_dir.pb.h>
#include <libcockatrice/protocol/pb/command_deck_download.pb.h>
@ -192,6 +195,77 @@ void AbstractServerSocketInterface::logDebugMessage(const QString &message)
logger->logMessage(message, this);
}
void AbstractServerSocketInterface::processCommandContainer(const CommandContainer &cont)
{
QElapsedTimer timer;
timer.start();
Server_ProtocolHandler::processCommandContainer(cont);
const qint64 elapsedMs = timer.nsecsElapsed() / 1000000;
// The base dispatch is an if/else-if chain — at most one family is
// actually processed. Recording every family in the container would
// let an unauthenticated client stampforge developer/moderator/admin
// samples by batching them alongside a session command the server
// actually runs. Mirror the base's selection and skip entirely when
// deleted or when no family matched.
if (deleted) {
return;
}
// When getPbExtension returns -1 (no extension set) and the kind is
// non-zero, typeIdFor wraps into the previous kind's range instead of
// hitting the typeId < 0 guard in observeCommand. Skip such entries.
int kind = -1;
if (cont.game_command_size()) {
kind = 2;
} else if (cont.room_command_size()) {
kind = 1;
} else if (cont.session_command_size()) {
kind = 0;
} else if (cont.moderator_command_size()) {
kind = 3;
} else if (cont.admin_command_size()) {
kind = 4;
} else if (cont.developer_command_size()) {
kind = 5;
}
if (kind >= 0) {
auto recordDispatched = [&](int familyKind, const auto &cmds) {
for (const auto &cmd : cmds) {
const int ext = getPbExtension(cmd);
if (ext >= 0) {
servatrice->getMetricsRegistry().observeCommand(MetricsRegistry::typeIdFor(familyKind, ext),
elapsedMs);
}
}
};
if (kind == 0) {
recordDispatched(kind, cont.session_command());
} else if (kind == 1) {
recordDispatched(kind, cont.room_command());
} else if (kind == 2) {
recordDispatched(kind, cont.game_command());
} else if (kind == 3) {
recordDispatched(kind, cont.moderator_command());
} else if (kind == 4) {
recordDispatched(kind, cont.admin_command());
} else {
recordDispatched(kind, cont.developer_command());
}
}
const int slowCommandMs = servatrice->getMetricsSlowCommandMs();
if (slowCommandMs > 0 && elapsedMs >= slowCommandMs) {
const ServerInfo_User *info = getUserInfo();
const QString user = authState == PasswordRight && info ? QString::fromStdString(info->name())
: QStringLiteral("unauthenticated");
qCWarning(AbstractServerSocketInterfaceLog) << "slow command container from" << user << "processed in"
<< elapsedMs << "ms (" << cont.ByteSizeLong() << "bytes)";
}
}
Response::ResponseCode AbstractServerSocketInterface::processExtendedSessionCommand(int cmdType,
const SessionCommand &cmd,
ResponseContainer &rc)
@ -1704,6 +1778,51 @@ Response::ResponseCode AbstractServerSocketInterface::cmdGetServerStats(const Co
re->set_uptime_secs(snapshot.uptimeSecs);
re->set_timest(snapshot.timest);
// Live metrics from the in-process MetricsRegistry (resets on server restart)
re->set_cards_in_games(static_cast<google::protobuf::uint64>(servatrice->getCardsInGamesTotal()));
re->set_eventloop_stalls_total(static_cast<google::protobuf::uint64>(servatrice->getEventLoopStallsTotal()));
re->set_eventloop_last_stall_ms(static_cast<google::protobuf::uint64>(servatrice->getEventLoopLastStallMs()));
re->set_eventloop_max_stall_ms(static_cast<google::protobuf::uint64>(servatrice->getEventLoopMaxStallMs()));
re->set_total_commands(static_cast<google::protobuf::uint64>(servatrice->getMetricsRegistry().totalCommands()));
re->set_total_command_time_ms(
static_cast<google::protobuf::uint64>(servatrice->getMetricsRegistry().totalTimeMs()));
re->set_active_command_types(servatrice->getMetricsRegistry().activeTypeCount());
const auto gameStart = servatrice->getMetricsRegistry().getGameStartSnapshot();
re->set_game_start_count(static_cast<google::protobuf::uint64>(gameStart.count));
re->set_game_start_total_ms(static_cast<google::protobuf::uint64>(gameStart.totalMs));
// Per-command breakdown: resolve protobuf extension names via the descriptor pool
static const char *messageNames[] = {"SessionCommand", "RoomCommand", "GameCommand",
"ModeratorCommand", "AdminCommand", "DeveloperCommand"};
const auto activeStats = servatrice->getMetricsRegistry().collectActiveStats();
for (const auto &stat : activeStats) {
const int kind = stat.typeId / MetricsRegistry::KindStride;
const int number = stat.typeId % MetricsRegistry::KindStride;
QString label;
if (kind >= 0 && kind < MetricsRegistry::NumKinds) {
const google::protobuf::DescriptorPool *pool = google::protobuf::DescriptorPool::generated_pool();
const google::protobuf::Descriptor *message = pool->FindMessageTypeByName(messageNames[kind]);
const google::protobuf::FieldDescriptor *extension =
message ? pool->FindExtensionByNumber(message, number) : nullptr;
if (extension) {
label = QString::fromLatin1(MetricsRegistry::KindNames[kind]) + QStringLiteral("/") +
QString::fromStdString(std::string(extension->message_type()->name()));
}
}
if (label.isEmpty()) {
label = QString::number(stat.typeId);
}
CommandStats *cs = re->add_command_stats();
cs->set_kind_index(static_cast<google::protobuf::uint32>(kind));
cs->set_extension_number(static_cast<google::protobuf::uint32>(number));
cs->set_command_name(label.toStdString());
cs->set_count(static_cast<google::protobuf::uint64>(stat.count));
cs->set_total_ms(static_cast<google::protobuf::uint64>(stat.totalMs));
}
rc.setResponseExtension(re);
return Response::RespOk;
}