Cockatrice/cockatrice/src/interface/widgets/tabs/tab_developer.cpp
BruebachL 202a5ac958
[Client/Server/Protocol] Surface live metrics in the Developer tab (#7212)
* [Server] Instrument command processing, game starts, and event loops

Add a lock-free MetricsRegistry that accumulates per-command processing
times in preallocated histogram slots (one per protobuf command type,
bucketed at 1/5/10/25/50/100/250/500/1000/2500/5000 ms +Inf). The
hot-path observeCommand() uses only relaxed atomic adds — no locks,
no allocations, no cache-line ping-pong beyond the unavoidable counter
updates.

Wire the registry into AbstractServerSocketInterface::processCommandContainer()
so every processed command is attributed with its container's wall-clock
time. When a container exceeds metrics/slow_command_ms (default 500),
a warning is logged including the connected username.

Add an EventLoopWatchdog heartbeat that runs on every socket pool thread.
If a heartbeat overshoots metrics/stall_warn_ms (default 2000 ms), the
overshoot is recorded in atomic counters and a warning is logged. Both
thresholds are configurable in servatrice.ini; setting stall_warn_ms to 0
disables the watchdogs entirely.

Track game-start durations via a separate histogram in MetricsRegistry.
Server_Game::startGameNow() measures the time from zone creation through
player materialization and reports it via Server::observeGameStartDurationMs().

Add a live card-count gauge: Server_Game exposes getCardsInGame() and
Servatrice::getCardsInGamesTotal() sums across all running games under
the appropriate read locks.

Include a standalone metrics_registry_test (Google Test) that validates
empty registries, single/multi-sample histograms, kind encoding,
overflow-slot collapse, negative-duration clamping, gauge rendering,
and the game-start histogram separation.

Took 10 minutes

* [Client/Server/Protocol] Surface live metrics in the Developer tab

Extend Response_GetServerStats with live counters from the in-process
MetricsRegistry: cards in games, event loop stall totals/worst,
total commands processed, average command time, active command types,
and game-start count/duration. Add a repeated CommandStats message
carrying per-command breakdowns (kind, extension number, resolved
protobuf name, count, total ms) for every type that has seen at
least one sample.

Server-side cmdGetServerStats() populates all new fields after the
existing DB uptime snapshot query, resolving protobuf extension names
via the descriptor pool for human-readable labels like
session/Command_Ping.

Expand TabDeveloper with two tables: an overview section (existing
DB stats plus the new live metrics) and a per-command breakdown table
(Command / Count / Total ms / Avg ms) sorted by total_ms descending
so the hottest commands surface first.

Took 55 minutes

Took 47 seconds

* [Server] Drop dead Prometheus histogram, add developer command metrics, fix watchdog init order

- metrics_registry: remove toPrometheusText/appendCumulativeBuckets and the time-bucket histogram that nothing in production ever emitted (the future /metrics exporter can bring it back); keep counts/totals read by the Developer tab
- Fix +Inf bucket routing that never incremented, and its test that locked the bug in
- Instrument developer_command container (kind 6) in processCommandContainer and stats label resolution
- Read metrics/{slow_command_ms,stall_warn_ms} at the top of initServer() so stall_warn_ms=0 disables the watchdogs before pool threads start
- Shrink KindStride to 1280 (largest extension in use is 1206) with a static_assert; document scrape cost of getCardsInGamesTotal; note slow_command logging has no rate limit in servatrice.ini.example

* [Tests] Give metrics_registry_test an explicit main

* [Server] Record only the dispatched command family; drop unused totals

processCommandContainer recorded every family in a container even though
the base if/else-if dispatch processes at most one. An unauthenticated
client could batch a session command (login) with fabricated developer,
moderator, and admin entries and forge genuine-looking samples that were
never executed or authorized. Mirror the base's selection, skip when the
handler was already deleted, and skip entries whose extension number is
-1 (which would otherwise wrap into the previous kind's id range).

[Server] Drop dead process-lifetime byte/uptime counters

txBytesTotal/rxBytesTotal added an atomic RMW to every socket write and
read for counters nothing consumes (cmdGetServerStats fills tx_bytes,
rx_bytes, and uptime_secs from the DB snapshot). Remove the two atomics
and the getTxBytesTotal/getRxBytesTotal/getUptimeSeconds getters; the
incTxBytes/incRxBytes slots and mutexes remain for the ISL legacy
counters.

[Protocol] Document kind 5 as developer in CommandStats

NumKinds is 6 and the server emits kind_index = 5 for developer
commands; the comment stopped at 4.

* [Client] Togglable auto-refresh for Developer stats tab

* [Oracle] Fix clang-format alignment of card type priority list

---------

Co-authored-by: Lukas Brübach <Bruebach.Lukas@bdosecurity.de>
2026-09-11 17:18:56 +02:00

261 lines
11 KiB
C++

/**
* @file tab_developer.cpp
* @ingroup ServerTabs
*/
//! \todo Document this file.
#include "tab_developer.h"
#include <QCheckBox>
#include <QDateTime>
#include <QHBoxLayout>
#include <QHeaderView>
#include <QLabel>
#include <QPushButton>
#include <QSpinBox>
#include <QTableWidget>
#include <QTimer>
#include <QVBoxLayout>
#include <algorithm>
#include <libcockatrice/network/client/abstract/abstract_client.h>
#include <libcockatrice/protocol/pb/command_get_server_stats.pb.h>
#include <libcockatrice/protocol/pb/response_get_server_stats.pb.h>
#include <libcockatrice/protocol/pending_command.h>
static constexpr int DEFAULT_AUTO_REFRESH_INTERVAL_SECS = 30;
TabDeveloper::TabDeveloper(TabSupervisor *_tabSupervisor, AbstractClient *_client)
: Tab(_tabSupervisor), client(_client)
{
statsTable = new QTableWidget(0, 2);
statsTable->setSizePolicy(QSizePolicy::Expanding, QSizePolicy::Preferred);
statsTable->setEditTriggers(QAbstractItemView::NoEditTriggers);
statsTable->setSelectionBehavior(QAbstractItemView::SelectRows);
statsTable->setSelectionMode(QAbstractItemView::SingleSelection);
statsTable->verticalHeader()->setVisible(false);
statsTable->horizontalHeader()->setSectionResizeMode(0, QHeaderView::Interactive);
statsTable->horizontalHeader()->setSectionResizeMode(1, QHeaderView::Interactive);
statsTable->horizontalHeader()->setStretchLastSection(true);
commandTable = new QTableWidget(0, 4);
commandTable->setSizePolicy(QSizePolicy::Expanding, QSizePolicy::Expanding);
commandTable->setEditTriggers(QAbstractItemView::NoEditTriggers);
commandTable->setSelectionBehavior(QAbstractItemView::SelectRows);
commandTable->setSelectionMode(QAbstractItemView::SingleSelection);
commandTable->verticalHeader()->setVisible(false);
commandTable->horizontalHeader()->setSectionResizeMode(0, QHeaderView::Interactive);
commandTable->horizontalHeader()->setSectionResizeMode(1, QHeaderView::Interactive);
commandTable->horizontalHeader()->setSectionResizeMode(2, QHeaderView::Interactive);
commandTable->horizontalHeader()->setSectionResizeMode(3, QHeaderView::Interactive);
statusLabel = new QLabel;
autoRefreshCheckBox = new QCheckBox;
autoRefreshCheckBox->setChecked(false);
refreshIntervalSpinBox = new QSpinBox;
refreshIntervalSpinBox->setRange(5, 3600);
refreshIntervalSpinBox->setValue(DEFAULT_AUTO_REFRESH_INTERVAL_SECS);
refreshIntervalSpinBox->setEnabled(false);
autoRefreshTimer = new QTimer(this);
connect(autoRefreshTimer, &QTimer::timeout, this, &TabDeveloper::refreshClicked);
connect(autoRefreshCheckBox, &QCheckBox::toggled, this, &TabDeveloper::autoRefreshToggled);
connect(refreshIntervalSpinBox, QOverload<int>::of(&QSpinBox::valueChanged), this,
&TabDeveloper::refreshIntervalChanged);
refreshButton = new QPushButton;
refreshButton->setAutoDefault(true);
connect(refreshButton, &QPushButton::clicked, this, &TabDeveloper::refreshClicked);
auto *buttonLayout = new QHBoxLayout;
buttonLayout->addWidget(statusLabel, 1, Qt::AlignLeft);
buttonLayout->addWidget(autoRefreshCheckBox, 0, Qt::AlignRight);
buttonLayout->addWidget(refreshIntervalSpinBox, 0, Qt::AlignRight);
buttonLayout->addWidget(refreshButton, 0, Qt::AlignRight);
auto *tableLayout = new QHBoxLayout;
tableLayout->addWidget(statsTable, 1);
tableLayout->addWidget(commandTable, 2);
auto *mainLayout = new QVBoxLayout;
mainLayout->addLayout(tableLayout, 1);
mainLayout->addLayout(buttonLayout);
auto *central = new QWidget;
central->setLayout(mainLayout);
setCentralWidget(central);
retranslateUi();
}
void TabDeveloper::retranslateUi()
{
autoRefreshCheckBox->setText(tr("Auto-refresh"));
autoRefreshCheckBox->setToolTip(tr("Automatically request fresh server statistics at a fixed interval."));
refreshIntervalSpinBox->setSuffix(tr(" s"));
refreshIntervalSpinBox->setToolTip(tr("Seconds between automatic refreshes."));
refreshButton->setText(tr("Refresh server stats"));
statsTable->setHorizontalHeaderLabels(QString(tr("Statistic;Value")).split(";"));
commandTable->setHorizontalHeaderLabels(QString(tr("Command;Count;Total ms;Avg ms")).split(";"));
if (statsTable->rowCount() == 0) {
statusLabel->clear();
}
}
QString TabDeveloper::formatBytes(quint64 bytes)
{
const quint64 kib = 1024;
const quint64 mib = 1024 * kib;
const quint64 gib = 1024 * mib;
if (bytes >= gib) {
return tr("%1 GiB").arg(QString::number(bytes / static_cast<double>(gib), 'f', 2));
}
if (bytes >= mib) {
return tr("%1 MiB").arg(QString::number(bytes / static_cast<double>(mib), 'f', 2));
}
if (bytes >= kib) {
return tr("%1 KiB").arg(QString::number(bytes / static_cast<double>(kib), 'f', 2));
}
return tr("%1 bytes").arg(bytes);
}
QString TabDeveloper::formatDurationMs(qint64 ms)
{
if (ms >= 1000) {
return tr("%1 s").arg(QString::number(ms / 1000.0, 'f', 2));
}
return tr("%1 ms").arg(ms);
}
void TabDeveloper::appendStatRow(const QString &name, const QString &value)
{
const int row = statsTable->rowCount();
statsTable->insertRow(row);
statsTable->setItem(row, 0, new QTableWidgetItem(name));
statsTable->setItem(row, 1, new QTableWidgetItem(value));
}
void TabDeveloper::appendSeparatorRow(const QString &sectionTitle)
{
const int row = statsTable->rowCount();
statsTable->insertRow(row);
auto *labelItem = new QTableWidgetItem(sectionTitle);
auto font = labelItem->font();
font.setBold(true);
labelItem->setFont(font);
labelItem->setFlags(labelItem->flags() & ~Qt::ItemIsSelectable);
statsTable->setItem(row, 0, labelItem);
statsTable->setItem(row, 1, new QTableWidgetItem(QString()));
}
void TabDeveloper::refreshClicked()
{
if (requestPending) {
return;
}
requestPending = true;
Command_GetServerStats cmd;
PendingCommand *pend = client->prepareDeveloperCommand(cmd);
connect(pend, &PendingCommand::finished, this, &TabDeveloper::serverStatsResponse);
client->sendCommand(pend);
}
void TabDeveloper::autoRefreshToggled(bool checked)
{
refreshIntervalSpinBox->setEnabled(checked);
if (checked) {
refreshIntervalChanged();
refreshClicked();
} else {
autoRefreshTimer->stop();
}
}
void TabDeveloper::refreshIntervalChanged()
{
if (autoRefreshCheckBox->isChecked()) {
autoRefreshTimer->start(refreshIntervalSpinBox->value() * 1000);
}
}
void TabDeveloper::serverStatsResponse(const Response &resp)
{
requestPending = false;
if (resp.response_code() != Response::RespOk) {
statusLabel->setText(tr("No server statistics available yet."));
return;
}
const Response_GetServerStats &response = resp.GetExtension(Response_GetServerStats::ext);
statsTable->setRowCount(0);
// Overview section
appendStatRow(tr("Registered users online"), QString::number(response.users_count()));
appendStatRow(tr("Moderators online"), QString::number(response.mods_count()));
appendStatRow(tr("Games running"), QString::number(response.games_count()));
appendStatRow(tr("Traffic sent (last tick)"), formatBytes(response.tx_bytes()));
appendStatRow(tr("Traffic received (last tick)"), formatBytes(response.rx_bytes()));
const qint64 uptime = static_cast<qint64>(response.uptime_secs());
const int days = static_cast<int>(uptime / 86400);
const int hours = static_cast<int>((uptime % 86400) / 3600);
const int minutes = static_cast<int>((uptime % 3600) / 60);
appendStatRow(tr("Server uptime"), days > 0 ? tr("%1d %2h %3m").arg(days).arg(hours).arg(minutes)
: tr("%1h %2m").arg(hours).arg(minutes));
const QDateTime snapshotTime = QDateTime::fromSecsSinceEpoch(static_cast<qint64>(response.timest()));
appendStatRow(tr("Snapshot taken"), snapshotTime.toLocalTime().toString("yyyy-MM-dd HH:mm"));
// Live metrics section
appendSeparatorRow(tr("Live Metrics"));
appendStatRow(tr("Cards in live games"), QString::number(response.cards_in_games()));
appendStatRow(tr("Total commands processed"), QString::number(response.total_commands()));
if (response.total_commands() > 0) {
const double avgMs = static_cast<double>(response.total_command_time_ms()) / response.total_commands();
appendStatRow(tr("Avg command time"), QString::number(avgMs, 'f', 2) + " ms");
}
appendStatRow(tr("Active command types"), QString::number(response.active_command_types()));
appendStatRow(tr("Event loop stalls"), QString::number(response.eventloop_stalls_total()));
appendStatRow(tr("Last stall overshoot"), formatDurationMs(response.eventloop_last_stall_ms()));
appendStatRow(tr("Worst stall overshoot"), formatDurationMs(response.eventloop_max_stall_ms()));
if (response.game_start_count() > 0) {
appendStatRow(tr("Game starts"), QString::number(response.game_start_count()));
const double avgStartMs = static_cast<double>(response.game_start_total_ms()) / response.game_start_count();
appendStatRow(tr("Avg game start time"), QString::number(avgStartMs, 'f', 1) + " ms");
}
// Per-command breakdown table
QList<CommandStats> sortedStats(response.command_stats().begin(), response.command_stats().end());
std::sort(sortedStats.begin(), sortedStats.end(),
[](const auto &a, const auto &b) { return a.total_ms() > b.total_ms(); });
commandTable->setRowCount(0);
for (const auto &cs : sortedStats) {
const int row = commandTable->rowCount();
commandTable->insertRow(row);
commandTable->setItem(row, 0, new QTableWidgetItem(QString::fromStdString(cs.command_name())));
auto *countItem = new QTableWidgetItem(QString::number(cs.count()));
countItem->setTextAlignment(Qt::AlignRight | Qt::AlignVCenter);
commandTable->setItem(row, 1, countItem);
auto *totalItem = new QTableWidgetItem(QString::number(cs.total_ms()));
totalItem->setTextAlignment(Qt::AlignRight | Qt::AlignVCenter);
commandTable->setItem(row, 2, totalItem);
const double avg = cs.count() > 0 ? static_cast<double>(cs.total_ms()) / cs.count() : 0.0;
auto *avgItem = new QTableWidgetItem(QString::number(avg, 'f', 2));
avgItem->setTextAlignment(Qt::AlignRight | Qt::AlignVCenter);
commandTable->setItem(row, 3, avgItem);
}
commandTable->resizeColumnsToContents();
statsTable->resizeColumnsToContents();
commandTable->resizeColumnsToContents();
statusLabel->setText(tr("Updated %1").arg(QDateTime::currentDateTime().toString("yyyy-MM-dd HH:mm")));
}