mirror of
https://github.com/Cockatrice/Cockatrice.git
synced 2026-09-22 09:35:08 -07:00
[Client/Server/Protocol] Surface live metrics in the Developer tab (#7212)
* [Server] Instrument command processing, game starts, and event loops
Add a lock-free MetricsRegistry that accumulates per-command processing
times in preallocated histogram slots (one per protobuf command type,
bucketed at 1/5/10/25/50/100/250/500/1000/2500/5000 ms +Inf). The
hot-path observeCommand() uses only relaxed atomic adds — no locks,
no allocations, no cache-line ping-pong beyond the unavoidable counter
updates.
Wire the registry into AbstractServerSocketInterface::processCommandContainer()
so every processed command is attributed with its container's wall-clock
time. When a container exceeds metrics/slow_command_ms (default 500),
a warning is logged including the connected username.
Add an EventLoopWatchdog heartbeat that runs on every socket pool thread.
If a heartbeat overshoots metrics/stall_warn_ms (default 2000 ms), the
overshoot is recorded in atomic counters and a warning is logged. Both
thresholds are configurable in servatrice.ini; setting stall_warn_ms to 0
disables the watchdogs entirely.
Track game-start durations via a separate histogram in MetricsRegistry.
Server_Game::startGameNow() measures the time from zone creation through
player materialization and reports it via Server::observeGameStartDurationMs().
Add a live card-count gauge: Server_Game exposes getCardsInGame() and
Servatrice::getCardsInGamesTotal() sums across all running games under
the appropriate read locks.
Include a standalone metrics_registry_test (Google Test) that validates
empty registries, single/multi-sample histograms, kind encoding,
overflow-slot collapse, negative-duration clamping, gauge rendering,
and the game-start histogram separation.
Took 10 minutes
* [Client/Server/Protocol] Surface live metrics in the Developer tab
Extend Response_GetServerStats with live counters from the in-process
MetricsRegistry: cards in games, event loop stall totals/worst,
total commands processed, average command time, active command types,
and game-start count/duration. Add a repeated CommandStats message
carrying per-command breakdowns (kind, extension number, resolved
protobuf name, count, total ms) for every type that has seen at
least one sample.
Server-side cmdGetServerStats() populates all new fields after the
existing DB uptime snapshot query, resolving protobuf extension names
via the descriptor pool for human-readable labels like
session/Command_Ping.
Expand TabDeveloper with two tables: an overview section (existing
DB stats plus the new live metrics) and a per-command breakdown table
(Command / Count / Total ms / Avg ms) sorted by total_ms descending
so the hottest commands surface first.
Took 55 minutes
Took 47 seconds
* [Server] Drop dead Prometheus histogram, add developer command metrics, fix watchdog init order
- metrics_registry: remove toPrometheusText/appendCumulativeBuckets and the time-bucket histogram that nothing in production ever emitted (the future /metrics exporter can bring it back); keep counts/totals read by the Developer tab
- Fix +Inf bucket routing that never incremented, and its test that locked the bug in
- Instrument developer_command container (kind 6) in processCommandContainer and stats label resolution
- Read metrics/{slow_command_ms,stall_warn_ms} at the top of initServer() so stall_warn_ms=0 disables the watchdogs before pool threads start
- Shrink KindStride to 1280 (largest extension in use is 1206) with a static_assert; document scrape cost of getCardsInGamesTotal; note slow_command logging has no rate limit in servatrice.ini.example
* [Tests] Give metrics_registry_test an explicit main
* [Server] Record only the dispatched command family; drop unused totals
processCommandContainer recorded every family in a container even though
the base if/else-if dispatch processes at most one. An unauthenticated
client could batch a session command (login) with fabricated developer,
moderator, and admin entries and forge genuine-looking samples that were
never executed or authorized. Mirror the base's selection, skip when the
handler was already deleted, and skip entries whose extension number is
-1 (which would otherwise wrap into the previous kind's id range).
[Server] Drop dead process-lifetime byte/uptime counters
txBytesTotal/rxBytesTotal added an atomic RMW to every socket write and
read for counters nothing consumes (cmdGetServerStats fills tx_bytes,
rx_bytes, and uptime_secs from the DB snapshot). Remove the two atomics
and the getTxBytesTotal/getRxBytesTotal/getUptimeSeconds getters; the
incTxBytes/incRxBytes slots and mutexes remain for the ISL legacy
counters.
[Protocol] Document kind 5 as developer in CommandStats
NumKinds is 6 and the server emits kind_index = 5 for developer
commands; the comment stopped at 4.
* [Client] Togglable auto-refresh for Developer stats tab
* [Oracle] Fix clang-format alignment of card type priority list
---------
Co-authored-by: Lukas Brübach <Bruebach.Lukas@bdosecurity.de>
This commit is contained in:
BruebachL
GitHub
parent
d5d99e4dfb
commit
202a5ac958
22 changed files with 818 additions and 7 deletions
|
|
@ -6,40 +6,80 @@
|
|||
|
||||
#include "tab_developer.h"
|
||||
|
||||
#include <QCheckBox>
|
||||
#include <QDateTime>
|
||||
#include <QHBoxLayout>
|
||||
#include <QHeaderView>
|
||||
#include <QLabel>
|
||||
#include <QPushButton>
|
||||
#include <QSpinBox>
|
||||
#include <QTableWidget>
|
||||
#include <QTimer>
|
||||
#include <QVBoxLayout>
|
||||
#include <algorithm>
|
||||
#include <libcockatrice/network/client/abstract/abstract_client.h>
|
||||
#include <libcockatrice/protocol/pb/command_get_server_stats.pb.h>
|
||||
#include <libcockatrice/protocol/pb/response_get_server_stats.pb.h>
|
||||
#include <libcockatrice/protocol/pending_command.h>
|
||||
|
||||
static constexpr int DEFAULT_AUTO_REFRESH_INTERVAL_SECS = 30;
|
||||
|
||||
TabDeveloper::TabDeveloper(TabSupervisor *_tabSupervisor, AbstractClient *_client)
|
||||
: Tab(_tabSupervisor), client(_client)
|
||||
{
|
||||
statsTable = new QTableWidget(0, 2);
|
||||
statsTable->setSizePolicy(QSizePolicy::Expanding, QSizePolicy::Expanding);
|
||||
statsTable->setSizePolicy(QSizePolicy::Expanding, QSizePolicy::Preferred);
|
||||
statsTable->setEditTriggers(QAbstractItemView::NoEditTriggers);
|
||||
statsTable->setSelectionBehavior(QAbstractItemView::SelectRows);
|
||||
statsTable->setSelectionMode(QAbstractItemView::SingleSelection);
|
||||
statsTable->horizontalHeader()->setStretchLastSection(true);
|
||||
statsTable->verticalHeader()->setVisible(false);
|
||||
statsTable->horizontalHeader()->setSectionResizeMode(0, QHeaderView::Interactive);
|
||||
statsTable->horizontalHeader()->setSectionResizeMode(1, QHeaderView::Interactive);
|
||||
statsTable->horizontalHeader()->setStretchLastSection(true);
|
||||
|
||||
commandTable = new QTableWidget(0, 4);
|
||||
commandTable->setSizePolicy(QSizePolicy::Expanding, QSizePolicy::Expanding);
|
||||
commandTable->setEditTriggers(QAbstractItemView::NoEditTriggers);
|
||||
commandTable->setSelectionBehavior(QAbstractItemView::SelectRows);
|
||||
commandTable->setSelectionMode(QAbstractItemView::SingleSelection);
|
||||
commandTable->verticalHeader()->setVisible(false);
|
||||
commandTable->horizontalHeader()->setSectionResizeMode(0, QHeaderView::Interactive);
|
||||
commandTable->horizontalHeader()->setSectionResizeMode(1, QHeaderView::Interactive);
|
||||
commandTable->horizontalHeader()->setSectionResizeMode(2, QHeaderView::Interactive);
|
||||
commandTable->horizontalHeader()->setSectionResizeMode(3, QHeaderView::Interactive);
|
||||
|
||||
statusLabel = new QLabel;
|
||||
|
||||
autoRefreshCheckBox = new QCheckBox;
|
||||
autoRefreshCheckBox->setChecked(false);
|
||||
|
||||
refreshIntervalSpinBox = new QSpinBox;
|
||||
refreshIntervalSpinBox->setRange(5, 3600);
|
||||
refreshIntervalSpinBox->setValue(DEFAULT_AUTO_REFRESH_INTERVAL_SECS);
|
||||
refreshIntervalSpinBox->setEnabled(false);
|
||||
|
||||
autoRefreshTimer = new QTimer(this);
|
||||
connect(autoRefreshTimer, &QTimer::timeout, this, &TabDeveloper::refreshClicked);
|
||||
connect(autoRefreshCheckBox, &QCheckBox::toggled, this, &TabDeveloper::autoRefreshToggled);
|
||||
connect(refreshIntervalSpinBox, QOverload<int>::of(&QSpinBox::valueChanged), this,
|
||||
&TabDeveloper::refreshIntervalChanged);
|
||||
|
||||
refreshButton = new QPushButton;
|
||||
refreshButton->setAutoDefault(true);
|
||||
connect(refreshButton, &QPushButton::clicked, this, &TabDeveloper::refreshClicked);
|
||||
|
||||
auto *buttonLayout = new QHBoxLayout;
|
||||
buttonLayout->addWidget(statusLabel, 1, Qt::AlignLeft);
|
||||
buttonLayout->addWidget(autoRefreshCheckBox, 0, Qt::AlignRight);
|
||||
buttonLayout->addWidget(refreshIntervalSpinBox, 0, Qt::AlignRight);
|
||||
buttonLayout->addWidget(refreshButton, 0, Qt::AlignRight);
|
||||
|
||||
auto *tableLayout = new QHBoxLayout;
|
||||
tableLayout->addWidget(statsTable, 1);
|
||||
tableLayout->addWidget(commandTable, 2);
|
||||
|
||||
auto *mainLayout = new QVBoxLayout;
|
||||
mainLayout->addWidget(statsTable);
|
||||
mainLayout->addLayout(tableLayout, 1);
|
||||
mainLayout->addLayout(buttonLayout);
|
||||
|
||||
auto *central = new QWidget;
|
||||
|
|
@ -51,8 +91,13 @@ TabDeveloper::TabDeveloper(TabSupervisor *_tabSupervisor, AbstractClient *_clien
|
|||
|
||||
void TabDeveloper::retranslateUi()
|
||||
{
|
||||
autoRefreshCheckBox->setText(tr("Auto-refresh"));
|
||||
autoRefreshCheckBox->setToolTip(tr("Automatically request fresh server statistics at a fixed interval."));
|
||||
refreshIntervalSpinBox->setSuffix(tr(" s"));
|
||||
refreshIntervalSpinBox->setToolTip(tr("Seconds between automatic refreshes."));
|
||||
refreshButton->setText(tr("Refresh server stats"));
|
||||
statsTable->setHorizontalHeaderLabels(QString(tr("Statistic;Value")).split(";"));
|
||||
commandTable->setHorizontalHeaderLabels(QString(tr("Command;Count;Total ms;Avg ms")).split(";"));
|
||||
if (statsTable->rowCount() == 0) {
|
||||
statusLabel->clear();
|
||||
}
|
||||
|
|
@ -75,6 +120,14 @@ QString TabDeveloper::formatBytes(quint64 bytes)
|
|||
return tr("%1 bytes").arg(bytes);
|
||||
}
|
||||
|
||||
QString TabDeveloper::formatDurationMs(qint64 ms)
|
||||
{
|
||||
if (ms >= 1000) {
|
||||
return tr("%1 s").arg(QString::number(ms / 1000.0, 'f', 2));
|
||||
}
|
||||
return tr("%1 ms").arg(ms);
|
||||
}
|
||||
|
||||
void TabDeveloper::appendStatRow(const QString &name, const QString &value)
|
||||
{
|
||||
const int row = statsTable->rowCount();
|
||||
|
|
@ -83,16 +136,52 @@ void TabDeveloper::appendStatRow(const QString &name, const QString &value)
|
|||
statsTable->setItem(row, 1, new QTableWidgetItem(value));
|
||||
}
|
||||
|
||||
void TabDeveloper::appendSeparatorRow(const QString §ionTitle)
|
||||
{
|
||||
const int row = statsTable->rowCount();
|
||||
statsTable->insertRow(row);
|
||||
auto *labelItem = new QTableWidgetItem(sectionTitle);
|
||||
auto font = labelItem->font();
|
||||
font.setBold(true);
|
||||
labelItem->setFont(font);
|
||||
labelItem->setFlags(labelItem->flags() & ~Qt::ItemIsSelectable);
|
||||
statsTable->setItem(row, 0, labelItem);
|
||||
statsTable->setItem(row, 1, new QTableWidgetItem(QString()));
|
||||
}
|
||||
|
||||
void TabDeveloper::refreshClicked()
|
||||
{
|
||||
if (requestPending) {
|
||||
return;
|
||||
}
|
||||
requestPending = true;
|
||||
Command_GetServerStats cmd;
|
||||
PendingCommand *pend = client->prepareDeveloperCommand(cmd);
|
||||
connect(pend, &PendingCommand::finished, this, &TabDeveloper::serverStatsResponse);
|
||||
client->sendCommand(pend);
|
||||
}
|
||||
|
||||
void TabDeveloper::autoRefreshToggled(bool checked)
|
||||
{
|
||||
refreshIntervalSpinBox->setEnabled(checked);
|
||||
if (checked) {
|
||||
refreshIntervalChanged();
|
||||
refreshClicked();
|
||||
} else {
|
||||
autoRefreshTimer->stop();
|
||||
}
|
||||
}
|
||||
|
||||
void TabDeveloper::refreshIntervalChanged()
|
||||
{
|
||||
if (autoRefreshCheckBox->isChecked()) {
|
||||
autoRefreshTimer->start(refreshIntervalSpinBox->value() * 1000);
|
||||
}
|
||||
}
|
||||
|
||||
void TabDeveloper::serverStatsResponse(const Response &resp)
|
||||
{
|
||||
requestPending = false;
|
||||
if (resp.response_code() != Response::RespOk) {
|
||||
statusLabel->setText(tr("No server statistics available yet."));
|
||||
return;
|
||||
|
|
@ -101,6 +190,8 @@ void TabDeveloper::serverStatsResponse(const Response &resp)
|
|||
const Response_GetServerStats &response = resp.GetExtension(Response_GetServerStats::ext);
|
||||
|
||||
statsTable->setRowCount(0);
|
||||
|
||||
// Overview section
|
||||
appendStatRow(tr("Registered users online"), QString::number(response.users_count()));
|
||||
appendStatRow(tr("Moderators online"), QString::number(response.mods_count()));
|
||||
appendStatRow(tr("Games running"), QString::number(response.games_count()));
|
||||
|
|
@ -117,6 +208,54 @@ void TabDeveloper::serverStatsResponse(const Response &resp)
|
|||
const QDateTime snapshotTime = QDateTime::fromSecsSinceEpoch(static_cast<qint64>(response.timest()));
|
||||
appendStatRow(tr("Snapshot taken"), snapshotTime.toLocalTime().toString("yyyy-MM-dd HH:mm"));
|
||||
|
||||
// Live metrics section
|
||||
appendSeparatorRow(tr("Live Metrics"));
|
||||
appendStatRow(tr("Cards in live games"), QString::number(response.cards_in_games()));
|
||||
appendStatRow(tr("Total commands processed"), QString::number(response.total_commands()));
|
||||
|
||||
if (response.total_commands() > 0) {
|
||||
const double avgMs = static_cast<double>(response.total_command_time_ms()) / response.total_commands();
|
||||
appendStatRow(tr("Avg command time"), QString::number(avgMs, 'f', 2) + " ms");
|
||||
}
|
||||
appendStatRow(tr("Active command types"), QString::number(response.active_command_types()));
|
||||
|
||||
appendStatRow(tr("Event loop stalls"), QString::number(response.eventloop_stalls_total()));
|
||||
appendStatRow(tr("Last stall overshoot"), formatDurationMs(response.eventloop_last_stall_ms()));
|
||||
appendStatRow(tr("Worst stall overshoot"), formatDurationMs(response.eventloop_max_stall_ms()));
|
||||
|
||||
if (response.game_start_count() > 0) {
|
||||
appendStatRow(tr("Game starts"), QString::number(response.game_start_count()));
|
||||
const double avgStartMs = static_cast<double>(response.game_start_total_ms()) / response.game_start_count();
|
||||
appendStatRow(tr("Avg game start time"), QString::number(avgStartMs, 'f', 1) + " ms");
|
||||
}
|
||||
|
||||
// Per-command breakdown table
|
||||
QList<CommandStats> sortedStats(response.command_stats().begin(), response.command_stats().end());
|
||||
std::sort(sortedStats.begin(), sortedStats.end(),
|
||||
[](const auto &a, const auto &b) { return a.total_ms() > b.total_ms(); });
|
||||
|
||||
commandTable->setRowCount(0);
|
||||
for (const auto &cs : sortedStats) {
|
||||
const int row = commandTable->rowCount();
|
||||
commandTable->insertRow(row);
|
||||
commandTable->setItem(row, 0, new QTableWidgetItem(QString::fromStdString(cs.command_name())));
|
||||
|
||||
auto *countItem = new QTableWidgetItem(QString::number(cs.count()));
|
||||
countItem->setTextAlignment(Qt::AlignRight | Qt::AlignVCenter);
|
||||
commandTable->setItem(row, 1, countItem);
|
||||
|
||||
auto *totalItem = new QTableWidgetItem(QString::number(cs.total_ms()));
|
||||
totalItem->setTextAlignment(Qt::AlignRight | Qt::AlignVCenter);
|
||||
commandTable->setItem(row, 2, totalItem);
|
||||
|
||||
const double avg = cs.count() > 0 ? static_cast<double>(cs.total_ms()) / cs.count() : 0.0;
|
||||
auto *avgItem = new QTableWidgetItem(QString::number(avg, 'f', 2));
|
||||
avgItem->setTextAlignment(Qt::AlignRight | Qt::AlignVCenter);
|
||||
commandTable->setItem(row, 3, avgItem);
|
||||
}
|
||||
commandTable->resizeColumnsToContents();
|
||||
statsTable->resizeColumnsToContents();
|
||||
commandTable->resizeColumnsToContents();
|
||||
|
||||
statusLabel->setText(tr("Updated %1").arg(QDateTime::currentDateTime().toString("yyyy-MM-dd HH:mm")));
|
||||
}
|
||||
|
|
|
|||
|
|
@ -10,9 +10,12 @@
|
|||
#include "tab.h"
|
||||
|
||||
class AbstractClient;
|
||||
class QCheckBox;
|
||||
class QLabel;
|
||||
class QPushButton;
|
||||
class QSpinBox;
|
||||
class QTableWidget;
|
||||
class QTimer;
|
||||
class Response;
|
||||
|
||||
class TabDeveloper : public Tab
|
||||
|
|
@ -21,15 +24,24 @@ class TabDeveloper : public Tab
|
|||
private:
|
||||
AbstractClient *client;
|
||||
QTableWidget *statsTable;
|
||||
QTableWidget *commandTable;
|
||||
QPushButton *refreshButton;
|
||||
QLabel *statusLabel;
|
||||
QCheckBox *autoRefreshCheckBox;
|
||||
QSpinBox *refreshIntervalSpinBox;
|
||||
QTimer *autoRefreshTimer;
|
||||
bool requestPending = false;
|
||||
|
||||
void appendStatRow(const QString &name, const QString &value);
|
||||
void appendSeparatorRow(const QString §ionTitle);
|
||||
static QString formatBytes(quint64 bytes);
|
||||
static QString formatDurationMs(qint64 ms);
|
||||
|
||||
private slots:
|
||||
void refreshClicked();
|
||||
void serverStatsResponse(const Response &resp);
|
||||
void autoRefreshToggled(bool checked);
|
||||
void refreshIntervalChanged();
|
||||
|
||||
public:
|
||||
explicit TabDeveloper(TabSupervisor *_tabSupervisor, AbstractClient *_client);
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue