mirror of
https://github.com/Cockatrice/Cockatrice.git
synced 2026-09-21 09:05:10 -07:00
* [Server] Instrument command processing, game starts, and event loops
Add a lock-free MetricsRegistry that accumulates per-command processing
times in preallocated histogram slots (one per protobuf command type,
bucketed at 1/5/10/25/50/100/250/500/1000/2500/5000 ms +Inf). The
hot-path observeCommand() uses only relaxed atomic adds — no locks,
no allocations, no cache-line ping-pong beyond the unavoidable counter
updates.
Wire the registry into AbstractServerSocketInterface::processCommandContainer()
so every processed command is attributed with its container's wall-clock
time. When a container exceeds metrics/slow_command_ms (default 500),
a warning is logged including the connected username.
Add an EventLoopWatchdog heartbeat that runs on every socket pool thread.
If a heartbeat overshoots metrics/stall_warn_ms (default 2000 ms), the
overshoot is recorded in atomic counters and a warning is logged. Both
thresholds are configurable in servatrice.ini; setting stall_warn_ms to 0
disables the watchdogs entirely.
Track game-start durations via a separate histogram in MetricsRegistry.
Server_Game::startGameNow() measures the time from zone creation through
player materialization and reports it via Server::observeGameStartDurationMs().
Add a live card-count gauge: Server_Game exposes getCardsInGame() and
Servatrice::getCardsInGamesTotal() sums across all running games under
the appropriate read locks.
Include a standalone metrics_registry_test (Google Test) that validates
empty registries, single/multi-sample histograms, kind encoding,
overflow-slot collapse, negative-duration clamping, gauge rendering,
and the game-start histogram separation.
Took 10 minutes
* [Client/Server/Protocol] Surface live metrics in the Developer tab
Extend Response_GetServerStats with live counters from the in-process
MetricsRegistry: cards in games, event loop stall totals/worst,
total commands processed, average command time, active command types,
and game-start count/duration. Add a repeated CommandStats message
carrying per-command breakdowns (kind, extension number, resolved
protobuf name, count, total ms) for every type that has seen at
least one sample.
Server-side cmdGetServerStats() populates all new fields after the
existing DB uptime snapshot query, resolving protobuf extension names
via the descriptor pool for human-readable labels like
session/Command_Ping.
Expand TabDeveloper with two tables: an overview section (existing
DB stats plus the new live metrics) and a per-command breakdown table
(Command / Count / Total ms / Avg ms) sorted by total_ms descending
so the hottest commands surface first.
Took 55 minutes
Took 47 seconds
* [Server] Drop dead Prometheus histogram, add developer command metrics, fix watchdog init order
- metrics_registry: remove toPrometheusText/appendCumulativeBuckets and the time-bucket histogram that nothing in production ever emitted (the future /metrics exporter can bring it back); keep counts/totals read by the Developer tab
- Fix +Inf bucket routing that never incremented, and its test that locked the bug in
- Instrument developer_command container (kind 6) in processCommandContainer and stats label resolution
- Read metrics/{slow_command_ms,stall_warn_ms} at the top of initServer() so stall_warn_ms=0 disables the watchdogs before pool threads start
- Shrink KindStride to 1280 (largest extension in use is 1206) with a static_assert; document scrape cost of getCardsInGamesTotal; note slow_command logging has no rate limit in servatrice.ini.example
* [Tests] Give metrics_registry_test an explicit main
* [Server] Record only the dispatched command family; drop unused totals
processCommandContainer recorded every family in a container even though
the base if/else-if dispatch processes at most one. An unauthenticated
client could batch a session command (login) with fabricated developer,
moderator, and admin entries and forge genuine-looking samples that were
never executed or authorized. Mirror the base's selection, skip when the
handler was already deleted, and skip entries whose extension number is
-1 (which would otherwise wrap into the previous kind's id range).
[Server] Drop dead process-lifetime byte/uptime counters
txBytesTotal/rxBytesTotal added an atomic RMW to every socket write and
read for counters nothing consumes (cmdGetServerStats fills tx_bytes,
rx_bytes, and uptime_secs from the DB snapshot). Remove the two atomics
and the getTxBytesTotal/getRxBytesTotal/getUptimeSeconds getters; the
incTxBytes/incRxBytes slots and mutexes remain for the ISL legacy
counters.
[Protocol] Document kind 5 as developer in CommandStats
NumKinds is 6 and the server emits kind_index = 5 for developer
commands; the comment stopped at 4.
* [Client] Togglable auto-refresh for Developer stats tab
* [Oracle] Fix clang-format alignment of card type priority list
---------
Co-authored-by: Lukas Brübach <Bruebach.Lukas@bdosecurity.de>
261 lines
11 KiB
C++
261 lines
11 KiB
C++
/**
|
|
* @file tab_developer.cpp
|
|
* @ingroup ServerTabs
|
|
*/
|
|
//! \todo Document this file.
|
|
|
|
#include "tab_developer.h"
|
|
|
|
#include <QCheckBox>
|
|
#include <QDateTime>
|
|
#include <QHBoxLayout>
|
|
#include <QHeaderView>
|
|
#include <QLabel>
|
|
#include <QPushButton>
|
|
#include <QSpinBox>
|
|
#include <QTableWidget>
|
|
#include <QTimer>
|
|
#include <QVBoxLayout>
|
|
#include <algorithm>
|
|
#include <libcockatrice/network/client/abstract/abstract_client.h>
|
|
#include <libcockatrice/protocol/pb/command_get_server_stats.pb.h>
|
|
#include <libcockatrice/protocol/pb/response_get_server_stats.pb.h>
|
|
#include <libcockatrice/protocol/pending_command.h>
|
|
|
|
static constexpr int DEFAULT_AUTO_REFRESH_INTERVAL_SECS = 30;
|
|
|
|
TabDeveloper::TabDeveloper(TabSupervisor *_tabSupervisor, AbstractClient *_client)
|
|
: Tab(_tabSupervisor), client(_client)
|
|
{
|
|
statsTable = new QTableWidget(0, 2);
|
|
statsTable->setSizePolicy(QSizePolicy::Expanding, QSizePolicy::Preferred);
|
|
statsTable->setEditTriggers(QAbstractItemView::NoEditTriggers);
|
|
statsTable->setSelectionBehavior(QAbstractItemView::SelectRows);
|
|
statsTable->setSelectionMode(QAbstractItemView::SingleSelection);
|
|
statsTable->verticalHeader()->setVisible(false);
|
|
statsTable->horizontalHeader()->setSectionResizeMode(0, QHeaderView::Interactive);
|
|
statsTable->horizontalHeader()->setSectionResizeMode(1, QHeaderView::Interactive);
|
|
statsTable->horizontalHeader()->setStretchLastSection(true);
|
|
|
|
commandTable = new QTableWidget(0, 4);
|
|
commandTable->setSizePolicy(QSizePolicy::Expanding, QSizePolicy::Expanding);
|
|
commandTable->setEditTriggers(QAbstractItemView::NoEditTriggers);
|
|
commandTable->setSelectionBehavior(QAbstractItemView::SelectRows);
|
|
commandTable->setSelectionMode(QAbstractItemView::SingleSelection);
|
|
commandTable->verticalHeader()->setVisible(false);
|
|
commandTable->horizontalHeader()->setSectionResizeMode(0, QHeaderView::Interactive);
|
|
commandTable->horizontalHeader()->setSectionResizeMode(1, QHeaderView::Interactive);
|
|
commandTable->horizontalHeader()->setSectionResizeMode(2, QHeaderView::Interactive);
|
|
commandTable->horizontalHeader()->setSectionResizeMode(3, QHeaderView::Interactive);
|
|
|
|
statusLabel = new QLabel;
|
|
|
|
autoRefreshCheckBox = new QCheckBox;
|
|
autoRefreshCheckBox->setChecked(false);
|
|
|
|
refreshIntervalSpinBox = new QSpinBox;
|
|
refreshIntervalSpinBox->setRange(5, 3600);
|
|
refreshIntervalSpinBox->setValue(DEFAULT_AUTO_REFRESH_INTERVAL_SECS);
|
|
refreshIntervalSpinBox->setEnabled(false);
|
|
|
|
autoRefreshTimer = new QTimer(this);
|
|
connect(autoRefreshTimer, &QTimer::timeout, this, &TabDeveloper::refreshClicked);
|
|
connect(autoRefreshCheckBox, &QCheckBox::toggled, this, &TabDeveloper::autoRefreshToggled);
|
|
connect(refreshIntervalSpinBox, QOverload<int>::of(&QSpinBox::valueChanged), this,
|
|
&TabDeveloper::refreshIntervalChanged);
|
|
|
|
refreshButton = new QPushButton;
|
|
refreshButton->setAutoDefault(true);
|
|
connect(refreshButton, &QPushButton::clicked, this, &TabDeveloper::refreshClicked);
|
|
|
|
auto *buttonLayout = new QHBoxLayout;
|
|
buttonLayout->addWidget(statusLabel, 1, Qt::AlignLeft);
|
|
buttonLayout->addWidget(autoRefreshCheckBox, 0, Qt::AlignRight);
|
|
buttonLayout->addWidget(refreshIntervalSpinBox, 0, Qt::AlignRight);
|
|
buttonLayout->addWidget(refreshButton, 0, Qt::AlignRight);
|
|
|
|
auto *tableLayout = new QHBoxLayout;
|
|
tableLayout->addWidget(statsTable, 1);
|
|
tableLayout->addWidget(commandTable, 2);
|
|
|
|
auto *mainLayout = new QVBoxLayout;
|
|
mainLayout->addLayout(tableLayout, 1);
|
|
mainLayout->addLayout(buttonLayout);
|
|
|
|
auto *central = new QWidget;
|
|
central->setLayout(mainLayout);
|
|
setCentralWidget(central);
|
|
|
|
retranslateUi();
|
|
}
|
|
|
|
void TabDeveloper::retranslateUi()
|
|
{
|
|
autoRefreshCheckBox->setText(tr("Auto-refresh"));
|
|
autoRefreshCheckBox->setToolTip(tr("Automatically request fresh server statistics at a fixed interval."));
|
|
refreshIntervalSpinBox->setSuffix(tr(" s"));
|
|
refreshIntervalSpinBox->setToolTip(tr("Seconds between automatic refreshes."));
|
|
refreshButton->setText(tr("Refresh server stats"));
|
|
statsTable->setHorizontalHeaderLabels(QString(tr("Statistic;Value")).split(";"));
|
|
commandTable->setHorizontalHeaderLabels(QString(tr("Command;Count;Total ms;Avg ms")).split(";"));
|
|
if (statsTable->rowCount() == 0) {
|
|
statusLabel->clear();
|
|
}
|
|
}
|
|
|
|
QString TabDeveloper::formatBytes(quint64 bytes)
|
|
{
|
|
const quint64 kib = 1024;
|
|
const quint64 mib = 1024 * kib;
|
|
const quint64 gib = 1024 * mib;
|
|
if (bytes >= gib) {
|
|
return tr("%1 GiB").arg(QString::number(bytes / static_cast<double>(gib), 'f', 2));
|
|
}
|
|
if (bytes >= mib) {
|
|
return tr("%1 MiB").arg(QString::number(bytes / static_cast<double>(mib), 'f', 2));
|
|
}
|
|
if (bytes >= kib) {
|
|
return tr("%1 KiB").arg(QString::number(bytes / static_cast<double>(kib), 'f', 2));
|
|
}
|
|
return tr("%1 bytes").arg(bytes);
|
|
}
|
|
|
|
QString TabDeveloper::formatDurationMs(qint64 ms)
|
|
{
|
|
if (ms >= 1000) {
|
|
return tr("%1 s").arg(QString::number(ms / 1000.0, 'f', 2));
|
|
}
|
|
return tr("%1 ms").arg(ms);
|
|
}
|
|
|
|
void TabDeveloper::appendStatRow(const QString &name, const QString &value)
|
|
{
|
|
const int row = statsTable->rowCount();
|
|
statsTable->insertRow(row);
|
|
statsTable->setItem(row, 0, new QTableWidgetItem(name));
|
|
statsTable->setItem(row, 1, new QTableWidgetItem(value));
|
|
}
|
|
|
|
void TabDeveloper::appendSeparatorRow(const QString §ionTitle)
|
|
{
|
|
const int row = statsTable->rowCount();
|
|
statsTable->insertRow(row);
|
|
auto *labelItem = new QTableWidgetItem(sectionTitle);
|
|
auto font = labelItem->font();
|
|
font.setBold(true);
|
|
labelItem->setFont(font);
|
|
labelItem->setFlags(labelItem->flags() & ~Qt::ItemIsSelectable);
|
|
statsTable->setItem(row, 0, labelItem);
|
|
statsTable->setItem(row, 1, new QTableWidgetItem(QString()));
|
|
}
|
|
|
|
void TabDeveloper::refreshClicked()
|
|
{
|
|
if (requestPending) {
|
|
return;
|
|
}
|
|
requestPending = true;
|
|
Command_GetServerStats cmd;
|
|
PendingCommand *pend = client->prepareDeveloperCommand(cmd);
|
|
connect(pend, &PendingCommand::finished, this, &TabDeveloper::serverStatsResponse);
|
|
client->sendCommand(pend);
|
|
}
|
|
|
|
void TabDeveloper::autoRefreshToggled(bool checked)
|
|
{
|
|
refreshIntervalSpinBox->setEnabled(checked);
|
|
if (checked) {
|
|
refreshIntervalChanged();
|
|
refreshClicked();
|
|
} else {
|
|
autoRefreshTimer->stop();
|
|
}
|
|
}
|
|
|
|
void TabDeveloper::refreshIntervalChanged()
|
|
{
|
|
if (autoRefreshCheckBox->isChecked()) {
|
|
autoRefreshTimer->start(refreshIntervalSpinBox->value() * 1000);
|
|
}
|
|
}
|
|
|
|
void TabDeveloper::serverStatsResponse(const Response &resp)
|
|
{
|
|
requestPending = false;
|
|
if (resp.response_code() != Response::RespOk) {
|
|
statusLabel->setText(tr("No server statistics available yet."));
|
|
return;
|
|
}
|
|
|
|
const Response_GetServerStats &response = resp.GetExtension(Response_GetServerStats::ext);
|
|
|
|
statsTable->setRowCount(0);
|
|
|
|
// Overview section
|
|
appendStatRow(tr("Registered users online"), QString::number(response.users_count()));
|
|
appendStatRow(tr("Moderators online"), QString::number(response.mods_count()));
|
|
appendStatRow(tr("Games running"), QString::number(response.games_count()));
|
|
appendStatRow(tr("Traffic sent (last tick)"), formatBytes(response.tx_bytes()));
|
|
appendStatRow(tr("Traffic received (last tick)"), formatBytes(response.rx_bytes()));
|
|
|
|
const qint64 uptime = static_cast<qint64>(response.uptime_secs());
|
|
const int days = static_cast<int>(uptime / 86400);
|
|
const int hours = static_cast<int>((uptime % 86400) / 3600);
|
|
const int minutes = static_cast<int>((uptime % 3600) / 60);
|
|
appendStatRow(tr("Server uptime"), days > 0 ? tr("%1d %2h %3m").arg(days).arg(hours).arg(minutes)
|
|
: tr("%1h %2m").arg(hours).arg(minutes));
|
|
|
|
const QDateTime snapshotTime = QDateTime::fromSecsSinceEpoch(static_cast<qint64>(response.timest()));
|
|
appendStatRow(tr("Snapshot taken"), snapshotTime.toLocalTime().toString("yyyy-MM-dd HH:mm"));
|
|
|
|
// Live metrics section
|
|
appendSeparatorRow(tr("Live Metrics"));
|
|
appendStatRow(tr("Cards in live games"), QString::number(response.cards_in_games()));
|
|
appendStatRow(tr("Total commands processed"), QString::number(response.total_commands()));
|
|
|
|
if (response.total_commands() > 0) {
|
|
const double avgMs = static_cast<double>(response.total_command_time_ms()) / response.total_commands();
|
|
appendStatRow(tr("Avg command time"), QString::number(avgMs, 'f', 2) + " ms");
|
|
}
|
|
appendStatRow(tr("Active command types"), QString::number(response.active_command_types()));
|
|
|
|
appendStatRow(tr("Event loop stalls"), QString::number(response.eventloop_stalls_total()));
|
|
appendStatRow(tr("Last stall overshoot"), formatDurationMs(response.eventloop_last_stall_ms()));
|
|
appendStatRow(tr("Worst stall overshoot"), formatDurationMs(response.eventloop_max_stall_ms()));
|
|
|
|
if (response.game_start_count() > 0) {
|
|
appendStatRow(tr("Game starts"), QString::number(response.game_start_count()));
|
|
const double avgStartMs = static_cast<double>(response.game_start_total_ms()) / response.game_start_count();
|
|
appendStatRow(tr("Avg game start time"), QString::number(avgStartMs, 'f', 1) + " ms");
|
|
}
|
|
|
|
// Per-command breakdown table
|
|
QList<CommandStats> sortedStats(response.command_stats().begin(), response.command_stats().end());
|
|
std::sort(sortedStats.begin(), sortedStats.end(),
|
|
[](const auto &a, const auto &b) { return a.total_ms() > b.total_ms(); });
|
|
|
|
commandTable->setRowCount(0);
|
|
for (const auto &cs : sortedStats) {
|
|
const int row = commandTable->rowCount();
|
|
commandTable->insertRow(row);
|
|
commandTable->setItem(row, 0, new QTableWidgetItem(QString::fromStdString(cs.command_name())));
|
|
|
|
auto *countItem = new QTableWidgetItem(QString::number(cs.count()));
|
|
countItem->setTextAlignment(Qt::AlignRight | Qt::AlignVCenter);
|
|
commandTable->setItem(row, 1, countItem);
|
|
|
|
auto *totalItem = new QTableWidgetItem(QString::number(cs.total_ms()));
|
|
totalItem->setTextAlignment(Qt::AlignRight | Qt::AlignVCenter);
|
|
commandTable->setItem(row, 2, totalItem);
|
|
|
|
const double avg = cs.count() > 0 ? static_cast<double>(cs.total_ms()) / cs.count() : 0.0;
|
|
auto *avgItem = new QTableWidgetItem(QString::number(avg, 'f', 2));
|
|
avgItem->setTextAlignment(Qt::AlignRight | Qt::AlignVCenter);
|
|
commandTable->setItem(row, 3, avgItem);
|
|
}
|
|
commandTable->resizeColumnsToContents();
|
|
statsTable->resizeColumnsToContents();
|
|
commandTable->resizeColumnsToContents();
|
|
|
|
statusLabel->setText(tr("Updated %1").arg(QDateTime::currentDateTime().toString("yyyy-MM-dd HH:mm")));
|
|
}
|