mirror of
https://github.com/Cockatrice/Cockatrice.git
synced 2026-09-21 09:05:10 -07:00
* [Server] Instrument command processing, game starts, and event loops
Add a lock-free MetricsRegistry that accumulates per-command processing
times in preallocated histogram slots (one per protobuf command type,
bucketed at 1/5/10/25/50/100/250/500/1000/2500/5000 ms +Inf). The
hot-path observeCommand() uses only relaxed atomic adds — no locks,
no allocations, no cache-line ping-pong beyond the unavoidable counter
updates.
Wire the registry into AbstractServerSocketInterface::processCommandContainer()
so every processed command is attributed with its container's wall-clock
time. When a container exceeds metrics/slow_command_ms (default 500),
a warning is logged including the connected username.
Add an EventLoopWatchdog heartbeat that runs on every socket pool thread.
If a heartbeat overshoots metrics/stall_warn_ms (default 2000 ms), the
overshoot is recorded in atomic counters and a warning is logged. Both
thresholds are configurable in servatrice.ini; setting stall_warn_ms to 0
disables the watchdogs entirely.
Track game-start durations via a separate histogram in MetricsRegistry.
Server_Game::startGameNow() measures the time from zone creation through
player materialization and reports it via Server::observeGameStartDurationMs().
Add a live card-count gauge: Server_Game exposes getCardsInGame() and
Servatrice::getCardsInGamesTotal() sums across all running games under
the appropriate read locks.
Include a standalone metrics_registry_test (Google Test) that validates
empty registries, single/multi-sample histograms, kind encoding,
overflow-slot collapse, negative-duration clamping, gauge rendering,
and the game-start histogram separation.
Took 10 minutes
* [Client/Server/Protocol] Surface live metrics in the Developer tab
Extend Response_GetServerStats with live counters from the in-process
MetricsRegistry: cards in games, event loop stall totals/worst,
total commands processed, average command time, active command types,
and game-start count/duration. Add a repeated CommandStats message
carrying per-command breakdowns (kind, extension number, resolved
protobuf name, count, total ms) for every type that has seen at
least one sample.
Server-side cmdGetServerStats() populates all new fields after the
existing DB uptime snapshot query, resolving protobuf extension names
via the descriptor pool for human-readable labels like
session/Command_Ping.
Expand TabDeveloper with two tables: an overview section (existing
DB stats plus the new live metrics) and a per-command breakdown table
(Command / Count / Total ms / Avg ms) sorted by total_ms descending
so the hottest commands surface first.
Took 55 minutes
Took 47 seconds
* [Server] Drop dead Prometheus histogram, add developer command metrics, fix watchdog init order
- metrics_registry: remove toPrometheusText/appendCumulativeBuckets and the time-bucket histogram that nothing in production ever emitted (the future /metrics exporter can bring it back); keep counts/totals read by the Developer tab
- Fix +Inf bucket routing that never incremented, and its test that locked the bug in
- Instrument developer_command container (kind 6) in processCommandContainer and stats label resolution
- Read metrics/{slow_command_ms,stall_warn_ms} at the top of initServer() so stall_warn_ms=0 disables the watchdogs before pool threads start
- Shrink KindStride to 1280 (largest extension in use is 1206) with a static_assert; document scrape cost of getCardsInGamesTotal; note slow_command logging has no rate limit in servatrice.ini.example
* [Tests] Give metrics_registry_test an explicit main
* [Server] Record only the dispatched command family; drop unused totals
processCommandContainer recorded every family in a container even though
the base if/else-if dispatch processes at most one. An unauthenticated
client could batch a session command (login) with fabricated developer,
moderator, and admin entries and forge genuine-looking samples that were
never executed or authorized. Mirror the base's selection, skip when the
handler was already deleted, and skip entries whose extension number is
-1 (which would otherwise wrap into the previous kind's id range).
[Server] Drop dead process-lifetime byte/uptime counters
txBytesTotal/rxBytesTotal added an atomic RMW to every socket write and
read for counters nothing consumes (cmdGetServerStats fills tx_bytes,
rx_bytes, and uptime_secs from the DB snapshot). Remove the two atomics
and the getTxBytesTotal/getRxBytesTotal/getUptimeSeconds getters; the
incTxBytes/incRxBytes slots and mutexes remain for the ISL legacy
counters.
[Protocol] Document kind 5 as developer in CommandStats
NumKinds is 6 and the server emits kind_index = 5 for developer
commands; the comment stopped at 4.
* [Client] Togglable auto-refresh for Developer stats tab
* [Oracle] Fix clang-format alignment of card type priority list
---------
Co-authored-by: Lukas Brübach <Bruebach.Lukas@bdosecurity.de>
354 lines
12 KiB
C++
354 lines
12 KiB
C++
/***************************************************************************
|
|
* Copyright (C) 2008 by Max-Wilhelm Bruker *
|
|
* brukie@laptop *
|
|
* *
|
|
* This program is free software; you can redistribute it and/or modify *
|
|
* it under the terms of the GNU General Public License as published by *
|
|
* the Free Software Foundation; either version 2 of the License, or *
|
|
* (at your option) any later version. *
|
|
* *
|
|
* This program is distributed in the hope that it will be useful, *
|
|
* but WITHOUT ANY WARRANTY; without even the implied warranty of *
|
|
* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the *
|
|
* GNU General Public License for more details. *
|
|
* *
|
|
* You should have received a copy of the GNU General Public License *
|
|
* along with this program; if not, write to the *
|
|
* Free Software Foundation, Inc., *
|
|
* 59 Temple Place - Suite 330, Boston, MA 02111-1307, USA. *
|
|
***************************************************************************/
|
|
#ifndef SERVATRICE_H
|
|
#define SERVATRICE_H
|
|
|
|
#include "metrics_registry.h"
|
|
|
|
#include <QDateTime>
|
|
#include <QHostAddress>
|
|
#include <QMetaType>
|
|
#include <QMutex>
|
|
#include <QReadWriteLock>
|
|
#include <QSqlDatabase>
|
|
#include <QSslCertificate>
|
|
#include <QSslKey>
|
|
#include <QTcpServer>
|
|
#include <QWebSocketServer>
|
|
#include <atomic>
|
|
#include <libcockatrice/protocol/pb/response_report_stats.pb.h>
|
|
#include <memory>
|
|
#include <server.h>
|
|
#include <utility>
|
|
|
|
Q_DECLARE_METATYPE(QSqlDatabase)
|
|
|
|
class QSqlQuery;
|
|
class QTimer;
|
|
|
|
class GameReplay;
|
|
class Servatrice;
|
|
class Servatrice_ConnectionPool;
|
|
class Servatrice_DatabaseInterface;
|
|
class AbstractServerSocketInterface;
|
|
class IslInterface;
|
|
class FeatureSet;
|
|
|
|
class Servatrice_GameServer : public QTcpServer
|
|
{
|
|
Q_OBJECT
|
|
private:
|
|
Servatrice *server;
|
|
QList<Servatrice_ConnectionPool *> connectionPools;
|
|
|
|
public:
|
|
Servatrice_GameServer(Servatrice *_server,
|
|
int _numberPools,
|
|
const QSqlDatabase &_sqlDatabase,
|
|
QObject *parent = nullptr);
|
|
~Servatrice_GameServer() override;
|
|
|
|
protected:
|
|
void incomingConnection(qintptr socketDescriptor) override;
|
|
Servatrice_ConnectionPool *findLeastUsedConnectionPool();
|
|
};
|
|
|
|
class Servatrice_WebsocketGameServer : public QWebSocketServer
|
|
{
|
|
Q_OBJECT
|
|
private:
|
|
Servatrice *server;
|
|
QList<Servatrice_ConnectionPool *> connectionPools;
|
|
|
|
public:
|
|
Servatrice_WebsocketGameServer(Servatrice *_server,
|
|
int _numberPools,
|
|
const QSqlDatabase &_sqlDatabase,
|
|
QObject *parent = nullptr);
|
|
~Servatrice_WebsocketGameServer() override;
|
|
|
|
protected:
|
|
Servatrice_ConnectionPool *findLeastUsedConnectionPool();
|
|
protected slots:
|
|
void onNewConnection();
|
|
};
|
|
|
|
class Servatrice_IslServer : public QTcpServer
|
|
{
|
|
Q_OBJECT
|
|
private:
|
|
Servatrice *server;
|
|
QSslCertificate cert;
|
|
QSslKey privateKey;
|
|
|
|
public:
|
|
Servatrice_IslServer(Servatrice *_server,
|
|
const QSslCertificate &_cert,
|
|
QSslKey _privateKey,
|
|
QObject *parent = nullptr)
|
|
: QTcpServer(parent), server(_server), cert(_cert), privateKey(std::move(_privateKey))
|
|
{
|
|
}
|
|
|
|
protected:
|
|
void incomingConnection(qintptr socketDescriptor) override;
|
|
};
|
|
|
|
class ServerProperties
|
|
{
|
|
public:
|
|
int id;
|
|
QSslCertificate cert;
|
|
QString hostname;
|
|
QHostAddress address;
|
|
int gamePort;
|
|
int controlPort;
|
|
|
|
ServerProperties(int _id,
|
|
const QSslCertificate &_cert,
|
|
QString _hostname,
|
|
const QHostAddress &_address,
|
|
int _gamePort,
|
|
int _controlPort)
|
|
: id(_id), cert(_cert), hostname(std::move(_hostname)), address(_address), gamePort(_gamePort),
|
|
controlPort(_controlPort)
|
|
{
|
|
}
|
|
};
|
|
|
|
class Servatrice : public Server
|
|
{
|
|
Q_OBJECT
|
|
public:
|
|
enum AuthenticationMethod
|
|
{
|
|
AuthenticationNone,
|
|
AuthenticationSql,
|
|
AuthenticationPassword
|
|
};
|
|
private slots:
|
|
void statusUpdate();
|
|
void shutdownTimeout();
|
|
|
|
protected:
|
|
void doSendIslMessage(const IslMessage &msg, int _serverId) override;
|
|
|
|
private:
|
|
enum DatabaseType
|
|
{
|
|
DatabaseNone,
|
|
DatabaseMySql
|
|
};
|
|
AuthenticationMethod authenticationMethod;
|
|
DatabaseType databaseType;
|
|
QTimer *pingClock, *statusUpdateClock;
|
|
Servatrice_GameServer *gameServer;
|
|
Servatrice_WebsocketGameServer *websocketGameServer;
|
|
Servatrice_IslServer *islServer;
|
|
mutable QMutex loginMessageMutex;
|
|
QString loginMessage;
|
|
mutable QMutex shutdownStateMutex;
|
|
QString dbPrefix;
|
|
QMap<QString, bool> serverRequiredFeatureList;
|
|
QString officialWarnings;
|
|
Servatrice_DatabaseInterface *servatriceDatabaseInterface;
|
|
int serverId;
|
|
int uptime;
|
|
QMutex txBytesMutex, rxBytesMutex;
|
|
quint64 txBytes, rxBytes;
|
|
MetricsRegistry metricsRegistry;
|
|
int metricsSlowCommandMs = 500;
|
|
int metricsStallWarnMs = 2000;
|
|
std::atomic<qint64> eventLoopStallsTotal{0}; ///< heartbeat overshoots past the warn threshold
|
|
std::atomic<qint64> eventLoopLastStallMs{0}; ///< overshoot of the most recent stall
|
|
std::atomic<qint64> eventLoopMaxStallMs{0}; ///< worst overshoot seen since process start
|
|
|
|
QString shutdownReason;
|
|
int shutdownMinutes;
|
|
int nextShutdownMessageMinutes;
|
|
QTimer *shutdownTimer;
|
|
|
|
mutable QMutex reportStatsMutex;
|
|
QDateTime reportStatsTimestamp;
|
|
std::shared_ptr<const Response_ReportStats> reportStatsCache;
|
|
static constexpr int reportStatsCacheTtlSeconds = 60;
|
|
|
|
mutable QMutex serverListMutex;
|
|
QList<ServerProperties> serverList;
|
|
void updateServerList();
|
|
|
|
QMap<int, IslInterface *> islInterfaces;
|
|
|
|
QString getDBPrefixString() const;
|
|
QString getDBHostNameString() const;
|
|
QString getDBDatabaseNameString() const;
|
|
QString getDBUserNameString() const;
|
|
QString getDBPasswordString() const;
|
|
QString getRoomsMethodString() const;
|
|
QString getISLNetworkSSLCertFile() const;
|
|
QString getISLNetworkSSLKeyFile() const;
|
|
int getServerStatusUpdateTime() const;
|
|
int getNumberOfTCPPools() const;
|
|
int getServerTCPPort() const;
|
|
int getNumberOfWebSocketPools() const;
|
|
int getServerWebSocketPort() const;
|
|
int getISLNetworkPort() const;
|
|
bool getISLNetworkEnabled() const;
|
|
bool getEnableInternalSMTPClient() const;
|
|
QHostAddress getServerTCPHost() const;
|
|
QHostAddress getServerWebSocketHost() const;
|
|
|
|
public slots:
|
|
void scheduleShutdown(const QString &reason, int minutes);
|
|
void updateLoginMessage();
|
|
void setRequiredFeatures(const QString &featureList);
|
|
|
|
public:
|
|
explicit Servatrice(QObject *parent = nullptr);
|
|
~Servatrice() override;
|
|
bool initServer();
|
|
QMap<QString, bool> getServerRequiredFeatureList() const override
|
|
{
|
|
return serverRequiredFeatureList;
|
|
}
|
|
QString getServerName() const;
|
|
QString getLoginMessage() const override
|
|
{
|
|
QMutexLocker locker(&loginMessageMutex);
|
|
return loginMessage;
|
|
}
|
|
SessionEvent *getLoginSessionEvent() const override;
|
|
SessionEvent *makeShutdownEvent() const;
|
|
QString getRequiredFeatures() const override;
|
|
QString getAuthenticationMethodString() const;
|
|
QString getDBTypeString() const;
|
|
QString getDbPrefix() const
|
|
{
|
|
return dbPrefix;
|
|
}
|
|
QString getEmailBlackList() const;
|
|
QString getEmailWhiteList() const;
|
|
AuthenticationMethod getAuthenticationMethod() const
|
|
{
|
|
return authenticationMethod;
|
|
}
|
|
bool permitUnregisteredUsers() const override
|
|
{
|
|
return authenticationMethod != AuthenticationNone;
|
|
}
|
|
bool getGameShouldPing() const override
|
|
{
|
|
return true;
|
|
}
|
|
bool getClientIDRequiredEnabled() const override;
|
|
bool getRegOnlyServerEnabled() const override;
|
|
bool getMaxUserLimitEnabled() const override;
|
|
bool getStoreReplaysEnabled() const override;
|
|
bool getRegistrationEnabled() const;
|
|
bool getRequireEmailForRegistrationEnabled() const;
|
|
bool getRequireEmailActivationEnabled() const;
|
|
bool getEnableLogQuery() const override;
|
|
bool getEnableForgotPassword() const;
|
|
bool getEnableForgotPasswordChallenge() const;
|
|
bool getEnableAudit() const;
|
|
bool getEnableRegistrationAudit() const;
|
|
bool getEnableForgotPasswordAudit() const;
|
|
int getMinPasswordLength() const;
|
|
int getIdleClientTimeout() const override;
|
|
int getServerID() const override;
|
|
int getMaxGameInactivityTime() const override;
|
|
int getMaxPlayerInactivityTime() const override;
|
|
int getClientKeepAlive() const override;
|
|
int getMaxUsersPerAddress() const;
|
|
int getMessageCountingInterval() const override;
|
|
int getMaxMessageCountPerInterval() const override;
|
|
int getMaxMessageSizePerInterval() const override;
|
|
int getMaxGamesPerUser() const override;
|
|
int getCommandCountingInterval() const override;
|
|
int getMaxCommandCountPerInterval() const override;
|
|
int getMaxUserTotal() const override;
|
|
bool permitCreateGameAsJudge() const override;
|
|
int getMaxTcpUserLimit() const;
|
|
int getMaxWebSocketUserLimit() const;
|
|
int getUsersWithAddress(const QHostAddress &address) const;
|
|
int getMaxAccountsPerEmail() const;
|
|
int getForgotPasswordTokenLife() const;
|
|
QList<AbstractServerSocketInterface *> getUsersWithAddressAsList(const QHostAddress &address) const;
|
|
void incTxBytes(quint64 num);
|
|
void incRxBytes(quint64 num);
|
|
void addDatabaseInterface(QThread *thread, Servatrice_DatabaseInterface *databaseInterface);
|
|
|
|
// Metrics (see [metrics] section in servatrice.ini.example)
|
|
MetricsRegistry &getMetricsRegistry()
|
|
{
|
|
return metricsRegistry;
|
|
}
|
|
/**
|
|
* Sums cards across all zones of all running games. Each game takes its
|
|
* own gameMutex -- the hot per-game lock every game action contends on --
|
|
* and then iterates every player's zones, so the scrape cost is really
|
|
* O(total cards in play) plus one mutex acquisition per live game. Keep
|
|
* scrapes infrequent in big multiplayer rooms.
|
|
*/
|
|
qint64 getCardsInGamesTotal() const;
|
|
int getMetricsSlowCommandMs() const
|
|
{
|
|
return metricsSlowCommandMs;
|
|
}
|
|
/// Heartbeat overshoot that counts as a stall. A value of 0 disables the watchdogs.
|
|
int getMetricsStallWarnMs() const
|
|
{
|
|
return metricsStallWarnMs;
|
|
}
|
|
void observeGameStartDurationMs(qint64 elapsedMs) override;
|
|
qint64 getEventLoopStallsTotal() const
|
|
{
|
|
return eventLoopStallsTotal.load(std::memory_order_relaxed);
|
|
}
|
|
qint64 getEventLoopLastStallMs() const
|
|
{
|
|
return eventLoopLastStallMs.load(std::memory_order_relaxed);
|
|
}
|
|
qint64 getEventLoopMaxStallMs() const
|
|
{
|
|
return eventLoopMaxStallMs.load(std::memory_order_relaxed);
|
|
}
|
|
/// Records one heartbeat overshoot and logs a single warning for it.
|
|
void observeEventLoopStall(const QString &threadName, qint64 overshootMs);
|
|
/**
|
|
* Installs an EventLoopWatchdog in @p thread. Called once per socket pool
|
|
* thread right after it starts.
|
|
*/
|
|
void watchWorkerThread(QThread *thread);
|
|
|
|
bool islConnectionExists(int _serverId) const;
|
|
void addIslInterface(int _serverId, IslInterface *interface);
|
|
void removeIslInterface(int _serverId);
|
|
QReadWriteLock islLock;
|
|
|
|
// The moderation queue statistics are shared between all connected moderators and
|
|
// cached briefly to avoid re-running several full-table queries on every refresh.
|
|
std::shared_ptr<const Response_ReportStats> getCachedReportStats() const;
|
|
void cacheReportStats(const Response_ReportStats &stats);
|
|
|
|
QList<ServerProperties> getServerList() const;
|
|
};
|
|
|
|
#endif
|