Cockatrice/libcockatrice_settings/libcockatrice/settings/download_settings.cpp
BruebachL 14acf3bf64
[PictureLoader] Add user-configurable per-host request caps (#7287)
* [PictureLoader] Add user-configurable per-host request caps

Picture downloads were throttled to a uniform 10 requests/second per host
with no way to tune a specific server. A rate-limited API host (Scryfall
caps at 10 req/s) can trip 429s during bursts, and CDN hosts with no rate
limit were throttled needlessly.

Introduce developer-owned per-host caps that users can only ever lower,
never raise, exposed in the download settings page:
- DownloadSettings::DEVELOPER_HOST_CAPS sets the ceiling per host
  (api.scryfall.com 9, cards.scryfall.io unlimited, others 10).
- A new hostRequestLimits setting stores user overrides in downloads.ini;
  clampHostRequestLimit() bounds them to [1, devCap] so a user can reduce
  api.scryfall.com to 5 but never raise it above 9.
- The picture worker seeds, halves on 429, and recovers its sustained
  per-host allowance against the effective ceiling instead of the global
  maximum, and skips per-host accounting entirely for unlocked hosts
  (cards.scryfall.io) while global pacing and 429 backoff still apply.
- The deck editor settings page gains one spinbox per known host, each
  clamped to its developer cap.

* [PictureLoader] Let unlocked hosts skip dispatch pacing; adjust limits per URL

Two refinements to the per-host request caps:

- Unlocked hosts (UNLIMITED_HOST_QUOTA, e.g. cards.scryfall.io) no longer
  wait on the 100ms dispatch pacing or consume the global per-second quota.
  dispatchQueuedRequest fires their queued requests back-to-back, bounded
  only by their 429 backoff window and Qt's per-host connection pool, so
  an unthrottled CDN is not artificially slowed.
- The deck editor download settings page replaces the static grid of one
  spinbox per known host with an "Adjust Rate Limit" toolbar action on the
  URL list. It picks the host out of the selected URL and clamps the entry
  against the developer cap table (including for user-added URLs).

Also fixes a review finding: resetRequestQuota could write the
UNLIMITED_HOST_QUOTA sentinel (-1) into the sustained per-host quota when a
host became unlocked mid-run, permanently poisoning its allowance. Stale
entries for unlocked hosts are now dropped, and the per-second seed is
clamped against the effective ceiling so a lowered limit applies immediately.

* [PictureLoader] Cap unlocked host bursts and adapt them to 429s

* [PictureLoader] Store per-host limits readably and show them per URL

* [PictureLoader] Make dispatch and rate-limit bookkeeping key on the real host

Addresses ZeldaZach's round-4 review nits:

- Dispatch now resolves the cached-redirect chain before the in-flight gate,
  so a redirect learned after a URL was queued can no longer bypass the
  MAX_IN_FLIGHT_PER_HOST cap and drain the whole queue onto the redirect
  target, which may carry its own developer cap. processSingleRequest does
  the same so the allowance math keys on the host that is actually hit.
- The per-host in-flight slot is released when the reply is destroyed (with
  the worker as the connection context) rather than on a 'finished'
  connection bound to the work object, so an aborted reply or a work object
  deleted while a reply is pending can never permanently shrink the fast
  path's concurrency.
- storeSettings only prunes limits for hosts with neither a URL nor a
  developer cap, so throttles on redirect targets (api.scryfall.com ->
  cards.scryfall.io) survive URL removal.
- Unlocked hosts are offered 0..UNLOCKED_HOST_LIMIT_MAX (50) in the rate
  limit dialog, matching clampHostRequestLimit() and the documented
  hand-editable range, so values written into downloads.ini are no longer
  silently rewritten on the next edit.

---------

Co-authored-by: Lukas Brübach <Bruebach.Lukas@bdosecurity.de>
2026-09-21 18:30:03 +02:00

137 lines
4.6 KiB
C++

#include "download_settings.h"
#include "settings_manager.h"
const QStringList DownloadSettings::DEFAULT_DOWNLOAD_URLS = {
"https://cards.scryfall.io/large/!prop:side!/!set:uuid_substr_0_1!/!set:uuid_substr_1_1!/!set:uuid!.jpg",
"https://api.scryfall.com/cards/!set:uuid!?format=image&face=!prop:side!&lang=!sflang!",
"https://api.scryfall.com/cards/multiverse/!set:muid!?format=image&lang=!sflang!",
"https://gatherer.wizards.com/Handlers/Image.ashx?multiverseid=!set:muid!&type=card",
"https://gatherer.wizards.com/Handlers/Image.ashx?name=!name!&type=card"};
const QString DownloadSettings::SCRYFALL_NAMED_LOCALIZED_URL =
"https://api.scryfall.com/cards/named?fuzzy=!localizedName!&lang=!sflang!&format=image&face=!prop:side!";
// Developer-set ceilings for the per-host request allowance. Users may lower a host's
// allowance via the download settings, but can never raise it above these values. Hosts
// not listed default to DEFAULT_HOST_REQUEST_LIMIT. A cap of UNLIMITED_HOST_QUOTA marks a
// host that is never throttled per host and skips the dispatch pacing (429 backoff still applies).
const QHash<QString, int> DownloadSettings::DEVELOPER_HOST_CAPS = {
// The Scryfall API enforces 10 requests/second; stay one under so a burst can't trip 429s.
{"api.scryfall.com", 9},
// The Scryfall image CDN has no documented per-client rate limit.
{"cards.scryfall.io", UNLIMITED_HOST_QUOTA},
};
const QHash<QString, int> &DownloadSettings::getDeveloperHostCaps()
{
return DEVELOPER_HOST_CAPS;
}
DownloadSettings::DownloadSettings(const QString &settingPath, QObject *parent = nullptr)
: SettingsManager(settingPath + "downloads.ini", "downloads", QString(), parent)
{
}
void DownloadSettings::setDownloadUrls(const QStringList &downloadURLs)
{
setValue(QVariant::fromValue(downloadURLs), "urls");
}
QStringList DownloadSettings::getAllURLs() const
{
return getValue("urls").toStringList();
}
void DownloadSettings::resetToDefaultURLs()
{
setValue(QVariant::fromValue(DEFAULT_DOWNLOAD_URLS), "urls");
}
bool DownloadSettings::addLocalizedScryfallUrl()
{
const QStringList urls = getAllURLs();
if (urls.contains(SCRYFALL_NAMED_LOCALIZED_URL)) {
return false;
}
QStringList updated = urls;
updated.prepend(SCRYFALL_NAMED_LOCALIZED_URL);
setDownloadUrls(updated);
return true;
}
bool DownloadSettings::getPicDownload() const
{
return getValue("pictureDownload", QString(), QString(), true).toBool();
}
void DownloadSettings::setPicDownload(bool _picDownload)
{
setValue(_picDownload, "pictureDownload");
emit picDownloadChanged();
}
bool DownloadSettings::getDownloadSpoilersStatus() const
{
return getValue("downloadSpoilers", QString(), QString(), false).toBool();
}
void DownloadSettings::setDownloadSpoilerStatus(bool _spoilerStatus)
{
setValue(_spoilerStatus, "downloadSpoilers");
emit downloadSpoilerStatusChanged();
}
QHash<QString, int> DownloadSettings::getHostRequestLimits() const
{
auto settings = getSettings();
if (!defaultGroup.isEmpty()) {
settings.beginGroup(defaultGroup);
}
settings.beginGroup("hostRequestLimits");
QHash<QString, int> hostRequestLimits;
const QStringList hosts = settings.childKeys();
for (const QString &host : hosts) {
hostRequestLimits.insert(host, settings.value(host).toInt());
}
settings.endGroup();
if (!defaultGroup.isEmpty()) {
settings.endGroup();
}
return hostRequestLimits;
}
void DownloadSettings::setHostRequestLimits(const QHash<QString, int> &hostRequestLimits)
{
auto settings = getSettings();
if (!defaultGroup.isEmpty()) {
settings.beginGroup(defaultGroup);
}
// Drop the legacy single-key form (an opaque @Variant blob) written by earlier builds so each
// host is stored as a plain, hand-editable key in its own subgroup.
settings.remove("hostRequestLimits");
settings.beginGroup("hostRequestLimits");
settings.remove(QString());
for (auto it = hostRequestLimits.cbegin(); it != hostRequestLimits.cend(); ++it) {
settings.setValue(it.key(), it.value());
}
settings.endGroup();
if (!defaultGroup.isEmpty()) {
settings.endGroup();
}
settings.sync();
emit hostRequestLimitsChanged();
}
int DownloadSettings::clampHostRequestLimit(const QString &host, int requested) const
{
const int devCap = DEVELOPER_HOST_CAPS.value(host, DEFAULT_HOST_REQUEST_LIMIT);
if (devCap == UNLIMITED_HOST_QUOTA) {
return qMax(MIN_HOST_REQUEST_LIMIT, requested);
}
return qBound(MIN_HOST_REQUEST_LIMIT, requested, devCap);
}