jeirmeister/Cockatrice
1
0
Fork 0
mirror of https://github.com/Cockatrice/Cockatrice.git synced 2026-11-08 19:13:25 -08:00
Code Issues Projects Releases Packages Wiki Activity Actions 5

[Card] Add Scryfall Tagger tag search and filtering (#7337)

Browse source 
* [Card] Import Scryfall Tagger tags from MTGJson

* [Card] Search cards by Scryfall Tagger tag

* [Card] Add a Tag attribute to the card filter tree

* [Card] Add a Tags filter to the visual card database

* [Oracle] Add a script to inject Scryfall tags into AllPrintings

* [Card] Elide long tag lists in the card info and drop the Scryfall label from the tag filter tooltip

* [Oracle] Load card tags from a tag database during import

Add a TagDatabase loader that understands Scryfall's Oracle Tags bulk data and user-supplied tag files (tag objects, flat oracle-id or name maps, optional manifest, JSON or JSONL, gzip/xz/zip). The importer joins the slugs on identifiers.scryfallOracleId while the identifiers are still available, falls back to the card name, and unions them with any tags the source already provides. Split-card faces merge through the existing tag union path.

This replaces oracle/scripts/inject_scryfall_tags.py, which patched AllPrintings.json before import. Loading during import keeps tags on by default for every source and lets users pair Oracle with any tag database.

* [Oracle] Add a tag data step to the import wizard

Insert a Tags page between set selection and the import. It downloads Scryfall's Oracle Tags by default (or a user-supplied file/URL), can include parent tags, and persists tagsenabled, tagssource and tagsparents in oracle.ini. The page loads asynchronously and soft-fails in background mode, so a tag download or parse error never aborts the card database import.

* [Help] Describe tag search without naming Scryfall

Tags can come from Scryfall's Oracle Tags or any user-supplied tag database, so the search help should not claim a single source.

* [Oracle] Fix Scryfall tag requests and the retry loop

api.scryfall.com rejects requests that lack an Accept header with HTTP 400, and QNetworkAccessManager does not add one. Send application/json for the bulk metadata request and */* for the tag file, and surface the URL plus Scryfall's JSON error body when a request fails.

A failed tag load used to call next(), which re-ran validatePage() and started the same failing download again, trapping the user in an error loop. Stay on the page instead, drop the unusable database, and tell the user they can retry, change the source, or untick the option to import without tags. Also explain parent tags inline with a real Scryfall hierarchy example.

* [Oracle] Give the parent-tag explainer room to breathe

The wrapped explainer sat flush under the checkbox with no separation from the status label, which made the paragraph look cramped. Indent it under the checkbox and add a little vertical margin.

* [Oracle] Stop the parent-tag explainer from being clipped

A word-wrapped QLabel reports a one-line sizeHint, so the wizard page reserved too little height and the explanation lost its top and bottom lines. Pin the vertical size policy to Minimum on the explainer and on the status label, so the wrapped height becomes part of the page minimum. Verified with a layout probe: the label went from 28 px tall (needed 56) to its full height.

---------

Co-authored-by: Lukas Brübach <Bruebach.Lukas@bdosecurity.de>
This commit is contained in:
BruebachL 2026-11-08 20:57:45 +01:00 • committed by GitHub
parent c7e8422808
commit 6138fbd0a0
No known key found for this signature in database
GPG key ID: B5690EEEBB952194
31 changed files with 2242 additions and 37 deletions
Download patch file Download diff file Expand all files Collapse all files

1
oracle/CMakeLists.txt
View file

@ -25,6 +25,7 @@ set(oracle_SOURCES
src/pagetemplates.cpp
src/parsehelpers.cpp
src/raw_json_scanner.cpp
src/tag_database.cpp
../cockatrice/src/client/settings/cache_settings.cpp
../cockatrice/src/client/settings/card_counter_settings.cpp
../cockatrice/src/client/settings/shortcuts_settings.cpp

34
oracle/src/oracleimporter.cpp
View file

@ -6,6 +6,7 @@
#include "libcockatrice/card/relation/card_relation_type.h"
#include "libcockatrice/interfaces/noop_card_set_priority_controller.h"
#include "parsehelpers.h"
#include "tag_database.h"
#include <QChar>
#include <QDebug>
@ -521,6 +522,31 @@ int OracleImporter::importCardsFromSet(const CardSetPtr &currentSet, const QJson
properties.insert("coloridentity", colorIdentity);
}
// Card tags: the source's own `tags` array is used when present (MTGJson
// may ship one eventually), then the configured tag database is joined in
// while the card's identifiers are still available. Stored space-separated
// so `tags:` can match each slug as a discrete token; slugs never contain
// whitespace.
QStringList tags;
for (const QJsonValue &tag : card.value("tags").toArray()) {
const QString tagSlug = tag.toString().trimmed().toLower();
if (!tagSlug.isEmpty() && !tags.contains(tagSlug)) {
tags.append(tagSlug);
}
}
if (tagDatabase) {
const QString oracleId = getJsonString(identifiers, "scryfallOracleId");
for (const QString &tagSlug : tagDatabase->tagsForCard(oracleId, name)) {
if (!tags.contains(tagSlug)) {
tags.append(tagSlug);
}
}
}
if (!tags.isEmpty()) {
tags.sort();
properties.insert("tags", tags.join(" "));
}
const auto &mainCardType = getMainCardType(card.value("types").toVariant().toStringList());
if (mainCardType.isEmpty()) {
qDebug() << "warning: no mainCardType for card:" << name;
@ -667,6 +693,14 @@ int OracleImporter::importCardsFromSet(const CardSetPtr &currentSet, const QJson
if (newPriority >= 0 && (currentPriority < 0 || newPriority < currentPriority)) {
properties.insert(prop, thisCardPropertyValue);
}
} else if (prop == "tags") {
// Tags are oracle-level: union both faces instead of
// concatenating them with the split-card separator.
QStringList merged =
(originalPropertyValue + " " + thisCardPropertyValue).split(" ", Qt::SkipEmptyParts);
merged.removeDuplicates();
merged.sort();
properties.insert(prop, merged.join(" "));
} else {
properties.insert(prop,
originalPropertyValue + splitCardPropSeparator + thisCardPropertyValue);

18
oracle/src/oracleimporter.h
View file

@ -17,12 +17,14 @@
#include <QString>
#include <QStringList>
#include <libcockatrice/card/card_info.h>
#include <memory>
#include <qnamespace.h>
#include <qtmetamacros.h>
#include <utility>
class CardRelation;
class QRegularExpression;
class TagDatabase;
// many users prefer not to see these sets with non english arts
// they will given priority PriorityLowest
@ -215,6 +217,12 @@ private:
*/
bool localizationEnabled = false;
/**
* Optional tag lookup applied to every imported card. Owned by the wizard
* so it outlives the import worker; null when tags are disabled.
*/
std::shared_ptr<const TagDatabase> tagDatabase;
/**
* Localized name/text collected per imported card key while parsing sets,
* applied to the CardInfo objects by applyLocalizedData() once all
@ -269,6 +277,16 @@ public:
{
progressReporting = enabled;
}
/**
* @brief Supplies the tag database applied to every imported card.
*
* Pass nullptr to import without tags. The wizard keeps the database alive
* for the whole import, so the worker sees a stable pointer.
*/
void setTagDatabase(std::shared_ptr<const TagDatabase> database)
{
tagDatabase = std::move(database);
}
/**
* Selects the ISO-639 language code whose foreignData is imported.
* English (the default) and unsupported codes disable localization.

3
oracle/src/oraclewizard.cpp
View file

@ -57,7 +57,8 @@ OracleWizard::OracleWizard(QWidget *parent) : QWizard(parent)
QList<OracleWizardPage *> pages;
if (!isSpoilersOnly) {
pages << new IntroPage << new LoadSetsPage << new SaveSetsPage << new LoadTokensPage << new OutroPage;
pages << new IntroPage << new LoadSetsPage << new TagsPage << new SaveSetsPage << new LoadTokensPage
<< new OutroPage;
} else {
pages << new LoadSpoilersPage << new OutroPage;
}

4
oracle/src/oraclewizard.h
View file

@ -4,6 +4,7 @@
#include <QByteArray>
#include <QString>
#include <QWizard>
#include <memory>
#include <qtmetamacros.h>
#include <utility>
@ -11,6 +12,7 @@ class QNetworkAccessManager;
class OracleImporter;
class QLocalServer;
class QSettings;
class TagDatabase;
class QWidget;
class OracleWizard : public QWizard
@ -68,6 +70,8 @@ public:
OracleImporter *importer;
QSettings *settings;
QNetworkAccessManager *nam;
/** @brief Tag data loaded by the Tags page, consumed by the importer. */
std::shared_ptr<TagDatabase> tagDatabase;
bool downloadedPlainXml = false;
QByteArray xmlData;
bool backgroundMode = false;

505
oracle/src/pages.cpp
View file

@ -6,6 +6,7 @@
#include "oracleimporter.h"
#include "oraclewizard.h"
#include "pagetemplates.h"
#include "tag_database.h"
#include "version_string.h"
#include <QBuffer>
@ -19,7 +20,9 @@
#include <QGridLayout>
#include <QIODevice>
#include <QJsonDocument>
#include <QJsonObject>
#include <QJsonParseError>
#include <QJsonValue>
#include <QLabel>
#include <QLineEdit>
#include <QList>
@ -36,6 +39,7 @@
#include <QScrollBar>
#include <QSettings>
#include <QSharedPointer>
#include <QSizePolicy>
#include <QTextEdit>
#include <QTextStream>
#include <QTimer>
@ -72,6 +76,7 @@ class QWidget;
#define MTGJSON_V4_URL_COMPONENT "mtgjson.com/files/"
#define ALLSETS_URL_FALLBACK "https://www.mtgjson.com/api/v5/AllPrintings.json"
#define MTGJSON_VERSION_URL "https://www.mtgjson.com/api/v5/Meta.json"
#define SCRYFALL_TAGS_URL "https://api.scryfall.com/bulk-data/oracle-tags"
#ifdef HAS_LZMA
#define ALLSETS_URL "https://www.mtgjson.com/api/v5/AllPrintings.json.xz"
@ -707,6 +712,502 @@ void LoadSetsPage::importFinished()
});
}
namespace
{
/**
* @brief True when a URL points at a tag file rather than Scryfall's bulk metadata.
*/
bool looksLikeTagFileUrl(const QUrl &url)
{
const QString path = url.path().toLower();
return path.endsWith(".json") || path.endsWith(".jsonl") || path.endsWith(".gz") || path.endsWith(".xz") ||
path.endsWith(".zip");
}
/**
* @brief Extends Qt's terse network error with a JSON error body.
*
* Scryfall explains a rejected request in the response body (for example a
* missing Accept header); QNetworkReply::errorString() only says "status code
* 400". Non-JSON bodies (CDN HTML pages) are ignored so the dialog stays short.
*/
QString networkErrorDetail(QNetworkReply *reply)
{
QString detail = reply->errorString();
const QString body = QString::fromUtf8(reply->readAll()).trimmed();
if (body.startsWith('{') && body.size() <= 300) {
detail += QStringLiteral(" ") + body;
}
return detail;
}
} // namespace
TagsPage::TagsPage(QWidget *parent) : OracleWizardPage(parent)
{
enableCheckBox = new QCheckBox(this);
enableCheckBox->setChecked(true);
connect(enableCheckBox, &QCheckBox::toggled, this, &TagsPage::updateSourceEnabled);
urlRadioButton = new QRadioButton(this);
fileRadioButton = new QRadioButton(this);
urlRadioButton->setChecked(true);
connect(urlRadioButton, &QRadioButton::toggled, this, &TagsPage::updateSourceEnabled);
connect(fileRadioButton, &QRadioButton::toggled, this, &TagsPage::updateSourceEnabled);
urlLineEdit = new QLineEdit(this);
fileLineEdit = new QLineEdit(this);
urlButton = new QPushButton(this);
connect(urlButton, &QPushButton::clicked, this, &TagsPage::actRestoreDefaultSource);
fileButton = new QPushButton(this);
connect(fileButton, &QPushButton::clicked, this, &TagsPage::actChooseFile);
parentsCheckBox = new QCheckBox(this);
parentsExplainer = new QLabel(this);
parentsExplainer->setWordWrap(true);
// Indent under the checkbox and give the wrapped text room above and below,
// so it reads as an explanation of the option rather than a caption glued
// to the controls around it.
parentsExplainer->setContentsMargins(24, 4, 0, 10);
// A word-wrapped label reports a one-line sizeHint, so a short page squeezes
// it and clips the text top and bottom. Pinning the vertical policy makes
// the wrapped height part of the page's minimum size.
parentsExplainer->setSizePolicy(QSizePolicy::Preferred, QSizePolicy::Minimum);
statusLabel = new QLabel(this);
statusLabel->setWordWrap(true);
statusLabel->setTextInteractionFlags(Qt::TextSelectableByMouse);
statusLabel->setSizePolicy(QSizePolicy::Preferred, QSizePolicy::Minimum);
progressLabel = new QLabel(this);
progressBar = new QProgressBar(this);
progressLabel->hide();
progressBar->hide();
auto *layout = new QGridLayout(this);
layout->addWidget(enableCheckBox, 0, 0, 1, 2);
layout->addWidget(urlRadioButton, 1, 0);
layout->addWidget(urlLineEdit, 1, 1);
layout->addWidget(urlButton, 2, 1, Qt::AlignRight);
layout->addWidget(fileRadioButton, 3, 0);
layout->addWidget(fileLineEdit, 3, 1);
layout->addWidget(fileButton, 4, 1, Qt::AlignRight);
layout->addWidget(parentsCheckBox, 5, 0, 1, 2);
layout->addWidget(parentsExplainer, 6, 0, 1, 2);
layout->addWidget(statusLabel, 7, 0, 1, 2);
layout->addWidget(progressLabel, 8, 0);
layout->addWidget(progressBar, 8, 1);
layout->setRowStretch(9, 1);
setLayout(layout);
connect(&watcher, &QFutureWatcher<LoadResult>::finished, this, &TagsPage::actLoadFinished);
updateSourceEnabled();
retranslateUi();
}
void TagsPage::retranslateUi()
{
setTitle(tr("Tag data"));
setSubTitle(tr("Card tags let you search and filter by tag. Oracle downloads Scryfall's Oracle Tags by "
"default; you can also point it at your own tag database."));
enableCheckBox->setText(tr("Download card tags"));
enableCheckBox->setToolTip(
tr("Apply tag slugs to the card database so cards can be searched and filtered by tag."));
urlRadioButton->setText(tr("Download URL:"));
urlButton->setText(tr("Restore default URL"));
fileRadioButton->setText(tr("Local file:"));
fileButton->setText(tr("Choose file..."));
parentsCheckBox->setText(tr("Include parent tags"));
parentsCheckBox->setToolTip(tr("Also apply the broader tags that this tag belongs to."));
parentsExplainer->setText(
tr("Tags form a hierarchy: \"tutor-creature-giant\" belongs to \"tutor-creature\", which belongs to "
"\"tutor\". With this on, a card tagged with the specific tag also matches searches for its broader "
"parent tags. It adds many more tags to each card."));
}
bool TagsPage::isComplete() const
{
return !loadActive;
}
void TagsPage::initializePage()
{
enableCheckBox->setChecked(wizard()->settings->value("tagsenabled", true).toBool());
parentsCheckBox->setChecked(wizard()->settings->value("tagsparents", false).toBool());
const QString source = wizard()->settings->value("tagssource").toString();
if (source.isEmpty() || source.startsWith("http://") || source.startsWith("https://")) {
urlRadioButton->setChecked(true);
urlLineEdit->setText(source.isEmpty() ? QString(SCRYFALL_TAGS_URL) : source);
fileLineEdit->clear();
} else {
fileRadioButton->setChecked(true);
fileLineEdit->setText(source);
urlLineEdit->setText(SCRYFALL_TAGS_URL);
}
statusLabel->clear();
progressLabel->hide();
progressBar->hide();
updateSourceEnabled();
if (wizard()->backgroundMode && isEnabled()) {
// In background mode runInBackground() calls initializePage() instead of
// next(), so a page that has nothing to do must advance itself.
if (validatePage()) {
emit readyToContinue();
}
}
}
QString TagsPage::selectedSource() const
{
return urlRadioButton->isChecked() ? urlLineEdit->text().trimmed() : fileLineEdit->text().trimmed();
}
void TagsPage::updateSourceEnabled()
{
const bool enabled = enableCheckBox->isChecked();
const bool usingUrl = urlRadioButton->isChecked();
const bool usingFile = fileRadioButton->isChecked();
urlRadioButton->setEnabled(enabled);
fileRadioButton->setEnabled(enabled);
urlLineEdit->setEnabled(enabled && usingUrl);
urlButton->setEnabled(enabled && usingUrl);
fileLineEdit->setEnabled(enabled && usingFile);
fileButton->setEnabled(enabled && usingFile);
parentsCheckBox->setEnabled(enabled);
parentsExplainer->setEnabled(enabled);
}
void TagsPage::saveSettings()
{
wizard()->settings->setValue("tagsenabled", enableCheckBox->isChecked());
wizard()->settings->setValue("tagsparents", parentsCheckBox->isChecked());
QString source = selectedSource();
if (urlRadioButton->isChecked() && source == SCRYFALL_TAGS_URL) {
source.clear();
}
if (source.isEmpty()) {
wizard()->settings->remove("tagssource");
} else {
wizard()->settings->setValue("tagssource", source);
}
}
bool TagsPage::validatePage()
{
if (!enableCheckBox->isChecked()) {
wizard()->tagDatabase.reset();
saveSettings();
return true;
}
const auto rejectValidation = [this](const QString &message) {
if (wizard()->backgroundMode) {
qWarning() << message;
wizard()->tagDatabase.reset();
emit readyToContinue();
} else {
QMessageBox::critical(this, tr("Error"), message);
}
return false;
};
const QString source = selectedSource();
const bool includeParents = parentsCheckBox->isChecked();
// Going back and forth through the wizard must not reload an unchanged source.
if (wizard()->tagDatabase && source == loadedSource && includeParents == loadedParents) {
return true;
}
if (loadActive) {
return false;
}
if (urlRadioButton->isChecked()) {
const QUrl url = QUrl::fromUserInput(source);
if (!url.isValid() || url.isEmpty()) {
return rejectValidation(tr("The provided tag URL is not valid."));
}
} else if (source.isEmpty() || !QFile::exists(source)) {
return rejectValidation(tr("Please choose an existing tag database file."));
}
saveSettings();
loadedSource = source;
loadedParents = includeParents;
wizard()->disableButtons();
setEnabled(false);
loadActive = true;
emit completeChanged();
progressBar->setRange(0, 0);
progressBar->setValue(0);
progressLabel->setText(tr("Fetching tag data..."));
progressLabel->show();
progressBar->show();
statusLabel->clear();
if (urlRadioButton->isChecked()) {
const QUrl url = QUrl::fromUserInput(source);
if (looksLikeTagFileUrl(url)) {
downloadTagFile(url);
} else {
downloadMetadata(url);
}
} else {
loadFromFile(source);
}
return false;
}
void TagsPage::actChooseFile()
{
QFileDialog dialog(this, tr("Load tag database"));
dialog.setFileMode(QFileDialog::ExistingFile);
QString extensions = "*.json *.jsonl";
#ifdef HAS_ZLIB
extensions += " *.gz *.zip";
#endif
#ifdef HAS_LZMA
extensions += " *.xz";
#endif
dialog.setNameFilter(tr("Tag database (%1)").arg(extensions));
if (!fileLineEdit->text().isEmpty() && QFile::exists(fileLineEdit->text())) {
dialog.selectFile(fileLineEdit->text());
}
if (!dialog.exec()) {
return;
}
fileLineEdit->setText(dialog.selectedFiles().at(0));
fileRadioButton->setChecked(true);
updateSourceEnabled();
}
void TagsPage::actRestoreDefaultSource()
{
urlLineEdit->setText(SCRYFALL_TAGS_URL);
urlRadioButton->setChecked(true);
updateSourceEnabled();
}
void TagsPage::downloadMetadata(const QUrl &url)
{
QNetworkRequest request(url);
request.setHeader(QNetworkRequest::UserAgentHeader, QString("Cockatrice %1").arg(VERSION_STRING));
// api.scryfall.com rejects requests without an Accept header (HTTP 400);
// QNetworkAccessManager does not add one on its own.
request.setRawHeader("Accept", "application/json");
request.setAttribute(QNetworkRequest::RedirectPolicyAttribute, QNetworkRequest::NoLessSafeRedirectPolicy);
activeReply = wizard()->nam->get(request);
connect(activeReply, &QNetworkReply::finished, this, &TagsPage::actMetadataFinished);
}
void TagsPage::actMetadataFinished()
{
auto *reply = qobject_cast<QNetworkReply *>(sender());
if (!reply) {
return;
}
activeReply = nullptr;
reply->deleteLater();
if (wizard()->wasCancelled()) {
return;
}
if (reply->error() != QNetworkReply::NoError) {
failLoad(
tr("Could not fetch tag metadata from %1: %2").arg(reply->url().toString(), networkErrorDetail(reply)));
return;
}
QJsonParseError parseError{};
const QJsonDocument document = QJsonDocument::fromJson(reply->readAll(), &parseError);
if (parseError.error != QJsonParseError::NoError || !document.isObject()) {
failLoad(tr("The tag metadata could not be read."));
return;
}
const QJsonObject metadata = document.object();
QString downloadUrl = metadata.value("jsonl_download_uri").toString();
if (downloadUrl.isEmpty()) {
downloadUrl = metadata.value("download_uri").toString();
}
if (downloadUrl.isEmpty()) {
failLoad(tr("The tag data source did not provide a download URL."));
return;
}
downloadTagFile(QUrl::fromUserInput(downloadUrl));
}
void TagsPage::downloadTagFile(const QUrl &url)
{
QNetworkRequest request(url);
request.setHeader(QNetworkRequest::UserAgentHeader, QString("Cockatrice %1").arg(VERSION_STRING));
request.setRawHeader("Accept", "*/*");
request.setAttribute(QNetworkRequest::RedirectPolicyAttribute, QNetworkRequest::NoLessSafeRedirectPolicy);
activeReply = wizard()->nam->get(request);
connect(activeReply, &QNetworkReply::finished, this, &TagsPage::actTagFileFinished);
connect(activeReply, &QNetworkReply::downloadProgress, this, &TagsPage::actDownloadProgress);
}
void TagsPage::actDownloadProgress(qint64 received, qint64 total)
{
emitProgress("download", received, total);
if (total > 0) {
progressBar->setRange(0, static_cast<int>(total));
progressBar->setValue(static_cast<int>(received));
}
progressLabel->setText(tr("Downloading tag data (%1 MB)").arg(received / (1024 * 1024)));
}
void TagsPage::actTagFileFinished()
{
auto *reply = qobject_cast<QNetworkReply *>(sender());
if (!reply) {
return;
}
activeReply = nullptr;
reply->deleteLater();
if (wizard()->wasCancelled()) {
return;
}
if (reply->error() != QNetworkReply::NoError) {
failLoad(tr("Could not download tag data from %1: %2").arg(reply->url().toString(), networkErrorDetail(reply)));
return;
}
const QByteArray data = reply->readAll();
if (data.isEmpty()) {
failLoad(tr("The tag download was empty."));
return;
}
beginLoad(data);
}
void TagsPage::loadFromFile(const QString &path)
{
QFile file(path);
if (!file.open(QIODevice::ReadOnly)) {
failLoad(tr("Cannot open file '%1'.").arg(path));
return;
}
beginLoad(file.readAll());
}
void TagsPage::beginLoad(const QByteArray &data)
{
progressBar->setRange(0, 0);
progressLabel->setText(tr("Reading tag data..."));
const bool includeParents = parentsCheckBox->isChecked();
future = QtConcurrent::run([data, includeParents]() {
LoadResult result;
result.database = TagDatabase::loadFromData(data, TagDatabase::LoadOptions{.includeParentTags = includeParents},
&result.errorMessage);
return result;
});
watcher.setFuture(future);
}
void TagsPage::actLoadFinished()
{
if (wizard()->wasCancelled()) {
return;
}
const LoadResult result = watcher.result();
loadActive = false;
emit completeChanged();
wizard()->enableButtons();
setEnabled(true);
progressLabel->hide();
progressBar->hide();
if (!result.database) {
wizard()->tagDatabase.reset();
failLoad(result.errorMessage.isEmpty() ? tr("The tag data could not be read.") : result.errorMessage);
return;
}
wizard()->tagDatabase = result.database;
statusLabel->setText(tr("Loaded %1 tags for %2 cards.")
.arg(wizard()->tagDatabase->tagCount())
.arg(wizard()->tagDatabase->taggedCardCount()));
qInfo() << "Loaded tag database:" << wizard()->tagDatabase->tagCount() << "tags for"
<< wizard()->tagDatabase->taggedCardCount() << "cards";
advance();
}
void TagsPage::failLoad(const QString &message)
{
loadActive = false;
emit completeChanged();
wizard()->enableButtons();
setEnabled(true);
progressLabel->hide();
progressBar->hide();
qWarning() << message;
// The failed load leaves no usable database; never keep tags from a
// previous, different source.
wizard()->tagDatabase.reset();
if (wizard()->backgroundMode) {
statusLabel->setText(message);
emit readyToContinue();
return;
}
// Do not advance here. Advancing runs validatePage() again, which starts
// the same failing download and traps the user in an error loop. Staying
// on the page lets them fix the source, retry with Next, or opt out.
const QString guidance = tr("You can try again, choose another source, or untick \"Download card tags\" "
"to import without tags.");
statusLabel->setText(message + "\n\n" + guidance);
QMessageBox::warning(this, tr("Tags not loaded"), message + "\n\n" + guidance);
}
void TagsPage::advance()
{
if (wizard()->backgroundMode) {
emit readyToContinue();
} else {
wizard()->next();
}
}
void TagsPage::cancelWork()
{
if (activeReply) {
activeReply->abort();
activeReply = nullptr;
}
if (future.isRunning()) {
future.cancel();
watcher.cancel();
future.waitForFinished();
}
}
SaveSetsPage::SaveSetsPage(QWidget *parent) : OracleWizardPage(parent)
{
pathLabel = new QLabel(this);
@ -770,6 +1271,10 @@ void SaveSetsPage::initializePage()
Qt::UniqueConnection);
connect(&importWatcher, &QFutureWatcher<int>::finished, this, &SaveSetsPage::importFinished, Qt::UniqueConnection);
// The Tags page has already loaded (or deliberately skipped) the tag data.
// The wizard keeps it alive past this page; the importer only borrows it.
wizard()->importer->setTagDatabase(wizard()->tagDatabase);
wizard()->disableButtons();
importActive = true;
emit completeChanged();

70
oracle/src/pages.h
View file

@ -6,21 +6,25 @@
#include <QByteArray>
#include <QFuture>
#include <QFutureWatcher>
#include <QPointer>
#include <QString>
#include <QStringList>
#include <QtGlobal>
#include <memory>
#include <qtmetamacros.h>
class QCheckBox;
class QComboBox;
class QLabel;
class QLineEdit;
class QNetworkReply;
class QRadioButton;
class QProgressBar;
class QTextEdit;
class QPushButton;
class QUrl;
class QWidget;
class TagDatabase;
class IntroPage : public OracleWizardPage
{
@ -109,6 +113,72 @@ private slots:
void zipDownloadFailed(const QString &message);
};
/**
* @brief Optional wizard step that loads the tag database used by the importer.
*
* Reloading is skipped when the source and parent-tag setting are unchanged, so
* going back and forth through the wizard does not re-download the tag data.
*/
class TagsPage : public OracleWizardPage
{
Q_OBJECT
public:
explicit TagsPage(QWidget *parent = nullptr);
void retranslateUi() override;
bool isComplete() const override;
protected:
void initializePage() override;
bool validatePage() override;
void cancelWork() override;
private:
struct LoadResult
{
std::shared_ptr<TagDatabase> database;
QString errorMessage;
};
QCheckBox *enableCheckBox;
QRadioButton *urlRadioButton;
QRadioButton *fileRadioButton;
QLineEdit *urlLineEdit;
QLineEdit *fileLineEdit;
QPushButton *urlButton;
QPushButton *fileButton;
QCheckBox *parentsCheckBox;
QLabel *parentsExplainer;
QLabel *statusLabel;
QLabel *progressLabel;
QProgressBar *progressBar;
QFutureWatcher<LoadResult> watcher;
QFuture<LoadResult> future;
QPointer<QNetworkReply> activeReply;
bool loadActive = false;
QString loadedSource;
bool loadedParents = false;
[[nodiscard]] QString selectedSource() const;
void updateSourceEnabled();
void saveSettings();
void beginLoad(const QByteArray &data);
void loadFromFile(const QString &path);
void downloadMetadata(const QUrl &url);
void downloadTagFile(const QUrl &url);
void finishLoad();
void failLoad(const QString &message);
void advance();
private slots:
void actChooseFile();
void actRestoreDefaultSource();
void actDownloadProgress(qint64 received, qint64 total);
void actMetadataFinished();
void actTagFileFinished();
void actLoadFinished();
};
class SaveSetsPage : public OracleWizardPage
{
Q_OBJECT

643
oracle/src/tag_database.cpp Normal file
View file

@ -0,0 +1,643 @@
#include "tag_database.h"
#include <QBuffer>
#include <QJsonArray>
#include <QJsonDocument>
#include <QJsonObject>
#include <QJsonParseError>
#include <QJsonValue>
#include <QList>
#include <QSet>
#include <QUuid>
#ifdef HAS_LZMA
#include "lzma/decompress.h"
#endif
#ifdef HAS_ZLIB
#include "zip/unzip.h"
#include <zlib.h>
#endif
#include <utility>
namespace
{
using TagSets = QHash<QString, QSet<QString>>;
struct ParsedTag
{
QString slug;
QStringList parentIds;
QList<QString> oracleIds;
};
struct ParseContext
{
bool includeParentTags = false;
QHash<QString, ParsedTag> tagsById;
TagSets oracleTags;
TagSets nameTags;
};
const QByteArray &gzipMagic()
{
static const QByteArray magic = QByteArray::fromHex("1f8b");
return magic;
}
const QByteArray &xzMagic()
{
static const QByteArray magic = QByteArray::fromHex("fd377a585a00");
return magic;
}
const QByteArray &zipMagic()
{
static const QByteArray magic = QByteArray::fromHex("504b0304");
return magic;
}
QString normalizedSlug(const QString &slug)
{
return slug.trimmed().toLower();
}
/**
* @brief Rejects slugs that cannot survive the space-separated `tags` property.
*/
bool isValidSlug(const QString &slug)
{
if (slug.isEmpty()) {
return false;
}
for (const QChar &character : slug) {
if (character.isSpace()) {
return false;
}
}
return true;
}
QString normalizedName(const QString &name)
{
return name.trimmed().toLower();
}
/**
* @brief True when a map key has the shape of a Scryfall oracle id.
*
* A manifest with an explicit `key` removes the ambiguity for card names that
* happen to look like UUIDs.
*/
bool looksLikeOracleId(const QString &key)
{
return !QUuid::fromString(key.trimmed()).isNull();
}
void addTags(TagSets &target, const QString &key, const QStringList &slugs)
{
// Both key spaces are case-insensitive: Scryfall UUIDs are hex, and card
// names must match regardless of the source's capitalization.
const QString normalizedKey = key.trimmed().toLower();
if (normalizedKey.isEmpty()) {
return;
}
QSet<QString> &tags = target[normalizedKey];
for (const QString &rawSlug : slugs) {
const QString slug = normalizedSlug(rawSlug);
if (isValidSlug(slug)) {
tags.insert(slug);
}
}
if (tags.isEmpty()) {
target.remove(normalizedKey);
}
}
#ifdef HAS_ZLIB
bool decompressGzip(const QByteArray &input, QByteArray &output, QString *errorMessage)
{
z_stream stream{};
if (inflateInit2(&stream, 16 + MAX_WBITS) != Z_OK) {
if (errorMessage) {
*errorMessage = TagDatabase::tr("Gzip support could not be initialized.");
}
return false;
}
stream.next_in = reinterpret_cast<Bytef *>(const_cast<char *>(input.constData()));
stream.avail_in = static_cast<uInt>(input.size());
int result = Z_OK;
char buffer[64 * 1024];
while (result == Z_OK) {
stream.next_out = reinterpret_cast<Bytef *>(buffer);
stream.avail_out = sizeof(buffer);
result = inflate(&stream, Z_NO_FLUSH);
const uInt produced = static_cast<uInt>(sizeof(buffer)) - stream.avail_out;
if (produced > 0) {
output.append(buffer, static_cast<int>(produced));
}
if (result != Z_OK && result != Z_STREAM_END) {
inflateEnd(&stream);
if (errorMessage) {
*errorMessage = TagDatabase::tr("Gzip decompression failed.");
}
return false;
}
if (produced == 0 && stream.avail_in == 0 && result == Z_OK) {
inflateEnd(&stream);
if (errorMessage) {
*errorMessage = TagDatabase::tr("The gzip stream is truncated.");
}
return false;
}
}
inflateEnd(&stream);
return result == Z_STREAM_END;
}
#endif
#ifdef HAS_LZMA
bool decompressXz(const QByteArray &input, QByteArray &output, QString *errorMessage)
{
QByteArray inputCopy = input;
QBuffer inputBuffer(&inputCopy);
QBuffer outputBuffer(&output);
inputBuffer.open(QBuffer::ReadOnly);
outputBuffer.open(QBuffer::WriteOnly);
XzDecompressor decompressor;
if (!decompressor.decompress(&inputBuffer, &outputBuffer)) {
if (errorMessage) {
*errorMessage = TagDatabase::tr("Xz decompression failed.");
}
return false;
}
return true;
}
#endif
#ifdef HAS_ZLIB
bool decompressZip(const QByteArray &input, QByteArray &output, QString *errorMessage)
{
QByteArray inputCopy = input;
QBuffer inputBuffer(&inputCopy);
UnZip archive;
const UnZip::ErrorCode openError = archive.openArchive(&inputBuffer);
if (openError != UnZip::Ok) {
if (errorMessage) {
*errorMessage = TagDatabase::tr("Failed to open the Zip archive: %1.").arg(archive.formatError(openError));
}
return false;
}
const QStringList files = archive.fileList();
if (files.size() != 1) {
archive.closeArchive();
if (errorMessage) {
*errorMessage = TagDatabase::tr("The Zip archive must contain exactly one tag database file.");
}
return false;
}
QBuffer outputBuffer(&output);
outputBuffer.open(QBuffer::ReadWrite);
const UnZip::ErrorCode extractError = archive.extractFile(files.first(), &outputBuffer);
archive.closeArchive();
if (extractError != UnZip::Ok) {
if (errorMessage) {
*errorMessage = TagDatabase::tr("Zip extraction failed: %1.").arg(archive.formatError(extractError));
}
return false;
}
return true;
}
#endif
/**
* @brief Unwraps gzip/xz/zip payloads, including nested archives.
*/
bool decompressData(QByteArray &data, QString *errorMessage)
{
// Nested compression is legitimate (a zip around a gzip), but a handful of
// layers is enough; the cap keeps a hostile file from looping forever.
for (int layer = 0; layer < 4; ++layer) {
if (data.startsWith(gzipMagic())) {
#ifdef HAS_ZLIB
QByteArray decompressed;
if (!decompressGzip(data, decompressed, errorMessage)) {
return false;
}
data = std::move(decompressed);
continue;
#else
if (errorMessage) {
*errorMessage = TagDatabase::tr("This version of Oracle does not support gzip compressed files.");
}
return false;
#endif
}
if (data.startsWith(xzMagic())) {
#ifdef HAS_LZMA
QByteArray decompressed;
if (!decompressXz(data, decompressed, errorMessage)) {
return false;
}
data = std::move(decompressed);
continue;
#else
if (errorMessage) {
*errorMessage = TagDatabase::tr("This version of Oracle does not support xz compressed files.");
}
return false;
#endif
}
if (data.startsWith(zipMagic())) {
#ifdef HAS_ZLIB
QByteArray decompressed;
if (!decompressZip(data, decompressed, errorMessage)) {
return false;
}
data = std::move(decompressed);
continue;
#else
if (errorMessage) {
*errorMessage = TagDatabase::tr("This version of Oracle does not support zipped files.");
}
return false;
#endif
}
return true;
}
if (errorMessage) {
*errorMessage = TagDatabase::tr("The tag database is compressed in too many layers.");
}
return false;
}
bool parseKeyedMap(const QJsonObject &map, const QString &explicitKey, ParseContext &context, QString *errorMessage)
{
if (map.isEmpty()) {
if (errorMessage) {
*errorMessage = TagDatabase::tr("The tag database contains no tag entries.");
}
return false;
}
for (auto it = map.constBegin(); it != map.constEnd(); ++it) {
if (!it.value().isArray()) {
if (errorMessage) {
*errorMessage = TagDatabase::tr("The tag list for \"%1\" must be an array of slugs.").arg(it.key());
}
return false;
}
QStringList slugs;
for (const QJsonValue &value : it.value().toArray()) {
if (!value.isString()) {
if (errorMessage) {
*errorMessage =
TagDatabase::tr("The tag list for \"%1\" must contain only slug strings.").arg(it.key());
}
return false;
}
slugs.append(value.toString());
}
const bool oracleKey = explicitKey.isEmpty() ? looksLikeOracleId(it.key())
: (explicitKey == "oracle_id" || explicitKey == "oracleid");
if (oracleKey) {
addTags(context.oracleTags, it.key().trimmed(), slugs);
} else {
addTags(context.nameTags, normalizedName(it.key()), slugs);
}
}
return true;
}
bool parseTagRow(const QJsonObject &object, ParseContext &context)
{
QStringList slugs;
for (const QJsonValue &value : object.value("tags").toArray()) {
if (value.isString()) {
slugs.append(value.toString());
}
}
const QString oracleId = object.value("oracle_id").toString().trimmed();
if (!oracleId.isEmpty()) {
addTags(context.oracleTags, oracleId, slugs);
}
const QString name = object.value("name").toString().trimmed();
if (!name.isEmpty()) {
addTags(context.nameTags, normalizedName(name), slugs);
}
return true;
}
bool parseTagObject(const QJsonObject &object, ParseContext &context)
{
// Art and illustration tags describe a printing's artwork, not the oracle
// card, so they never belong in the card-level tag property.
const QString type = object.value("type").toString().trimmed().toLower();
if (!type.isEmpty() && type != "oracle") {
return true;
}
const QString slug = normalizedSlug(object.value("slug").toString());
if (!isValidSlug(slug)) {
return true;
}
QList<QString> oracleIds;
for (const QJsonValue &value : object.value("taggings").toArray()) {
const QString oracleId = value.toObject().value("oracle_id").toString().trimmed();
if (!oracleId.isEmpty() && !oracleIds.contains(oracleId)) {
oracleIds.append(oracleId);
}
}
const QString id = object.value("id").toString().trimmed();
if (!context.includeParentTags || id.isEmpty()) {
for (const QString &oracleId : oracleIds) {
addTags(context.oracleTags, oracleId, QStringList{slug});
}
return true;
}
ParsedTag &tag = context.tagsById[id];
if (tag.slug.isEmpty()) {
tag.slug = slug;
}
for (const QJsonValue &parent : object.value("parent_ids").toArray()) {
const QString parentId = parent.toString().trimmed();
if (!parentId.isEmpty() && !tag.parentIds.contains(parentId)) {
tag.parentIds.append(parentId);
}
}
for (const QString &oracleId : oracleIds) {
if (!tag.oracleIds.contains(oracleId)) {
tag.oracleIds.append(oracleId);
}
}
return true;
}
bool handleObject(const QJsonObject &object, ParseContext &context, QString *errorMessage)
{
if (object.contains("slug")) {
return parseTagObject(object, context);
}
const QJsonValue tagsValue = object.value("tags");
if (tagsValue.isArray() && (object.contains("oracle_id") || object.contains("name"))) {
return parseTagRow(object, context);
}
if (tagsValue.isObject()) {
const QString key = object.value("key").toString().trimmed().toLower();
return parseKeyedMap(tagsValue.toObject(), key, context, errorMessage);
}
bool allArrays = !object.isEmpty();
for (auto it = object.constBegin(); it != object.constEnd(); ++it) {
if (!it.value().isArray()) {
allArrays = false;
break;
}
}
if (allArrays) {
return parseKeyedMap(object, QString(), context, errorMessage);
}
if (errorMessage) {
*errorMessage = TagDatabase::tr("Unrecognized tag database format.");
}
return false;
}
bool parseJsonArray(const QJsonArray &array, ParseContext &context, QString *errorMessage)
{
if (array.isEmpty()) {
if (errorMessage) {
*errorMessage = TagDatabase::tr("The tag database contains no tag entries.");
}
return false;
}
for (const QJsonValue &value : array) {
if (!value.isObject()) {
if (errorMessage) {
*errorMessage = TagDatabase::tr("The tag database array must contain only objects.");
}
return false;
}
if (!handleObject(value.toObject(), context, errorMessage)) {
return false;
}
}
return true;
}
bool parseJsonLines(const QByteArray &payload, ParseContext &context, QString *errorMessage)
{
const QList<QByteArray> lines = payload.split('\n');
bool parsedAny = false;
for (int index = 0; index < lines.size(); ++index) {
const QByteArray line = lines.at(index).trimmed();
if (line.isEmpty()) {
continue;
}
QJsonParseError lineError{};
const QJsonDocument lineDocument = QJsonDocument::fromJson(line, &lineError);
if (lineError.error != QJsonParseError::NoError || !lineDocument.isObject()) {
if (errorMessage) {
*errorMessage = TagDatabase::tr("Malformed tag database at line %1: %2")
.arg(index + 1)
.arg(lineError.errorString());
}
return false;
}
if (!handleObject(lineDocument.object(), context, errorMessage)) {
return false;
}
parsedAny = true;
}
if (!parsedAny) {
if (errorMessage) {
*errorMessage = TagDatabase::tr("The tag database contains no tag entries.");
}
return false;
}
return true;
}
QHash<QString, QStringList> sortedLists(const TagSets &sets)
{
QHash<QString, QStringList> result;
result.reserve(sets.size());
for (auto it = sets.constBegin(); it != sets.constEnd(); ++it) {
if (it.value().isEmpty()) {
continue;
}
QStringList slugs = it.value().values();
slugs.sort();
result.insert(it.key(), slugs);
}
return result;
}
} // namespace
std::shared_ptr<TagDatabase>
TagDatabase::loadFromData(const QByteArray &data, const LoadOptions &options, QString *errorMessage)
{
if (errorMessage) {
errorMessage->clear();
}
if (data.trimmed().isEmpty()) {
if (errorMessage) {
*errorMessage = tr("The tag database is empty.");
}
return nullptr;
}
QByteArray payload = data;
if (!decompressData(payload, errorMessage)) {
return nullptr;
}
if (payload.trimmed().isEmpty()) {
if (errorMessage) {
*errorMessage = tr("The tag database is empty.");
}
return nullptr;
}
ParseContext context;
context.includeParentTags = options.includeParentTags;
QJsonParseError parseError{};
const QJsonDocument document = QJsonDocument::fromJson(payload, &parseError);
if (parseError.error == QJsonParseError::NoError) {
if (document.isArray()) {
if (!parseJsonArray(document.array(), context, errorMessage)) {
return nullptr;
}
} else if (document.isObject()) {
if (!handleObject(document.object(), context, errorMessage)) {
return nullptr;
}
} else {
if (errorMessage) {
*errorMessage = tr("The tag database is not a JSON object or array.");
}
return nullptr;
}
} else if (payload.trimmed().startsWith('[')) {
// A broken array is not JSONL; report the parser error instead of
// blaming an arbitrary line.
if (errorMessage) {
*errorMessage = tr("Malformed tag database: %1").arg(parseError.errorString());
}
return nullptr;
} else if (!parseJsonLines(payload, context, errorMessage)) {
return nullptr;
}
if (context.includeParentTags) {
// Expand upward only after every tag is known: a parent object may
// appear after its children in the file.
for (auto it = context.tagsById.constBegin(); it != context.tagsById.constEnd(); ++it) {
QStringList slugs{it->slug};
QSet<QString> visited;
QList<QString> pending = it->parentIds;
while (!pending.isEmpty()) {
const QString parentId = pending.takeLast();
if (visited.contains(parentId)) {
continue;
}
visited.insert(parentId);
const auto parent = context.tagsById.constFind(parentId);
if (parent == context.tagsById.constEnd()) {
continue;
}
slugs.append(parent->slug);
pending.append(parent->parentIds);
}
for (const QString &oracleId : it->oracleIds) {
addTags(context.oracleTags, oracleId, slugs);
}
}
}
auto database = std::make_shared<TagDatabase>();
database->tagsByOracleId = sortedLists(context.oracleTags);
database->tagsByName = sortedLists(context.nameTags);
QSet<QString> distinctSlugs;
for (const QStringList &slugs : database->tagsByOracleId) {
for (const QString &slug : slugs) {
distinctSlugs.insert(slug);
}
}
for (const QStringList &slugs : database->tagsByName) {
for (const QString &slug : slugs) {
distinctSlugs.insert(slug);
}
}
database->distinctTagCount = distinctSlugs.size();
if (database->isEmpty()) {
if (errorMessage) {
*errorMessage = tr("The tag database contains no usable tags.");
}
return nullptr;
}
return database;
}
QStringList TagDatabase::tagsForCard(const QString &oracleId, const QString &name) const
{
QStringList result;
if (!oracleId.isEmpty()) {
result = tagsByOracleId.value(oracleId.trimmed().toLower());
}
const QString normalizedCardName = normalizedName(name);
if (!normalizedCardName.isEmpty()) {
const QStringList byName = tagsByName.value(normalizedCardName);
for (const QString &slug : byName) {
if (!result.contains(slug)) {
result.append(slug);
}
}
}
if (!result.isEmpty()) {
result.sort();
}
return result;
}
bool TagDatabase::isEmpty() const
{
return tagsByOracleId.isEmpty() && tagsByName.isEmpty();
}

69
oracle/src/tag_database.h Normal file
View file

@ -0,0 +1,69 @@
#ifndef TAG_DATABASE_H
#define TAG_DATABASE_H
#include <QByteArray>
#include <QCoreApplication>
#include <QHash>
#include <QString>
#include <QStringList>
#include <memory>
/**
* @brief Tag slugs keyed by Scryfall oracle id and/or card name.
*
* Loaded from Scryfall's Oracle Tags bulk data or from a user-supplied tag
* database. The data is import-time only: Oracle applies the slugs to the
* `tags` card property before writing cards.xml, so the client needs no
* tag-specific code.
*
* Accepted inputs, auto-detected:
* - Scryfall tag objects, either a JSON array or one object per JSONL line;
* - a flat map of `oracle_id` (or card name) to an array of slugs;
* - a manifest object `{"key": "oracle_id"|"name", "tags": { ... }}`;
* - JSONL rows `{"oracle_id": "...", "tags": [...]}` or the `name` equivalent.
*
* Gzip, xz and zip inputs are decompressed transparently when the build has
* the matching library. Scryfall serves its bulk data as gzipped JSONL.
*/
class TagDatabase
{
Q_DECLARE_TR_FUNCTIONS(TagDatabase)
public:
struct LoadOptions
{
bool includeParentTags = false;
};
/**
* @brief Parses and decompresses a tag database.
* @return The database, or nullptr with @p errorMessage set on failure.
*/
static std::shared_ptr<TagDatabase>
loadFromData(const QByteArray &data, const LoadOptions &options, QString *errorMessage = nullptr);
/**
* @brief Returns the sorted slugs for one card.
*
* Oracle ids are the primary key. A name-keyed database is consulted as a
* fallback, case-insensitively. An empty list means the card has no tags.
*/
[[nodiscard]] QStringList tagsForCard(const QString &oracleId, const QString &name) const;
[[nodiscard]] bool isEmpty() const;
/** @brief Number of distinct slugs in the database. */
[[nodiscard]] int tagCount() const
{
return distinctTagCount;
}
/** @brief Number of cards that received at least one tag. */
[[nodiscard]] int taggedCardCount() const
{
return tagsByOracleId.size() + tagsByName.size();
}
private:
QHash<QString, QStringList> tagsByOracleId;
QHash<QString, QStringList> tagsByName; ///< keys are lowercased
int distinctTagCount = 0;
};
#endif // TAG_DATABASE_H