S1 · DefinitionsS2 · PipelineS3 · InstructionS4 · Output schemaS5 · ValidationS6 · ReproducibilityOther registries
S1. Operational definitions of the EULAR 2021 criteria
The EULAR definition was written for clinicians, not databases. To apply it to a registry, each criterion was translated into a verifiable rule and agreed with the chief rheumatologist of the city before the analysis. Rules refined after adjudication are noted in the text.
| Criterion | Operational rule | Data source |
|---|---|---|
| A. Failure of ≥ 2 b/tsDMARDs with different mechanisms of action | Discontinuation for inefficacy or an adverse event of ≥ 2 drugs from different classes (TNFi, IL-6i, JAKi, anti-CD20, co-stimulation blockade, IL-1i) at the third or later line. Drugs, not classes, are counted: a class remains “failed” even if the patient later responds to another drug of the same class. Not counted as failure: remission, pregnancy, malignancy, death, technical or administrative reasons, reason not established. | Registry (deterministic) where coded discontinuation reasons exist; otherwise history in notes (LLM) |
| B. Signs of active or progressive disease (≥ 1 of 4) | B1: DAS28-ESR > 3.2 / CDAI > 10 / SDAI > 11. B2: current systemic manifestations from 11 categories; rheumatoid nodules count only when the text indicates new, enlarging or active pulmonary nodules, stable or historical nodules do not; Sjögren's syndrome always counts. B3: inability to taper GC < 7.5 mg/day. B4: radiographic progression. Assessed at the last visit; joint symptoms without indices are excluded from the main analysis. | Note text (LLM), every “yes” with a quotation |
| C. Management perceived as problematic | Two layers: free-text judgement on the note as a whole and a structured checklist of 8 patient-side + 17 physician-side items. Not part of the status definition; descriptive layer. | Note text (LLM), with quotation |
| Final status | D2T = A. Persistent D2T = A and B; controlled D2T = A and not B. Sensitivity analysis: A and B and C. | — |
Discontinuation reasons in the registry. Failure = “inefficacy” or “adverse event”. Not failure = remission, death, malignancy, pregnancy, technical or administrative reasons, reason not established.
S2. Processing pipeline
De-identificationNames, insurance numbers and direct identifiers are replaced by codes before any processing.
Registry chainSequence of b/tsDMARDs with dates and coded discontinuation reasons; criterion A is computed from it deterministically.
Quotation-grounded extractionThe model receives the note text and drug chain and returns JSON of a fixed schema; each positive decision carries a verbatim quotation ≤ 120 characters.
Two-model cascadeAll patients with the compact model; borderline and conflicting cases re-assessed with the larger model, whose result takes precedence.
Deterministic rulesStatus is assembled in code from the JSON fields and the registry; rules are pre-specified and versioned.
Blinded validation and adjudication200 records annotated blind by four rheumatologists; disagreements adjudicated by the lead rheumatologist with coded reasons.
Key principle: the model does not decide the status. It extracts features with quotations, and the status is assembled by transparent code under pre-specified rules. Any rule can therefore be changed and the cohort recomputed without re-querying the model, as was done when the nodule rule was refined.
S3. Model instruction (system prompt v2.1)
Reproduced verbatim in the source language (English instructions with Russian clinical terms, as the notes are in Russian). The instruction is passed as the system message; the note text and registry chain as the user message.
# ROLE
You are an expert Rheumatologist and Clinical Data Scientist adjudicating "Difficult-to-Treat Rheumatoid Arthritis" (D2TRA) per EULAR 2021, extended with a structured perception checklist (version v2.1, September 2026).
# INPUT
NOTE_TEXT: one or more clinical documents (выписки / осмотры) of ONE patient, separated by "---". Use NOTE_TEXT only. Never invent facts. Every true value must be backed by a short verbatim quote (≤120 chars) from the text. Use null when the text gives no usable evidence either way; use false for explicit negation or clear absence in the most recent visit.
# CRITERION A — treatment failure history
Failed ≥2 distinct mechanisms of action (MOA) among bDMARD/tsDMARD. MOA classes: TNF-i (адалимумаб, этанерцепт, инфликсимаб, голимумаб, цертолизумаб), IL6-i (тоцилизумаб, левилимаб, олокизумаб, сарилумаб), JAK-i (тофацитиниб, барицитиниб, упадацитиниб, филготиниб), CD20-i (ритуксимаб), T-costim (абатацепт), IL1-i (анакинра, канакинумаб). Count MOAs, not drugs (adalimumab + etanercept = 1 MOA). "Failure" = stopped/switched for inefficacy (нф, неэффективность, без эффекта, первичная/вторичная неэффективность) or adverse event (ня, непереносимость). Stopped for cost, pregnancy, administrative, remission, infection-only interruption = NOT a failure. The current drug counts as failed only if the text says it is ineffective.
Also list EVERY bDMARD/tsDMARD ever mentioned for this patient in bts_drugs with its MOA and outcome: failed_inefficacy / failed_adverse_event / stopped_other (cost, pregnancy, administrative, remission, infection interruption, unknown reason) / ongoing / unknown. One entry per drug course; a quote for each.
# CRITERION B — active / progressive disease. Sub-items are reported separately; the overall B is computed downstream (narrow: b1/b2b/b3/b4; wide: also b2a).
- b1_activity: moderate or higher activity — DAS28-ESR > 3.2 OR CDAI > 10 OR SDAI > 11 at the most recent visit with data. Report the numbers found.
- b2a_joint_signs: articular signs/symptoms of active RA at the most recent visit — explicit swollen/tender joints (ЧПС/ЧБС > 0, синовит, припухлость, болезненность при пальпации), morning stiffness ≥30 min, physician statement of "активность заболевания" without indices. Report the quote.
- b2b_systemic: any RA systemic (extra-articular) manifestation CURRENTLY present at the most recent visit: ревматоидные узелки; кожный васкулит (язвенно-некротический, инфаркты ногтевого ложа, дигитальный артериит); нейропатия (мононеврит, полинейропатия); плеврит; перикардит; миокардит; синдром Шегрена (incl. сухой синдром); поражение глаз (склерит, эписклерит, увеит); интерстициальное заболевание лёгких (ИЗЛ, НСИП, пневмонит); генерализованная миопатия; поражение системы крови как проявление РА (анемия, нейтропения, тромбоцитоз названные врачом как проявление РА). Historical mentions ("в анамнезе", "ранее") do not count. List the manifestations found with the exact key names; value=true only if the list is non-empty.
- b3_gc: inability to taper glucocorticoids below 7.5 mg/day prednisone-equivalent (метилпреднизолон 6 мг = преднизолон 7.5 мг) because of disease activity, or a stated need for ongoing GC ≥7.5 mg. Report the dose.
- b4_radiographic: rapid radiographic progression — new erosions, "прогрессирование", "отрицательная динамика" on X-ray/MRI/CT of joints, or Sharp/van der Heijde ≥5 points/year.
# CRITERION C — perception. Two independent assessments.
## C_free (as before): the physician and/or the patient explicitly perceive the disease management as problematic — "трудноконтролируемый", "рефрактерный", "проблемный", "стандартная терапия неэффективна", "сложный пациент", "не удаётся достичь ремиссии/низкой активности", repeated escalations without effect. Physician and patient statements separately.
## C_struct: for EACH item below, true if explicitly supported by the text, false if explicitly negated/absent, null if not mentioned. Quote when true.
Patient-side (P):
P1 pain VAS/ВАШ ≥ 4 (of 10) at the most recent visit.
P2 functional impairment or ankylosis (ФН/ФК III–IV, выраженные функциональные нарушения, анкилоз, self-care limitation).
P3 joint surgery of any kind (артропластика, эндопротезирование, синовэктомия, артродез).
P4 need for continuous NSAID intake (постоянный/ежедневный приём НПВП).
P5 combination therapy with more than 3 drugs for RA simultaneously (DMARDs + GC + NSAID etc. counted as antirheumatic drugs).
P6 systemic manifestations present (same list as b2b).
P7 severe kidney involvement: ХБП 4–5, амилоидоз, гемодиализ.
P8 osteoporosis with pathological (low-energy) fractures.
Physician-side (D):
D1 radiographic stage 4 (IV стадия, Штейнброкер IV).
D2 endoprosthesis or other joint surgery performed.
D3 failure of three MOA classes of bDMARD/tsDMARD (≥3 distinct failed MOAs).
D4 patient non-adherence / non-compliance (некомплаентность, пропуски, самостоятельная отмена).
D5 impossibility to withdraw glucocorticoids (невозможность отмены ГКС, зависимость от ГКС).
D6 infectious events limiting therapeutic options (recurrent/serious infections that forced stopping or avoiding DMARDs).
D7 incorrect diagnostic concept (диагноз пересмотрен, сомнения в диагнозе РА, overlap-синдром).
D8 comorbidities limiting therapeutic options (онкология, ХСН, гепатит, ХОБЛ, etc. explicitly limiting DMARD choice).
D9 systemic manifestations complicating the disease course (systemic manifestation explicitly described as complicating management).
D10 need for psychiatric consultation (депрессия, тревога, консультация психиатра/психотерапевта).
D11 polyvalent allergy (поливалентная аллергия, multiple drug allergies).
D12 therapy schedule disruptions due to infectious events (перерывы в терапии из-за инфекций, COVID-19 etc.).
D13 absence of therapy escalation per expert opinion (therapy not escalated despite activity; "рекомендована эскалация", "не проводилась смена терапии").
D14 tuberculosis history or latent TB (туберкулёз в анамнезе, ЛТИ, положительный Диаскинтест/квантиферон, профилактика изониазидом).
D15 renal amyloidosis or end-stage renal failure.
D16 secondary osteoarthritis grade 3–4 (вторичный остеоартроз 3–4 ст.).
D17 joint contractures (контрактуры).
# OUTPUT
Return JSON exactly matching the provided schema. Keep quotes short and verbatim. No prose outside JSON.
S4. Structured output schema (JSON Schema, strict)
Each feature is returned as an object {value, quote}: value ∈ {true, false, null}, quote is a verbatim excerpt from the text. A response violating the schema is rejected by the API; a positive value without a quotation is rejected by our code and re-requested.
{
"type": "object",
"properties": {
"A": {
"type": "object",
"properties": {
"value": {
"type": [
"boolean",
"null"
]
},
"quote": {
"type": [
"string",
"null"
]
},
"failed_moa": {
"type": "array",
"items": {
"type": "string",
"enum": [
"TNF-i",
"IL6-i",
"JAK-i",
"CD20-i",
"T-costim",
"IL1-i"
]
}
},
"bts_drugs": {
"type": "array",
"items": {
"type": "object",
"properties": {
"drug": {
"type": "string"
},
"moa": {
"type": "string",
"enum": [
"TNF-i",
"IL6-i",
"JAK-i",
"CD20-i",
"T-costim",
"IL1-i",
"IL17-i",
"IL23-i",
"other"
]
},
"outcome": {
"type": "string",
"enum": [
"failed_inefficacy",
"failed_adverse_event",
"stopped_other",
"ongoing",
"unknown"
]
},
"quote": {
"type": [
"string",
"null"
]
}
},
"required": [
"drug",
"moa",
"outcome",
"quote"
],
"additionalProperties": false
}
}
},
"required": [
"value",
"quote",
"failed_moa",
"bts_drugs"
],
"additionalProperties": false
},
"B": {
"type": "object",
"properties": {
"b1_activity": {
"type": "object",
"properties": {
"value": {
"type": [
"boolean",
"null"
]
},
"quote": {
"type": [
"string",
"null"
]
},
"das28": {
"type": [
"number",
"null"
]
},
"cdai": {
"type": [
"number",
"null"
]
},
"sdai": {
"type": [
"number",
"null"
]
}
},
"required": [
"value",
"quote",
"das28",
"cdai",
"sdai"
],
"additionalProperties": false
},
"b2a_joint_signs": {
"type": "object",
"properties": {
"value": {
"type": [
"boolean",
"null"
]
},
"quote": {
"type": [
"string",
"null"
]
}
},
"required": [
"value",
"quote"
],
"additionalProperties": false
},
"b2b_systemic": {
"type": "object",
"properties": {
"value": {
"type": [
"boolean",
"null"
]
},
"quote": {
"type": [
"string",
"null"
]
},
"systemic": {
"type": "array",
"items": {
"type": "string",
"enum": [
"nodules",
"vasculitis",
"neuropathy",
"pleuritis",
"pericarditis",
"myocarditis",
"sjogren",
"eye",
"ild",
"myopathy",
"hematologic"
]
}
}
},
"required": [
"value",
"quote",
"systemic"
],
"additionalProperties": false
},
"b3_gc": {
"type": "object",
"properties": {
"value": {
"type": [
"boolean",
"null"
]
},
"quote": {
"type": [
"string",
"null"
]
},
"dose_mg_pred_eq": {
"type": [
"number",
"null"
]
}
},
"required": [
"value",
"quote",
"dose_mg_pred_eq"
],
"additionalProperties": false
},
"b4_radiographic": {
"type": "object",
"properties": {
"value": {
"type": [
"boolean",
"null"
]
},
"quote": {
"type": [
"string",
"null"
]
}
},
"required": [
"value",
"quote"
],
"additionalProperties": false
}
},
"required": [
"b1_activity",
"b2a_joint_signs",
"b2b_systemic",
"b3_gc",
"b4_radiographic"
],
"additionalProperties": false
},
"C_free": {
"type": "object",
"properties": {
"physician": {
"type": "object",
"properties": {
"value": {
"type": [
"boolean",
"null"
]
},
"quote": {
"type": [
"string",
"null"
]
}
},
"required": [
"value",
"quote"
],
"additionalProperties": false
},
"patient": {
"type": "object",
"properties": {
"value": {
"type": [
"boolean",
"null"
]
},
"quote": {
"type": [
"string",
"null"
]
}
},
"required": [
"value",
"quote"
],
"additionalProperties": false
},
"value": {
"type": [
"boolean",
"null"
]
}
},
"required": [
"physician",
"patient",
"value"
],
"additionalProperties": false
},
"C_struct": {
"type": "object",
"properties": {
"P1": {
"type": "object",
"properties": {
"value": {
"type": [
"boolean",
"null"
]
},
"quote": {
"type": [
"string",
"null"
]
}
},
"required": [
"value",
"quote"
],
"additionalProperties": false
},
"P2": {
"type": "object",
"properties": {
"value": {
"type": [
"boolean",
"null"
]
},
"quote": {
"type": [
"string",
"null"
]
}
},
"required": [
"value",
"quote"
],
"additionalProperties": false
},
"P3": {
"type": "object",
"properties": {
"value": {
"type": [
"boolean",
"null"
]
},
"quote": {
"type": [
"string",
"null"
]
}
},
"required": [
"value",
"quote"
],
"additionalProperties": false
},
"P4": {
"type": "object",
"properties": {
"value": {
"type": [
"boolean",
"null"
]
},
"quote": {
"type": [
"string",
"null"
]
}
},
"required": [
"value",
"quote"
],
"additionalProperties": false
},
"P5": {
"type": "object",
"properties": {
"value": {
"type": [
"boolean",
"null"
]
},
"quote": {
"type": [
"string",
"null"
]
}
},
"required": [
"value",
"quote"
],
"additionalProperties": false
},
"P6": {
"type": "object",
"properties": {
"value": {
"type": [
"boolean",
"null"
]
},
"quote": {
"type": [
"string",
"null"
]
}
},
"required": [
"value",
"quote"
],
"additionalProperties": false
},
"P7": {
"type": "object",
"properties": {
"value": {
"type": [
"boolean",
"null"
]
},
"quote": {
"type": [
"string",
"null"
]
}
},
"required": [
"value",
"quote"
],
"additionalProperties": false
},
"P8": {
"type": "object",
"properties": {
"value": {
"type": [
"boolean",
"null"
]
},
"quote": {
"type": [
"string",
"null"
]
}
},
"required": [
"value",
"quote"
],
"additionalProperties": false
},
"D1": {
"type": "object",
"properties": {
"value": {
"type": [
"boolean",
"null"
]
},
"quote": {
"type": [
"string",
"null"
]
}
},
"required": [
"value",
"quote"
],
"additionalProperties": false
},
"D2": {
"type": "object",
"properties": {
"value": {
"type": [
"boolean",
"null"
]
},
"quote": {
"type": [
"string",
"null"
]
}
},
"required": [
"value",
"quote"
],
"additionalProperties": false
},
"D3": {
"type": "object",
"properties": {
"value": {
"type": [
"boolean",
"null"
]
},
"quote": {
"type": [
"string",
"null"
]
}
},
"required": [
"value",
"quote"
],
"additionalProperties": false
},
"D4": {
"type": "object",
"properties": {
"value": {
"type": [
"boolean",
"null"
]
},
"quote": {
"type": [
"string",
"null"
]
}
},
"required": [
"value",
"quote"
],
"additionalProperties": false
},
"D5": {
"type": "object",
"properties": {
"value": {
"type": [
"boolean",
"null"
]
},
"quote": {
"type": [
"string",
"null"
]
}
},
"required": [
"value",
"quote"
],
"additionalProperties": false
},
"D6": {
"type": "object",
"properties": {
"value": {
"type": [
"boolean",
"null"
]
},
"quote": {
"type": [
"string",
"null"
]
}
},
"required": [
"value",
"quote"
],
"additionalProperties": false
},
"D7": {
"type": "object",
"properties": {
"value": {
"type": [
"boolean",
"null"
]
},
"quote": {
"type": [
"string",
"null"
]
}
},
"required": [
"value",
"quote"
],
"additionalProperties": false
},
"D8": {
"type": "object",
"properties": {
"value": {
"type": [
"boolean",
"null"
]
},
"quote": {
"type": [
"string",
"null"
]
}
},
"required": [
"value",
"quote"
],
"additionalProperties": false
},
"D9": {
"type": "object",
"properties": {
"value": {
"type": [
"boolean",
"null"
]
},
"quote": {
"type": [
"string",
"null"
]
}
},
"required": [
"value",
"quote"
],
"additionalProperties": false
},
"D10": {
"type": "object",
"properties": {
"value": {
"type": [
"boolean",
"null"
]
},
"quote": {
"type": [
"string",
"null"
]
}
},
"required": [
"value",
"quote"
],
"additionalProperties": false
},
"D11": {
"type": "object",
"properties": {
"value": {
"type": [
"boolean",
"null"
]
},
"quote": {
"type": [
"string",
"null"
]
}
},
"required": [
"value",
"quote"
],
"additionalProperties": false
},
"D12": {
"type": "object",
"properties": {
"value": {
"type": [
"boolean",
"null"
]
},
"quote": {
"type": [
"string",
"null"
]
}
},
"required": [
"value",
"quote"
],
"additionalProperties": false
},
"D13": {
"type": "object",
"properties": {
"value": {
"type": [
"boolean",
"null"
]
},
"quote": {
"type": [
"string",
"null"
]
}
},
"required": [
"value",
"quote"
],
"additionalProperties": false
},
"D14": {
"type": "object",
"properties": {
"value": {
"type": [
"boolean",
"null"
]
},
"quote": {
"type": [
"string",
"null"
]
}
},
"required": [
"value",
"quote"
],
"additionalProperties": false
},
"D15": {
"type": "object",
"properties": {
"value": {
"type": [
"boolean",
"null"
]
},
"quote": {
"type": [
"string",
"null"
]
}
},
"required": [
"value",
"quote"
],
"additionalProperties": false
},
"D16": {
"type": "object",
"properties": {
"value": {
"type": [
"boolean",
"null"
]
},
"quote": {
"type": [
"string",
"null"
]
}
},
"required": [
"value",
"quote"
],
"additionalProperties": false
},
"D17": {
"type": "object",
"properties": {
"value": {
"type": [
"boolean",
"null"
]
},
"quote": {
"type": [
"string",
"null"
]
}
},
"required": [
"value",
"quote"
],
"additionalProperties": false
}
},
"required": [
"P1",
"P2",
"P3",
"P4",
"P5",
"P6",
"P7",
"P8",
"D1",
"D2",
"D3",
"D4",
"D5",
"D6",
"D7",
"D8",
"D9",
"D10",
"D11",
"D12",
"D13",
"D14",
"D15",
"D16",
"D17"
],
"additionalProperties": false
}
},
"required": [
"A",
"B",
"C_free",
"C_struct"
],
"additionalProperties": false
}
S5. Blinded validation and adjudication
Annotation form
The annotator sees the note text and a de-identified registry extract; model output is hidden and access to the rest of the system is blocked for the duration. Records are presented in shuffled order; the stratum is not shown.
| A | Criterion met? (yes / no / cannot determine) + failed classes |
| B1–B4 | Four items separately, each yes / no / cannot determine; DAS28 and GC dose fields |
| Systemic | Checkboxes for 11 categories |
| C | Free-text judgement; explicit physician wording present; 8 + 17 checklist |
| Outcome | Status: persistent D2T / controlled D2T / not D2T; documentation quality; comment |
Adjudication form
For each discordant record the model's assessment with quotations and the expert's answers are shown side by side with discordant items highlighted. Verdict: model correct / expert correct / other status. The reason is coded from a list: drug only in history (absent from registry); short course; adverse-event stop; non-medical switch; clinical-trial drug; two drugs of the same class; activity during interruption; diagnosis other than RA; insufficient data; model misread the text; expert annotation error.
Sample of 200 records: stratified by model status and data source (cohort 1 — 35 persistent / 25 controlled / 40 not D2T; cohort 2 — 25 / 25 / 50), fixed random seed. Four rheumatologists, 50 records each. Metrics: agreement, sensitivity, specificity, PPV/NPV with Wilson 95% CIs and Cohen's κ, per criterion and for the final status, before and after adjudication.
S6. Run parameters and reproducibility
| Models | GPT-5.6 Luna (all patients) · GPT-5.6 Terra (1,232 borderline) · OpenAI Responses API |
| Mode | zero-shot; reasoning effort = medium; strict JSON schema; verbosity = low |
| Calibration | 80 records manually assessed by the lead rheumatologist in a pilot; instructions frozen afterwards |
| Volume | 2,748 patients, 3,299 documents; median length 6,890 (cohort 1) and 11,720 (cohort 2) characters |
| Processing time | under one day for the whole cohort |
| Rule version | v2.1 (September 2026) + B2 nodule refinement of 1 October 2026 |
Individual patient data are not shared. Aggregate tables, status-assembly code and forms are available from the authors on request.
The same method for your registry
MAG applies this approach to registries and clinical-record collections in rheumatology and beyond: operationalisation of criteria together with the clinical lead, quotation-grounded extraction, blinded validation and manuscript preparation. The main work is methodological: rules, validation and interpretation.
- Any criteria: D2T, remission, phenotypes, adverse events, discontinuation reasons
- Russian, English and other note languages
- Data stay with the owner; work on de-identified text
- Web validation interface for your experts