?review=1 appended to the URL, click the “💬” button at the bottom right and select the passage. Items marked [to confirm] need an authors' decision.1 Moscow Healthcare Department, Moscow, Russia · 2 Pirogov Russian National Research Medical University, Department of General Therapy, Moscow, Russia · 3 Medical Advisers Group (MAG), [city/country]
Objectives. To estimate the prevalence of difficult-to-treat rheumatoid arthritis (D2T RA) in a population-based registry of patients receiving biologic (b) and targeted synthetic (ts) DMARDs, using large language models (LLMs) to extract the EULAR criteria from unstructured clinical notes, and to validate this approach against blinded rheumatologist review.
Methods. Retrospective study of the unified b/tsDMARD registry of Moscow (2,785 patients from three rheumatology centres, status as of 31 December 2024). In 2,748 patients with available notes, criterion A (failure of ≥2 b/tsDMARD classes) was derived deterministically from the registry drug sequence and coded discontinuation reasons (n = 2,068) or extracted from the text where reasons were not coded (n = 680); criterion B (current activity: composite indices, systemic manifestations, glucocorticoids ≥7.5 mg/day, radiographic progression) and criterion C (management perceived as problematic: free-text judgement and a 25-item checklist) were extracted by LLMs, each positive decision requiring a verbatim supporting quotation. D2T was defined by criterion A; patients with current activity were classified as persistent and those without as controlled D2T. For validation, 200 records stratified by status and data source were independently annotated by four rheumatologists blinded to model output; sensitivity, specificity and Cohen's κ were computed per criterion and for the final status.
Results. 364 of 2,748 patients met the D2T criteria (13.2%; 95% CI 12.0–14.6): 196 persistent (7.1%) and 168 controlled (6.1%). Prevalence was similar across centres (12.2–14.1%) and ICD-10 groups (11.8–16.0%). Agreement between LLM and experts for the final D2T classification was 82.5%, sensitivity 93.1% (95% CI 85.8–96.8), specificity 74.3% (65.6–81.5), κ = 0.65; after adjudication of disagreements by the lead rheumatologist, sensitivity was 96.8% (91.0–98.9), specificity 82.1% (73.7–88.2) and κ = 0.78; κ for the three-class status was 0.63 before and 0.77 after adjudication. Registry-derived criterion A was reproduced almost perfectly (κ = 0.90), text-derived A moderately (κ = 0.55); index-based activity κ = 0.87; free-text “problematic management” κ = 0.16. Adjudication of 46 disagreements upheld the expert in 26 and the model in 17 cases (annotation error acknowledged in 10); 6 of the remaining disagreements reflected a diagnosis other than RA rather than an extraction error. Adjudication led to a refined rule for rheumatoid nodules: stable nodules are not counted as current activity. An explicit statement of D2T or refractory disease was present in the notes of 2 of 364 patients (0.5%).
Conclusions. One in eight patients in a city-wide b/tsDMARD registry meets the D2T RA criteria, and this status is almost never documented. LLMs with quotation-grounded output reproduce expert assessment with substantial agreement; reliability is highest for criteria based on structured data (drug registry, activity indices) and lowest for the subjective criterion C, which argues for an operational specification of the EULAR definition for registry use.
Keywords: rheumatoid arthritis; difficult-to-treat rheumatoid arthritis; D2T RA; large language models; natural language processing; registry; electronic health records; validation.
The concept of difficult-to-treat (D2T) rheumatoid arthritis (RA) was introduced by a EULAR task force to describe patients in whom active or progressive disease persists despite sequential switching between drugs with different mechanisms of action, and whose management is perceived as problematic by the rheumatologist or the patient [1, 3–5]. The 2021 definition comprises three mandatory components: (A) failure of at least two biologic (b) or targeted synthetic (ts) DMARDs with different mechanisms of action after failure of conventional synthetic DMARDs; (B) signs suggestive of active or progressive disease according to at least one of four items; and (C) management perceived as problematic [1]. EULAR points to consider for the management of such patients followed in 2022 [2].
Prevalence estimates of D2T RA vary widely, from 5–7% in Japanese and Dutch cohorts to 15–20% in biologics registries [6–13]. The variation reflects not only differences in populations and drug access but also in how the criteria are operationalised: criterion A can be reconstructed from structured prescription data, whereas criteria B and, above all, C reside mainly in free-text clinical notes [14]. Consequently, most population-based estimates rely either on a simplified “biologic-refractory” definition based on the number of treatment lines [6, 7] or on manual review of small samples.
Large language models (LLMs) can extract clinical features from unstructured text without task-specific training and, when instructed appropriately, can support every decision with a verbatim quotation from the source document that is open to verification [16–18]. This makes it feasible to apply the full EULAR definition to entire registries; however, we are not aware of publications in which such an approach has been validated against blinded expert review at the level of individual criteria. In the Russian literature, the D2T RA concept has so far been discussed mainly at the level of clinical series [15].
The aims of this study were to estimate the prevalence and structure of D2T RA in a unified city-wide registry of patients receiving b/tsDMARDs by applying LLMs to clinical notes and registry data; to validate this approach by blinded annotation of a representative sample by four rheumatologists; and to assess how far D2T status is reflected in the medical documentation.
This retrospective observational study used the unified registry of patients with inflammatory arthritis receiving b/tsDMARDs through centralised drug provision in Moscow. The registry covers three municipal rheumatology centres (City Clinical Hospital No. 52, Pirogov City Clinical Hospital No. 1 and the Loginov Moscow Clinical Scientific Centre) and, as of 31 December 2024, contained 2,785 patients with ICD-10 codes M05, M06 and M08; the diagnosis of RA was established by the treating rheumatologists according to the 2010 ACR/EULAR criteria [19], and treatment followed current recommendations [20]. For each patient, the registry holds sex, date of birth, ICD-10 code, centre, the sequence of b/tsDMARDs with start and stop dates, coded discontinuation reasons (inefficacy, adverse event, remission, death, malignancy, pregnancy, technical or administrative reasons, reason not established) and concomitant csDMARD and glucocorticoid therapy in 2024.
Rheumatologist notes were obtained from the electronic health record in two parts. The first part (cohort 1) comprised 680 patients with 1,231 notes from 2022–2025 (up to two per patient; median length 6,890 characters); for these patients the registry drug sequence did not include discontinuation reasons. The second part (cohort 2) comprised 2,068 patients with a single note from the last visit in 2024 (median length 11,720 characters) and a complete registry sequence with coded reasons. Thirty-seven registry patients had no notes and were excluded. The final analytic cohort comprised 2,748 patients (figure 1). All texts were de-identified before processing: names, insurance numbers and other direct identifiers were replaced by codes.
The EULAR criteria [1] were operationalised for registry data in advance of the main analysis and agreed with the chief rheumatologist of the city (table 1). Criterion A was met when at least two drugs with different mechanisms of action (TNF inhibitors, IL-6 receptor inhibitors, JAK inhibitors, anti-CD20, co-stimulation blockade, IL-1 inhibitors) had failed at the third or later line. In line with the EULAR definition, drugs rather than classes were counted: a class remained “failed” even if the patient subsequently responded to another drug of the same class. Failure was defined as discontinuation for inefficacy or for an adverse event; discontinuation for remission, pregnancy, malignancy, death, technical or administrative reasons, or with an unestablished reason, was not counted as failure. In cohort 2 criterion A was computed deterministically from the registry sequence; in cohort 1, where reasons were not coded, it was extracted from the history in the notes.
Criterion B was assessed at the last visit and met if at least one of four items was present: (B1) activity above moderate by composite index (DAS28-ESR > 3.2, CDAI > 10 or SDAI > 11 [24]); (B2) current systemic manifestations of RA from a predefined list of 11 categories (rheumatoid nodules, cutaneous vasculitis, neuropathy, pleuritis, pericarditis, myocarditis, Sjögren's syndrome, ocular involvement, interstitial lung disease, generalised myopathy, haematological manifestations), with rheumatoid nodules counted as a current manifestation only when the text indicated new or enlarging nodules or active pulmonary nodules, whereas stable or historical nodules were not regarded as a sign of activity (rule refined after adjudication and applied to the whole cohort); (B3) inability to taper glucocorticoids below 7.5 mg/day prednisolone equivalent; (B4) radiographic progression. Joint symptoms without composite indices were not included in the main analysis (narrow interpretation).
Criterion C was assessed in two ways: a free-text judgement (“management perceived as problematic by the physician and/or the patient” based on the note as a whole) and a structured checklist of 8 patient-perspective items (pain VAS ≥ 4, functional class III–IV, joint surgery, continuous NSAID use, more than three drugs for RA, systemic manifestations, severe renal involvement, osteoporosis with fractures) and 17 physician-perspective items (radiographic stage IV, joint replacement, failure of three classes, non-adherence, inability to stop glucocorticoids, therapy-limiting infections, diagnostic revision, limiting comorbidity, complicating systemic manifestations, need for psychiatric care, polyvalent allergy, infection-related treatment interruptions, absence of escalation despite activity, history of tuberculosis, amyloidosis, secondary osteoarthritis grade III–IV, contractures).
Following a preliminary analysis and discussion with the clinical lead, it was agreed that in a population-based cohort, where past activity is implied by the sequential switching itself, D2T status is defined by criterion A; patients with current activity (criterion B at the last visit) were classified as persistent D2T and those without as controlled D2T. Criterion C was treated as a descriptive layer. Prevalence under stricter operationalisations (A and B and C) is reported as a sensitivity analysis.
For each patient, the note text (all available notes in cohort 1) together with the registry drug sequence was submitted to an LLM with instructions reproducing the operational definitions above. The model returned a structured response of fixed schema: for each criterion and sub-item, a value (yes/no/insufficient data), the list of failed drug classes with the outcome of each course, composite index values, glucocorticoid dose, systemic manifestations identified, answers to the 25 checklist items, and a mandatory verbatim quotation from the note supporting each positive decision. Responses that violated the schema or lacked quotations for positive decisions were rejected and re-requested.
Two general-purpose models of the same family were used (GPT-5.6 Luna and GPT-5.6 Terra; OpenAI) [wording and model versions to confirm for publication]. The smaller model processed all patients; the larger model re-assessed patients with borderline or conflicting results (discordance between text-derived and registry-derived criterion A, positive criterion A without composite indices, schema errors): 398 of 680 in cohort 1 and 834 of 2,068 in cohort 2. In case of disagreement the larger model's result was retained. Extraction was zero-shot; the instructions were calibrated on 80 records previously assessed manually by the clinical lead in a pilot phase and were not modified thereafter. Processing of the whole cohort took less than one day.
A stratified random validation sample of 200 patients was drawn with a fixed random seed: 100 from each cohort; within cohort 1, 35 persistent, 25 controlled and 40 non-D2T by model output, and within cohort 2, 25, 25 and 50, respectively. Record order was shuffled; neither stratum nor model output was shown to annotators. Four rheumatologists (authors AIZ, [initials]) independently annotated 50 records each in a purpose-built web interface showing the note text and a de-identified registry extract (drug sequence, dates, discontinuation reasons). The annotation form mirrored the extraction structure: criterion A with the list of failed classes, items B1–B4, free-text criterion C, the 25-item checklist, final status (persistent D2T / controlled D2T / not D2T), a rating of documentation completeness and a free comment; “cannot be determined from the documents” was allowed for each item. Annotators' access to model output elsewhere in the system was technically blocked for the duration of annotation. No overlap of records between annotators was planned.
Expert assessment served as the reference standard. For each binary item and for the final D2T classification, percentage agreement, sensitivity, specificity, positive and negative predictive values with 95% confidence intervals (Wilson method) and Cohen's κ were computed; for the three-class status, percentage agreement and κ. Pairs with a “cannot be determined” answer were excluded for the corresponding item. κ was interpreted according to Landis and Koch [21]. All records with a discordant final status were subsequently adjudicated by the lead rheumatologist (AIZ) in a dedicated interface displaying the model's assessment with quotations side by side with the expert's answers; for each record a verdict was recorded (model correct / expert correct / other status) and the reason for disagreement was coded (annotation error, text misread, diagnosis other than RA, two drugs of the same class, activity during treatment interruption, short course, clinical-trial drug, insufficient data). Metrics were computed twice: against the original expert annotation and against the adjudicated status.
To assess how far D2T status is reflected in the documentation, the notes of all patients meeting the criteria were searched with regular expressions for the terms “D2T”, “difficult-to-treat”, and Russian equivalents of “refractory”, “difficult to control/treat”, “treatment-resistant” and “torpid”; all matches were reviewed manually.
Continuous variables are presented as median and interquartile range (IQR), categorical variables as counts and percentages. Prevalence is reported with Wilson 95% CIs. Comparisons between centres and diagnostic groups are descriptive. Analyses were performed in Python 3.12 (pandas, statsmodels).
The study used de-identified retrospective data within the organisational and methodological remit of the chief rheumatologist of the Moscow Healthcare Department. [Local ethics committee approval: number and date to be confirmed; required for submission.]
The analysis included 2,748 patients (figure 1): median age at 31 December 2024 was 60 years [IQR 43–68] and 2,216 (80.6%) were women. Seropositive RA (M05) was coded in 2,017 (73.4%), seronegative RA (M06.0) in 261 (9.5%), juvenile arthritis (M08) in 416 (15.1%) and adult-onset Still's disease (M06.1) in 50 (1.8%). The most frequent drugs in 2024 were etanercept (571), rituximab (420), olokizumab (332), tofacitinib (255), levilimab (253), adalimumab (228), abatacept (227) and baricitinib (213). Of 1,100 documented b/tsDMARD discontinuations, 478 (43%) were for inefficacy, 278 (25%) for an adverse event, 177 (16%) had no established reason, 51 were for remission, 40 for death, 28 for malignancy and 37 for technical or administrative reasons. Patient characteristics by D2T status are shown in table 2.
Criterion A was met by 364 patients, 13.2% (95% CI 12.0–14.6). In 196 (7.1%; 6.2–8.2) disease activity was present at the last visit (persistent D2T) and in 168 (6.1%; 5.3–7.1) it was absent (controlled D2T). The proportion of D2T was similar across the three centres (13.4%, 12.2% and 14.1%) and across diagnostic groups: 13.4% for M05, 13.8% for M06.0, 11.8% for M08 and 16.0% for M06.1; after excluding patients coded M08 and M06.1, prevalence was 307/2,282, 13.5% (12.1–14.9). The difference between cohorts was pronounced: 264/680 (38.8%) in cohort 1, drawn from patients with multiple notes and long follow-up, versus 100/2,068 (4.8%) in cohort 2 (table 3).
Components of criterion B at the last visit were identified in 1,169 patients (42.5%): index-based activity in 808 (29.4%), current systemic manifestations in 508 (18.5%), glucocorticoids ≥ 7.5 mg/day in 109 (4.0%) and radiographic progression in 67 (2.4%). Among D2T patients, at least one checklist item of criterion C was present in 348 (96%; median 4 items), whereas the free-text judgement of “problematic management” was positive in 171 (47%). Under stricter operationalisations prevalence fell to 193 (7.0%) when A, B and at least one checklist item were required, and to 127 (4.6%) when A, B and a positive free-text judgement were required (table 3). In cohort 2, agreement between text-derived and registry-derived criterion A was 97.3% (κ = 0.75): the text yielded 47 positive calls with a negative registry and 9 negative calls with a positive registry.
All 200 records were annotated in full; median annotation time per record ranged from 3.0 to 5.6 minutes between experts. Documentation completeness was rated good in 135 records, partial in 45 and poor in 8. Agreement metrics are shown in table 4 and figure 2. For the final D2T classification, agreement was 82.5%, sensitivity 93.1% (95% CI 85.8–96.8), specificity 74.3% (65.6–81.5) and κ = 0.65; for the three-class status, agreement was 77.0% and κ = 0.63. The model missed 6 of 87 patients judged D2T by experts and produced 29 excess positive classifications.
Agreement depended strongly on the data source for criterion A: in cohort 2, where A was registry-derived, κ for A was 0.90 (sensitivity 100%, specificity 90.9%) and κ for the final classification 0.78; in cohort 1, where A was text-derived, the corresponding values were 0.55 and 0.53. Among the B items, index-based activity showed the best agreement (κ = 0.87), followed by systemic manifestations (κ = 0.70) and glucocorticoids (κ = 0.65 with 9 expert-positive records). Radiographic progression could be assessed by experts in only 93 records; of 22 expert-positive records the model reproduced 3 (κ = 0.13). The free-text criterion C was not reproducible (κ = 0.16): experts judged management problematic in 89 records and the model in 56, largely in different patients; the checklist at a ≥ 1-item threshold was positive in 93% of records according to the model and 80% according to experts (κ = 0.28). Between experts, κ for the final classification ranged from 0.47 to 0.76.
Forty-six records were adjudicated: 35 discordant on D2T and 11 discordant only on status (persistent or controlled). The verdict upheld the expert in 26 records and the model in 17 (including 4 records in which an initial verdict for the expert was revised after checking against the EULAR definition, which counts drugs rather than classes), and assigned another status in 3; in 10 records the lead rheumatologist acknowledged an annotation error. Against the adjudicated status, agreement for the final D2T classification was 89.0%, sensitivity 96.8% (95% CI 91.0–98.9), specificity 82.1% (73.7–88.2) and κ = 0.78; for the three-class status, 85.5% and κ = 0.77; κ was 0.88 in cohort 2 (sensitivity 100%, specificity 89.3%) and 0.68 in cohort 1; by expert, κ ranged from 0.65 to 0.84 (table 4, figure 2).
Of the 29 excess positive classifications, 22 arose from disagreement on criterion A: the model counted two failed classes, the expert one or none. In 11 of these the disputed drug was mentioned only in the history and was absent from the registry sequence (prescriptions before registry inclusion, clinical-trial drugs, courses of one month or a single infusion); in 17 the model interpreted a switch described in the text as inefficacy while the expert did not; in 4 the disputed class had been stopped for an adverse event after a short course. The hypothesis that experts systematically discount discontinuation for adverse events was not supported: among concordant positive calls on A, experts counted a class stopped solely for an adverse event in 42 of 85 cases. The remaining 7 disagreements occurred with concordant criterion A: in 5 records the expert questioned the diagnosis of RA (psoriatic arthritis, Still's disease), in 2 the expert considered the switch non-medical or the inefficacy doubtful. Overall, experts flagged a diagnosis other than RA in 8 of 200 records (4%). Of the 6 cases missed by the model, in 4 the failed class was present in both text and registry but not counted by the model; in 2 the expert assigned D2T with a single failed class in their own list.
After adjudication, 19 excess and 3 missed classifications remained. Among the excess classifications, 6 were associated with a diagnosis other than RA (psoriatic arthritis, Still's disease, spondyloarthritis, a switch indicated by Crohn's disease) with criterion A formally met; 3 with text misreading (an adalimumab biosimilar assigned to IL-6 inhibitors; certolizumab missed; unconfirmed Sjögren's syndrome); 2 with activity during a treatment interruption; and one each with a short course and a clinical-trial drug. Of the 11 status-only disagreements, 6 concerned an isolated rheumatoid nodule counted by the model as a current systemic manifestation (item B2) in the absence of joint activity; in these records the lead rheumatologist classified the status as controlled, noting that the model was “formally correct”; this prompted the refinement of the B2 rule that was then applied to the whole cohort (see Discussion).
Among the 364 patients meeting the D2T criteria, an explicit statement of difficult-to-treat or refractory disease was found in the notes of 2 (0.5%; 95% CI 0.2–2.0): “enrolled in the D2TRA follow-up group” in one case and “refractoriness to therapy in the setting of adverse effects” in the other. Two further string matches referred to contexts unrelated to RA.
In a population-based registry of patients receiving b/tsDMARDs in a large city, 13.2% met the D2T RA criteria; in 46% of them the disease was controlled at the time of assessment. LLMs with mandatory quotation grounding made it possible to apply the full EULAR definition to all 2,748 patients, and blinded validation by four rheumatologists showed substantial agreement for the final classification (κ 0.65) and near-perfect agreement for criteria based on structured data. Yet D2T status was explicitly documented in only two of 364 patients.
The prevalence we observed lies in the middle of the published range: 5.7–6.4% in Japanese cohorts [9–11], about 6–10% in European clinical cohorts [7, 8, 12, 13], and 15–20% among patients in biologics registries under the simplified “biologic-refractory” definition [6, 7]. Direct comparison is hampered by differences in the denominator (all RA patients or only those on b/tsDMARDs) and in the operationalisation of criteria. Our data illustrate the second source of variability clearly: within the same cohort, prevalence ranges from 5.0% to 13.2% depending on whether current activity is required and how criterion C is handled. We suggest that registry studies should separate two entities: biologic refractoriness (criterion A), a stable feature of the treatment history that is reliably recoverable from prescription data, and current state (persistent or controlled activity), which changes over time. Distinguishing persistent from controlled D2T is consistent with the view of D2T as a state rather than a lifelong label [5, 8, 13].
Adjudication showed that more than a third of the initial disagreements (17 of 46, or 23 when records with another diagnosis are included) were not model errors, and it surfaced two operationalisation questions. First, the model counted an isolated rheumatoid nodule without signs of inflammatory activity as a current systemic manifestation; by decision of the lead rheumatologist, stable and historical nodules were excluded from item B2, whereas new or active nodules, Sjögren's syndrome and other extra-articular manifestations are counted. Applying this rule to the whole cohort moved 29 patients from persistent to controlled status (225/139 before refinement, 196/168 after); in 4 patients with pulmonary nodules and no indication of their activity the status was classified as controlled. Second, in 4 records the expert initially did not count a class as failed because the patient currently responded to another drug of the same class; at adjudication this was revised in favour of the EULAR definition. Had the alternative rule been applied (“a class with a currently effective drug is not counted as failed”), criterion A would no longer have been met in 67 of 364 patients and prevalence would have fallen from 13.2% to 10.8%, illustrating the sensitivity of the estimate to the wording of the rule.
Validation revealed a clear hierarchy of criterion reliability. Criterion A derived from the registry sequence with coded discontinuation reasons was reproduced almost perfectly (κ 0.90), whereas extraction of the same criterion from free-text history yielded only moderate agreement (κ 0.55). Review of the disagreements showed that most were caused not by reading errors but by differences in which treatment episodes count as “failure”: drugs prescribed before registry inclusion or within clinical trials, courses of a single infusion, switches for a mix of clinical and organisational reasons. The EULAR definition does not resolve these situations, and explicit rules are needed. Among the B items, index-based activity was reproduced reliably (κ 0.87) and systemic manifestations acceptably, whereas radiographic progression is practically impossible to assess from the text of the last visit, for models and experts alike, reflecting the absence of formalised radiographic follow-up in routine practice. Finally, the free-text criterion C was not reproducible between the model and experts (κ 0.16) and, judging by the spread between experts, probably not between experts either; the structured checklist at a one-item threshold was positive in the vast majority of patients regardless of D2T status and therefore lacked discriminative ability. This empirically confirms previously expressed concerns about the subjectivity of criterion C [3, 22, 23] and points to the need for its operational specification if the definition is to serve epidemiological and registry research.
The finding that an explicit mention of D2T or refractory disease was present in the documentation of only 0.5% of patients meeting the criteria deserves attention. This is not undiagnosed disease: the diagnosis of RA and the treatment history are documented in these patients. It is an unrecognised or unrecorded status which, according to the EULAR points to consider [2], should trigger a review of the diagnostic concept, adherence, comorbidity and the appropriateness of further escalation. Automated registry screening producing a list of patients with supporting quotations could serve as a tool for such targeted review; with 93% sensitivity and 74% specificity in an enriched sample, it should be regarded as a triage instrument for expert verification rather than a final classification.
A further finding is the presence in the RA registry of patients with other diagnoses: experts questioned the diagnosis of RA in 4% of validation records, and by ICD-10 code 15% of registry patients have juvenile arthritis. D2T prevalence in these groups did not differ from the main group, but for clinical conclusions on D2T RA it is advisable to restrict the cohort to codes M05 and M06.0, which did not change our estimate (13.5%).
Strengths of the study include the population-based nature of the registry, which covers all patients in the city on b/tsDMARDs through centralised provision; pre-specified operational definitions; full transparency of model decisions through verbatim quotations; blinded validation at the level of individual criteria by four experts; and reproducibility of the approach in other registries. Limitations include the retrospective design and single region; assessment of activity at the last visit only, which precludes conclusions about its persistence; incomplete documentation (composite indices were not recorded for all patients and radiographic follow-up rarely); different sources for criterion A between cohorts 1 and 2, reflecting the evolution of the registry and requiring separate analysis; absence of record overlap between experts, so that inter-rater agreement was assessed only indirectly; enrichment of the validation sample with D2T patients, so that predictive values are not transferable to the population; and the use of commercial models whose versions may change, partly mitigated by publication of the instructions and output schema in the supplementary materials. Adjudication was performed by a single lead rheumatologist who also took part in annotation and in formulating the operational definitions, which may have biased verdicts towards the agreed rules.
In a city-wide b/tsDMARD registry, one in eight patients meets the D2T RA criteria, and this status is almost never reflected in the documentation. Large language models with quotation grounding allow the full EULAR definition to be applied to an entire registry and reproduce blinded expert review with substantial agreement. Criterion reliability is heterogeneous: criterion A from coded treatment data and index-based activity are reproduced almost perfectly, whereas the subjective criterion C and radiographic progression from text are not. Use of the D2T definition in registries requires its operational specification; the proposed approach may serve as a basis for automated identification of patients in need of expert treatment review.
[To confirm.] Concept and clinical lead: AIZ; operationalisation of criteria: AIZ, YP; development of the extraction pipeline and validation interface, statistical analysis: YP; blinded annotation: AIZ, [initials of three experts]; manuscript drafting: YP, AIZ; all authors approved the final version.
[To confirm. Funding sources and competing interests.]
Individual patient data cannot be shared. The model instructions, structured output schema, annotation and adjudication forms and run parameters are published in the supplementary materials; aggregate tables are available from the authors on request.
| EULAR 2021 criterion | Operational definition in this study | Data source |
|---|---|---|
| A. Failure of ≥ 2 b/tsDMARDs with different mechanisms of action after csDMARD failure | Discontinuation for inefficacy or adverse event of ≥ 2 drugs from different classes (TNFi, IL-6i, JAKi, anti-CD20, co-stimulation blockade, IL-1i); discontinuation for remission, pregnancy, malignancy, death, technical/administrative reasons or with unestablished reason not counted | Cohort 2: registry (deterministic). Cohort 1: history in notes (LLM) |
| B. Signs of active/progressive disease (≥ 1 of 4) | B1: DAS28-ESR > 3.2 / CDAI > 10 / SDAI > 11. B2: current systemic manifestations (11 categories). B3: inability to taper GC < 7.5 mg/day. B4: radiographic progression. Assessed at the last visit | Note text (LLM), with quotation |
| C. Management perceived as problematic | Free-text judgement on the note as a whole and structured checklist: 8 patient items + 17 physician items (threshold ≥ 1); descriptive layer | Note text (LLM), with quotation |
| Final status | D2T = A. Persistent D2T = A and B; controlled D2T = A and not B. Sensitivity analyses: A and B and C (free-text); A and B and C (checklist ≥ 1) | — |
b/ts/csDMARD — biologic/targeted synthetic/conventional synthetic disease-modifying antirheumatic drugs; GC — glucocorticoids; LLM — large language model.
| Characteristic | Persistent D2T (n = 196) | Controlled D2T (n = 168) | Not D2T (n = 2,384) | All (n = 2,748) |
|---|---|---|---|---|
| Women, n (%) | 169 (86) | 141 (84) | 1906 (80) | 2216 (81) |
| Seropositive RA (M05), n (%) | 148 (76) | 123 (73) | 1746 (73) | 2017 (73) |
| DAS28 where recorded, median [IQR] | 4.1 [3.4–4.9] | 2.5 [1.8–2.9] | 2.6 [1.8–3.3] | 2.6 [1.9–3.4] |
| B1 — index-based activity, n (%) | 150 (77) | 0 (0) | 658 (28) | 808 (29) |
| B2 — current systemic manifestations, n (%) | 76 (39) | 0 (0) | 432 (18) | 508 (18) |
| rheumatoid nodules (active) | 15 (8) | 0 (0) | 80 (3) | 95 (3) |
| Sjögren's syndrome | 46 (23) | 0 (0) | 262 (11) | 308 (11) |
| interstitial lung disease | 13 (7) | 0 (0) | 92 (4) | 105 (4) |
| neuropathy | 15 (8) | 0 (0) | 56 (2) | 71 (3) |
| ocular involvement | 14 (7) | 0 (0) | 75 (3) | 89 (3) |
| B3 — GC ≥ 7.5 mg/day, n (%) | 34 (17) | 0 (0) | 75 (3) | 109 (4) |
| B4 — radiographic progression, n (%) | 16 (8) | 0 (0) | 51 (2) | 67 (2) |
| C — “problematic management”, free-text, n (%) | 127 (65) | 44 (26) | 356 (15) | 527 (19) |
| C — checklist ≥ 1 item, n (%) | 193 (98) | 155 (92) | 2031 (85) | 2379 (87) |
| C — number of checklist items, median [IQR] | 5 [3–7] | 3 [2–5] | 3 [1–5] | 3 [1–5] |
Median age of the whole cohort at 31 December 2024 was 60 years [43–68]; sex was missing for 66 patients. B and C items refer to the last visit; B2 applies the nodule rule (stable nodules not counted; the “nodules” row shows active nodules or nodules together with other manifestations). B components are zero in controlled D2T by definition of the status. [Add age by group and number of treatment lines from the registry.]
| Definition / subgroup | n | D2T, n | % (95% CI) |
|---|---|---|---|
| Main definition (criterion A) | 2,748 | 364 | 13.2 (12.0–14.6) |
| persistent (A and B) | 2,748 | 196 | 7.1 (6.2–8.2) |
| controlled (A and not B) | 2,748 | 168 | 6.1 (5.3–7.1) |
| A and B and C (checklist ≥ 1) | 2,748 | 193 | 7.0 (6.1–8.0) |
| A and B and C (free-text) | 2,748 | 127 | 4.6 (3.9–5.5) |
| Cohort 1 (multiple notes, 2022–2025; A from text) | 680 | 264 | 38.8 (35.2–42.5) |
| Cohort 2 (last visit 2024; A from registry) | 2,068 | 100 | 4.8 (4.0–5.8) |
| City Clinical Hospital No. 52 | 988 | 132 | 13.4 (11.4–15.6) |
| Pirogov City Clinical Hospital No. 1 | 813 | 99 | 12.2 (10.1–14.6) |
| Loginov Moscow Clinical Scientific Centre | 881 | 124 | 14.1 (11.9–16.5) |
| M05 seropositive RA | 2,017 | 271 | 13.4 (12.0–15.0) |
| M06.0 seronegative RA | 261 | 36 | 13.8 (10.1–18.5) |
| M08 juvenile arthritis | 416 | 49 | 11.8 (9.0–15.2) |
| M06.1 adult-onset Still's disease | 50 | 8 | 16.0 (8.3–28.5) |
| M05 + M06.0 only | 2,282 | 307 | 13.5 (12.1–14.9) |
Wilson CIs. Centre was missing for 66 patients.
| Criterion | n | Agreement, % | Sens., % | Spec., % | PPV, % | NPV, % | κ |
|---|---|---|---|---|---|---|---|
| A — failure of ≥ 2 classes | 195 | 86.2 | 94.4 | 79.0 | 79.4 | 94.3 | 0.73 |
| cohort 2 (registry-derived) | 99 | 94.9 | 100 | 90.9 | — | — | 0.90 |
| cohort 1 (text-derived) | 96 | 77.1 | 89.1 | 66.0 | — | — | 0.55 |
| B1 — index-based activity | 192 | 93.8 | 88.9 | 96.7 | 94.1 | 93.5 | 0.87 |
| B2 — systemic manifestations | 193 | 87.6 | 86.3 | 88.0 | 72.1 | 94.7 | 0.70 |
| B3 — GC ≥ 7.5 mg/day | 197 | 95.9 | 88.9 | 96.3 | 53.3 | 99.5 | 0.65 |
| B4 — radiographic progression | 93 | 76.3 | 13.6 | 95.8 | 50.0 | 78.2 | 0.13 |
| B — any of B1–B4 | 141 | 86.5 | 87.6 | 83.3 | 93.9 | 69.8 | 0.67 |
| C — free-text judgement | 195 | 59.5 | 37.1 | 78.3 | 58.9 | 59.7 | 0.16 |
| C — checklist ≥ 1 item | 200 | 82.0 | 96.9 | 24.4 | 83.2 | 66.7 | 0.28 |
| D2T — final classification | 200 | 82.5 | 93.1 | 74.3 | 73.6 | 93.3 | 0.65 |
| Status (3 classes) | 200 | 77.0 | — | — | — | — | 0.63 |
| D2T — after adjudication | 200 | 89.0 | 96.8 | 82.1 | 82.7 | 96.7 | 0.78 |
| cohort 2 (registry-derived) | 100 | 94.0 | 100 | 89.3 | — | — | 0.88 |
| cohort 1 (text-derived) | 100 | 84.0 | 94.0 | 74.0 | — | — | 0.68 |
| Status (3 classes) — after adjudication | 200 | 85.5 | — | — | — | — | 0.77 |
n — records with a definite expert answer for the item. PPV/NPV — positive/negative predictive value; not transferable to the population because the sample was enriched with D2T patients (50%). κ for the final classification by expert: 0.64; 0.76; 0.47; 0.75 before adjudication and 0.80; 0.84; 0.65; 0.83 after. Adjudication: 46 disagreements; verdict “expert correct” 26, “model correct” 17, other status 3.