betaFeedback

WorksComputer Science

LABEL: Literature Asymmetry between Efficacy and Limits in Drug Knowledge and Language Models

Abstract

Every medication comes with two types of medical facts: indications, which describe what conditions the drug treats (benefits), and contraindications, which describe who must avoid it to prevent serious harm (safety limits). Although contraindications are critical for patient safety and FDA regulations require listing them as rigorously as indications, we find that contraindications are far less documented than indications in general prose. In this paper, we measure how often these texts contain both facts across 574 drugs from PrimeKG by comparing a drug's indications against its own contraindications. Across Wikipedia articles and PubMed abstracts, we find that reference prose names a drug's benefits about twice as often as its safety limits (within-drug odds ratio 2.19). We call this documentation gap LABEL (Literature Asymmetry between Efficacy and Limits). Language models mirror this gap. As we scale Qwen2.5 from 0.5B to 14B, accuracy on indications rises by 18.3 points, whereas accuracy on contraindications remains flat near random guessing. However, when we provide models with a retrieved passage that states the answer, they use safety facts just as effectively as benefits. Furthermore, routing safety questions to official FDA labels boosts accuracy by 15.8 points whenever the label states the fact. Overall, our findings show that models fail on medication safety not because they cannot reason about risk, but because free-text corpora lack safety facts. Clinical AI systems should therefore route medication-safety questions to official regulatory labels and structured databases rather than relying on general public records. Code and data are available at https://github.com/SMRayeed/LABEL.

Introduction

If we look up a medication online, its Wikipedia page usually tells us what it treats, but it is much less likely to mention who must never take it. In medicine, the condition a drug treats is called an indication. For example, metformin treats type 2 diabetes. Conversely, a condition that makes a drug dangerous to take is called a contraindication. For example, ciprofloxacin, a common antibiotic, must not be given to people with myasthenia gravis, a disease that weakens the muscles, because the drug can worsen muscle weakness. Structured clinical databases such as DrugCentral record both kinds of facts (Et al. 2017). However, free text records like Wikipedia or PubMed abstracts do not document them equally. This gap matters because large language models (LLMs) learn from these text corpora, and patients

and clinicians increasingly ask the LLMs about medications. If the corpora record contraindications less often than indications, LLMs may be less reliable on questions where a wrong answer can cause patients the greatest harm. Recognizing this gap also changes how we diagnose and fix the issue. When a model misses a contraindication, the model itself is not necessarily bad at reasoning about safety; rather, the safety fact may simply be missing from the prose on which it was trained. And in that case, writing better prompts or building better retrievers will not help. Instead, clinical systems must route safety questions to structured databases and regulatory documents that record contraindications.

Prior works in two research directions explored pieces of this problem. First, pharmacy and medical informatics studies audit reference websites like Wikipedia by checking a small set of drugs against clinical textbooks (Clauson et al. [4]; Kräenbring et al. [15]). Second, AI benchmarks built from biomedical knowledge graphs (structured networks that store facts as subject-relation-object links) test how well LLMs answer clinical questions (Liu et al. [20]), but they do not check whether the underlying facts actually appear in text corpora. Moreover, neither line of work controls for drug popularity: widely prescribed drugs have longer articles and thousands of research papers, so comparing facts across different drugs mixes a drug’s overall popularity with the type of fact being measured. In this work, we bridge these two directions by evaluating text coverage and model knowledge on the same drug-disease facts (Figure 1). Our main contributions are as follows:

Figure 1. Overview of our study pipeline. (1) We sample paired benefit facts (indications) and safety facts (contraindications) for the exact same drugs from the PrimeKG knowledge graph. (2) We retrieve text from Wikipedia, PubMed abstracts, and official FDA drug labels, and check both whether the text mentions the target disease and whether the sentence states the correct medical relationship. (3) We test language models on the same facts both closed book (from memory alone) and open book (with a retrieved passage). (4) Comparing benefit and safety facts within each drug reveals the LABEL asymmetry.

  • Within-drug text audit. We design a paired, within-drug evaluation across 574 drugs from PrimeKG, comparing how often text documents each drug’s indications versus its own contraindications so that article length, drug fame, and research volume cancel out.

  • Cross-corpus audit. We evaluate three text sources: English Wikipedia articles, PubMed research abstracts (using a balanced search budget for benefits and harms), and official FDA drug labels. We show that both Wikipedia and PubMed state benefits more than twice as often as safety limits, whereas FDA labels document both equally. We also verify with a language-model judge (Qwen2.5-7B) and four human annotators that matched sentences genuinely mention the medical relationship rather than just mentioning the disease.

  • Scaling and distractor difficulty. We test 1,800 drug-disease facts across five Qwen2.5 models from 0.5B to 14B parameters. Scaling improves indication accuracy by 18.3 points but contraindication accuracy by only 5.0. We repeat the test with harder, closely related wrong options in four models from three families (Qwen2.5, OLMo-2 and Granite-3.1), and the gap holds in every model.

  • Separating retrieval from reading. We test models with retrieved passages and separate performance by whether a passage explicitly contains the target condition. We show that when a passage states a contraindication, LLMs use it just as well as they use a stated indication.

  • Negative control and clinical routing. We validate our pipeline on Hetionet, a negative control whose two drug-disease relations both describe benefit, and find no gap. Finally, we demonstrate a relation-aware routing strategy that redirects safety questions to the FDA labels, boosting accuracy by 15.8 points when the label states the fact.

dated within a year of an FDA safety warning. These audits cover at most about a hundred drugs, grade articles against a reference text rather than a drug’s full set of facts, and do not link missing facts to model errors.

Related Work

Drug information in reference text. Wikipedia is one of the most widely read health resources online [7], and pharmacy studies have audited its drug articles for more than a decade. Clauson et al. [4] compare Wikipedia with a professional drug compendium and find a narrower scope and more omissions. Kräenbring et al. (2014) compare 100 drug articles with pharmacology textbooks and find high accuracy but incomplete coverage. Other audits compare Wikipedia with Micromedex, Lexicomp and FDA medication guides, and find it less complete [18, 30, 2, 9]. Two findings are close to ours. Kupferberg and Protus [17] examine the Wikipedia articles of five statins, a class of cholesterol-lowering drugs, and find no incorrect statements but frequent omissions of contraindications and drug interactions. Hwang, Bourgeois, and Seeger [10] find that more than a third of drug articles are not updated within a year of an FDA safety warning.

Structured drug databases and knowledge graphs. Unlike general prose, regulatory and clinical databases are built specifically to track drug safety. DrugCentral curates approved indications and contraindications from regulatory records [37], SIDER extracts adverse drug reactions from package inserts [16], and openFDA provides programmatic access to the full text of official FDA drug labels [14]. Biomedical knowledge graphs such as PrimeKG and Hetionet integrate these curated databases with standardized disease ontologies [3, 8]. We use these structured graphs as ground truth to test, fact by fact, how much of this clinical knowledge actually makes it to the free text.

Medical knowledge in LLMs. LLMs achieve high scores on standard medical licensing exams [34, 35]. However, evaluations built from biomedical knowledge graphs reveal that model performance varies sharply depending on the type of clinical relation. For instance, MH-GraphBench [20] tests models on questions derived from PrimeKG and finds that predicting specific relations between entities is much harder than identifying entity types. While existing benchmarks document what LLMs get wrong, they do not trace those failures back to what is missing in reference text.

Corpus frequency and model recall. A separate line of research shows that LLMs remember facts better when facts appear frequently in pretraining text. Kandpal et al. [13] demonstrate that question-answering accuracy scales with number of relevant documents in the training corpus, and Mallen et al. [22] show that LLMs struggle with rare entities unless assisted by retrieval. Similarly, LMEent [6] indexes entity mentions in pretraining data to study how models acquire factual knowledge. However, these studies measure popularity at the level of an entire entity (such as a drug name). Because a drug’s benefits and safety limits share the same drug name, entity-level popularity cannot explain why a model knows what a popular drug treats but not who must avoid it. We resolve this by measuring coverage at the level of individual facts and comparing benefit versus safety relations within the same drug.

RAG in medicine. Retrieval-augmented generation (RAG) is a standard way to help models answer questions when parametric memory fails [19]. In biomedicine, the MIRAGE benchmark [39] shows that retrieving passages from PubMed, textbooks, and Wikipedia improves overall accuracy. Yet existing medical RAG benchmarks report aggregate accuracy across entire question sets without separating indication questions from contraindication questions and without checking whether the retrieved text actually states the answer. Recent works on context sufficiency [12, 29] show that RAG helps only when the retrieved passage contains the needed fact. We evaluate this condition for drug safety and show that retrieval fails far more often for contraindications than for indications. Our findings complement recent systems like Bang et al. [1], which answer contraindication queries by querying structured drug databases.

How data limits what models learn. Our investigation builds on a broader observation from our prior work in scientific AI: gaps in underlying data limit what downstream models can learn. In Bio-VQA, model accuracy drops from 0.72 at phylum to 0.12 at species [28]. We observe similar granularity drops in beetle classification [26, 27], where specimen imaging protocols constrain what computer vision models can extract [5]. Likewise, knowledge-based VQA benchmarks often include questions that their own reference sources fail to support (Ma et al. 2026), and targeted evaluation sets regularly expose blind spots hidden by broad benchmarks [36]. Here, we ask the same question for medication safety: what does reference text actually record before any LLM reads it?

Data and Study Setup

Three types of drug-disease facts. We study three clinical relationships between a drug and a disease. An indication is a disease that the drug is officially approved to treat. A contraindication is a condition in which the drug must not be given because the risk of patient harm outweighs any possible benefit. An off-label use is a condition that doctors treat with the drug in clinical practice even though the use is not part of its formal regulatory approval. Indications and off-label uses describe therapeutic benefits, whereas contraindications define safety limits that protect patients from harm.

Knowledge graphs and disease ontologies. We draw our primary drug-disease facts from PrimeKG [3], a large biomedical knowledge graph that integrates 20 clinical resources. In a knowledge graph, each

fact is stored as a link, called an edge. An edge connects a drug node to a disease node under a specific relation. PrimeKG takes its drug-disease edges from DrugCentral [37], comprising 9,388 indication edges, 30,675 contraindication edges, and 2,568 off-label use edges. Every disease in PrimeKG is linked to the MONDO Disease Ontology [38], a tree-like hierarchy that organizes conditions from broad categories (such as diabetes mellitus) down to specific subtypes (type 1 diabetes mellitus). Removing overly broad top-level categories with more than 400 descendants leaves 17,801 parent-child links across 11,788 diseases. As a negative control, we evaluate Hetionet [8], an independent biomedical knowledge graph built from different underlying sources. Hetionet contains 755 treats edges (curative therapy) and 390 palliates edges (where a drug relieves symptoms without curing the underlying disease). Because both relations in Hetionet describe therapeutic benefits, it serves as a control to verify that our text-matching pipeline does not manufacture artificial gaps between relations.

Text corpora. We test how often drug-disease facts appear across three distinct text sources:

  1. English Wikipedia: We retrieve the full English Wikipedia article for each drug, representing general-purpose reference prose written for a broad audience.

  2. PubMed abstracts: We retrieve 40 relevance-ranked biomedical research abstracts per drug via the NCBI E-utilities API [32]. To make sure our search does not favor treatment papers over safety papers, we split the query budget evenly: we retrieve 20 abstracts using a therapeutic-use query and 20 abstracts using an adverse-effect and contraindication query.

  3. Official FDA drug labels: We retrieve official manufacturer prescribing labels from openFDA [14]. We search for drugs by generic name, substance name, and brand name, pooling up to five labels per drug so that medications sold by multiple manufacturers are thoroughly covered. From each label, we extract three text

views: the Indications and Usage section alone, the Contraindications section alone, and the whole label (which includes all sections, such as the Boxed Warning, Warnings and Precautions, Adverse Reactions, and Use in Specific Populations). We find official FDA labels for 472 of our 574 sampled drugs (82.2%).

Evaluation samples. To compare indications and contraindications within the same drug, we identify all 595 drugs in PrimeKG that have at least three indication edges and at least three contraindication edges, and we randomly sample exactly three of each per drug. Among these, 574 drugs have an English Wikipedia article of sufficient length, yielding a balanced text-audit sample of 1,722 indication facts and 1,722 contraindication facts. The 472 drugs with an FDA label provide 1,416 matched facts per relation. For our closed-book model scaling experiments, we draw a separate sample of 600 facts for each of the three relations (1,800 facts total). Table 1 summarizes all datasets and sample sizes.

Methodology

We run four connected experiments on the same drug facts. We (1) measure how often each corpus names each fact, (2) compare benefit coverage against safety coverage within each drug, (3) test what LLMs know from memory across model sizes and question difficulty, and (4) test how much a retrieved passage from prose or the FDA label helps.

Measuring Whether Text States a Fact

Granularity of disease mentions. For each drug-disease edge, we check whether the drug’s text mentions the disease at three levels (Table 2). Level 0 is the disease’s exact MONDO label (type 1 diabetes mellitus). Level 1 adds synonyms, lay terms and abbreviations taken from the titles of Wikipedia pages that redirect to the disease’s article (juvenile diabetes). Level 2 adds the names of the disease’s MONDO ancestors up to two levels above it (diabetes). The matcher ignores case, respects word boundaries so that asthma does not match asthmatic, and skips names shorter than four characters. We apply the same matcher to Wikipedia, PubMed and FDA labels (Appendix § 13).

LevelText containsExample
0exact name“type 1 diabetes mellitus”
1an alternative name“juvenile diabetes”
2only the family“diabetes”
3nothing-

Table 2. Levels at which a text can name a disease. Levels 0 and 1 count as named.

What counts as a named fact. We count a fact as named only at Level 0 or Level 1. A family mention (Level 2) does not count, because saying that a drug is risky in liver disease does not identify a specific condition such as cirrhosis.

Verifying medical relationships in text. A disease name in a drug’s article does not guarantee that the sentence states the clinical relationship. We extract the first sentences that name the disease and ask an instruction-tuned judge (Qwen2.5-7B) to classify each into one of four categories: (A) the drug treats or manages the disease, (B) the drug must not be used in patients with the disease, (C) the drug may cause or worsen the disease or requires caution, or (D) the sentence names both without a clinical link. A sentence supports an indication when the judge selects (A), and a contraindication when it selects (B) or (C). Four human annotators independently labeled 160 matched sentences with the graph relation hidden, to validate the judge (Appendix J).

Control for random matches. A disease name may appear in a drug’s article by coincidence. To measure this random floor, we pair each drug with three unrelated diseases from the same relation and run the same matcher. On Wikipedia, unrelated diseases match in only 1.22% of indication pairs, 0.72% of off-label pairs and 0.78% of contraindication pairs. Full FDA labels are much longer, so their random floor is higher (2.10% and 2.82%), but still an order of magnitude below real match rates.

Comparing Each Drug Against Itself

Within-drug odds ratio. Commonly prescribed drugs have long Wikipedia articles, dozens of PubMed papers and extensive FDA labels. Comparing indications from famous drugs against contraindications from obscure drugs would confound relation type with drug popularity. We remove this confounder by comparing indications and contraindications within each drug. For each drug dd, we form a 2×22 \times2 table of its named indications (Id+I_d^{+}), unnamed indications (Id−I_d^{-}), named contraindications (Cd+C_d^{+}) and unnamed contraindications (Cd−C_d^{-}), where NdN_d is its number of sampled facts. We pool these tables over the set of drugs D\mathcal{D} with a Mantel-Haenszel odds ratio, ORMH_{\mathrm{MH}} [23]:

ORMH=∑d∈DwdORd∑d∈Dwd,ORd=Id+Cd−Id−Cd+,wd=Id+Cd−+Id−Cd+Nd.(1)\mathrm{OR}_{\mathrm{MH}} = \frac{\sum_{d \in\mathcal{D}} w_d \mathrm{OR}_d}{\sum_{d \in\mathcal{D}} w_d}, \qquad\mathrm{OR}_d = \frac{I_d^{+} C_d^{-}}{I_d^{-} C_d^{+}}, \qquad w_d = \frac{I_d^{+} C_d^{-} + I_d^{-} C_d^{+}}{N_d}. \tag*{(1)}

The odds of a fact being named are the probability that the text names it divided by the probability that it does not. ORMH_{\mathrm{MH}}=1 means that a drug’s benefits and safety limits are named at equal rates. ORMH_{\mathrm{MH}}=2 means that, for the same drug, the odds of naming a benefit are twice the odds of naming a safety limit. The Cochran-Mantel-Haenszel test gives the p-value.

Control for the number of facts per drug. In PrimeKG, drugs have about 3× as many contraindication edges as indication edges. A drug with dozens of contraindications may have some omitted simply because no summary lists every risk. We control for this in two ways. First, a logistic regression predicts whether a fact is named from its relation type, holding fixed the logarithm of the drug’s edge count in that relation. Second, we group drugs by their edge count in the relation (3 to 5, 6 to 12, and 13 or more) and measure the gap within each group.

Testing What LLMs Know

Closed-book knowledge probe. To test what LLMs know without retrieved text (closed book), we turn the 1,800 probe facts into five-option questions. Each question pairs the true disease with four random diseases from the same relation that have no link to the drug. We append each option to a relation stem, such as “X is a drug used to treat the disease called”, and pick the option with the highest mean token log-probability. Random guessing gives 20% accuracy. We evaluate five instruction-tuned Qwen2.5 models: 0.5B, 1.5B, 3B, 7B and 14B (Qwen Team 2024) (Appendix § 20). Because all five models answer the same questions, we compare their gains from scale fact by fact, and test the difference with a paired bootstrap over drugs.

Harder Distractors, Open-Book Retrieval, and Routing

Harder options with an abstention choice. Random distractors can sometimes be ruled out by basic medical intuition. We therefore build a harder set of 450 indication and 406 contraindication questions whose four wrong options are MONDO siblings of the true disease first, then cousins. A sixth option, “Not enough information to tell”, lets the model abstain instead of guessing. We stratify the questions (up to 150 per group and relation) by whether the drug’s prose passage names the exact disease, names only its family, or is silent. We evaluate Qwen2.5-0.5B, Qwen2.5-7B, OLMo-2-7B, and Granite-3.1-8B models.

Open-book evaluation with retrieved prose. We ask the 856 harder questions twice: once closed book, and once with a retrieved passage in the prompt. The passage is a cleaned excerpt of at most 1,800 characters from the drug’s Wikipedia article and PubMed abstracts, built around any mention of the disease or its family (Appendix § 21).

Routing questions to FDA labels. Finally, we test whether routing safety questions to the FDA label helps. For the 726 questions whose drug has a label, we cut a passage of the same length from the whole label. The routed setup uses the prose passage for indication questions and the label passage for contraindication questions (Appendix § 22).

Results and Analysis

Text Favors Benefits 2× over Safety Facts

The Label gap. Wikipedia names 20.33% of a drug’s indications but only 9.81% of its contraindications (Table 3 and Figure 2). Comparing each drug against itself, the odds that Wikipedia names a benefit are 2.43 times the odds that it names a safety limit. PubMed abstracts show a wider gap, 26.31% against 11.44%, with an odds ratio of 2.77. Even with Wikipedia and PubMed combined, the odds ratio stays

CorpusInd.Contra.ORMH\mathrm{OR}_{\mathrm{MH}}pp
Wikipedia20.339.812.432×10−182 \times10^{-18}
PubMed26.3111.442.775×10−295 \times10^{-29}
Both29.0415.912.192×10−202 \times10^{-20}

Table 3. Share of facts (%) that text names, within-drug odds ratio of indication against contraindication and its Cochran-Mantel-Haenszel (pp): 574 drugs, 1722 edges per relation.

Figure 2. Percentage of facts named in each prose corpus for the same 574 drugs (1,722 facts per relation). Numbers above each pair of bars give the within-drug odds ratio (ORMH\mathrm{OR}_{\mathrm{MH}}); values above 1 mean the corpus names benefits more often than safety limits.

at 2.19. A reader or retriever that checks both sources finds nearly three in ten indications (29.04%) but fewer than one in six contraindications (15.91%).

Dedicated searches do not close the contraindication gap. Our PubMed search spends half of its budget on adverse effects and contraindications. Compared with a single general query of 40 abstracts, this balanced search raises contraindication coverage by only 1.22 points (10.22% to 11.44%) and indication coverage by 2.97 points (23.34% to 26.31%). The odds ratio is 2.77 under both searches. Toxicity searches retrieve safety papers, but those papers rarely name the specific conditions that rule a drug out.

The safety gap is widest for drugs with few facts. Figure 3 splits the gap by how many facts a drug has in PrimeKG. In Wikipedia, the gap is +13.94 points for drugs with 3 to 5 facts, +6.04 points for 6 to 12 facts, and +1.64 points for 13 or more. Wikipedia and PubMed combined show the same pattern (+11.81, +6.90 and +2.23 points). With the

Figure 3. Coverage gap (indication minus contraindication naming rate, in points) grouped by the drug’s number of facts in that relation. Positive bars mean the text favors indications. Prose favors indications in every group, whereas the FDA label does not.

TextInd.Con.ORMH_{\mathrm{MH}}Deg.
Wikipedia22.2510.95
Wiki + PubMed30.5117.23
Label, own section19.072.619.85ind.
Label, whole23.7322.881.05contra.
All three34.0426.691.43none

Table 5. Share of facts (%) named on the same 1,416 edges per relation (472 drugs with a label). Deg. is the relation that text favors once the drug’s edge count is fixed in the logistic model: ind. and contra. are significant at p<0.01p<0.01, and none has p=0.86p=0.86. Within-drug pp-values: 10−4510^{-45}, 0.620.62 and 2×10−52 \times10^{-5}.

edge count held fixed in the logistic model, contraindications remain less likely to be named (p=2.4×10−4p=2.4 \times10^{-4} in Wikipedia, 1.4×10−51.4 \times10^{-5} in PubMed and 9.7×10−49.7 \times10^{-4} combined).

Benefits are often written at the family level, safety limits are not. Relaxing the matcher shows where the missing mentions go. On Wikipedia, alternative names add only 1.39 points on average. Family names up to two levels add 14.06 points for indications but only 1.72 points for contraindications, after subtracting the random floor. Prose often says that a drug treats a broad group of diseases, but it rarely names a contraindication even at the family level.

Matched sentences state clinical relationships. Table 4 counts a fact only when its sentence states the relation, as read by the Qwen2.5-7B judge. The four annotators agree closely with one another (Fleiss’ κ=0.89\kappa=0.89), and the judge matches their majority vote on 95.5% of sentences (Cohen’s κ=0.92\kappa=0.92), where κ=1\kappa=1 is perfect agreement and κ=0\kappa=0 is chance. Under this stricter count, the prose gap remains (odds ratios of 2.70 in Wikipedia and 2.09 in PubMed), and the whole FDA label remains balanced (0.88). The annotators also show how safety facts are phrased: 82% to 83% of matched contraindication sentences are warnings or precautions, and only 10% to 11% are formal prohibitions. This is why a keyword filter agrees poorly with the annotators (κ=0.43\kappa=0.43).

CorpusNamedStatedShare (ind./con.)
Wikipedia2.682.7093.4 / 92.0
PubMed2.832.0969.6 / 86.6
Whole label1.050.8879.5 / 90.7

Table 4. Within-drug odds ratio when a fact counts as named, and when its sentence must also state the relation, as read by the model judge. Here we recompute named facts sentence by sentence; they agree with Tables 3 and 5 on 97.9% to 100% of edges. The last column gives the share of named facts (%) whose sentence states the relation.

FDA Labels Show No Coverage Gap

The FDA label names both relations. If contraindications were missing from prose because they are obscure edge cases, they would be missing from regulatory documents too. They are not. The whole FDA label names 23.73% of indications and 22.88% of contraindications on the same facts (Figure 4

Figure 4. Percentage of facts named for the 472 drugs with an FDA label (1,416 facts per relation). Own section checks only Indications and Usage for indications and only Contraindications for contraindications. Numbers above the bars give ORMH_{\mathrm{MH}}. The whole label shows no gap (ORMH_{\mathrm{MH}}=1.05).

and Table 5), an odds ratio of 1.05 (p=0.62p=0.62). With the edge count held fixed, the label slightly favors contraindications (coefficient +0.30, p=0.005p=0.005).

Safety facts sit across the whole label. The Contraindications section alone names only 2.61% of PrimeKG’s contraindications, and it is blank for 9.5% of contraindication edges. Most safety facts sit in the Boxed Warning, Warnings and Precautions and Use in Specific Populations sections. To check that the matcher did not miss reworded text, four annotators read 50 contraindications that the section failed to match. In 45 of the 49 cases with a majority vote, the condition is absent from the section, and only 3 state it in other words. A clinical retriever must therefore index the whole label, not just its Contraindications section.

The label rescues safety facts that prose misses. Adding the whole label to Wikipedia and PubMed raises coverage to 34.04% for indications and 26.69% for contraindications, and the odds ratio falls to 1.43. With the edge count held fixed, the gap disappears (p=0.86p=0.86). Of the 1,172 contraindications that both prose corpora miss, the label names 11.4%, more than twice its rate for missed indications (5.1% of 984). The label asymmetry is therefore a property of general prose, not of the facts themselves.

Scaling Helps Benefits, Not Safety Facts

Closed-book accuracy across five model sizes. Across a 28-fold increase in parameters, indication accuracy climbs from 31.50% at 0.5B to 50.83% at 7B and 49.83% at 14B (Figure 5; Appendix H). Contraindication accuracy stays between 22.17% and 27.17% at every size, barely above the 20% chance rate. Off-label uses fall in between, rising from 28.50% to 35.83%. At 7B and 14B, models answer about half of the indications correctly but do little better than chance on contraindications, a gap of 22.7 to 27.0 points. Paired fact by fact from Qwen2.5-0.5B to Qwen2.5-7B, indications gain 19.33 points and contraindications only 1.67 points (p=4.8×10−12p=4.8 \times10^{-12}). From 0.5B to 14B, the gains are 18.33 points for indications, 7.33 for off-label uses and 5.00 for contraindications.

Figure 5. Closed-book five-choice accuracy (%) across five Qwen2.5 sizes (0.5B to 14B) on 600 facts per relation. Shaded bands show 95% intervals, and the dashed line marks the 20% chance rate. Scale improves accuracy on indications but leaves contraindications near chance.

Accuracy tracks text coverage. The three relations rank in the same order in model accuracy as in prose coverage: indications first, off-label uses second and contraindications last. Split by whether Wikipedia names the fact, scaling from 0.5B to 14B improves accuracy by 17.5 to 26.7 points on named facts but only 4.1 to 16.6 points on unnamed facts. Wikipedia names only 40 of the 600 probe contraindications, against 101 of the 600 indications, so larger models have few written safety facts to learn from. Even among named facts, Qwen2.5-14B reaches 84.2% on indications but 40.0% on contraindications. One likely reason is that an article states a drug’s uses up front, while its warnings sit in long lists.

The gap persists with harder options. With the MONDO siblings as wrong options and an abstention choice, every model shows a large closed-book gap favoring indications (Table 6): +7.9 points for Qwen2.5-0.5B, +30.9 for Qwen2.5-7B, +23.0 for OLMo-2-7B and +45.9 for Granite-3.1-8B.

ModelInd.Contra.GapWithinpp
Qwen2.5-0.5B36.028.1+7.9+8.20.009
Qwen2.5-7B56.025.1+30.9+31.3<10−15<10^{-15}
OLMo-2-7B56.033.0+23.0+23.62×10−132 \times10^{-13}
Granite-3.1-8B58.712.8+45.9+46.3<10−15<10^{-15}

Table 6. Closed-book accuracy (%) with wrong options from the answer’s MONDO neighborhood and a Not enough information option; 450 indication and 406 contraindication questions. Within is the gap computed within groups of questions whose page names the answer, its family or nothing. The pp-value tests this within-group gap.

LLM failures: wrong guesses or abstention. The abstention option reveals two failure styles on contraindications

(Figure 6). Qwen2.5-0.5B and OLMo-2-7B rarely abstain and answer 71.4% and 64.0% of contraindication questions wrongly. Qwen2.5-7B and Granite-3.1-8B abstain on more than half of them (50.7% and 67.5%), against only 22.0% and 12.7% of indication questions. Either way, precision on the contraindication questions a model does answer is 9 to 28 points lower than on indications (Appendix I).

Figure 6. Closed-book outcomes on the harder questions (450 indication and 406 contraindication questions), split into correct answers, wrong guesses and abstentions (Declines). On contraindications, Qwen2.5-0.5B and OLMo-2-7B mostly guess wrong, whereas Qwen2.5-7B and Granite-3.1-8B mostly abstain.

RAG helps only when the fact is written

Models read written safety facts just as well as benefits. If LLMs fail on contraindication questions because they could not handle safety phrasing, a passage stating a safety fact would help less than one stating a benefit. It does not (Figure 7). When the passage names the disease, accuracy rises by 14.2 points on contraindications and 11.1 points on indications across the three 7B to 8B models. Every model shows the same pattern (Appendix I).

Figure 7. Change in accuracy (points) when a retrieved prose passage is added to the prompt, averaged over Qwen2.5-7B, OLMo-2-7B and Granite-3.1-8B, grouped by whether the passage names the disease, names only its MONDO family, or is silent. When the passage names the disease, models gain as much on safety limits (+14.2) as on benefits (+11.1).

Silent passages versus family mentions. When a passage is silent, accuracy falls by 11.3 points on indications and 8.9 points on contraindications, because the models abstain more (Not enough information rises by 26.0 and 32.9 points). This is the right behavior when the context lacks the answer. When a passage names only the disease family, the models abstain less than with a silent passage and often pick a sibling disease instead. Indication accuracy falls by 8.2 points, and contraindication accuracy does not change (+0.3).

Routing safety questions to FDA labels. Table 7 replaces the prose passage with a label passage. On the 135 contraindication questions whose label names the condition, accuracy rises from 40.0% to 55.8%, a gain of 15.8 points, and abstentions fall by 12.1 points. When the label does not name it (211 questions), the label passage lowers accuracy slightly (−4.9 points). For indications, the label adds little when it names the disease (+3.4) and hurts when it does not (−11.0). Prose is the better source for benefit questions, and label is better for safety questions whose answer it states.

Rel.In labelnnProseLabelDiff.
Contra.yes13540.055.8+15.8
no21117.112.2−4.9
Ind.yes11784.387.7+3.4
no26344.933.8−11.0

Table 7. Accuracy (%) with a prose page and with a whole-label page, pooled over the three 7B to 8B models, split by whether the whole label names the answer (In label).

Negative Control on Hetionet Shows No Gap

Hetionet’s treats and palliates relations both describe benefit, so a sound matcher should find no gap between them. On Wikipedia, it names 57.33% of treats facts and 56.10% of palliates facts, an odds ratio of 1.05 (p=0.70p=0.70). Among the 51 drugs that have both relations, the within-drug odds ratio is 2.48, but this is not significant once the edge count is held fixed (p=0.16p=0.16), and its sign flips across edge-count groups (Appendix § 23). Under the same controls, the PrimeKG gap stays significant in every prose corpus. With the FDA label result, this control shows that the Label asymmetry comes from the prose, not from the measurement.

Discussion

Medication safety is the blind spot of general prose. Two independent prose corpora, Wikipedia articles written for the public and PubMed abstracts written by researchers, show the same bias: the facts that matter most for preventing harm are named about half as often as the facts about what a drug treats. FDA labels, which must document both efficacy and safety, show no such gap. One explanation is that prose is written around a drug’s purpose. An encyclopedia article introduces a drug by what it treats, and most clinical papers study whether a therapy works. Even papers on toxicity rarely list the preexisting conditions that make a drug unsafe. Scale does not fill this gap: across a 28-fold increase in size, contraindication accuracy stays near chance, and larger models change how they fail, from wrong guesses to abstention, rather than whether they know (Figure 6).

Design principles for clinical AI agents. Any clinical agent that answers medication questions from its own weights or from general prose inherits the Label asymmetry. Figure 8 shows a relation-aware design that follows from our results, of which we test the routing step.

Figure 8. A relation-aware clinical agent workflow that follows from our findings. (1) A planner classifies whether the question asks about a drug’s benefits (indications) or its safety limits (contraindications), and routes it to general prose for benefits, or to structured databases and the whole FDA label for safety limits. (2) A verifier checks whether the retrieved passage names the exact condition, names only its family, or is silent, and the agent then answers with a citation, expands the family through the ontology, or abstains.

  1. Relation-aware routing. A planner first decides whether the question is about benefit or about safety. Benefit questions can go to general prose, which names 29% to 31% of indications. Safety questions should go to structured drug databases such as DrugCentral and to the whole FDA label. The label’s Contraindications section alone names only 2.6% of PrimeKG’s contraindications, against 22.9% for the whole label.

  1. Verification before answering. A lightweight verifier checks how the retrieved text names the condition. Our string and ontology matcher is fast enough for this check. The routing test shows why it matters: the label passage raises contraindication accuracy by +15.8 when it names the condition, and does not help when it is silent (Table 7).

  1. Handling coarse or silent evidence. With an exact or alternative-name match, the agent answers and cites the sentence. With only a family mention, it treats the passage as insufficient and uses the ontology to check specific subtypes. With silent text, it abstains and tells the user which source it checked.

Limitations

Corpora and model weights. Model developers do not release their exact pretraining corpora, so the link between prose coverage and closed-book accuracy is a correlation, not a trace of pretraining tokens. The same gap in two prose sources, its absence in FDA labels, and model accuracy that follows prose coverage across three relations make pretraining exposure a plausible mechanism, but we do not establish it. Our PubMed analysis uses abstracts only, so its coverage is a lower bound for the full literature.

String matching and paraphrase. String matching miss clinical paraphrases and match a disease mentioned in passing. The random floor bounds false positives at under 1.3% in prose and 2.9% in FDA labels. The LLM judge, validated by annotators, finds that 87% to 93% of named contraindications and 70% to 93% of named indications state the relation.

Models and question format. The scaling curve uses one model family (Qwen2.5). The harder-option and open-book tests add two more families (OLMo-2 and Granite-3.1) at the 7B to 8B scale. Multiple choice measures whether a model recognizes the correct disease, which is an upper bound on its ability to recall it in free-form answers.

Granularity of graph facts. Even the whole FDA label names only 22.88% of PrimeKG’s sampled contraindications, because PrimeKG often records a specific subtype

(adult-onset myasthenia gravis) while text states the parent condition (myasthenia gravis). A clinical review of unmatched facts would separate such granularity differences from true omissions.

Conclusion and Future Work

General prose names a drug’s benefits about twice as often as its safety limits, the FDA label does not. And LLMs follow the former. Several directions follow from this result. First, OLmo-2 releases its pretraining data, so the link between corpus and model can be traced directly by counting how often each fact appears in the tokens a model was trained on. Second, the audit can extend to full-text articles, clinical guidelines and other languages, and to safety relations beyond contraindications, such as drug interactions. Third, free-form answers and medical LLMs would show whether the gap holds beyond medical choice and general-purpose models. Fourth, a clinical review of facts that no text names would separate graph granularity from true omission. Finally, we plan to build and evaluate the full relation-aware agent of Figure 8 end to end, with a verifier that checks every safety answer against the label before the agent responds.

Supplementary Material

This supplement gives every detail needed to read, check and rebuild the results in the main paper. It holds the full data counts, the exact search queries and prompts, the matching rule and its pseudocode, every statistic, every table behind each figure, the annotation study, the routing test, the negative control and worked case studies. Each appendix stands on its own, so a reader can go straight to the part they need.

How to read the tables. Unless a caption says otherwise, a percentage is the share of edges or questions, a difference is in percentage points, brackets hold a 95% interval, and Ind. and Contra. stand for indication and contraindication. A gap is always indication minus contraindication, so a positive gap means text or models do better on benefit facts.

A note on terms. An edge is one drug-disease fact in a knowledge graph. A fact is named when the drug’s text contains the disease’s exact name or an alternative name (levels 0 and 1 in Table 2). Family only means the text names only a broader MONDO term. Silent means it names neither. A model declines when it picks the option Not enough information to tell. The code numbers the levels 0 (exact), 2 (alternative name), 3 (family) and 4 (none), and keeps level 1 for a synonym list that we do not use, so these four code levels map to levels 0, 1, 2 and 3 of the paper.

Overview and Reproducibility

This appendix gives the workflow of the whole study, the scripts behind each stage, the data versions, and the commands that rebuild every number. It ends with a summary of every check we run against the main finding.

Workflow

The study has one idea: hold the drug fixed and compare its two kinds of fact. The pseudocode below shows the whole workflow. Each step is a function described in a later appendix.

# 1. Facts (Appendix B)
G   = PrimeKG drug-disease edges
H   = MONDO hierarchy, broad terms removed
S   = 3 indications + 3 contraindications
    for every drug that has 3 of each

# 2. Text (Appendices C, D)
for drug d in S:
    T["wiki"]  = wikipedia(d)
    T["pm"]    = pubmed(d, "treat", 20)
              + pubmed(d, "harm", 20)
    T["label"] = openfda(d, max_labels=5)
    for edge (d, r, y) in S[d]:
        for c in T:
            level[c] = name_level(T[c], y, H)
            floor[c] = mean(name_level(T[c], y2, H)
                            for 3 unrelated y2)

# 3. Within-drug test (Appendices E, F, G)
for corpus c:
    OR, CI = mantel_haenszel(level[c] <= 1,
                             strata = drugs)
    coef   = logistic(named ~ relation
                      + log(edges of d in r))

# 4. Models (Appendices H, I, K)
for model m, edge e:
    closed[m, e] = answer(m, question(e))
    prose[m, e] = answer(m, page_prose(e)
                           + question(e))
    label[m, e] = answer(m, page_label(e)
                          + question(e))
compare by relation and by naming group

# 5. Checks (Appendices J, L)
judge(sentence) states the relation?
same test on Hetionet: two benefit relations

Pipeline

The analysis runs as six stages. Each stage is a shell function in pipeline.sh that calls standalone Python scripts and reads only the files of the stages before it. Table 8 lists the stages and their scripts. One command, bash run_all.sh, runs all six from the public downloads, submits the model jobs to the cluster scheduler, waits for them and writes every table. A second script, relcheck.sh, runs the relation check, scores the annotation sheets and runs the routing test.

StageScriptsWhat it does
download-Downloads PrimeKG, MONDO, the Disease Ontology and Hetionet, and records their checksums.
prepareprep_primekg, mondo_parents, prep_hetionetSplits the graphs by relation and builds the disease hierarchy tables.
textgonogo_p1, drug_matched_sample, second_corpus, retier, name_tiersDraws the samples, fetches Wikipedia and PubMed, and assigns every edge a naming level and a floor.
extendfetch_labels, label_coverageFetches FDA labels and measures naming in each label section.
modelsrun_probe_cpu, abstain_probeRuns the closed-book probe and the harder-option and page tests as cluster jobs.
analysisdilution_test, mh_rbg, w3_stats, w3_extras, w3_failure, w3_summaryComputes every odds ratio, strata table, paired gain and page effect.
relcheckrelcheck, route_reportRuns the judge, the keyword check, the annotation scoring and the routing test.

Table 8. Pipeline stages and the scripts each stage runs.

Caching. Every network request is cached on disk. A rerun reads the cache and makes no request. A failed request is never cached, so a network error is never stored as an empty result. Each stage writes a log with its request count, throttle events and unrecoverable failures. Every stage in our run reports zero unrecoverable failures.

Data Versions and Compute

Table 9 gives the files of our run with the first characters of their SHA-256 checksums. We downloaded all files on 2026-10-28. Wikipedia, PubMed and openFDA change over time, so we release our cached copies of every article, abstract and label.

FileVersionSHA-256 (start)
PrimeKG kg.csvDataverse0a5a010fcd88
MONDO mondo.obo2026-10-062631ee8165d4
DOID doid.obo2026-09-3051e717b6b9f5
Hetionet edgesv1.0611f0411ac66
Hetionet nodesv1.03ad9eb86c031

Table 9. Data files of our run and the start of their checksums.

Compute. All model scoring runs on CPU nodes of a university supercomputer, with PyTorch 2.14 and Transformers 5.20 in 32-bit floating point. No GPU is needed. Each model job has its own time limit by size, and the analysis job starts only after every model job has finished. The statistics use only the Python standard library, and every random draw uses seed 0.

Released files. PrimeKG, Hetionet, MONDO, Wikipedia, PubMed and openFDA are public. We release the sampled edge lists, the naming level of every edge in every corpus, every model score, the annotation sheets and the scripts, so every table in the paper and in this supplement can be rebuilt from the released files without any network access.

Summary of Checks

Table 10 lists the alternative explanations for the gap and the tests that address them. Each row points to the appendix with the full result. The FDA label rules out the explanation that the facts are too rare to write down in any source.

ExplanationTestResult
Famous drugs dominateWithin-drug odds ratio (Appendix F)2.19-2.77 in prose
Many contraindications per drugLogistic model, edge-count strata (Appendix F)Gap holds, p≤10−3p \le10^{-3}
One editorial processWikipedia and PubMed separately (Appendix F)2.43 and 2.77
Retrieval favors efficacySplit PubMed budget (Appendix C)OR 2.77 both ways
Matcher favors a relationHetionet, two benefit relations (Appendix L)No gap, p=0.16p = 0.16
Mention without relationJudge, four annotators (Appendix J)Prose 2.09-2.70, label 0.88
Matcher misses wordingRecall check, 50 facts (Appendix J)6% of misses
Coincidental matchesFloor on unrelated diseases (Appendix D)≤1.2%\le1.2\% prose, ≤2.8%\le2.8\% label
Facts too rare to writeWhole FDA label (Appendix G)OR 1.05, p=0.62p = 0.62
Small models onlyFive sizes, 0.5B-14B (Appendix H)Contra. 22-27% at every size
Easy wrong optionsMONDO siblings, 4 models (Appendix I)Gap 7.9-45.9 points
Models cannot use safety textPage in the prompt (Appendix I)+14.2 vs. +11.1 points
Label pages help everywhereRouting split (Appendix K)Only when the label names it

Table 10. Alternative explanations for the gap, the tests that address them and where to find each result.

Data and Samples

This appendix gives the counts behind Table 1, explains how we build the disease hierarchy, and shows how each sample is drawn.

SourceItemCount
PrimeKGindication edges9,388
contraindication edges30,675
off-label use edges2,568
Hetionettreats / palliates edges755 / 390
MONDOparent links / diseases17,801 / 11,788
Text sampledrugs574
edges per relation1,722
FDA label sampledrugs with a label472
edges per relation1,416
Probe sampleedges per relation600
model sizes (Qwen2.5)5
Page testind. / contra. questions450 / 406
models (families)4 (3)

Table 1. Data in this study. An edge is one drug-disease fact in a knowledge graph. The lower block gives the samples we draw for each test.

Knowledge Graphs

PrimeKG. PrimeKG [3] ships as one edge file of 8,100,498 rows. We keep its 、】【 three drug-disease relations and its disease-disease links (Table 11). The drug-disease relations come from DrugCentral (Ursu et al. [37]). A drug has, on average, about three times as many contraindication edges as indication edges, which is why we test the edge count directly (Appendix § 18).

PrimeKG relationEdges
indication9,388
contraindication30,675
off-label use2,568
disease-disease64,388
all rows in kg.csv8,100,498

Table 11. PrimeKG relations we use. The contraindication to indication ratio is 3.27.

Hetionet. Hetionet v1.0 (Himmelstein et al. [8]) has 47,031 nodes. Its treats relation has 755 edges over 387 drugs and 77 diseases, and its palliates relation has 390 edges over 221 drugs and 50 diseases. Both relations describe benefit, which makes Hetionet a negative control (Appendix § 23).

Disease Hierarchy

MONDO. We read the MONDO release of 2,026-10-6, which has 32,134 terms and 46,732 is a links. We keep the links whose child carries a name in PrimeKG. Of the 17,008 MONDO identifiers named in PrimeKG, 14,591 (85.8%) exist in this release. A family name is only useful if it is specific enough to point to a small group of diseases. We therefore drop every link whose parent has more than 400 descendants. This removes 5,857 links and leaves 17,801 links over 11,788 diseases, with at most 9 parents per disease. Table 12 lists the broadest terms that the cut removes. Without the cut, the phrase “disease” would make almost every disease count as named at the family level.

MONDO termDescendants
disease or disorder31,582
human disease23,262
disease by body system or component18,875
disease by etiologic mechanism17,810
disease of genetic or genomic mechanism12,256
Mendelian disease11,784
non-human animal disease8,318
disease by developmental or physiological process7,322

Table 12. The broadest MONDO terms, all removed as family names by the cut at 400 descendants.

Disease Ontology. Hetionet names diseases with the Disease Ontology (Schriml et al. [33]), so we build its hierarchy the same way from the release of 2,026-9-30. It has 12,335 terms. After the same cut, it gives 103 links over 74 of Hetionet’s diseases. Examples are Crohn’s disease under inflammatory bowel disease and Parkinson’s disease under synucleinopathy.

Samples

Text sample. A within-drug comparison needs drugs that have both relations. We keep the 595 drugs with at least three indication and three contraindication edges and draw exactly three of each, so every drug contributes a balanced table. This gives 1,785 edges per relation. Of these drugs, 574 have an English Wikipedia article long enough to measure, which leaves 1,722 edges per relation. At this size, and at naming rates near 20% and 7%, the within-drug test detects an odds ratio of about 1.6 or larger. The 472 drugs with an FDA label give 1,416 edges per relation.

Probe sample. The model probe uses a separate draw of 600 edges for each of indication, contraindication and off-label use, 1,800 edges in all over 817 Wikipedia articles. This draw is not balanced within drugs. The probe compares models on the same edges, so the pairing comes from the edges themselves.

Harder-option sample. The harder-option test reuses the question set of our companion study on declining. It draws up to 150 questions for each relation and naming group, which gives 450 indication and 406 contraindication questions. Only 106 contraindications in the pool are named only at the family level, so that group is smaller.

Hetionet sample. We draw up to 600 edges per relation. Palliates has only 390 edges, of which 385 have a measurable article, so the sample has 600 treats and 385 palliates edges.

How plainly the samples are named. Table 13 gives the naming level of every edge in each sample, measured on Wikipedia. Most PrimeKG facts are silent, while most Hetionet facts are named. Hetionet’s diseases are fewer and broader, so its articles name them more often. This difference does not matter for the control, because the control compares two relations inside Hetionet.

SampleEdgesExactAlt.FamilyNone
Text (PrimeKG)3,44413.21.89.875.1
Probe (PrimeKG)1,80010.31.010.178.6
Hetionet98554.52.35.337.9

Table 13. Naming level of every sampled edge in Wikipedia (%), both relations together.

Corpora

This appendix explains how we fetch each corpus and how we check that retrieval does not create the gap.

Wikipedia

For each drug, we fetch the current English Wikipedia article by title through the MediaWiki API and cache its source text. We use the article as fetched, with its markup. The relation check removes markup before it splits sentences (Appendix J). The 574 drugs of the text sample have 574 articles, and the 1,800 probe edges come from 817 articles.

PubMed

Queries. We retrieve abstracts through the NCBI E-utilities ([32]), sorted by relevance. Each drug gets a budget of 40 abstracts, split evenly between two searches with 20 abstracts each. Both searches require the drug’s name in the title or abstract. One adds a fragment about therapeutic use, and the other adds a fragment about harm:

"<drug>"[Title/Abstract] AND
    (treat*[tiab] OR therap*[tiab]
    OR efficac*[tiab])

"<drug>"[Title/Abstract] AND
    (contraindicat*[tiab] OR adverse[tiab]
    OR toxicit*[tiab] OR "side effect"[tiab]
    OR "side effects"[tiab] OR induced[tiab])

We fetch the union of returned records once, join the titles and abstracts into one text per drug, and cache it. A text shorter than 400 characters counts as absent, which affects 12 of 3,444 edges. The balanced run makes 1,721 requests with no unrecoverable failure.

Why a balanced search. A search for a drug’s name alone, sorted by relevance, could return mostly efficacy papers and so create the gap by retrieval. The first version of the corpus used exactly such a search, with 40 abstracts per drug. The balanced search gives the harm fragment the same budget as the treatment fragment. If retrieval created the gap, the balanced search would shrink it. It does not: both versions give a within-drug odds ratio of 2.77 (Table 18).

CorpusInd.Contra.OR [RBG 95% CI]CMH pp
Wikipedia20.339.812.43 [1.98, 2.98]1.9×10−181.9 \times10^{-18}
PubMed, balanced26.3111.442.77 [2.30, 3.33]5.5×10−295.5 \times10^{-29}
PubMed, sgl.23.3410.222.77 [2.27, 3.37]1.9×10−251.9 \times10^{-25}
Wiki + PubMed29.0415.912.19 [1.85, 2.59]1.7×10−201.7 \times10^{-20}
Wiki + PubMed, sgl.26.6615.102.10 [1.76, 2.49]3.0×10−173.0 \times10^{-17}
Label, own section19.072.619.85 [6.71, 14.45]1.3×10−451.3 \times10^{-45}
Label, whole23.7322.881.05 [0.88, 1.25]0.62
All three34.0426.691.43 [1.21, 1.68]2.2×10−52.2 \times10^{-5}
Hetionet57.3356.102.48 [1.21, 5.08]0.017

Table 18. Share of facts named (%) and the within-drug odds ratio. Sgl. is the first, single-query PubMed corpus. Prose rows use 1,722 edges per relation over 574 drugs, label rows 1,416 edges over 472 drugs. For Hetionet, the columns are treats and palliates, and the odds ratio uses the 51 drugs that have both. Test-based intervals: Wikipedia [2.01, 2.94], PubMed [2.33, 3.28], PubMed single [2.31, 3.32], Wiki + PubMed [1.86, 2.57], Wiki + PubMed single [1.77, 2.48], label own [7.75, 12.51], label whole [0.88, 1.25], all three [1.21, 1.68], Hetionet [1.29, 4.75].

The silent share. Table 14 gives the share of facts that a corpus does not name, under each version. Adding PubMed to Wikipedia shrinks the silent share by 7.40 points with the balanced search and 5.81 points with the single search. The silent share stays far higher for contraindications in every column.

SearchRelationWikipediaPubMedUnion
Balancedcontraindication90.1988.5684.09
indication79.6773.6970.96
both84.9381.1377.53
Singlecontraindication90.1989.7884.90
indication79.6776.6673.34
both84.9383.2279.12

Table 14. Share of facts (%) that a corpus does not name (level above 1), 1,722 edges per relation.

FDA Label

Finding a label. We query the openFDA drug label service ([14]) by generic name, then by substance name, then by brand name, and keep the first search that returns a label. A drug sold by several manufacturers has one label from each, so we pool up to five labels per drug. We find a label for 472 of the 574 drugs (82.2%).

Sections. From each label we keep three texts. The Indications and Usage section is the indication’s own section. The Contraindications section is the contraindication’s own section. The whole label joins the text of every section, including the Boxed Warning, Warnings and Precautions, Adverse Reactions and Use in Specific Populations. We measure indications and contraindications in their own section and in the whole label (Appendix § 19).

Combination products. A search by generic name can return the label of a combination product. For example, a search for tretinoin can return a label for clindamycin and tretinoin, whose warnings about colitis come from clindamycin. We keep these labels, because a retriever would return them too. Appendix § 24 shows one such case.

Measuring Whether Text Names a Fact

This appendix explains how we decide whether a drug’s text names a disease. The same rules apply to Wikipedia, PubMed and the FDA label.

Names and Matching

Three sets of names. Each disease has three sets of names. The exact name is its MONDO label [38]. The alternative names are the titles of Wikipedia pages that redirect to the disease’s article, which bring in synonyms, lay terms and abbreviations. For example, juvenile diabetes redirects to type 1 diabetes. The family names are the labels of the disease’s MONDO ancestors up to two levels above it, after the cut of Appendix B. On the draw of Table 15, a disease has 1.5 to 2.1 family names on average, depending on the relation. The sets are nested: a text that names a disease at a finer level also counts at every coarser level, and each edge gets the finest level it reaches (Table 2).

Matching rule. A phrase matches when it appears on word boundaries, ignoring case, with an optional trailing “s” for plurals. A multi-word phrase matches across any run of white space, so a line break inside a name does not break the match. We never match phrases shorter than four characters, so short abbreviations do not create chance hits. Under this rule, “asthma” does not match “asthmatic”, and “colitis” does not match inside a longer word.

def matches(text, names):
    for n in names:
        if len(n) < 4:
            continue
        words = n.split()
        pat = (r"\b" + r"\s+".join(
                    re.escape(w) for w in words)
                    + r"s?\b")
        if re.search(pat, text, re.I):
            return True
    return False
def name_level(text, y, H):
    if matches(text, [label(y)]):       return 0
    if matches(text, redirects(y)):     return 1
    if matches(text, ancestors(y, H,
                               depth=2)):       return 2
    return 3
named = name_level(text, y, H) <= 1

What counts as named. In the main paper, only levels 0 and 1 count as named. A family match does not count, because “diabetes” does not say which diabetes the drug treats or harms. We report the family level separately, because it shows where text points in the right direction without naming the fact.

Chance Floor

For every edge, we draw three diseases from the same relation’s targets that have no edge to the drug in that relation. We run the same matcher on the drug’s text for these unrelated diseases. The share that matches is the floor. It estimates how often a disease name appears in a drug’s text by coincidence, not as a statement of the relation. We compute the floor at every level, because wider name sets also match unrelated diseases more often.

def floor(edges, text, level):
    hits = 0
    for (d, r, y) in edges:
        pool = [y2 for y2 in targets(r)
                if (d, r, y2) not in G]
        for y2 in sample(pool, 3, seed=0):
            hits += name_level(text[d], y2, H)
                       <= level
    return hits / (3 × len(edges))

Table 15 gives the real rate and the floor at each level on a draw of 600 edges per relation from the full graph. At the exact level, real edges match about 7 to 15 times more often than unrelated ones. At the family level, the floor rises to 1.83–3.28%, still far below the threshold of about 15% at which a level would only measure how common its words are.

RelationReal edgeFloor
exact+alt.+familyexact+alt.+family
Indication17.1719.3334.671.221.392.67
Off-label10.5011.8323.670.720.943.28
Contraind.5.506.679.330.780.891.83

Table 15. Cumulative match rate (%) in Wikipedia for real edges and for three unrelated diseases per edge, 600 edges per relation. Each column adds a set of names to the one before.

What Each Level Adds

We split the lift of each level into two parts. Nomenclature is the floor-corrected gain from alternative names: it measures how often text uses another word for the same disease. Granularity is the floor-corrected gain from family names: it measures how often text names only a broader group. Table 16 gives both parts.

RelationExact, correctedNomenclatureGranularity
Indication15.942.0014.06
Off-label use9.781.119.50
Contraindication4.721.061.72
Mean1.398.43

Table 16. Floor-corrected naming rate (%) at the exact level, and the floor-corrected gain (points) from alternative names and from family names up to two levels.

Three things stand out. First, alternative names add little for every relation, so the gap is not a matter of synonyms. Second, family names add a lot for benefit facts: text often says that a drug treats “bacterial infections” rather than a specific infection. Third, family names add almost nothing for contraindications. Text does not name most contraindications even at the level of their family, so no name table or ontology lookup can recover them from prose.

One level up. With only the immediate parent as the family name, the granularity gain is 9.33 points for indications, 5.83 for off-label uses and 1.33 for contraindications, with a mean of 5.50. The order of the relations is the same at both depths.

Examples

Table 17 lists contraindications that Wikipedia does not name exactly but names at a coarser level. These are the cases that the family level exists to catch. A family match says that the text names a broader disease that contains the fact, not the fact itself. An alternative-name match is a real naming, because a redirect is another name for the same disease.

DrugContraindicated diseaseFound as
Isoniazidacute liver failurefamily
Entecavirnon-alcoholic steatohepatitisfamily
Gancicloviradult idiopathic neutropeniafamily
Prednisoloneocular tuberculosisfamily
Clonidinechronic renal failure syndromefamily
Sumatriptancoronary thrombosisfamily
CiprofloxacinAchilles bursitisfamily
Salbutamolhypertensionalt. name
AlmotriptanPrinzmetal anginaalt. name
Pazopaniblong QT syndromealt. name
Clomifenevision disorderalt. name

Table 17. Contraindications that Wikipedia names only by an alternative name or by their family.

Statistical Procedures

This appendix defines every statistic in the paper and in this supplement. The statistics use only the Python standard library, and every random draw uses seed 0.

Within-drug odds ratio. For drug dd, let ada_d and bdb_d be its named and unnamed indication edges, cdc_d and ede_d its named and unnamed contraindication edges, and ndn_d their total. The Mantel-Haenszel estimate [23] is

OR⁡MH=∑daded/nd∑dbdcd/nd.(2)\operatorname{OR}_{\mathrm{MH}} = \frac{\sum_d a_d e_d/n_d}{\sum_d b_d c_d/n_d}. \tag*{(2)}

With Pd=(ad+ed)/ndP_d=(a_d+e_d)/n_d, Qd=(bd+cd)/ndQ_d=(b_d+c_d)/n_d, Rd=aded/ndR_d=a_d e_d/n_d and Sd=bdcd/ndS_d=b_d c_d/n_d, we use the Robins-Breslow-Greenland variance [31],

Var⁡^(log⁡OR⁡MH)=∑dPdRd2R2+∑dQdSd2S2+∑d(PdSd+QdRd)2RS,(3)\widehat{\operatorname{Var}}(\log\operatorname{OR}_{\mathrm{MH}})=\frac{\sum_d P_dR_d}{2R^2}+\frac{\sum_d Q_dS_d}{2S^2}+\frac{\sum_d(P_dS_d+Q_dR_d)}{2RS}, \tag*{(3)}

where R=∑dRdR=\sum_d R_d and S=∑dSdS=\sum_d S_d. The 95% interval is exp⁡(log⁡OR⁡MH±1.96 SE^)\exp(\log\operatorname{OR}_{\mathrm{MH}}\pm1.96\,\widehat{\mathrm{SE}}). This variance stays valid when each drug contributes only a few edges, as ours do. We test the null hypothesis of no association with the continuity-corrected Cochran-Mantel-Haenszel test.

Edge-count control. A logistic model predicts whether a fact is named from an indicator for its relation and from the logarithm of the drug’s number of edges in that relation in the full graph. The relation coefficient tests the gap with the edge count held fixed.

Wilson intervals. For kk successes in nn trials, with p^=k/n\hat{p}=k/n and z=1.96z=1.96, the interval is

p^+z22n±zp^(1−p^)n+z24n21+z2n.(4)\frac{\hat{p}+\frac{z^2}{2n}\pm z\sqrt{\frac{\hat{p}(1-\hat{p})}{n}+\frac{z^2}{4n^2}}}{1+\frac{z^2}{n}}. \tag*{(4)}

Unlike the simple normal interval, it stays inside [0,1][0,1] and behaves well for small shares such as the chance floor.

Paired gain from scale. For each edge ii, the gain is gi=yi7B−yi0.5Bg_i=y_i^{7\mathrm{B}}-y_i^{0.5\mathrm{B}}, where y=1y=1 for a correct answer and 0 otherwise. The statistic is the mean gain on indications minus the mean gain on contraindications. Its interval comes from 5,000 bootstrap resamples in which we draw drugs, not edges, with replacement, so the edges of one drug move together. The pp-value uses the normal approximation with the bootstrap standard error.

Gap within naming groups. In the harder-option test, we compute the indication-minus-contraindication difference δg\delta_g separately within the questions whose page names the answer, names only its family, or names nothing. We combine the groups as ∑gwgδg/∑gwg\sum_g w_g\delta_g/\sum_g w_g with wg=ng,1ng,2/(ng,1+ng,2)w_g=n_{g,1}n_{g,2}/(n_{g,1}+n_{g,2}), where ng,1n_{g,1} and ng,2n_{g,2} are the group’s indication and contraindication counts. The standard error follows from the binomial variances of the group proportions.

Page effect. For the page test, the effect of the page on a question is its outcome with the page minus its outcome closed book. We average this effect over questions and draw 95% intervals from 5,000 bootstrap resamples of questions. For pooled results, we first average each question’s effect over the three 7–8B models, so each question counts once.

Agreement between annotators. For NN items, each labeled by nn annotators, let nijn_{ij} be the number of annotators who give item ii label jj. Fleiss’ kappa is κ=(Pˉ−Pˉe)/(1−Pˉe)\kappa=(\bar{P}-\bar{P}_e)/(1-\bar{P}_e), where Pˉ=1N∑i∑jnij2−nn(n−1)\bar{P}=\frac{1}{N}\sum_i\frac{\sum_j n_{ij}^2-n}{n(n-1)} is the observed agreement and Pˉe=∑jpj2\bar{P}_e=\sum_j p_j^2, with pjp_j the overall share of label $j, is the agreement expected by chance. Cohen’s kappa is the same quantity for two raters.

Bootstrap over drugs. The pseudocode below shows the paired bootstrap of the gain from scale. Resampling drugs rather than edges keeps the edges of one drug together, so a drug with many edges cannot make the interval look narrower than it is.

def boot_gain(edges, B=5000, seed=0):
    by_drug = group(edges, key="drug")
    drugs = list(by_drug)
    stats = []
    for b in range(B):
        pick = choices(drugs, k=len(drugs))
        E = [e for d in pick for e in by_drug[d]]
        g_ind = mean(e.y7 - e.y05 for e in E
                     if e.rel == "indication")
        g_con = mean(e.y7 - e.y05 for e in E
                     if e.rel == "contraindication")
        stats.append(g_ind - g_con)
    return percentile(stats, [2.5, 97.5])

Why not compare per-drug rates. A paired test of per-drug naming rates would look natural, but it is not valid here. With three edges per relation, a drug’s rate can only be 0, 1/3, 2/3 or 1, so the comparison would measure this rounding as much as the gap. The Mantel-Haenszel estimate uses the counts directly and has no such problem.

Within-Drug Tests in Every Corpus

This appendix gives the full within-drug results for every corpus: the raw naming rates, the Mantel-Haenszel odds ratio with both kinds of interval, the logistic edge-count control and the edge-count strata.

Odds Ratios

Table 18 gives the naming rates and the within-drug odds ratio for every corpus. The test-based interval comes from the Cochran-Mantel-Haenszel statistic. The Robins-Breslow-Greenland (RBG) interval comes from the variance of Appendix § 17. The two intervals agree closely in every corpus, so the result does not depend on the choice. Both prose corpora show a gap on their own, and so does each PubMed version. The whole label shows none.

Reading the label rows. The label’s own sections show a very large gap, but this is a property of the sections, not of the label. The Indications and Usage section is written to list what a drug treats. The Contraindications section lists only formal contraindications, which are a small part of what PrimeKG calls a contraindication (Appendix J). Most safety facts in the label sit in its warnings and adverse reactions. The whole label, which holds all of them, is the fair comparison with prose.

Why the Hetionet odds ratio is above one. The Hetionet within-drug odds ratio is 2.48, but it rests on only 51 drugs, and it does not survive the edge-count control below. Appendix § 23 explains this case.

Edge-Count Control

A drug with many contraindications may name each one less often, simply because no text lists them all. To test this, a logistic model predicts whether a fact is named from an indicator for contraindication and from the logarithm of the drug’s number of edges in that relation in the full graph. Table 19 gives the coefficients. A negative relation coefficient means that a contraindication is less likely to be named than an indication of a drug with the same edge count.

CorpusContra. coef.pplog edgespp
Wikipedia−0.4342.4×10−42.4 \times10^{-4}−0.4131.2×10−101.2 \times10^{-10}
PubMed, balanced−0.4731.4×10−51.4 \times10^{-5}−0.5541.2×10−191.2 \times10^{-19}
PubMed, sgl.−0.4723.5×10−53.5 \times10^{-5}−0.5173.6×10−163.6 \times10^{-16}
Wiki + PubMed−0.3299.7×10−49.7 \times10^{-4}−0.4378.2×10−168.2 \times10^{-16}
Label, own section−1.9266.0×10−226.0 \times10^{-22}−0.2306.9×10−36.9 \times10^{-3}
Label, whole+0.3045.0×10−35.0 \times10^{-3}−0.3201.5×10−81.5 \times10^{-8}
All three+0.0170.86−0.3351.2×10−101.2 \times10^{-10}
Hetionet−0.1910.16−0.4473.5×10−93.5 \times10^{-9}

Table 19. Logistic model of whether a fact is named, with the relation and the log of the drug’s edge count as predictors. For Hetionet, the relation term is palliates.

In every prose corpus, both terms matter. Drugs with more edges have each edge named less often, and contraindications are named less often even with the edge count held fixed. In the whole label, the sign flips: with the edge count held fixed, the label names a contraindication slightly more often than an indication. Once prose and label are pooled, the relation term vanishes.

Edge-Count Strata

Table 20 gives the gap within three strata of the drug’s edge count in the relation. In prose, the gap is largest for drugs with few edges and shrinks as drugs have more. For drugs with 13 or more edges, it is not distinguishable from zero in any prose corpus. These drugs have mostly contraindications and few indications, and their texts name few facts of either kind. In the whole label, the gap is close to zero or negative in every stratum.

Edges per drug3-56-1213+
Ind. edges (nn, prose)756678288
Contra. edges (nn, prose)1324051,185
Wikipedia+13.94***+6.04**+1.64
PubMed, balanced+18.33***+7.71***+2.37
PubMed, sgl.+17.88***+7.57***+1.92
Wiki + PubMed+11.81**+6.90**+2.23
Label, own section+20.77***+14.57***+10.10***
Label, whole−3.59−6.46*−1.50
All three+5.14−2.54+1.88

Table 20. Indication minus contraindication naming rate (points) within strata of the drug’s edge count. *p<0.05p < 0.05, **p<0.01p < 0.01, ***p<0.001p < 0.001. Label strata have 594, 591 and 231 indication edges and 102, 297 and 1,017 contraindication edges.

Table 21 gives the rates behind the Wikipedia and whole-label rows, so the reader can see that the shrinking gap in prose comes from indications, whose naming rate falls with edge count, while contraindications stay low in every stratum.

CorpusRelation3-56-1213+
WikipediaInd.26.0618.1410.42
WikipediaContra.12.1212.108.78
PubMedInd.36.5120.8012.50
PubMedContra.18.1813.0910.13
Label, wholeInd.27.7821.4919.05
Label, wholeContra.31.3727.9520.55

Table 21. Naming rate (%) by relation within strata of the drug’s edge count.

FDA Label Analysis

The FDA label is the one corpus that regulation requires to carry a drug’s safety facts. This appendix gives the full label results on the 472 drugs that have a label, with 1,416 edges per relation.

Coverage

Table 22 gives the share of facts named by each corpus and label section, with Wilson intervals. Every column uses the same drugs and the same edges, so the columns can be compared directly.

CorpusIndicationContraindication
Wikipedia22.25 [20.2, 24.5]10.95 [9.4, 12.7]
Wiki + PubMed30.51 [28.2, 33.0]17.23 [15.4, 19.3]
Label, own section19.07 [17.1, 21.2]2.61 [1.9, 3.6]
Label, whole23.73 [21.6, 26.0]22.88 [20.8, 25.1]
All three34.04 [31.6, 36.5]26.69 [24.5, 29.1]
Floor, whole label2.102.82

Table 22. Share of facts named (%) with 95% Wilson intervals, 1,416 edges per relation over 472 drugs with a label. The floor is the exact-name match rate of three unrelated diseases per edge.

Three results stand out. First, the whole label names contraindications as often as indications, at about 23% each, while Wikipedia names contraindications at half the rate of indications. Second, the whole label alone names more contraindications than Wikipedia and PubMed together. Third, the Contraindications section alone names only 2.61% of PrimeKG’s contraindications. A retriever that indexes only this section would miss almost every safety fact that the label carries.

What the Label Adds to Prose

Table 23 looks at the facts that prose misses, that is, facts that neither Wikipedia nor PubMed names. It asks how many of them the label names. The label rescues more than twice as many missed contraindications as missed indications, so it fills the gap that prose leaves.

RelationMissed by proseOwn sectionWhole label
Indication9843.45.1
Contraindication1,1721.411.4

Table 23. Facts that prose misses (count) and the share of them (%) that the label names, in the relation’s own section or anywhere in the label.

What the Contraindications Section Holds

The Contraindications section is short and formal. It is empty for 9.5% of the contraindication edges, because some labels list no contraindications at all, while the Indications section is empty for only 0.2% of indication edges. The section also rarely names even the family of a contraindicated disease: only 0.99% of contraindications are named at the family level there, against 11.23% of indications in the Indications section. The recall check of Appendix J confirms that the section does not state these facts in other words: annotators find 92% of the missed facts truly absent from it.

Where safety facts sit. In the relation check, most label sentences that name a contraindicated disease come from the Boxed Warning, Warnings and Precautions and Adverse Reactions sections. They read as warnings, such as an increased risk of a condition, rather than as formal contraindications (Table 35). This is why the whole label, not the Contraindications section, matches PrimeKG’s contraindication relation.

Chance Floor in the Label

Labels are longer than Wikipedia articles and list many conditions in their adverse reaction tables. Their floor is therefore higher: an unrelated disease’s exact name appears in the whole label for 2.10% of indication draws and 2.82% of contraindication draws. This floor is still an order of magnitude below the naming rate of real edges, and it is close for the two relations, so it cannot explain why the label shows no gap.

Label Strata

With the edge count held fixed, the whole label names contraindications slightly more often than indications (Table 19, coefficient +0.304, p=5.0×10−3p = 5.0 \times10^{-3}). In the strata, the gap is negative in all three (Table 20). The label is the only corpus in which a drug’s safety facts are written down at least as often as its uses.

Closed-Book Probe

This appendix gives the format of the closed-book probe and its full results across five model sizes.

Format

Question format. We score each edge as a completion. The prompt is a short stem for the relation that names the drug and stops before the disease (Table 24). We append each option as a continuation and score it by the mean log-probability of its tokens. The option with the highest score is the model’s answer. We use no chat template, passage or retrieval. The mean, rather than the sum, keeps long disease names from losing only because they have more tokens.

RelationPrompt stem (X is the drug’s name)
IndicationX is a drug used to treat the disease called
ContraindicationX is a drug that must not be given to patients who have
Off-label useX is a drug sometimes used off-label to treat
Treats (Hetionet)X is a drug used to treat the disease called
Palliates (Hetionet)X is a drug used to relieve the symptoms of

Table 24. Prompt stems for the closed-book probe. We score each option as the continuation of the stem.

def probe(model, stem, options):
    scores = []
    for o in options:
        ids = tok(stem + " " + o)
        n = len(tok(" " + o))
        lp = log_softmax(model(ids).logits)
        s = mean(lp[t-1, ids[t]]
                 for t in last(n, ids))
        scores.append(s)
    return options[argmax(scores)]

Options. Each question has five options: the true disease and four wrong diseases drawn at random from the same relation’s targets in the sample. The wrong options never include a disease that the drug is paired with in that relation within the sample. We cannot exclude true partners outside the sample. With about 600 targets per relation and a handful of partners per drug, the chance that a wrong option is an unrecorded true partner is about one percent, and it is the same for all relations. We shuffle the order of the options with a fixed seed. Chance accuracy is 20%.

Models. We probe Qwen2.5-0.5B, Qwen2.5-1.5B, Qwen2.5-3B, Qwen2.5-7B and Qwen2.5-14B, all instruction-tuned ([25]). All models run in 32-bit floating point on CPU with Hugging Face Transformers. Every model answers the same 1,800 questions with the same options in the same order, so we can pair them edge by edge.

Results

Accuracy at every size. Table 25 gives closed-book accuracy for every Qwen2.5 size, with the gap and its interval over drugs. Indication accuracy rises from 31.50% to about 50% and levels off from Qwen2.5-7B. Contraindication accuracy stays between 22.17% and 27.17%, close to chance at every size.

ModelInd.Off-labelContra.Gap [95% CI]
Qwen2.5-0.5B31.5028.5022.17+9.3 [4.0, 14.9]
Qwen2.5-1.5B45.6735.8325.33+20.33 [15.2, 26.2]
Qwen2.5-3B45.3332.6724.17+21.2 [16.1, 26.9]
Qwen2.5-7B50.8335.0023.83+27.0 [21.6, 32.4]
Qwen2.5-14B49.8335.8327.17+22.7 [17.1, 28.3]
Wikipedia names it16.8310.506.67

Table 25. Closed-book accuracy (%) on the probe sample, 600 edges per relation, with chance at 20%. Gap is indication minus contraindication accuracy, with a 95% bootstrap interval over drugs. The last row gives the share of the same edges that Wikipedia names (level 0 or 1).

Paired gain from scale. Table 26 gives the accuracy of the smallest and the 7B model with Wilson intervals, and the paired gain between them. Indications gain 19.33 points, off-label uses 6.50 points and contraindications only 1.67 points. The difference between the indication and contraindication gains is 17.67 points, with a bootstrap interval over 709 drugs that excludes zero by a wide margin.

Relation0.5B [95% CI]7B [95% CI]Gain
Indication31.50 [27.91, 35.32]50.83 [46.84, 54.82]+19.33
Off-label28.50 [25.03, 32.24]35.00 [31.29, 38.90]+6.50
Contraind.22.17 [19.03, 25.66]23.83 [20.60, 27.40]+1.67

Table 26. Probe accuracy (%) of Qwen2.5-0.5B and Qwen2.5-7B, and the paired gain between them (points), on 600 edges per relation. Indication gain minus contraindication gain: +17.67 [12.68, 22.70]; p=4.8×10−12p = 4.8 \times10^{-12}.

Accuracy by Whether Text Names the Fact

Table 27 splits accuracy by whether Wikipedia names the fact. Scale helps most where text names the fact. For named indications, accuracy climbs from 57.4% to 84.2%. For named contraindications, it climbs from 22.5% to 40.0%, a smaller gain from a lower start. For facts that Wikipedia does not name, accuracy rises much less in every relation, and silent contraindications stay at chance from the smallest model to the largest.

ModelIndication namedIndication notOff-label namedOff-label notContraind. namedContraind. not
Qwen2.5-0.5B57.426.341.327.022.522.1
Qwen2.5-1.5B74.339.961.932.827.525.2
Qwen2.5-3B80.238.349.230.730.023.8
Qwen2.5-7B84.244.157.132.432.523.2
Qwen2.5-14B84.242.963.532.640.026.2
nn1014996353740560

Table 27. Closed-book accuracy (%) split by whether Wikipedia names the fact (level 0 or 1). The last row gives the number of edges in each group.

From Qwen2.5-0.5B to Qwen2.5-14B, named contraindications gain 17.5 points [4.9, 30.8], while unnamed contraindications gain only 4.1 points [0.9, 7.2]. Even among named facts, Qwen2.5-14B gets 84.2% of indications but only 40.0% of contraindications. A named contraindication is usually named once, in a warning, while a named indication is usually the first thing an article says about the drug.

Why so few named contraindications. Only 40 of 600 sampled contraindications are named in Wikipedia, against 101 indications. Scale can only lift what text provides, and text provides few contraindications. This is why the contraindication curve in Figure 5 stays flat.

Harder Wrong Options and the Drug’s Page

This appendix describes the harder-option test and the page test, and gives every per-model result behind Figure 6 and Figure 7.

Format

Questions. This test reuses the question set of our companion study on declining. An indication question asks “Which disease is XX used to treat?”, and a contraindication question asks “In patients with which disease must XX not be given?”. Each question lists five diseases and the option Not enough information to tell, lettered A to F. The position of every option, including the decline option, is random. The prompt has the shape below. We show one real question, with its options in an illustrative order and two wrong options left out.

[Passage: <the drug’s page, if any>]
In patients with which disease must
Ciprofloxacin not be given?
A. myasthenia gravis with thymus
hyperplasia
B. <wrong option>
C. adult-onset myasthenia gravis
D. Not enough information to tell
E. myasthenia, limb-girdle, autoimmune
F. <wrong option>
Answer with the letter of one option
only.

Here the true answer, adult-onset myasthenia gravis, sits among other forms of myasthenia. A model that knows only that ciprofloxacin is risky in myasthenia gravis cannot tell these options apart, which is the point of the test.

Wrong options. We take the wrong options first from the true disease’s MONDO siblings, then from its cousins (other children of its grandparent), and only then from random diseases of the same relation. The wrong options never include the true disease’s own ancestors or descendants, or any other disease the drug has under the same relation, since those options would also be correct. In 481 of the 856 questions, all four wrong options are siblings or cousins.

def wrong_options(d, r, y, k=4):
    bad = ({y} | ancestors(y) | descendants(y)
           | partners(d, r))
    out = []
    for pool in (siblings(y), cousins(y),
                 targets(r)):
        for z in shuffle(pool, seed=0):
            if z not in bad and z not in out:
                out.append(z)
            if len(out) == k:
                return out
Naming groupIndicationContraindication
Names the disease39.729.5
Names only its family83.584.3
Silent60.744.8

Table 28. Share of questions (%) whose four wrong options are all MONDO siblings of the answer.

The page

A full drug article is too long for a prompt, so we cut a window around the evidence. The page starts with the first 800 characters of the drug’s cleaned text, which is its Wikipedia article followed by its PubMed abstracts. We then add a window of 280 characters on each side of every mention of the answer or its family, merge windows that overlap, and stop at 1,800 characters. A silent page has no mentions, so it is just the opening of the text. Every model sees the same page for the same question.

def page(text, names, head=800, w=280,
         cap=1800):
    spans = [(0, head)]
    for m in find_all(text, names):
        spans.append((m.start - w, m.end + w))
    spans = merge_overlaps(sorted(spans))
    out = " ... ".join(text[a:b]
                       for a, b in spans)
    return out[:cap]

The page does not use the wrong options. It mentions only the answer and its family, as a retriever centered on the drug would. A page could still name a sibling by chance, and this is part of what makes family-only pages hard.

Scoring. We format each prompt with the model’s chat template and read the answer from the next-token probability of each option letter. We score both spellings of each letter, with and without a leading space, keep the better one, and renormalize over the six letters. We batch prompts with left padding and rebuild the position indices from the attention mask, so batched and single scores agree.

def letter_answer(model, prompt):
    ids = chat_template(prompt)
    lp = log_softmax(model(ids).logits[-1])
    s = {L: max(lp[tok(L)], lp[tok(" " + L)])
         for L in "ABCDEF"}
p = softmax(list(s.values()))
return argmax(p), p

Models. We test Qwen2.5-0.5B and Qwen2.5-7B ([25]), OLMo-2-7B ((OLMo et al. 2024)) and Granite-3.1-8B ((IBM Granite Team 2024)), all instruction-tuned and in 32-bit floating point on CPU. The three 7-8B models come from three different families with different training data.

Closed Book

The gap with harder options. Table 29 repeats Table 6 and adds the gap computed within naming groups and its pp-value. The within-group gap removes the effect of the two relations having different mixes of named and silent questions. Both tests agree in every model.

ModelInd.Contra.GapWithinpp (within)
Qwen2.5-0.5B36.028.1+7.9+8.28.9 ×10−3\times10^{-3}
Qwen2.5-7B56.025.1+30.9+31.3< 10−1510^{-15}
OLMo-2-7B56.033.0+23.0+23.62.3 ×10−13\times10^{-13}
Granite-3.1-8B58.712.8+45.9+46.3< 10−1510^{-15}

Table 29. Harder-option test, closed book (%); 450 indication and 406 contraindication questions. The pp-values of the raw gap are 0.013 for Qwen2.5-0.5B, 1.5 ×10−11\times10^{-11} for OLMo-2-7B, and below 10−1510^{-15} for Qwen2.5-7B and Granite-3.1-8B.

How models fail. Table 30 gives the share of wrong answers and declines, and the precision on answered questions, for each model. Two kinds of failure appear. Qwen2.5-0.5B and OLMo-2-7B rarely decline and answer most contraindication questions wrongly. Qwen2.5-7B and Granite-3.1-8B decline on half or more of contraindication questions. Their precision on answered contraindications, 51.0% and 39.4%, is still far below their precision on indications.

ModelIndicationContraindication
wrongdecl.prec.wrongdecl.prec.
Qwen2.5-0.5B60.73.337.271.40.528.2
Qwen2.5-7B22.022.071.824.150.751.0
OLMo-2-7B34.010.062.264.03.034.0
Granite-3.1-8B28.712.767.219.767.539.4

Table 30. How models fail closed book with harder wrong options (%). Decl. is the share of questions answered Not enough information; prec. is the accuracy on the questions the model answers.

Closed book by naming group. The Closed column of Table 31 shows that closed-book accuracy also follows naming. For Qwen2.5-7B, named indications reach 78.7% and silent ones 39.3%. Named contraindications reach only 33.3% and silent ones 17.3%. The model does best on the facts that text names, and text names few contraindications.

Indication: ModelGroupnnClosedPageChange [95% CI]Contraindication: ModelGroupnnClosedPageChange [95% CI]
Qwen2.5-0.5Bnamed15047.386.7+39.3 [+31.3, +47.3]Qwen2.5-0.5Bnamed15030.070.7+40.7 [+32.0, +49.3]
family15034.044.7+10.7 [+2.0, +20.0]family10623.627.4+3.8 [−4.7, +12.3]
silent15026.734.0+7.3 [−0.7, +15.3]silent15029.323.3−6.0 [−13.3, +0.7]
Qwen2.5-7Bnamed15078.785.3+6.7 [0.0, +13.3]Qwen2.5-7Bnamed15033.344.0+10.7 [+0.7, +20.0]
family15050.038.0−12.0 [−18.7, −5.3]family10624.517.0−7.5 [−17.9, +2.8]
silent15039.322.7−16.7 [−24.0, −10.0]silent15017.33.3−14.0 [−20.0, −8.0]
OLMo-2-7Bnamed15075.388.0+12.7 [+5.3, +20.0]OLMo-2-7Bnamed15042.054.7+12.7 [+3.3, +22.0]
family15048.045.3−2.7 [−10.7, +5.3]family10630.230.20.0 [−8.5, +8.5]
silent15044.743.3−1.3 [−9.3, +6.7]silent15026.020.7−5.3 [−12.7, +1.3]
Granite-3.1-8Bnamed15076.790.7+14.0 [+7.3, +20.7]Granite-3.1-8Bnamed15018.738.0+19.3 [+10.7, +28.0]
family15054.044.0−10.0 [−17.3, −2.7]family1069.417.9+8.5 [0.0, +17.0]
silent15045.329.3−16.0 [−24.0, −8.0]silent1509.32.0−7.3 [−12.0, −2.7]
Pooled 7-8Bnamed150+11.1 [+6.7, +15.8]Pooled 7-8Bnamed150+14.2 [+7.3, +20.9]
family150−8.2 [−13.3, −3.1]family106+0.3 [−6.3, +6.9]
silent150−11.3 [−17.1, −6.0]silent150−8.9 [−12.9, −5.1]

Table 31. Accuracy (%) closed book and with the drug’s page, and the change (points) with a 95% bootstrap interval over questions, by what the page says about the answer. The pooled rows average each question over Qwen2.5-7B, OLMo-2-7B and Granite-3.1-8B first. In the named group, the indication change minus the contraindication change is −3.1 [−11.6, +5.1] points pooled, so a written safety fact helps at least as much as a written benefit fact.

With the Page

Table 31 gives accuracy closed book and with the page for every model and naming group, with bootstrap intervals over questions. Table 32 summarizes the changes. When the page names the answer, it helps both relations in every model, and it helps contraindications at least as much as indications. When the page names only the family or nothing, it helps little or hurts, because the larger models decline more.

ModelNames it: Ind.Names it: Con.Family only: Ind.Family only: Con.Silent: Ind.Silent: Con.
Qwen2.5-0.5B+39.3+40.7+10.7+3.8+7.3−6.0
Qwen2.5-7B+6.7+10.7−12.0−7.5−16.7−14.0
OLMo-2-7B+12.7+12.7−2.70.0−1.3−5.3
Granite-3.1-8B+14.0+19.3−10.0+8.5−16.0−7.3
Pooled 7-8B+11.1+14.2−8.2+0.3−11.3−8.9

Table 32. Change in accuracy (points) from closed book to the drug’s page, by what the page says about the answer. Each cell has 150 questions, except 106 family-only contraindication questions.

The smallest model. Qwen2.5-0.5B gains the most from a naming page, about 40 points for both relations. It rarely declines, so when the page contains the answer it copies it. It is also the only model that gains on indications from family-only and silent pages, by 10.7 and 7.3 points.

Declines with the page. Table 33 gives the share of questions that the three 7-8B models answer with Not enough information. A silent page raises declines the most, by 26.0 points on indications and 32.9 points on contraindications. A page that names the answer barely changes them. The models decline when the page gives them nothing and answer when it names the disease. With a silent page, Qwen2.5-7B declines on 94.0% of contraindication questions.

Indication: closedIndication: pageContraindication: closedContraindication: page
Page
Names the disease6.05.836.241.6
Names only its family9.126.042.154.4
Silent29.655.643.376.2

Table 33. Share of questions answered Not enough information (%), closed book and with the drug’s page, averaged over Qwen2.5-7B, OLMo-2-7B and Granite-3.1-8B.

Does the Sentence State the Relation?

The main measurement counts a fact as named when the drug’s text contains the disease’s name. This appendix checks whether the sentence that contains the name also states the relation, with a model judge, a keyword check and four annotators.

Sentences. For every indication and contraindication edge and each corpus, we split the drug’s text into sentences and keep up to three that contain the disease’s exact or alternative name. We strip Wikipedia markup first. This sentence-level call agrees with the main measurement on 97.9% of Wikipedia edges, 99.3% of PubMed edges and all label edges.

def naming_sentences(text, y, k=3):
    text = strip_markup(text)
    out = []
    for s in split_sentences(text):
        if matches(s, [label(y)] + redirects(y)):
            out.append(s)
        if len(out) == k:
            break
    return out

def supports(rel, sentence, d, y):
    L = judge(sentence, d, y)  # A, B, C or D
    if rel == "indication":
        return L == "A"
    return L in ("B", "C")    # strict: "B"

Model judge. Qwen2.5-7B reads the first naming sentence and picks one of four letters:

Sentence: <sentence>
What does this sentence say about X and Y?
A. X is used to treat or manage Y.
B. X must not be used in patients with

Y.

C. X may cause or worsen Y, or needs caution in patients with Y.

D. The sentence mentions both but states no link between them.

Answer with the letter of one option only.

We read the answer from the letter probabilities, as in Appendix I. A sentence supports an indication when the judge picks A. It supports a contraindication when the judge picks B or C, since a warning about harm is a safety statement. The strict reading (strict in Table 34) accepts only B.

CorpusDefinitionInd.Contra.OR [95% CI]
Wikipedianamed18.418.012.68 [2.16, 3.34]
keyword16.324.184.60 [3.49, 6.07]
judge17.197.382.70 [2.15, 3.39]
strict17.191.1018.68 [11.43, 30.54]
PubMednamed25.6110.802.83 [2.34, 3.41]
keyword13.592.735.64 [4.08, 7.78]
judge17.839.352.09 [1.71, 2.57]
strict17.830.06460.00 [58.01, 3,647.67]
Labelnamed23.7322.881.05 [0.88, 1.25]
keyword20.5516.811.30 [1.07, 1.57]
judge18.8620.760.88 [0.73, 1.07]
strict18.862.1910.32 [6.98, 15.25]

Table 34. Share of facts (%) counted as stated under each definition, and the within-drug odds ratio with Robins-Breslow-Greenland intervals. 1,722 edges per relation for Wikipedia and PubMed, 1,416 for the label.

Keyword check. As a cheap baseline, a sentence supports an indication when it contains a treatment word (treat, indicated, therapy, used for, management of, prevent and similar). It supports a contraindication when it contains a contraindication word (contraindicated, must not, should not, avoid, not recommended) or a warning word (caution, warning, worsen, exacerbate, risk, adverse, may cause).

Results. Table 34 gives every definition. With the judge, the prose gap stays at 2.09–2.70, and the label stays even. The strict reading gives very large odds ratios in every corpus, the label included, because formal contraindication wording is rare everywhere: even the label states it for only 2.19% of contraindications. Most safety facts appear as warnings, so we use the judge with warnings as the main definition.

Annotation. We draw 160 named sentences with seed 0: 40 for each relation from Wikipedia and 40 from the whole label. We shuffle them and hide the graph relation. Four annotators label each sentence on their own with one letter: T (the drug treats the disease), C (it must not be used), W (it may cause or worsen the disease, or needs caution) or M (a mention with no link). We take the majority label and leave out the 6 ties.

Agreement. The annotators agree closely: Fleiss’ κ\kappa = 0.89, pairwise Cohen’s κ\kappa from 0.86 to 0.93, and all four agree on 141 of 160 sentences. Against the majority label, the judge agrees on 95.5% of sentences (κ\kappa = 0.92) and on

98.1% of the supported or not supported calls (κ=0.88\kappa= 0.88). The keyword check agrees on 64.3% (κ=0.43\kappa= 0.43). It misses most safety statements because lists of adverse reactions, such as “Hematological: agranulocytosis, thrombocytopenia”, contain no relation word. By the majority label, 88.6% of named indications and 93.3% of named contraindications state the graph’s relation (Table 35).

RelationCorpusnnTCWMSupports
IndicationWikipedia3997.40.00.02.697.4
Label4080.00.010.010.080.0
Contra.Wikipedia395.110.382.12.692.3
Label362.811.183.32.894.4

Table 35. Majority label of four annotators (%) on named sentences. Supports is T for indications and C or W for contraindications.

What PrimeKG calls a contraindication. Only 10–11% of the contraindication sentences are formal contraindications (C). Most are warnings (W): an increased risk of myocardial infarction with ketorolac, or agranulocytosis in a list of adverse reactions. PrimeKG’s contraindication relation therefore covers formal contraindications and safety warnings, and our results apply to both.

Recall check. To test whether the matcher misses the wording of the label, we draw 50 of the 1,227 contraindications whose label has a non-empty Contraindications section that the matcher finds nothing in. The same four annotators read the section and mark each fact as Y (stated in other words), B (only a broader group is stated) or N (not there). They agree closely (Fleiss’ κ=0.92\kappa= 0.92, unanimous on 49 items). By the majority label, 45 facts (91.8%) are not in the section, 3 (6.1%) are stated in other words, and 1 is stated only as a broader group; 1 is a tie. The three misses are ulcerative colitis for clindamycin, cardiac valvular disorders for cabergoline, and marked hepatic damage for sulfamethoxazole. The low coverage of the Contraindications section is a property of the section, not of the matcher.

Annotation Examples

Table 36 shows sentences on which all four annotators agree, one for each label. The Diclofenac sentence comes from the label of a drug that treats rheumatoid arthritis, but the sentence itself only compares two groups of patients, so it does not state the relation. Such sentences are why we check the relation and not only the name.

Drug, diseaseSentenceLabel
Isocarboxazid, endogenous depression“It has demonstrated efficacy in various depressive subtypes, including atypical depression, melancholic depression, and endogenous depression.”T
Isocarboxazid, hypertension“Marplan (isocarboxazid) should not be administered to any patient with a confirmed or suspected cerebrovascular defect or to any patient with a cardiovascular disease, hypertension, …”C
Ketorolac, myocardial infarction“Nonsteroidal anti-inflammatory drugs (NSAIDs) cause an increased risk of serious cardiovascular thrombotic events, including myocardial infarction and stroke, …”W
Diclofenac, rheumatoid arthritis“Patients with osteoarthritis more often develop symptomatic liver disease than patients with rheumatoid arthritis.”M

Table 36. Sentences on which all four annotators agree. T: treats; C: must not be used; W: may cause or worsen, or needs caution; M: mention with no link.

Where annotators disagree. The 19 sentences without full agreement fall into three kinds. The first is a bare heading or list item with no verb, such as “Osteoporosis-related fractures” for omeprazole or “Kidney or liver problems” for rabeprazole. Most annotators read these as warnings, because they appear in warning sections, but one reads them as mentions. The second is the line between C and W, such as “Avoid use of fenoprofen calcium in patients with advanced renal disease unless benefits are expected to outweigh risk”. Both labels support a contraindication, so this is a disagreement that does not change any result. The third is a name used in another sense. For histerlin, hypertension appears inside idiopathic intracranial hypertension, a different disease. For norepinephrine, the sentence says that alpha blockers treat hypertension, not that norepinephrine does. The relation check exists to catch such cases.

Judge errors. The judge disagrees with the majority on 4.5% of sentences. Two cases show what can go wrong. For doripenem, a label fragment lists pyelonephritis among approved uses, and for ipratropium, a sentence states its use in the common cold, yet the judge answers none for both. These errors drop real indications, so they shrink the gap rather than create it.

Routing Test

Setup. We ask the harder-option questions of Appendix I a third time. The questions and options are built exactly as before, so they are the same questions. Only the passage changes: we cut it from the drug’s whole FDA label, with the same window rule of 800 opening characters and windows around each mention, up to 1,800 characters. We leave out the questions whose drug has no label after selection, which keeps the other questions identical. This leaves 726 questions: 380 indications and 346 contraindications. The routed condition takes the prose answer for indications and the label answer for contraindications. In a deployed agent, the planner would classify the relation from the question. Here the relation is known, so the test measures the value of the route, not the accuracy of the planner.

def routed_answer(model, q):
    if q.relation == "indication":
        page = page(prose(q.drug), names(q.y))
    else:
        page = page(label(q.drug), names(q.y))
    return letter_answer(model,
                         page + q.text)

Results. Table 37 gives the outcomes for each model. Routing raises correct answers in every model, by 0.7 to 2.7 points, and by 1.5 [−0.2, +3.2] points pooled. On contraindications alone, the label page raises correct answers by 3.2 [−0.4, +6.6] points and lowers declines by 3.0 points. On indications, it lowers correct answers by 6.6 [−9.7, −3.4] points and raises declines by 11.6 points, so routing indications to the label would hurt. The split in Table 7 and Table 38 explains the small overall gain. The label page helps safety questions by 15.8 points when the whole label names the answer, and it cannot help the 61% of contraindication questions whose label does not.

ModelConditionCorrectWrongDecline
Qwen2.5-7Bprose37.58.354.3
label32.05.162.9
routed38.27.954.0
OLMo-2-7Bprose49.329.521.2
label50.328.021.8
routed50.629.520.0
Granite-3.1-8Bprose39.912.847.2
label38.79.451.9
routed42.612.944.5
Pooledclosed book43.130.826.1
prose42.216.940.9
label40.314.145.5
routed43.816.839.5

Table 37. Outcomes (%) on all 726 questions with a label page.

RelationNamednnCorrectWrongDecline
Ind.yes117+3.4 [−1.7, 8.5]−1.7−1.7
no263−11.0 [−15.1, −7.1]−6.5+17.5
Contra.yes135+15.8 [8.6, 23.2]−3.7−12.1
no211−4.9 [−8.2, −1.7]+2.1+2.8

Table 38. Label page minus prose page (points), pooled over the three 7-8B models, split by whether the whole label names the answer (Named). Brackets give 95% bootstrap intervals over questions.

When the label names the answer. Table 38 gives the full split. When the whole label names a contraindication, the label page raises correct answers from 40.0% to 55.8% and cuts declines from 45.2% to 33.1%. When it does not, the label page helps nothing, and correct answers fall slightly. For indications, the label rarely adds anything that prose lacks: when it names the answer, prose usually names it too, and prose already reaches 84.3%. When it does not, the label page is mostly dosing and warnings, and the models decline more.

Negative Control on Hetionet

Hetionet’s treats and palliates relations both describe benefit. If our matcher created a gap between any two relations, it would create one here too. We run the same measurement on the same corpus, using the Disease Ontology instead of MONDO for the disease hierarchy.

Setup. We draw 600 treats edges and all 385 measurable palliates edges (Appendix B) and fetch the drugs’ Wikipedia articles, 497 in all. The matcher, the names, the floor and the tests are the same as for PrimeKG. The only change is the hierarchy: the Disease Ontology gives 103 family links over 74 diseases.

Result. Wikipedia names the disease for 57.33% of 600 sampled treats edges and 56.10% of 385 measurable palliates edges, an odds ratio of 1.05 (p=0.70p = 0.70). Within the 51 drugs that have both relations, the Mantel-Haenszel odds ratio is 2.48 [1.21, 5.08]. This within-drug difference does not survive the edge-count control. With the drug’s edge count held fixed, the relation term is not significant (p=0.16p = 0.16), and the strata change sign (Table 39). In PrimeKG, the relation term stays significant under the same control (p=2.4×10−4p = 2.4 \times10^{-4} in Wikipedia).

Why the within-drug ratio is above one. The within-drug odds ratio rests on only the 51 drugs that have both relations, and its interval is wide. It does not survive the edge-count control (p=0.16p = 0.16). Table 40 shows why: the gap is +8.16 points for drugs with one or two edges, close to zero for three to five, and reversed for six to twelve. A matcher bias would push the same way in every stratum. In PrimeKG, same control leaves the gap in place in every prose corpus (Table 19).

Edges per drugtreatsnnpalliatesnn
1-268.0631059.91217
3-551.619350.68146
6-1243.1713954.5522

Table 40. Hetionet naming rate (%) in Wikipedia within strata of the drug’s edge count.

treatspalliates
Named in Wikipedia (%)57.33 (n=600n = 600)56.10 (n=385n = 385)
Odds ratio, pooled1.05, p=0.70p = 0.70
Within drug (51 drugs)2.48 [1.21, 5.08]
Relation term, edges fixedp=0.16p = 0.16
Gap, 1-2 edges+8.16 (p=0.06p = 0.06)
Gap, 3-5 edges+0.93 (p=0.89p = 0.89)
Gap, 6-12 edges−11.38 (p=0.32p = 0.32)

Table 39. Negative control on Hetionet: two benefit relations, the same corpus and the same matcher. No gap survives the edge-count control, and the strata change sign.

Naming levels. Hetionet’s facts are named far more often than PrimeKG’s: 54.5% of its sampled edges are named exactly, against 13.2% in the PrimeKG text sample (Table 13). Hetionet covers fewer and broader diseases, such as hypertension and asthma, which articles name directly. This does not affect the control, which compares two relations on the same diseases and the same articles.

Case Studies

This appendix follows single facts through the study, so the reader can see what each measurement does on real text. All examples come from our released files.

One Drug through Every Stage

Ciprofloxacin is the running example of the paper. The text sample holds three of its indications and three of its contraindications. Table 41 shows how plainly each corpus names each fact.

FactWikipediaPubMedLabel
Indications
infectious diarrheal diseasenonenonenone
shigellosisnonenamednone
conjunctivitisnonenonenamed
Contraindications
peripheral neuropathynonenamednamed
adult-onset myasthenia gravisfamilyfamilyfamily
periarthritisfamilyfamilyfamily

Table 41. How plainly each corpus names six ciprofloxacin facts. Named is level 0 or 1; family is level 2.

Indications. No corpus names infectious diarrheal disease, because text names the specific infections instead. PubMed names shigellosis in a sentence that ends “. . . with endemic and epidemic shigellosis and invasive salmonellosis”, and the judge reads it as a treatment with probability 0.995. Only the label names conjunctivitis, in the Indications section of the ophthalmic solution.

Contraindications. PubMed names peripheral neuropathy in a warning, “. . . rupture, peripheral neuropathy and aortic aneurysm”, and the label names it in the Boxed Warning: “WARNING: SERIOUS ADVERSE REACTIONS INCLUDING TENDINITIS, TENDON RUPTURE, PERIPHERAL NEUROPATHY, CENTRAL NERVOUS SYSTEM EFFECTS

AND EXACERBATION OF MYASTHENIA GRAVIS”. The same warning names myasthenia gravis, but PrimeKG records the fact as adult-onset myasthenia gravis, a child of that term. Every corpus therefore names it only at the family level. Periarthritis is the same: text names only a broader term, not this specific disease.

What the models do. The harder-option question for adult-onset myasthenia gravis puts the answer next to other forms of myasthenia (Appendix § 21). Closed book, Qwen2.5-7B picks myasthenia gravis with thymus hyperplasia, and it keeps this answer with the prose page and with the label page. Closed book, Qwen2.5-0.5B picks myasthenia, limb-girdle, autoimmune. Granite-3.1-8B is wrong closed book and right with the prose page. OLMo-2-7B is wrong with the prose page and right with the label page. For periarthritis, Qwen2.5-7B and Granite-3.1-8B decline, and OLMo-2-7B and Qwen2.5-0.5B pick reactive arthritis. A family-level mention points the models to the right group but not to the right member, which is the behavior that the family-only group of Table 32 measures.

The lesson. A clinician would say that the label covers the myasthenia gravis risk. Our measurement agrees that the label points there, and it records the fact as family only because the graph is more specific than the text. This is the source of the graph-specificity limitation in § 7.

Safety Facts Only the Label Names

Table 42 lists contraindications that neither Wikipedia nor PubMed names but the whole label does. The judge reads the label sentences for the first three as warnings. These facts are among the 11.4% of prose misses that the label rescues (Table 23).

DrugContraindicated condition
Ketoconazoletorsades de pointes (QT prolongation)
Carbidopamyocardial infarction
Diflunisalhypertension
Desvenlafaxineangle-closure glaucoma
Lisinoprilneonatal jaundice

Table 42. Contraindications that prose misses and the whole label names.

These facts share one pattern. Each is a warning about a specific patient group or a rare harm: an arrhythmia, a heart attack, a rise in blood pressure, an eye emergency, a risk to a newborn. Prose about these drugs describes what they treat and how they work. The label lists who is at risk.

Questions That Routing Fixes

Table 43 lists contraindication questions that all three 7-8B models answer correctly with the label page and none answers correctly with the prose page. In the sumatriptan question, for example, all three models pick coronary atherosclerosis, a related heart disease, with the prose page. The label page names the answer, and all three models pick it.

DrugContraindicated condition
SumatriptanPrinzmetal angina
Acetylcysteineasthma
Apomorphineorthostatic hypotension
Bimatoprostuveitis
Fenoprofenkidney disease
Levocetirizinekidney disease

Table 43. Contraindication questions that all three 7-8B models answer correctly only with the label page.

Errors and Edge Cases

Combination labels. PrimeKG lists Crohn disease as a contraindication of tretinoin, and the label we retrieve names it. The label is for a gel that combines clindamycin with tretinoin, and the warning about inflammatory bowel disease comes from clindamycin. Our measurement counts this as named, as a retriever would. A clinical agent should check which ingredient a warning belongs to before it cites it.

Names in another sense. For histrelin, the label names hypertension only inside idiopathic intracranial hypertension, a different condition. For norepinephrine, a sentence says that alpha blockers treat hypertension. The matcher counts both as named. Three of four annotators mark the norepinephrine sentence as a mention with no link, and the histrelin sentence splits them evenly (Appendix J). The relation check exists to remove such cases.

Lists without verbs. Many label sentences are items in a list of adverse reactions, such as “Hematological: agranulocytosis, thrombocytopenia”. The keyword check misses them because they contain no relation word. The judge and the annotators read them as warnings, because the section heading gives the relation. This is the main reason the keyword check agrees poorly with the annotators.

Approval fragments. Some indication sentences are fragments of a list of approved uses whose verb sits in an earlier sentence, such as doripenem and pyelonephritis. The judge sometimes reads these as mentions with no link. These errors remove real indications, so they make the measured gap smaller, not larger.

References

  1. [1]Bang, B.; Yoon, J.; Chang, D.-J.; Park, S.; and Lee, Y. O. 2025. Retrieval Augmented Large Language Model System for Comprehensive Drug Contraindications. arXiv:2508.06145.arxiv.org/abs/2508.06145
  2. [2]Candelario, D. M.; Vazquez, V.; Jackson, W.; and Reilly, T. 2017. Completeness, accuracy, and readability of Wikipedia as a reference for patient medication information. Journal of the American Pharmacists Association, 57(2): 197–200.DOI
  3. [3]Chandak, P.; Huang, K.; and Zitnik, M. 2023. Building a Knowledge Graph to Enable Precision Medicine. Scientific Data, 10: 67.DOI
  4. [4]Clauson, K. A.; Polen, H. H.; Kamel Boulos, M. N.; and Dzenowagis, J. H. 2008. Scope, Completeness, and Accuracy of Drug Information in Wikipedia. Annals of Pharmacy and Pharmacotherapy, 42(12): 1814–1821.DOI
  5. [5]East, A.; Campolongo, E. G.; Meyers, L.; Rayeed, S. M.; Stevens, S.; Zarubiieva, I.; Fluck, I. E.; Girón, J. C.; Jousse, M.; Lowe, S.; Perry, K. I.; Betancourt, I.; Charney, N.; Donoso, E.; Fox, N.; Landsbergen, K. J.; Nepovinykh, E.; Ramirez, M.; Singh, P.; Thapa-Magar, K.; Thompson, M.; Waite, E.; Berger-Wolf, T.; Lapp, H.; Mabee, P.; Stewart, C.; Taylor, G.; and Record, S. 2025. Optimizing Image Capture for Computer Vision-Powered Taxonomic Identification and Trait Recognition of Biodiversity Specimens. Methods in Ecology and Evolution. Doi:10.1111/2041-210X.70140.DOI
  6. [6]Gottesman, D.; Gilaie-Dotan, A.; Cohen, I.; Gur-Arieh, Y.; Mobashsher, M.; Yoran, O.; and Geva, M. 2026. LLMent: A Suite for Analyzing Knowledge in Language Models from Pretraining Data to Representations. Transactions of the Association for Computational Linguistics, 14: 1685–1722.arxiv.org/abs/2509.03405
  7. [7]Heilman, J. M.; Kemmann, E.; Bornert, M.; Chatterjee, A.; Ragar, B.; Beards, G. M.; Iberri, D. J.; Harvey, M.; Thomas, B.; Stomp, W.; et al. 2011. Wikipedia: a key tool for global public health promotion. Journal of medical Internet research, 13(1): e1589.DOI
  8. [8]Himmelstein, D. S.; Lizee, A.; Hessler, C.; Brueggeman, L.; Chen, S. L.; Hadley, D.; Green, A.; Khankhanian, P.; and Baranzini, S. E. 2017. Systematic Integration of Biomedical Knowledge Prioritizes Drugs for Repurposing. eLife, 6: e26726.
  9. [9]Hunter, J. A.; Lee, T.; and Persaud, N. 2018. A comparison of the content and primary literature support for online medication information provided by Lexicomp and Wikipedia. Journal of the Medical Library Association, 106(3): 352–360.DOI
  10. [10]Hwang, T. J.; Bourgeois, F. T.; and Seeger, J. D. 2014. Drug safety in the digital age. New England Journal of Medicine, 370(26): 2460–2462.DOI
  11. [12]Joren, H.; Zhang, J.; Ferng, C.-S.; Juan, D.-C.; Taly, A.; and Rashtraichian, C. 2025. Sufficient Context: A New Lens on Retrieval Augmented Generation Systems. In International Conference on Learning Representations.arxiv.org/abs/2411.06037
  12. [13]Kandpal, N.; Deng, H.; Roberts, A.; Wallace, E.; and Raffel, C. 2023. Large Language Models Struggle to Learn Long-Tail Knowledge. In Proceedings of the 40th International Conference on Machine Learning (ICML), 15696–15707.arxiv.org/abs/2211.08411
  13. [14]Kass-Hout, T. A.; Xu, Z.; Mohebbi, M.; Nielsen, H.; Baker, A.; Levine, J.; Johanson, E.; and Bright, R. A. 2016. OpenFDA: an innovative platform providing access to a wealth of FDA’s publicly available data. Journal of the American Medical Informatics Association, 23(3): 596–600.
  14. [15]Kränbring, J.; Monzon Penza, T.; Gutmann, J.; Muehlich, S.; Zolk, O.; Wojnowksi, L.; Maas, R.; Engelhardt, S.; and Sarikas, A. 2014. Accuracy and completeness of drug information in Wikipedia: a comparison with standard textbooks of pharmacology. PloS one, 9(9): e106930.
  15. [16]Kuhn, M.; Letunic, I.; Jensen, L. J.; and Bork, P. 2016. The SIDER Database of Drugs and Side Effects. Nucleic Acids Research, 44(D1): D1075–D1079.DOI
  16. [17]Kupferberg, N.; and Protus, B. M. 2011. Accuracy and completeness of drug information in Wikipedia: an assessment. Journal of the Medical Library Association, 99(4): 310–313.DOI
  17. [18]Lavsa, S. M.; Corman, S. L.; Culley, C. M.; and Pummer, T. L. 2011. Reliability of Wikipedia as a medication information source for pharmacy students. Currents in Pharmacy Teaching and Learning, 3(2): 154–158.DOI
  18. [19]Lewis, P.; Perez, E.; Piktus, A.; Petroni, F.; Karpukhin, V.; Goyal, N.; Küttler, H.; Lewis, M.; Yih, W.-t.; Rocktäschel, T.; et al. 2020. Retrieval-augmented generation for knowledge-intensive nlp tasks. Advances in neural information processing systems, 33: 9459–9474.arxiv.org/abs/2005.11401
  19. [20]Liu, W.; Ni, C.; Mulvaney, S. A.; Rose, S. L.; Kantarcioglu, M.; Malin, B. A.; and Yin, Z. 2026. MHGraph-Bench: Knowledge Graph-Grounded Benchmarking of Mental Health Knowledge in Large Language Models. In Proceedings of the Fifth Workshop on Generation, Evaluation and Metrics (GEM), 393–409.
  20. [22]Mallen, A.; Asai, A.; Zhong, V.; Das, R.; Khashabi, D.; and Hajishirzi, H. 2023. When Not to Trust Language Models: Investigating Effectiveness of Parametric and Non-Parametric Memories. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (ACL), 9802–9822.
  21. [23]Mantel, N.; and Haenszel, W. 1959. Statistical Aspects of the Analysis of Data from Retrospective Studies of Disease. Journal of the National Cancer Institute, 22(4): 719–748.
  22. [25]Qwen Team. 2024. Qwen2.5 Technical Report. arXiv preprint arXiv:2412.15115.
  23. [26]Rayeed, S. M.; East, A.; Stevens, S.; Record, S.; and Stewart, C. V. 2025. BeetleVerse: A Study on Taxonomic Classification of Ground Beetles. arXiv preprint arXiv:2504.13393.arxiv.org/abs/2504.13393
  24. [27]Rayeed, S. M.; Khurana, M.; East, A.; Fluck, I. E.; Campolongo, E. G.; Stevens, S.; Zarubieva, I.; Lowe, S. C.; Denslow, M. W.; Donoso, E. D.; Wu, J.; Ramirez, M.; Baiser, B.; Stewart, C. V.; Mabee, P.; Berger-Wolf, T.; Karpatne, A.; Lapp, H.; Guralnick, R. P.; Taylor, G. W.; and Record, S. 2026a. A Continental-Scale Dataset of Ground Beetles with High-Resolution Images and Validated Morphological Trait Measurements. Scientific Data.arxiv.org/abs/2601.10687
  25. [28]Rayeed, S. M.; Ma, Q.; Stewart, C. V.; and Ma, Y. 2027. How Finely Can Vision-Language Models See Biology? Bio-VQA: A Benchmark for Taxonomic, Morphological, Spatial and Individual Reasoning. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision (WACV).arxiv.org/abs/2611.05559
  26. [29]Rayeed, S. M.; Stewart, C. V.; Ma, Y.; et al. 2026b. CLADE: Category-Level Answering from Deficient Evidence in Retrieval-Augmented Language Models. arXiv:2611.14355.arxiv.org/abs/2611.14355
  27. [30]Reilly, T.; Jackson, W.; Berger, V.; and Candelario, D. 2017. Accuracy and completeness of drug information in Wikipedia medication monographs. Journal of the American Pharmacists Association, 57(2): 193–196.DOI
  28. [31]Robins, J.; Breslow, N.; and Greenland, S. 1986. Estimators of the Mantel-Haenszel Variance Consistent in Both Sparse Data and Large-Strata Limiting Models. Biometrics, 42(2): 311–323.pubmed.ncbi.nlm.nih.gov/3741973
  29. [32]Sayers, E. 2009. The E-utilities In-Depth: Parameters, Syntax and More. Entrez Programming Utilities Help, National Center for Biotechnology Information.
  30. [33]Schriml, L. M.; Munro, J. B.; Schor, M.; Olley, D.; McCracken, C.; Felix, V.; Baron, J. A.; Jackson, R.; Bello, S. M.; Bearer, C.; et al. 2022. The Human Disease Ontology 2022 update. Nucleic Acids Research, 50(D1): D1255–D1261.DOI
  31. [34]Singhal, K.; Azizi, S.; Tu, T.; Mahdavi, S. S.; Wei, J.; Chung, H. W.; Scales, N.; Tanwani, A.; Cole-Lewis, H.; Pfohl, S.; et al. 2023. Large language models encode clinical knowledge. Nature, 620(7972): 172–180.DOI
  32. [35]Singhal, K.; Tu, T.; Gottweis, J.; Sayres, R.; Wulczyn, E.; Amin, M.; Hou, L.; Clark, K.; Pfohl, S. R.; Cole-Lewis, H.; et al. 2025. Toward expert-level medical question answering with large language models. Nature medicine, 31(3): 943–950.DOI
  33. [36]Stevens, S.; Kline, J.; Rayeed, S. M.; et al. 2025. Mind the (Data) Gap: Evaluating Vision Systems in Small Data Applications. arXiv preprint arXiv:2504.06486.arxiv.org/abs/2504.06486
  34. [37]Ursu, O.; Holmes, J.; Knockel, J.; Bologa, C. G.; Yang, J. J.; Mathias, S. L.; Nelson, S. J.; and Oprea, T. I. 2017. Drug-Central: Online Drug Compendium. Nucleic Acids Research, 45(D1): D932–D939.DOI
  35. [38]Vasilevsky, N. A.; Matentzoglu, N. A.; Toro, S.; Flack IV, J. E.; Hegde, H.; Unni, D. R.; Alyea, G. F.; Amberger, J. S.; Babb, L.; Balhoff, J. P.; et al. 2022. Mondo: Unifying diseases for the world, by the world. MedRxiv, 2022–04.DOI
  36. [39]Xiong, G.; Jin, Q.; Lu, Z.; and Zhang, A. 2024. Benchmarking Retrieval-Augmented Generation for Medicine. In Findings of the Association for Computational Linguistics: ACL 2024.arxiv.org/abs/2402.13178

Paper details

vlnc:W_uybztzk6
Version
v1 · latest
Published
Fresh — added to the Hub within the last 30 days
Added
License
CC BY 4.0
Usage

views · 0

downloads · 0

A view counts once per browser per day, a download each time the PDF is requested.

  • Computer Science

Not recorded for this paper

View statement
Looking for related papers…
Contents