betaFeedback

WorksComputer Science

VerifiedLab: Harnessing Knowns for Unknowns

Abstract

Across human history, knowledge has undergone several transitions, from orality to writing and then to digital archives. At each transition, humans established new conventions to preserve knowledge for the new era. Nowadays, as research agents are increasingly deployed for laboratory experiments, knowledge is crossing another boundary: from records written for humans to those for machines. In this work, we introduce VerifiedLab, a multi-agent harness framework for this human-to-machine transition of knowledge. VerifiedLab transforms research output from passive artifacts into active systems that accelerate downstream use, adoption and prediction of future trends. In VerifiedLab, a team of AI scientists with distinct roles reasons over records verified field by field: a principal investigator plans, a librarian computes the evidence directly from the records, specialists draft, and a gate and a critic check each claim against the measured reliability of its data before the harness decides whether to release or withhold it. We demonstrate the efficacy of VerifiedLab through extensive experiments on perovskite solar cells: VerifiedLab recovers known physical trends, anticipates devices reported a year later, and lets AI scientists inherit the literature together with a measure of how far it can be trusted. By scaling the inference-time verification compute, VerifiedLab introduces a new pathway for knowledge dissemination and machine harness for AI co-scientists.

For most of human history, knowledge was carried in human minds. How to knap a flint, read the weather or set a broken bone was held in memory, shown by hand and learned by watching. Across generations, social learning allowed knowledge to accumulate beyond individual discovery[1]. But a chain of minds is a fragile medium. Each time a skill passed from one mind to another, a step could be dropped. An instruction could be misremembered. A recipe could be repeated without the circumstance that had made it work. Preserving improvements through such handovers is central to cumulative culture[2, 3].

Writing gave knowledge a durable medium. A record on clay, papyrus or paper outlived its author, travelled beyond its author’s reach, and could be consulted unchanged a century later. Printing multiplied the copies and lowered the cost of spreading them. More scholars in different cities could consult and compare copies of the same text[4]. Journals such as the seventeenth-century Philosophical Transactions organized a continuing public record. Over subsequent centuries, conventions of authorship, methods reporting, editorial review and citation developed around that record[5]. Each of these steps carried knowledge across a boundary: from mind to page, from manuscript to press, from private letter to public record. At each boundary, meaning could leak away until a convention was invented to hold it in place. A manuscript without an author could not be questioned, an experiment without a method could not be repeated, and a claim without a citation could not be traced. Science became trustworthy because of these conventions, not only because of the medium.

For the literature, the present is a moment of déjà vu: it is crossing such a boundary again (Figure 1). Papers are now digital and searchable, while FAIR principles make the findability, provenance and reuse of scientific data explicit[6]. More and more of these records are

Figure 1. Scientific knowledge has crossed a boundary each time it changed medium. Left, the media that carried knowledge; right, the rule that made each trustworthy. The apex is the boundary being crossed now, where AI scientists read papers and this work supplies the rule. Icons: Heroicons (MIT), Font Awesome Free (CC BY 4.0).

read by machines, and the transition this time runs from papers written for human readers to structured records that machines act on. Research agents built on large language models (LLMs) read prior work, form hypotheses, plan and run experiments and choose the next experiment[7, 8, 9, 10, 11, 12, 13]. Where experiments are themselves code, an agent can write a script, run it and read the answer[7, 14, 15]. In chemistry and materials science the situation is different. The knowledge an agent needs is spread over tens of thousands of papers. Each paper mixes prose, tables, figures and supporting files. Turning those reports into structured data is a separate information-extraction task[16, 17]. Consider perovskite solar cells (PSCs), whose efficiency has advanced from early sensitized devices to certified research cells [18, 19]. Progress depends on composition[20, 21], interfaces [22, 23], processing chemistry[24, 25], tandem integration[26] and grain-boundary control[27]. The paper describes several devices; its best efficiency is in a table, the recipe for the light-absorbing layer in the methods, and the stability test in a supplementary figure with its own temperature, humidity and light. A solvent ratio mentioned in passing applies to one variant only. Each number belongs to one device, one step and one set of conditions; measurement and stability-reporting guidelines require those links for meaningful comparisons [28, 29]. A language model can recover the numbers while losing these links. It writes plausible wrong values: a reversed ratio, a layer attached to the wrong device, a stability figure separated from its protocol. Such a record passes a glance and fails in use, and its errors pass into every plan built on top of it. The alternative, curating the literature by hand, takes years of expert labour [30]. Automated databases have captured device properties and selected fabrication details [31, 32]; we extend this line of work to the complete record of each experiment, verified field by field.

This work addresses this challenge by converting each paper into a verified record that functions as a machine-readable account of the experiment, exposing its devices, procedures and measurements as fields that carry their evidence and their measured reliability. Language models extract materials data from scientific prose [16, 33, 34]; text-mining systems recover chemical entities and synthesis descriptions [35, 36], while literature embeddings capture associations between materials [37]. Long-context benchmarks nevertheless show that access to a document does not ensure reliable use of evidence throughout it [38]. Studies of LLM judges measure agreement with people and expose consistency and correctness limits [39, 40, 41]. Research-agent benchmarks assess experimentation, data-driven discovery and paper reproduction[14, 42, 43, 44, 45]. Scientific fact-checking and claim–evidence benchmarks assess support for claims, including evidence dispersed across full papers [46, 47]; attribution and citation benchmarks test the support of generated statements by identified sources [48, 49]. Each of these addresses a part of the task. We preserve device, procedure and measurement identity across an entire experimental record under field-specific scientific rules, with verifiers calibrated against experts and reliability estimates that stay with the records.

In this work, we introduce VerifiedLab, a multi-agent harness framework for this human-to-machine transition of knowledge (Figure 2). We built the harness for perovskite solar cells and tested it at each stage, from the reading of a paper to the use of the resulting records in scientific reasoning. Frontier models recover most of a record but lose its procedures: current agents misplace values that are tied to several procedural steps in experimental materials science and chemistry, and they share the same failure patterns. Written rules close much of this gap on the verifier side, bringing verifiers to expert-level agreement regardless of model size. Run over the recent literature of the field, the harness returns a database in which every category of field carries a judged accuracy. On this database we place a team of AI scientists with distinct roles. A principal investigator plans the study, a librarian computes the evidence directly from the records, specialists draft the claims, and a gate and a critic check each claim against the measured reliability of its data before the harness decides whether to release or withhold it. Reasoning only over the evidence, the team recovers known physical trends and anticipates devices reported a year later, and it withholds the claims, on stability for instance, whose data are not yet reliable enough to carry them.

Figure 2. Overview of the verified-literature harness. a, Two coupled agent loops. In the harness loop (blue), a generator agent fills one record per paper under the annotation Skill A\mathcal{A}, a verifier agent labels every field under the verification Skill B\mathcal{B}, and the audit re-versions both Skills. In the discovery loop (purple), a PI agent, a librarian and specialist agents reason over the verified database, and a critic rejects claims without provenance (dashed red). Published proposals re-enter the harness as papers (grey). b, The objects that a human laboratory, an LLM reading PDFs and the Verified Lab use for input, policy, evidence, checking and provenance. c, A recipe refined over three iterations with specialist feedback; nn and the share above 22% are computed over the matching verified records. d, A real record reconstructed with the annotation Skill (red, errors; green, released record), with the verifier’s feedback. e, Verifier decisions on candidate–reference pairs and the rule applied. f, A claim before and after the critic. Icons: Heroicons (MIT), Font Awesome Free (CC BY 4.0).

Verified Harness: Record, Policy and Check

The verified harness has three parts: a record, a policy and a check. The record is a schema; a generator agent fills it, and a verifier agent checks every filled value against evidence, both under written policies (Figure 2a and Methods). The schema was written by experimentalists. Its fields, nearly a hundred of them, describe a paper’s best-performing device in seven groups: reference data, cell definition, device stack, material synthesis, device fabrication, current–voltage (J–V) metrics and stability metrics. Each field has a fixed name and a convention for its value. Units stay attached to numbers. Layer stacks are written in order, as in TiO2/ZrO2/carbon. Null has one meaning: the paper does not say.

The scientific policy lives in versioned Skills rather than in prompts tied to one model. This makes an important dependency of the pipeline explicit, a concern also identified in work on technical debt in machine-learning systems[50]. Rigour is written down there, in the reviewing consensus of experts, the equivalence rules of chemistry and the boundary conditions of physics, and applied to every field. It is not left to a model that cannot be inspected or corrected. The annotation Skill gives a generator the rule for choosing the champion device, the priority among sources of evidence, the alignment of layers with values, the assignment of each measurement to its device and the conditions for writing Null. It forbids filling a field from general knowledge or from a different device. The verification Skill defines, for a verifier, the equivalence of two values. Chemical aliases, compatible units, numerical tolerances and ordered sequences are accepted. Reversed ratios, reordered layers and values borrowed from another device are rejected. For each field the verifier sees the schema entry, the candidate value, the expert reference and the rule, and it returns one of five labels: the values agree, both are empty, the candidate is missing, or the candidate is unsupported or wrong. Decisions are made field by field, so each can be audited on its own, and a wrong rule is caught by the audit and fixed in the next version. The score of a generator never assumes that the automated judge is right, because the judge is itself measured against expert decisions. The reference for those measurements comes from two expert-curated collections of PSC papers (Methods). One was annotated by us against the full schema; the other was drawn from a community-curated database[30]. Experts annotated independently and resolved disagreements before any automated verifier was run, separating annotation, adjudication and evaluation in line with established quality-management practice[51]. Many fields are legitimately Null, so a record left entirely empty already scores well above zero. This empty-record baseline is the floor against which generators are read. Figure 6 shows the whole workflow in use by a team of agents that reaches the literature only through the harness; we return to it after the benchmarks.

Figure 6. A team of AI scientists reasoning over verified records. a, Architecture, after the Virtual Lab [10]. A human sets the goal; a PI agent recruits specialists and a critic; every agent reaches the literature through the librarian, which returns records tagged by category reliability and provenance, and a forecaster trained on records to 2023 scores candidate devices. b, Six phases of a design question. Dialogue is illustrative and abridged; all numbers are computed from the released records and the forecaster. The recipes in Phase 5 are held-out 2025 devices chosen for illustration. c, One design round: the device at each iteration, the agent acting on it, the record the librarian returns with its verifier labels (TP, true positive), and the claim after the critic strikes the clause that rests on an unreliable category. nn and the share above 22% are computed over the matching verified records (Methods).

Procedural Information Hampers Recovery

The first test asks how much of a record today’s readers recover, and which part of it they lose. We ran eighteen generators, proprietary and open-weight, through the fixed harness, so that differences between runs reflect the model alone. A calibrated verifier, described in the next section, scored them (Figure 3a and Supplementary Fig. 1). With the annotation Skill, the strongest proprietary generators recover most of each record. The two open-weight generators sit close to the empty-record floor, because most of their errors are fields they leave blank. The proprietary generators fail differently. For most of them, the largest class of error is a value that is present but wrong, ahead of omissions and ahead of values that the paper never gave.

Figure 3. Research agents reconstruct experimental records. a, Field accuracy of 18 generators on Collection A (100 papers, 9,700 field decisions each) with (filled) and without (open) the annotation Skill; colours, model families; dashed line, empty-record baseline. b, Accuracy by subfield for three generators. c, Error composition for the five strongest generators, by subfield (left) and by the competence a field tests (right; localized scalars, 24 fields; relational structures, 30; conditions, 43). Omission, wrong value and over-prediction are defined in Methods. Decisions are those of the o3 verifier with the verification Skill.

The errors gather where the information is procedural (Figure 3b and Supplementary Fig. 2). Bibliographic fields are at ceiling for every model. Stability and synthesis fields are where models part company, and in the stability fields the usual mistake is to invent a condition that the paper left unspecified (Figure 3c). To separate knowledge from binding, we grouped the fields by what each one asks of the reader.

Localized scalars, which a paper states once in one place, are recovered best. Conditions tied to a single processing step or test come next. Relational fields, whose value is an ordered or multi-part structure attached to a layer or a device, are recovered worst, and they come back wrong rather than empty. A sizeable share of reported layer stacks and precursor-concentration lists do not match the reference. A spin-coating speed or time almost always does. These failures are shared, not model-specific. The ranking of fields by difficulty is nearly the same for every generator, and the same small set of fields sits at the bottom for strong and weak models alike: additive and

Prompting does not close it either. Switching the annotation Skill on, with everything else fixed, helps every generator (Figure 3a), but the help is uneven. Fields defined by an encoding convention, such

precursor concentrations, the atmosphere of each processing stage and the stability protocol. A stronger model lifts every category but does not remove these modes. Every frontier model knows perovskite chemistry. None of them reliably attaches a value to the device, the step and the condition that produced it, and a stronger reader narrows this gap without closing it. Procedural information, not chemical knowledge, is the part of the record that hampers recovery. as a flag for sequential deposition[52], improve sharply. Fields that require the model to align evidence across the paper, such as the list of precursor compounds or the atmosphere of a particular step, can get worse under the same instructions. Policy text tells an agent how to write a value down, but not which device or step the value belongs to. This is why the harness makes verification a separate role rather than trying to prompt its way to reliability. For deployment we chose the generator on the accuracy–cost frontier, a small proprietary model that gives up little accuracy against the best reader for a several-fold saving in cost per paper (Methods).

Verified Harness Brings AI Scientists to Expert Agreement

The second test moves from the agent that writes the record to the agent that checks it. A verifier is not ground truth, and its authority has to come from somewhere. In the verified harness that source is the written rule, and we tested whether rules, rather than the size of the model behind them, can bring a verifier to agreement with experts. We tested this in three steps: we calibrated proprietary verifiers against expert decisions, ran ten open-weight verifiers with and without the rules, and, within each model family, set the smallest checkpoint with rules against the largest without. Compared with the expert decisions on both collections (Figure 4a), three proprietary verifiers already agree with the experts most of the time without rules. With the verification Skill, their agreement rises to within a few decisions per thousand, and the gain is largest for the weakest model. The strongest configuration, audited field by field, is the reference verifier used for the generator benchmark above and for the open-weight study that follows.

Four-panel comparison of verifier agreement, equivalent-pair acceptance, non-equivalent-pair rejection, and accuracy differences

Figure 4. Written verification rules bring verifiers to expert agreement. a, Agreement with expert decisions for three proprietary verifiers, with and without the verification Skill, on Collection B (n=1,460n = 1{,}460 decisions) and Collection A (n=1,940n = 1{,}940). b, Agreement of ten open-weight verifiers with the audited o3 decisions (9,700 decisions) without (open) and with (filled) the Skill; Δ\Delta, paired difference in percentage points. c, Acceptance of equivalent pairs (n=4,620n = 4{,}620) against rejection of non-equivalent pairs (n=548n = 548), without (open) and with (filled) the Skill; inset, dashed region enlarged. d, Within each family, the largest minus the smallest checkpoint without rules (grey) and the smallest with rules minus the largest without (blue). Error bars, 95% record-clustered bootstrap intervals (2,000 resamples).

To see the mechanism, we ran ten open-weight verifiers from four model families on the same field decisions, with and without the Skill, and scored their agreement with the audited reference (Figure 4b and Supplementary Fig. 3). The Skill improves almost every verifier. It pulls the checkpoints closer together and helps most where the starting point is weakest. Most non-empty pairs are identical once trivially normalized and are decided correctly either way. The Skill acts on the remainder, where the candidate and the reference say the same thing in different words, and nearly every decision it changes turns a rejection into an acceptance (Figure 4c and Supplementary Table 1). The gains land where surface form misleads: journal aliases, synonyms for a deposition method, notation for humidity, units for a scan rate. Two fields regress because of defects in the policy, which the next version of the Skill corrects. Ordered layer stacks improve but remain the hardest decision. The models do not lack chemistry; without rules, a verifier rejects an equivalent value because it is written differently, and the Skill tells it that the two are the same.

Scale does not order verifier reliability (Figure 4d and Supplementary Table 2). In three of the four families, the larger checkpoint is less accurate without rules than the smaller one, and the smaller checkpoint with the Skill beats the larger one without it, by margins whose bootstrap intervals exclude zero. In the fourth family the largest model without rules still edges the smallest with rules, by a margin that cannot be told from zero. Two checkpoints that the Skill does not help show the limit of what rules can do. They already accept almost every equivalent pair and fail instead by accepting values that are among, and rules that license equivalences cannot make a verifier stricter (Figure 4c). Rules replace scale for verifiers whose errors are lexical rejections, which is the common case, but not for verifiers whose errors are permissive acceptances. The reference labels are the decisions of one proprietary model, so closer agreement may in part mean adopting that model’s conventions; the expert audit bounds this risk for the reference verifier itself. In practice, a laboratory can run an expert-grade verifier on a small open-weight model if the rules exist, and it can fix a wrong rule in the next version of the Skill, neither of which a larger model offers. The harness, not the model, is what brings these AI scientists to expert agreement.

Scaling Inference-time Compute Guarantees Reliability

Reliability at the scale of a field is bought with inference-time compute. After the generator has written a record, every field of it is checked again, by two independent judges of different capability, and the cost of these checks is the price of a measured reliability. A reliability measured on a hundred and fifty papers is useful only if it survives deployment and is delivered together with the data. We ran the harness over the peer-reviewed experimental PSC literature of the past five years, with the frontier-cost generator chosen above. The judges verified every field of every record under the verification Skill (Methods). Their disagreement is a cheap check on the evaluation itself. The harness returned one record per paper and a judged accuracy for every category of field (Supplementary Fig. 4a). The two judges differ by less than one accuracy point in any category.

The pattern across categories matches where the information sits in a paper, and it is the pattern the benchmark predicted. Fields that are local and lexically anchored are near ceiling: the reference data, the cell definition, the fabrication steps and the J–V metrics. Fields that require following a procedure across sections are harder: the device stack and the material synthesis. Stability fields are hardest, because a stability protocol mixes conditions, durations and relative values that are often reported only in a figure or in the supporting information. Recall exceeds precision overall and in every category but one (Supplementary Fig. 4b). The generator rarely misses information that is present; its usual failure is an incorrect value, the same profile the benchmark found on the expert-curated papers. Low-precision categories can propagate incorrect values into later analysis, an instance of the downstream effects of data-quality problems documented in other AI pipelines[53].

The database covers literature through April 2025 and complements the manually curated resource[30] and earlier automatically extracted datasets[31, 32]. Annotating the whole corpus cost a small fraction of the expert time behind the manually curated database. Category reliability is released alongside the records, extending the documentation of intended uses and evaluation conditions advocated by datasheets and model cards[54, 55]. These category scores do not provide confidence estimates for individual values. From it follows a rule for anyone who uses the data, person or agent: a field may enter automated analysis only if its category reliability is high enough. Our discovery experiments obey this rule, and neither uses stability, the least reliable category.

From Knowns to Unknowns

The last test is the use of the records in scientific reasoning, from the knowns of the field to its unknowns. We ran it at three levels: the recovery of trends that the field has established by other means, the prediction of devices that no model has seen, and the use of the records by a team of AI scientists under the reliability rule.

Figure 5a–c shows the J–V landscape read from the high-reliability categories alone. The distribution of reported efficiency has the shape the field would expect, and so do the distributions of open-circuit voltage, short-circuit current density and fill factor. High-efficiency devices cluster in a narrow band of bandgaps below the Shockley–Queisser limit (Figure 5b). Efficiency falls and spreads as the device area grows[56, 57, 58] (Figure 5c). Within this corpus, regular (nn–ii–pp) devices have higher peak efficiencies than inverted ones; this is a property of the sampled years, not a universal ordering of architectures[59]. Composition and surface-treatment associations are consistent with the broader literature on mixed-cation absorbers and interfacial modification[60, 61, 62]. Recovering these relationships supports aggregate use of the records, without establishing the accuracy of every value or a causal effect of treatment. Each record is a paper’s champion device, so these distributions describe what was reported, not every device that was made.

Figure 5. Reliable fields rediscover known trends and predict devices reported a year later. a–c, The 10,258 champion devices read from the high-reliability categories: a, reported PCE (dashed line, median); b, PCE against bandgap with the Shockley–Queisser limit (dashed) and the 1.5–1.7 eV window (shaded); c, PCE against cell area with the maximum, mean and median per logarithmic bin. d–g, Forecasting 2025 devices from recipe fields alone: d, reported PCE by publication year and the temporal split (box, interquartile range; whiskers, 1.5×); e–g, the fine-tuned T5 encoder on the 858 test devices, with predicted against reported PCE (band, ±2 points), the cumulative absolute error and the ROC curve for PCE > 20%; the expert-descriptor LightGBM and the training-mean predictor are shown for reference. h, Benchmark on the same split: MAE, the ratio of the mean absolute deviation of the targets to the MAE (MAD:MAE), R2R^2 and the AUC for PCE > 20%; mean ± s.d. over three seeds; bold, best per metric.

For an AI scientist, a literature-derived dataset matters only if the knowledge it encodes transfers to devices it has not seen. Following LLM4Mat-Bench’s comparison of task-specific and general-purpose predictors [63], we posed a temporally held-out task: predict a device’s efficiency from its recipe alone, the fields describing the cell, the stack, the synthesis and the fabrication. Models learn from papers published up to one year, are tuned on the next and are scored once on the year after that. Every performance, stability and bibliographic field is withheld to reduce target leakage (Methods) [64]. We compared gradient-boosted trees [65] and a tabular foundation model [66], text encoders fine-tuned on a plain-text serialization of the recipe [67, 68, 69], and a general-purpose LLM[70] whose training data end before any test paper appeared, used either through frozen embeddings or prompted with retrieved examples (Figure 5h). Every trained model clearly beats the predictor that always guesses the training mean, even though reported efficiencies drift upward from the training years to the test year. A tree model on a short list of expert-chosen descriptors already separates good devices from poor ones. Giving it every recipe field improves it further, so the full schema that the harness reconstructs carries information that expert descriptors leave out. The fine-tuned text encoders, which read the record as a few lines of text and need no feature engineering, do best of all, and most of their remaining error sits on low-efficiency devices (Figure 5d–Figure 5g; every model is shown in Supplementary Figs. 5 and 6). With the architecture held fixed, formamidinium content and bandgap rank highly among the descriptors. Studies of composition and

formamidinium absorbers make these associations plausible [20, 71], but feature importance alone does not establish a mechanism. A general-purpose LLM is no substitute for training on the records: prompted without examples it predicts nearly the same efficiency for every device, and the examples it is handed by retrieval account for most of what it gains. Residual error may reflect omitted processing choices, laboratory differences, extraction errors and model limitations; measurement conventions are another documented source of variability [28].

At the third level we sketched the Verified Lab, a team of LLM agents modelled on the Virtual Lab [10] in which every agent reaches the literature only through the harness (Figure 6a). A principal-investigator agent plans the study and convenes three specialists (a device physicist, a materials chemist and a data scientist) and a scientific critic. A librarian agent, which is the harness itself, answers their queries with records that carry their category reliability and their provenance, and computes the evidence directly from those records. A reliability gate admits only the categories above the threshold, and the critic checks each drafted claim against the evidence before the harness releases or withholds it. A forecaster, the text encoder trained above, scores candidate recipes. In a six-phase walk-through of a design question (Figure 6b), the team confines its reasoning to the categories the gate admits. It retrieves the champion devices with a reported efficiency, recovers that the best of them sit in a narrow bandgap window and that efficiency falls with area, and returns recipes from held-out papers whose forecast efficiencies lie close to the reported values. Each field is linked to its DOI, its rule and its verifier label. One design round is laid out as a pipeline (Figure 6c). Two specialists move a recipe from a single-cation absorber to a formamidinium-based one with an additive and a surface treatment. The librarian returns the matching records with their labels. The critic strikes the one clause that rests on an unreliable category. The dialogue is illustrative, but every number in it is computed from the released records. The demonstration shows the contract we propose for AI scientists, not a validated discovery; its running implementation, a research workspace released with the code, is shown in Supplementary Figs. 7–15. The same abstractions serve a human laboratory and an AI scientist (Figure 2b): a typed record in place of the PDF, a versioned Skill in place of a model-specific prompt, a reliability-scoped query in place of recall, and a verifier call and a provenance call in place of the colleague who re-derives a number by hand. Together, these results indicate that the verified records carry the knowns of the field in a form from which its unknowns can be approached, at all three levels, and that the reliability rule is what makes each level credible. The trends are read from categories whose accuracy is known, the forecaster is trained on them, and the critic enforces them.

Discussion

We set out to carry the experimental literature across its newest boundary with its meaning intact, and to measure the agents that do the carrying. Separating the writing of a record from its checking, governing both with written scientific policy and holding everything else fixed turned the output of a language model into a measured artefact. The result is a benchmark that shows where research agents fail, a verifier whose agreement with experts is known, and a field-wide database in which every category of field carries a judged accuracy. The same error profile appears on the benchmark and in the deployment. Agents recover isolated facts, fail when a value has to be tied to a device, a step or a condition, and fail by writing a plausible wrong value rather than by leaving a blank. These are the errors that an aggregate accuracy hides and that pass silently into an AI scientist’s plans. Three principles follow, and each runs against a common assumption about LLM-driven science. First, the bottleneck is binding, not knowledge. Existing systems demonstrate scientific text extraction and tool-augmented reasoning [16, 33, 72]. These successes leave a separate question: whether each extracted value remains attached to the correct experiment. Every frontier model fails in the same procedural places. A stronger model narrows the gap without closing it, and prompting fixes conventions but not alignment. Second, rules are worth more than scale, and rigour should be written down rather than hoped for. A verifier’s authority can come from calibration against experts under explicit, versioned rules rather than from the size of its backbone. The gain falls on exactly the decisions that string matching cannot settle, and the defects the audit exposed were fixed in the next version of the Skill. Writing the reviewing consensus of experts, the equivalence rules of chemistry and the boundary conditions of physics into versioned Skills makes verification inspectable, correctable and portable across models, none of which a larger model provides. The limit of rules is equally clear: they license equivalences, so they repair verifiers that are too strict and not verifiers that are too lenient. Third, reliability travels with the data. A reliability measured once on a benchmark and then set aside protects no one. Measured per category and attached to every record, it becomes the rule by which any later analysis, human or automated, decides what to trust. Seen against the history we began with, these principles are the rules for the newest boundary. They do for the machine-read record what review and citation did for print. The harness has limitations that we consider inherent in current LLM-based pipelines, and each points to a remedy. The reliability of the full database is judged by language models rather than by complete expert annotation, and one judge shares the generator’s backbone and may favour its outputs [73]. The expert audit bounds this risk for the calibrated verifier and the disagreement between the two judges keeps it visible, but a sample of deployed records should be re-audited as the database is reused. The Verification Skill describes equivalences in more detail than non-equivalences, which is why rules help over-strict verifiers and not lenient ones; explicit guardrails against false acceptance are the next addition. We studied one materials domain, so the schema and category scores need not transfer. Documenting the corpus and its intended use is necessary before generalizing to another setting [74]. Each record is a paper’s champion device and positive results are more likely to be published, so the database describes reported best devices. The trends and forecasts are associations tested on one held-out year; they suggest hypotheses and do not establish causes, consistent with cautions about interpreting data-driven materials models [75, 76]. The weakest categories, synthesis and stability, require following a procedure across a long document and reading figures and supplements. Multimodal inputs and targeted human review of these fields are the most direct ways to close the gap. Finally, the harness relies on commercial models whose versions change; we pin every version and release the records as the reference artefact.

Human expertise remains part of the design. Experts write the schema, author and version the Skills, adjudicate the reference records and audit the verifier. The harness scales their judgement to a whole field and reports, category by category, how far that scaling can be trusted. The error profile suggests a division of labour that we have motivated but not yet tested end to end. Localized scalars and exact matches can be accepted automatically, conditions and relational fields need verification under calibrated rules, and ordered sequences, multi-part concentrations and stability protocols stay with the experts.

Looking ahead, just as journals now expect data and code to be available, we anticipate that research agents will expect the literature to arrive as verified records with an explicit reliability. The ease with which a paper can be turned into such a record will itself become a measure of how reproducibly it was written. Every earlier medium of knowledge acquired its conventions after the fact, once the cost of unprotected transmission had become visible. A verified-literature harness offers AI scientists in the experimental sciences those conventions in advance: the data they reason over, and a measure of how far that data can be trusted. As agents move from answering questions to choosing what to synthesize and measure, that measure becomes a prerequisite.

Methods

Harness design

Task. Given a paper package dd (main text plus supporting information) and a schema S={sj}j=1mS = \{s_j\}_{j=1}^{m} of m=97m = 97 fields, the harness outputs one record r(d)r(d) describing the paper’s best-performing device, with every field filled by a string value or an explicit Null. Selecting one device per paper gives a comparable unit across heterogeneous reports; it also introduces a selection effect that we account for in the discovery analyses. The 97 fields fall into seven categories (reference data, cell definition, device stack, material synthesis, device fabrication, J–V metrics and stability metrics).

Versioned Skills. Scientific policy is separated from the code that executes it. The annotation Skill Aα=(S,C)A^{\alpha} = (S, C) (released as [email protected]) couples the schema with a policy CC for champion-device selection (rank by explicitly reported scan PCE, not stabilized output, certification or module efficiency), evidence priority (prefer explicit device-specific statements; map sample codes across article and supplement before combining fabrication and performance values), layer alignment (report the complete electrically active stack; align slash-delimited properties with layer order), measurement identity and the use of Null (when a value is absent, ambiguous between devices or only inferable). The verification Skill Vβ=(R,L)V^{\beta} = (R, L) ([email protected]) supplies field rules R={rj}R = \{r_j\} defining chemical aliases, compatible units, numerical tolerances, ordered-sequence semantics and device-specific comparisons, together with the allowed label set LL. A machine-readable registry pins the full semantic version used in every run; model changes preserve the evaluation contract, whereas changes to a schema, tolerance or decision rule require a new Skill version. The model-facing views materialized from both Skills are released with the code.

Execution boundary. The harness HH parses the paper package, materializes the annotation Skill for a model adapter, validates candidate keys and types (rejecting malformed records), pairs each candidate value with the expert reference and field rule into qj=(sj,y^j,yj∗,rj)q_j = (s_j, \hat{y}_j, y_j^{*}, r_j), invokes the verifier on each unit, validates the returned label against LL, aggregates metrics and writes an audit bundle containing document, candidate, reference, policy versions, model versions and decision provenance. Every generator and verifier traverses this same boundary.

Generator and verifier agents. A generator agent reads the full paper package, identifies the best-performing device and emits a JSON record conforming to SS. A verifier agent receives, for each field, the candidate value, the reference value and the field rule and returns one of TN (both empty), FP (reference empty, candidate filled), FN (reference present, candidate empty), TP (both present and scientifically equivalent) or FP mismatch (both present and non-equivalent). Accuracy, precision, recall and F1F_1 follow the standard definitions with the false-positive count equal to FP + FPmismatch_{\text{mismatch}}. Under this metric an entirely empty record scores 36.4% accuracy on Collection A, which we report as the empty-record baseline.

Expert-curated testbed

The reference comprises two collections. Collection A contributes 100 papers annotated by us against all 97 fields; Collection B contributes 50 papers with 73 overlapping fields drawn from the Perovskite Database [30]. For each paper, domain experts inspected the article and supplement, annotated independently against the pinned annotation Skill, identified field-wise disagreements and resolved them by discussion and re-inspection before any automated verifier was evaluated. The collections total 13,350 field decisions; an average paper package is 23,231 input tokens and an average record 1,586 output tokens.

Generator benchmark

We evaluated 16 proprietary and 2 open-weight generators (release date, pinned version and context length). Each generator produced one record per Collection A paper with and without the annotation Skill, with the harness, schema, source text, parser and output validator held fixed; the o3 verifier with the verification Skill, calibrated as described below, provided the field decisions. We report accuracy, precision, recall and F1F_1 over 9,700 decisions per generator Supplementary Fig. 1 and accuracy by subfield Supplementary Fig. 2. Error composition decomposes decisions into omissions (reference present, candidate Null), wrong values (both present, non-equivalent) and over-prediction (reference Null, candidate filled), each conditioned on the relevant reference state so that subfields with different null shares remain comparable. The three field classes (localized scalar, 24 fields; relational, 30; condition, 43) are an analysis grouping of the schema by the competence a field exposes, assigned from the schema definitions, not labels in the dataset. Per-field accuracy ranks were compared across generators with pairwise Spearman correlation. Request cost per paper for each proprietary generator was taken from the provider’s list price at the pinned version; the two open-weight generators have no API price and are shown without cost. Annotation-Skill variants with increasing field-specific detail were materialized for gpt-5-mini and gpt-4o-mini to measure how much domain policy contributes beyond the base model; these are experimental variants, not interchangeable versions of the released Skill.

Verifier calibration against experts

Three proprietary backbones (o3, gpt-5-mini and gpt-5-nano) verified generator records against the expert references on both collections, with and without the verification Skill, and each automated label was compared with the expert decision for the same field (Figure 4a). The o3 configuration with the Skill was then audited field by field and used as the reference verifier for the generator benchmark and the open-weight study.

Open-weight verifier study

Ten open-weight checkpoints (Qwen3-4B, -8B, -14B; Gemma-3-4B, -12B; Ministral-3-3B, -8B, -14B; Phi-4-mini (3.8B), Phi-4 (14B)) verified the same deterministic sample: a paper-round-robin pass selecting one evaluated generator record from each of the 100 Collection A papers, giving 9,700 field decisions per condition (Supplementary Fig. 3). Audited 203 decisions with the Skill provided the comparison labels (4,622 TP, 3,107 TN, 548 FP mismatch, 323 FP and 1,100 FN). Candidate–reference missingness assigns TN, FP or FN before inference, so the verifier decides only the 5,198 non-empty comparisons (TP versus FPmismatch_{\text{mismatch}}); the other 4,502 fields form the missingness stratum, whose accuracy (0.9636) is identical in both conditions. We report overall accuracy over all 9,700 decisions and semantic accuracy over the 5,198 non-empty decisions. All runs used temperature 0, seed 42, bfloat16 inference, a 16,384-token context and JSON-schema-constrained decoding through xgrammar in vLLM 0.12.0 [77] on NVIDIA A100 40GB GPUs, with Qwen3 thinking disabled; each run configuration pins model and tokenizer revisions, prompt hashes, source digests, package versions and GPU metadata, and all 200 outputs per backbone were recoverable with no invalid record or field. The Skill effect is the paired difference Δ=Accwith−Accwithout\Delta= \mathrm{Acc}_{\text{with}} - \mathrm{Acc}_{\text{without}}; we resampled the 100 records with replacement, retaining all 97 fields within each sampled record, and report 2.5th and 97.5th percentiles over 200 resamples for the per-model deltas and over 2,000 resamples for the within-family scale-versus-rules comparisons (Supplementary Table 2). Acceptance and rejection rates (Supplementary Table 1) are accuracies on the non-empty pairs the reference labels TP and FPmismatch_{\text{mismatch}}, respectively.

The field-wise audit pools each field across the ten checkpoints.

Corpus and deployment

We collected 10,258 peer-reviewed experimental PSC papers published between 2021 and April 2025 and indexed in Web of Science, each as main text plus supporting information. gpt-5-mini (gpt-5-mini-2025-08-07) served as the generator under the annotation Skill; it was chosen because it lies on the accuracy–cost frontier of the generator benchmark. Every record was then judged field by field by two independent verifier agents, gpt-5-mini and o3 (o3-2025-04-16), under the verification Skill. We report accuracy, precision, recall and F1F_1 per field category as the mean over the two judges, with their standard deviation as a check on the evaluation itself. The released records carry per-record provenance linking each record to its source digital object identifier (DOI), and category-level reliability scores. The benchmark builds on MavaLM, published as a NeurIPS 2026 AI4Science workshop poster [78].

Reliability-scoped discovery

Following the reliability rule, neither discovery analysis uses stability fields. The rediscovery analysis reads the high-reliability categories (cell definition, device stack, device fabrication and J–V metrics); the predictive models additionally read material-synthesis fields as inputs. Distributions in Figure 5 use all records with a parsable value for the plotted field; the Shockley–Queisser curve in Figure 5b is the detailed-balance limit for a 5,778 K blackbody Sun normalized to 1,000 W m−2^{-2} and a 300 K cell.

Qualitative demonstration

The Verified Lab (Figure 6) is a design sketch of how AI-scientist agents should consume the records, not an experiment; the agents, meetings and summaries are illustrative. Its statistics use the same parsing as Figure 5: records with a parsable reverse-scan PCE in (0, 30], bandgap from the perovskite bandgap field and cell area from the measured-area field (mm2^2 converted to cm2^2). Large-area devices are those with an area of at least 1 cm2^2; dominant electron-transport layers are the most frequent stack sequences among large-area devices above 20% PCE; a surface treatment is counted when the surface-treatment material field is not Null. Forecasts are those of the fine-tuned T5 encoder (seed with the median test MAE). The three recipes shown were drawn from test-set (2025) devices with non-empty composition, electron- and hole-transport fields and an absolute forecast error below 0.7 PCE points, so they illustrate the output format rather than typical accuracy, which Figure 5d–Figure 5h report. In the design round of Figure 6c, MA-only absorbers are records whose composition names MAPbI3_3 (or CH3_3NH3_3PbI3_3) without FA or Cs and whose electron- and hole-transport stacks contain SnO2_2 and Spiro-OMeTAD; FAPbI3_3 recipes are compositions beginning with FAPbI3_3 without Cs or MA, with MACl named in the additive field; a surface treatment is any non-Null surface-treatment material. The record shown there is reproduced from the database.

Temporal prediction benchmark

Data and split

The target is the reverse-scan PCE, parsed as the first number in the field. We kept the 8,450 records with a publication year and 0 < PCE ≤ 30%, which removes tandem devices and garbled values. Records from papers published up to 2023 form the training set (5,422), 2024 the validation set (2,170) and 2025 the test set (858). The validation set was used only for early stopping and for choosing the embedding layer and ridge penalty; the test set was scored once. The share of devices above 20% PCE rises from 43.5% (train) to 58.2% (validation) and 65.5% (test).

Inputs. Models see the 71 recipe fields of the schema (cell definition, device stack, synthesis and fabrication). Bibliographic and provenance fields, all J–V fields and all stability fields were removed, because PCE equals VOCV_{\mathrm{OC}} JSCJ_{\mathrm{SC}} FF /Pin/P_{\mathrm{in}}, forward-scan, stabilized and certified PCE nearly copy the target, and 610 titles contain a percentage, usually the efficiency.

Tabular features. The expert descriptors are 18 variables: presence of formamidinium (FA), methylammonium (MA) and Cs on the A site, Pb and Sn on the B site, and I, Br and Cl on the X site, parsed from the composition string; perovskite bandgap; absorber thickness; measured cell area; whether a perovskite additive, a surface treatment and an electron-transport-layer (ETL) additive are reported; substrate (ITO, FTO, other); ETL deposition method (spin coating, chemical bath, vapour phase or atomic layer deposition, other); hole-transport-layer (HTL) deposition atmosphere (N2_2, air, other); and sequential spin coating. For all recipe fields, fields with fewer than 10 training values were dropped; a field was treated as numeric if at least 60% of its training values contain exactly one number and was mapped to that number; otherwise its training values with at least 10 occurrences became categories and the rest mapped to “other”. The eight composition-site flags were appended, giving 77 features.

Models. LightGBM used a learning rate of 0.03, 31 leaves, row and column subsampling of 0.8 and early stopping on validation MAE (200 rounds). TabPFN v2 (v2.1.3) used default settings on the all-field table. Text models read a deterministic serialization of the non-empty recipe fields as “field name: value” lines, absorber fields first, built without any LLM. MatSciBERT (512 tokens) and the T5-base encoder (1,024 tokens) were fine-tuned with mean pooling, a linear head and L1 loss on the standardized target for 10 epochs (AdamW [79], batch size 16, learning rates 5×10−55 \times10^{-5} and 10−410^{-4}, 10% warm-up), keeping the epoch with the lowest validation MAE. For the frozen-embedding baseline, mean-pooled hidden states of Llama-3.1-8B-Instruct from layer 16 or 32 fed a ridge regression, with layer and penalty chosen on validation. Prompted Llama-3.1-8B-Instruct decoded greedily for up to 48 tokens; five-shot prompts added the five training records most similar to the query under TF-IDF. An answer was valid only if it was a bare number or ended with a statement of the form “. . . to be X%”; invalid answers were scored as the training mean and their rate is reported.

Metrics and statistics. We report MAE; the ratio of the mean absolute deviation of the test targets to the MAE (MAD:MAE, as in LLM4Mat-Bench [63]; values above 1 beat predicting the mean); R2R^2; and the ROC AUC for PCE > 20% computed with the predicted PCE as the score. Stochastically trained models were run with three seeds and are reported as mean ± s.d. Significance between models uses a one-sided paired bootstrap [80] on test MAE (10,000 resamples) over averaged predictions.

Controlled-subset analysis. As a complementary analysis holding the device architecture fixed, we modelled the 2,445 devices with an SnO2_2 ETL and a Spiro-OMeTAD HTL [81, 82], described by the 18 expert descriptors, with 5-fold cross-validation and grid-searched hyperparameters within each fold. Regression models were gradient boosting, XGBoost [83], LightGBM, random forest [84], extra trees, decision tree, AdaBoost and kk-nearest neighbours; classification models (PCE > 20%) were random forest, LightGBM, XGBoost, gradient boosting, extra trees and decision tree. Feature importance is gain-based for boosted and impurity-based for bagging ensembles; importance rankings from any single model should be read as hypotheses, not mechanisms.

Data availability

The verified device database (10,258 records, 97 core fields, per-record provenance to the source DOI and category-level reliability scores) is released as Parquet and gzipped JSON Lines files on the Hugging Face Hub at https://huggingface.co/datasets/jiayijdong/VerifiedLab and in the dataset/ directory of https://github.com/jiayijdong/VerifiedLab, together with the discovery code and per-run predictions. The expert-curated reference records, versioned annotation and verification Skills, model settings and field-level verification decisions are released with the code. Only structured factual values with provenance are released, not article text; full reproduction of the harness also requires model credentials and licensed article files.

Code availability

The harness runtime, the agentic research workspace, the prediction benchmark (with per-run predictions and raw LLM outputs) and the figure sources are available at https://github.com/jiayijdong/VerifiedLab; the manuscript source is kept separately at https://github.com/jiayijdong/Harnessing-Verified-literature-for-AI-scientist. A two-minute video of the VerifiedLab research workspace, recorded from the running system on local open-weight models, is available at youtu.be/P7We1Q_Xbyo.

Disclosure of Generative AI Usage

We used large language models to assist with writing clarity and grammar, with experimental, figure and analysis code, and with drafting parts of the appendix. All final wording, technical claims, code, results, figures’ scientific content, and conclusions were reviewed, validated, and produced by the authors.

Ethics statement

The database is built from peer-reviewed publications and does not involve human subjects or personal data. LLM-extracted scientific data can contain confident but incorrect values, and our results show that incorrect values, not omissions, are the dominant error. Users should consult the per-category reliability scores before relying on any field and should not treat model predictions as substitutes for experimental validation or safety assessment.

References

References

  1. [1]Boyd, R., Richerson, P. J. & Henrich, J. The cultural niche: Why social learning is essential for human adaptation. Proceedings of the National Academy of Sciences 108, 10918–10925 (2011).DOI
  2. [2]Dean, L. G., Vale, G. L., Laland, K. N., Flynn, E. & Kendal, R. L. Human cumulative culture: a comparative perspective. Biological Reviews 89, 284–301 (2014).
  3. [3]Mesoudi, A. & Thornton, A. What is cumulative cultural evolution? Proceedings of the Royal Society B 285, 20180712 (2018).
  4. [4]Eisenstein, E. L. The Printing Press as an Agent of Change: Communications and Cultural Transformations in Early-Modern Europe (Cambridge Univ. Press, Cambridge, 1979).
  5. [5]Fyfe, A., Moxham, N., McDougall-Waters, J. & Mørk Røstvik, C. A History of Scientific Journals: Publishing at the Royal Society, 1665–2015 (UCL Press, London, 2022).
  6. [6]Wilkinson, M. D. et al. The FAIR Guiding Principles for scientific data management and stewardship. Scientific Data 3, 160018 (2016).DOI
  7. [7]Lu, C. et al. The AI Scientist: Towards fully automated open-ended scientific discovery. arXiv preprint arXiv:2408.06092 (2024).
  8. [8]Yamada, Y. et al. The AI Scientist-v2: Workshop-level automated scientific discovery via agentic tree search. arXiv preprint arXiv:2504.08066 (2025).arxiv.org/abs/2504.08066
  9. [9]Gottweis, J. et al. Towards an AI co-scientist. arXiv preprint arXiv:2502.18864v1 (2025).
  10. [10]Swanson, K., Wu, W., Bulaong, N. L., Pak, J. E. & Zou, J. The virtual lab of AI agents designs new SARS-CoV-2 nanobodies. Nature 646, 716–723 (2025).DOI
  11. [11]Miao, J., Davis, J. R., Zhang, Y., Pritchard, J. K. & Zou, J. Reimagining research papers as interactive and reliable AI agents. Nature (2026).DOI
  12. [12]Boiko, D. A., MacKnight, R., Kline, B. & Gomes, G. Autonomous chemical research with large language models. Nature 624, 570–578 (2023).DOI
  13. [13]Szymanski, N. J. et al. An autonomous laboratory for the accelerated synthesis of inorganic materials. Nature 624, 86–91 (2023).DOI
  14. [14]Huang, Q., Vora, J., Liang, P. & Leskovec, J. MLAgentBench: Evaluating language agents on machine learning experimentation. In Proceedings of the 41st International Conference on Machine Learning, vol. 235 of Proceedings of Machine Learning Research, 20271–20309 (2024).
  15. [15]Chan, J. S. et al. MLE-bench: Evaluating machine learning agents on machine learning engineering. In International Conference on Learning Representations (2025).
  16. [16]Dagdelen, J. et al. Structured information extraction from scientific text with large language models. Nature Communications 15, 1418 (2024).
  17. [17]Xu, D. et al. Large language models for generative information extraction: a survey. Frontiers of Computer Science 18, 186357 (2024).
  18. [18]Kojima, A., Teshima, K., Shirai, Y. & Miyasaka, T. Organometal halide perovskites as visible-light sensitizers for photovoltaic cells. Journal of the American Chemical Society 131, 6050–6051 (2009).DOI
  19. [19]National Laboratory of the Rockies. Best research-cell efficiency chart (2026). Accessed 24 October 2026.DOI
  20. [20]Jeon, N. J. et al. Compositional engineering of perovskite materials for high-performance solar cells. Nature 517, 476–480 (2015).
  21. [21]Saliba, M. et al. Incorporation of rubidium cations into perovskite solar cells improves photovoltaic performance. Science 354, 206–209 (2016).
  22. [22]Zhou, H. et al. Interface engineering of highly efficient perovskite solar cells. Science 345, 542–546 (2014).DOI
  23. [23]Tan, S. et al. Stability-limiting heterointerfaces of perovskite photovoltaics. Nature 605, 268–273 (2022).DOI
  24. [24]Meng, H. et al. Inhibition of halide oxidation and deprotonation of organic cations with dimethylammonium formate for air-processed p-i-n perovskite solar cells. Nature Energy 9, 536–547 (2024).DOI
  25. [25]Wang, Y. et al. Improved solvent systems for commercially viable perovskite photovoltaic modules. Science 390, 1021–1028 (2025).
  26. [26]Li, N. et al. Optimal perovskite vapor partitioning on textured silicon for high-stability tandem solar cells. Science 390, ead z3698 (2025).
  27. [27]He, M. et al. Amorphous grain boundary engineering for scalable flexible perovskite photovoltaics with improved stability. Nature Energy 11, 262–273 (2026).
  28. [28]Christians, J. A., Manser, J. S. & Kamat, P. V. Best Practices in Perovskite Solar Cell Efficiency Measurements. Avoiding the Error of Making Bad Cells Look Good. The Journal of Physical Chemistry Letters 6, 852–857 (2015).
  29. [29]Khenkin, M. V. et al. Consensus statement for stability assessment and reporting for perovskite photovoltaics based on ISOS procedures. Nature Energy 5, 35–49 (2020).
  30. [30]Jacobsson, T. J. et al. An open-access database and analysis tool for perovskite solar cells based on the FAIR data principles. Nature Energy 7, 107–115 (2022).
  31. [31]Beard, E. J. & Cole, J. M. Perovskite- and dye-sensitized solar-cell device databases auto-generated using ChemDataExtractor. Scientific Data 9, 329 (2022).
  32. [32]Valencia, A. et al. Auto-generating a database on the fabrication details of perovskite solar devices. Scientific Data 12, 270 (2025).
  33. [33]Polak, M. P. & Morgan, D. Extracting accurate materials data from research papers with conversational language models and prompt engineering. Nature Communications 15, 1569 (2024).
  34. [34]Gupta, S., Mahmood, A., Shetty, P., Adeboye, A. & Ramprasad, R. Data extraction from polymer literature using large language models. Communications Materials 5, 269 (2024).
  35. [35]Swain, M. C. & Cole, J. M. ChemDataExtractor: A toolkit for automated extraction of chemical information from the scientific literature. Journal of Chemical Information and Modeling 56, 1894–1904 (2016).
  36. [36]Kim, E. et al. Materials synthesis insights from scientific literature via text extraction and machine learning. Chemistry of Materials 29, 9436–9444 (2017).
  37. [37]Tshitoyan, V. et al. Unsupervised word embeddings capture latent knowledge from materials science literature. Nature 571, 95–98 (2019).
  38. [38]Liu, N. F. et al. Lost in the Middle: How Language Models Use Long Contexts. Transactions of the Association for Computational Linguistics 12, 157–173 (2024).
  39. [39]Zheng, L. et al. Judging LLM-as-a-judge with MT-bench and chatbot arena. In Advances in Neural Information Processing Systems Datasets and Benchmarks Track (2023).
  40. [40]Tan, S. et al. JudgeBench: A benchmark for evaluating LLM-based judges. In International Conference on Learning Representations (2025).
  41. [41]Lee, N., Hong, J. & Thorne, J. Evaluating the consistency of LLM evaluators. arXiv preprint arXiv:2412.00543 (2024). Accepted to COLING 2025.arxiv.org/abs/2412.00543
  42. [42]Majumder, B. P. et al. DiscoveryBench: Towards data-driven discovery with large language models. In International Conference on Learning Representations (2025).
  43. [43]Chen, Z. et al. ScienceAgentBench: Toward rigorous assessment of language agents for data-driven scientific discovery. In International Conference on Learning Representations (2025).
  44. [44]Starace, G. et al. PaperBench: Evaluating AI’s ability to replicate AI research. arXiv preprint arXiv:2504.01848 (2025).arxiv.org/abs/2504.01848
  45. [45]Siegel, Z. S., Kapoor, S., Nadgir, N., Strobel, B. & Narayanan, A. CORE-Bench: Fostering the credibility of published research through a computational reproducibility agent benchmark. arXiv preprint arXiv:2409.11363 (2024).arxiv.org/abs/2409.11363
  46. [46]Wadden, D. et al. Fact or fiction: Verifying scientific claims. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing, 7534–7550 (2020).
  47. [47]Javaji, S. R. et al. Can AI validate science? benchmarking LLMs on claim–evidence reasoning in AI papers. In Proceedings of the 14th International Joint Conference on Natural Language Processing and the 4th Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics, 2355–2379 (2025).
  48. [48]Rashkin, H. et al. Measuring Attribution in Natural Language Generation Models. Computational Linguistics 49, 777–840 (2023).
  49. [49]Gao, T., Yen, H., Yu, J. & Chen, D. Enabling Large Language Models to Generate Text with Citations. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, 6465–6488 (2023).
  50. [50]Sculley, D. et al. Hidden technical debt in machine learning systems. In Advances in Neural Information Processing Systems, vol. 28 (2015).
  51. [51]Klie, J.-C., Castillo, R. E. d. & Gurevych, I. Analyzing Dataset Annotation Quality Management in the Wild. Computational Linguistics 50, 817–866 (2024).
  52. [52]Burschka, J. et al. Sequential deposition as a route to high-performance perovskite-sensitized solar cells. Nature 499, 316–319 (2013).
  53. [53]Sambasivan, N. et al. “Everyone wants to do the model work, not the data work”: Data Cascades in High-Stakes AI. In Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems, 1–15 (2021).
  54. [54]Gebru, T. et al. Datasheets for datasets. Communications of the ACM 64, 86–92 (2021).
  55. [55]Mitchell, M. et al. Model Cards for Model Reporting. In Proceedings of the Conference on Fairness, Accountability, and Transparency, 220–229 (2019).
  56. [56]Miah, M. H., Khandaker, M. U., Rahman, M. B., Nur-E-Alam, M. & Islam, M. A. Band gap tuning of perovskite solar cells for enhancing the efficiency and stability: issues and prospects. RSC Advances 14, 15876–15906 (2024).
  57. [57]Shockley, W. & Queisser, H. J. Detailed balance limit of efficiency of p-n junction solar cells. Journal of Applied Physics 32, 510–519 (1961).
  58. [58]Yang, C. et al. Achievements, challenges, and future prospects for industrialization of perovskite solar cells. Light: Science & Applications 13, 227 (2024).
  59. [59]Zhang, H. & Park, N.-G. Progress and issues in p–i–n type perovskite solar cells. DeCarbon 3, 100025 (2024).
  60. [60]Byranvand, M. M. et al. Recent progress in mixed a-site cation halide perovskite thin-films and nanocrystals for solar cells and light-emitting diodes. Advanced Optical Materials (2022).
  61. [61]Li, X. et al. Mixed perovskites (2d/3d)–based solar cells: a review on crystallization and surface modification for enhanced efficiency and stability. Advanced Composites and Hybrid Materials 6, 111 (2023).
  62. [62]Jiang, Q. et al. Surface passivation of perovskite film for efficient solar cells. Nature Photonics 13, 460–466 (2019).
  63. [63]Niyongabo Rubungo, A., Li, K., Hattrick-Simpers, J. & Bousso Dieng, A. LLM4Mat-Bench: Benchmarking large language models for materials property prediction. Machine Learning: Science and Technology 6, 020501 (2025).
  64. [64]Kapoor, S. & Narayanan, A. Leakage and the reproducibility crisis in machine-learning-based science. Patterns 4, 100804 (2023).
  65. [65]Ke, G. et al. LightGBM: A highly efficient gradient boosting decision tree. In Advances in Neural Information Processing Systems, vol. 30 (2017).
  66. [66]Hollmann, N. et al. Accurate predictions on small data with a tabular foundation model. Nature 637, 319–326 (2025).
  67. [67]Gupta, T., Zaki, M., Krishnan, N. M. A. & Mausam. MatSciBERT: A materials domain language model for text mining and information extraction. npj Computational Materials 8, 102 (2022).
  68. [68]Raffel, C. et al. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of Machine Learning Research 21, 1–67 (2020).
  69. [69]Rubungo, A. N., Arnold, C., Rand, B. P. & Dieng, A. B. LLM-Prop: Predicting physical and electronic properties of crystalline solids from their text descriptions. arXiv preprint arXiv:2310.14029 (2023).arxiv.org/abs/2310.14029
  70. [70]Grattafiori, A. et al. The Llama 3 herd of models. arXiv preprint arXiv:2407.21783 (2024).arxiv.org/abs/2407.21783
  71. [71]Min, H. et al. Efficient, stable solar cells by using inherent bandgap of α-phase formamidinium lead iodide. Science 366, 749–753 (2019).
  72. [72]M. Bran, A. et al. Augmenting large language models with chemistry tools. Nature Machine Intelligence 6, 525–535 (2024).
  73. [73]Panicksssery, A., Bowman, S. R. & Feng, S. LLM evaluators recognize and favor their own generations. arXiv preprint arXiv:2404.13076 (2024).
  74. [74]Bender, E. M. & Friedman, B. Data Statements for Natural Language Processing: Toward Mitigating System Bias and Enabling Better Science. Transactions of the Association for Computational Linguistics 6, 587–604 (2018).
  75. [75]Butler, K. T., Davies, D. W., Cartwright, H., Isayev, O. & Walsh, A. Machine learning for molecular and materials science. Nature 559, 547–555 (2018).
  76. [76]Schmidt, J., Marques, M. R. G., Botti, S. & Marques, M. A. L. Recent advances and applications of machine learning in solid-state materials science. npj Computational Materials 5, 83 (2019).
  77. [77]Kwon, W. et al. Efficient Memory Management for Large Language Model Serving with PagedAttention. In Proceedings of the 29th Symposium on Operating Systems Principles, 611–626 (2023).
  78. [78]Dong, J., Yao, Y. & Yang, Y. MavaLM: In search of a verifiable automated database harness for AI scientists. In NeurIPS 2026 Workshop on AI for Science (2026). Poster.
  79. [79]Loshchilov, I. & Hutter, F. Decoupled weight decay regularization. In International Conference on Learning Representations (2019).
  80. [80]Efron, B. Bootstrap Methods: Another Look at the Jackknife. The Annals of Statistics 7, 1–26 (1979).
  81. [81]Jiang, Q. et al. Enhanced electron extraction using SnO₂ for high-efficiency planar-structure HC(NH₂)₂PbI₃-based perovskite solar cells. Nature Energy 2, 16177 (2017).
  82. [82]Ávila López, A. et al. A Review of Perovskite-Based Solar Cells over the Last Decade: The Evolution of the Hole Transport Layer and the Use of WO₃ as an Electron Transport Layer. Coatings 15, 132 (2025).
  83. [83]Chen, T. & Guestrin, C. XGBoost: A scalable tree boosting system. In Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, 785–794 (2016).
  84. [84]Breiman, L. Random forests. Machine Learning 45, 5–32 (2001).

Paper details

vlnc:W_nfabcgeg
Version
v1 · latest
Published
Fresh — added to the Hub within the last 30 days
Added
License
CC BY 4.0
Usage

views · 0

downloads · 0

A view counts once per browser per day, a download each time the PDF is requested.

  • Computer Science

Not recorded for this paper

View statement
Looking for related papers…
Contents