betaFeedback

WorksComputer Science

The American Medical Directory Panel: Comparing AI-Assisted and LLM Transcription with Manual Data Entry at Scale

Abstract

How closely does machine transcription agree with manual double-entry data? We digitize the 1912 and 1921 editions of the American Medical Directory, a national register of US physicians with roughly 140,000 and 150,000 entries in the two volumes, and compare five machine-extraction workflows with a full-count double-entered manual transcription. Like many historical directories, the AMD is semi-structured rather than tabular, so errors arise from misreading the layout as well as the text. A script that parses commercial OCR text using rules written by coding agents recovers about 97 percent of manually transcribed records with similar precision, more than a multimodal language model workflow (gemini-2.0.flash) that reads the page images and the OCR text. When the two methods fail, they fail on partly overlapping sets of records, so reconciling their outputs raises both recall and precision to about 98 percent. An open-weight LLM pipeline run on a local GPU server falls between the parser and the multimodal workflow in recall, and replacing the commercial OCR model with an open-source alternative lowers its recall by only about 2 percentage points. As the manual transcription's own error rate is unknown, these rates measure agreement rather than accuracy. Direct processing charges for the commercial OCR and multimodal workflow were about 0.03 euro per page, compared with 14.70 euro per page for manual transcription.

JEL: C81, I11, N31, N32

Introduction

How closely does machine transcription agree with manual double-entry data? The time required to key and clean historical records limits both dataset size and the questions economic historians can investigate [17]. Advances in Optical Character Recognition (OCR) and large language models (LLMs) offer the potential to reduce the cost of digitizing historical sources, making large-scale transcription increasingly feasible [13, 15]. However, the benefits of automation depend on the accuracy of the resulting data. While automated transcription can reduce data-entry costs, assessing the quality of transcription still requires manual human review and raises new questions about validation. In particular, small character-level error rates can translate into substantial disagreement across records containing many fields [11]. Evaluating transcription accuracy therefore becomes critical.

Such evaluation raises a practical problem. Researchers may benchmark machine-generated data against a human-created reference dataset treated as ground truth, yet the reference dataset may itself contain transcription errors. Measures of agreement with a reference dataset therefore conflate machine transcription errors with errors in the benchmark, so agreement with a manual transcription does not necessarily establish accuracy [14]. This makes it difficult to determine which extraction strategy is most accurate. As AI-assisted digitization becomes more common, distinguishing agreement from accuracy becomes increasingly important for researchers releasing and using machine-generated historical datasets.

We study this question using the 1912 and 1921 editions of the American Medical Directory (AMD), a nationwide register of physicians in the United States published by the American Medical Association. The AMD records physician names, locations, medical schools, graduation years, licensing year, and professional credentials, making it one of the largest and most comprehensive individual-level sources on the early twentieth-century American medical profession. Despite its research potential, the directory remains difficult to use at scale because its information is embedded within compact, semi-structured entries rather than structured tables.

The AMD belongs to a broad class of historical sources that is common but difficult to digitize: directories and registers that list individuals under place or category headings in compact, abbreviated entries, such as city and trade directories, professional registers, and membership rolls. Unlike tables, such sources have no fixed rows or columns. Entries in the AMD are arranged beneath town headings and contain numerous abbreviations, irregular formatting conventions, and nested record components. Errors can therefore arise not only from misreading individual fields but also from failing to identify record boundaries or town headings. Because a town heading determines the location assigned to every physician listed beneath it, a single error can affect many subsequent records. The AMD therefore allows us to distinguish errors in reading individual fields from errors in reconstructing the structure of the directory itself, a distinction that does not arise in the tabular sources studied in recent work [6].

To examine both kinds of error, we compare five extraction workflows against a large double-entered manual transcription of both editions, using the same pages, record-linkage rule, and measures for each. Precision is the share of a workflow’s extracted records that link to a record in the manual transcription, and recall is the share of transcribed records that the workflow recovers; we compute both for whole records and for individual fields. Three workflows start from the same commercial OCR: a multimodal large language model that reads page images and OCR text, a deterministic parser of OCR text whose rules were written with coding agents, and a multi-stage open-weight LLM pipeline that reads OCR text alone. A fourth applies the same open-weight pipeline to OCR from an open-source model, which changes the OCR step (the engine and its column segmentation) and nothing downstream. The fifth, the combined table, reconciles the multimodal LLM and parser outputs field by field and forms the basis of the public release. Our data release includes 138,800 and 148,040 rows from the two editions, including cross-reference lines, covering 48 states and the District of Columbia. Because the case for automation rests on cost, we also compare the direct processing charges of the commercial OCR and MLLM workflow with those of the manual transcription.

Three results stand out. First, among the four single workflows, the deterministic parser has the highest recall in both editions. Its record-level precision and recall are 96.72 and 97.45 percent in 1921 and 97.54 and 97.36 percent in 1912, higher on both measures than the multimodal LLM in both editions (94.11 and 94.13 percent in 1921; 94.89 and 94.42 percent in 1912) and higher on recall than the open-weight LLM pipeline. Replacing the commercial OCR with open-source OCR in that pipeline lowers recall by 2.4 percentage points in 1921 and 2.3 in 1912. Second, when the methods fail, they fail on partly overlapping sets of records. The combined table recovers more reference records than either input alone, with precision and recall of 98.04 and 98.28 percent in 1921 and 97.72 and 97.80 percent in 1912. Among cells where only the parser agrees with the transcription, most for birth and graduation year are entries the MLLM workflow did not link, while most for license year and medical school are different values on linked records. The widest gaps between methods are in county in both editions and in city in 1921. Third, processing charges for the commercial OCR and MLLM workflow are far below the price of manual transcription: an estimated €0.03 per page, against €14.70 per page for double-entered transcription. This comparison excludes engineering, review, and model-development time. Even so, digitization projects previously considered prohibitively expensive may now be feasible.

These figures should be read in light of the problem raised above: agreement with a reference dataset is not the same as accuracy. Our reference is a double-entered transcription whose own error rate is unknown, so the reported measures quantify agreement rather than scan-verified accuracy. Entries missed by both the machine and the transcribers are unobserved, and three of the workflows share the same OCR, so agreement among them is weaker evidence than agreement among independent readings would be. We therefore view the paper as a methodological benchmark of alternative extraction strategies and a description of the released data. A subsequent study will adjudicate disagreements directly against the page images and estimate accuracy using an independently verified gold-standard sample.

This paper contributes to a growing literature on large-scale historical transcription. Existing workflows typically combine commercial OCR with pre- and post-processing to transform scanned sources into structured datasets (Correia and Luck 2023, see Beach and Hanlon 2023 for newspapers). Other projects have developed specialized layout-detection and OCR pipelines for large newspaper collections [7], tabular documents [12], and historical administrative directories [16]; Dell [13] surveys these approaches. More recently, large language models have been used to extract structured records directly from page images. Bäcker-Peral et al. [6] find high agreement with a human-transcribed gold standard for historical county tables at substantially lower cost, with regression estimates of the same sign and significance, and Ferrara [15] outlines a workflow that holds hand-coded validation data out of prompt development and model selection. We add to this literature in three ways. First, we compare multimodal, text-based LLM, and rule-based extraction on a common source and reference dataset, and we vary the OCR engine while holding the downstream pipeline fixed. Second, we study a directory-style source whose difficulty lies in its structure as much as in its characters, and we report which records and fields the methods fail to match. Third, we set out what agreement measures can and cannot establish when both machine-generated and human-generated transcriptions contain errors.

We also add to work that digitizes the AMD itself. Previous studies rely on hand-keyed samples for particular states and editions [21], vendor double-entry transcriptions of North Carolina [18] and of the 1906 edition for Southern and adjacent states (Chrisinger 2024), and OCR extractions of single editions checked against printed physician counts [25] or published state totals [1]. We provide national transcriptions of two editions validated against a double-entered reference. Appendix A summarizes earlier AMD datasets and their reported validation procedures.

Section 2 describes the directory. Section 3 describes the page images, the extraction workflows, and the public release. Section 4 describes the validation samples, the record-linkage rule, and the agreement measures, and Section 5 reports results for both editions. Section 6 reports robustness checks, and Section 7 discusses limitations and concludes with the planned adjudication against the page images. Appendix B documents the comparison protocol.

Background and Data: the American Medical Directory

The American Medical Directory (AMD) is a register of legally qualified and practicing physicians in the United States and Canada published every two to three years by the American Medical Association (AMA) starting in 1906 [2] (American Medical Association 1921). The AMA compiled the AMD volumes from an index card collection that was updated daily using reports from licensing boards and lists of newly graduated physicians [2].3

While the completeness of the AMD throughout its publication run remains an open question, the earliest editions may have omitted some physicians.4 Among sources on the medical profession during the early twentieth century, the directory uniquely captures individual physicians and their characteristics with nationwide coverage, and thus remains an invaluable resource for studying medical education and licensing, physician labor markets, and the professionalization of medical care. Appendix A provides an overview of prior hand-keyed transcriptions of AMD volumes and samples, and how these have been used in previous research.

Figure 1 shows a sample page of physician entries from the 1921 AMD volume, on the left, and closeups of two individual physician entries and one town heading with explanations of each field, on the right (American Medical Association 1921). The physician directory represents one section in a vast volume that includes listings of medical officers, medical schools, medical libraries, and hospitals. Each volume covers physicians in every state in the continental United States and the District of Columbia, as well as the US territories and Canada. The directory is organized hierarchically by state and town; within each state, towns are listed alphabetically. Each location heading reports the town name, its population, and county, and the physicians associated with that location are listed beneath it in alphabetical order. In some cases, a physician is listed in full under one town and appears elsewhere as a cross-reference.

Figure 1. Notes. Page 566 of the 1921 edition (Kansas), photographed from a library copy; the photograph is straightened, evenly lit and tinted, and each enlargement is straightened separately. The green box marks a town heading, which gives the town, its population and its county; every physician listed beneath it is assigned to that town. The orange box, marked with an asterisk, is the running head, which names the first town on the page and abbreviates the state (KAN., Kansas); town headings and entries do not repeat the state. The blue boxes mark two entries, enlarged on the right. Example 1, under Chanute, shows a home address, an office address and office hours; example 2, under Chetopa, shows the name, sect, birth year, school, graduation year, license year, Fellowship mark and specialty code. Labels follow the 1921 key (Appendix § 10) and the list of medical colleges printed in the same volume. Other pages of the 1921 volume print a second layout, with the birth year and license year in parentheses and the Fellowship mark after the birth year. Source: American Medical Association (1921).

The compact physician entries report medical education, credentials, professional affiliations, and details about the physician’s practice. A typical record follows a fixed order and format: the physician’s name, sect letter, two-digit birth year, medical school code and two-digit graduation year, two-digit license year, membership symbol, specialty code, home address, and office address with office hours. Depending on the edition and page layout, the birth year and license year may be printed in parentheses. The exact conventions and entry order vary slightly across and within editions and do not always match the printed edition-specific key included in the directory. The abbreviations that appear in the AMD physician entries are explained in detail for the 1921 edition in Appendix C.

This project draws on physician entries from every state in the continental United States and the District of Columbia in the 1912 and 1921 editions of the American Medical Directory ([2] (American Medical Association 1921)).5 In this data note, we focus on a physician’s name and location, the key dates (birth, graduation, and license year), and the medical school codes. Many of the remaining symbols require separate processing that lies outside the scope of this study.

Digitizing the Directory: OCR and Structured Extraction Pipelines

This section sets out the challenges with structured output extraction from the AMD, describes the page images used as source inputs and the derived OCR outputs, explains the extraction methods, and summarizes the structure of the public data release.

The semi-structured text of the physician entries and their organizational structure make the AMD difficult to digitize. As the town heading determines the location assigned to every physician listed beneath it, errors in identifying a heading can affect many subsequent records even when the individual entries themselves are read correctly. However, the relevant town may appear in the page header, in a different column on the same page, or on a separate page altogether. Thus, the assignment of a physician entry to the correct location requires judgment over what constitutes an entry or a town header, in addition to preserving context across images or image segments. These features make the AMD fundamentally different from tabular sources extracted at scale by other recent studies ([6]). Successful extraction therefore requires not only accurate character recognition but also correct identification of towns and record boundaries. For this reason, performance depends on recovering the structure of the document as well as the contents of individual entries.

Page images and OCR

Figure 2 provides an overview of the image pre-processing, OCR reading, structured output extraction, and validation workflows compared in this paper. Given that existing digitized versions of the AMD volumes available in the public domain consist of low-resolution color scans or poor-quality black and white scans, we obtained high-resolution digital images taken from physical copies of the AMD volumes as the source images for our extraction pipelines.6 The pre-processing

Figure 2. From scans to the release tables. 1 · OCR: image → text. Scan → pages: Split; prepare JPGs. Azure OCR: Cloud API (Azure), Jan 2026 · whole page. Column cut: Rule model: 3 columns. Chandra OCR: Open model, local GPU. 2 · Extraction: text → fields. Biography pass: Cloud LLM reads the image. Location pass: Cloud LLM reads OCR text. Match passes: Join names (rounds 1–2); carry place (round 3). Fill from OCR text: Names the image pass did not read (round 4). Rule-based parser: Rules by Claude Code. Open Model Pipelines: DeepSeek-V4-Flash on a local GPU, two calls. Names call: Towns + names. Cut per name: One snippet each. Fields call: Key fields per name. Azure text → Open LLM. Chandra text → Open OCR-LLM. 3 · Combine and release. Combine: Align; keep both values, choose one by rule. Clean and release: Edition datasets + flags. MLLM workflow. AI-assisted parser. Combined. Comparison with the manual transcription: same pages, record-linkage rule and measures for every dataset. Primary Data Release: MLLM workflow, AI-assisted parser and Combined. Open Model Pipelines: Open LLM and Open OCR-LLM. Solid arrows: data flow. Dashed arrow: pages selected for the image pass. Tags: the five datasets, named by workflow.

step consisted of cropping to ensure each image contained only the page itself, as well as a DPI adjustment.7 These images represent a common input across all our extraction workflows.

Five machine extraction workflows

We evaluate five machine extraction workflows that each produce a structured dataset derived from page images of the directory. Table D.2 (Appendix D) summarizes their inputs and outputs. All extraction methods begin with the same underlying page images. Four of the workflows rely on OCR output generated by Azure Document Intelligence, a proprietary cloud API service that extracts both text and layout information (bounding box coordinates). We processed every page image with physician entries in the 1912 and 1921 editions during January 2026.

Notes: Solid arrows indicate data flow and the dashed arrow indicates page selection for image-based extraction. The manual transcription is used for validation and for selecting release-cleaning procedures, but it does not directly supply reconciled field values. Cross-edition linkage and record collapse occur downstream of the workflow examined in this paper. Source: project workflow documentation.

Multimodal language model workflow

The multimodal language model (MLLM) workflow prompts the Google Gemini API and uses the gemini-2.0-flash model to extract information from the directory in two stages.8 The first stage reads the OCR text generated by Azure Document Intelligence to identify the town and county headings as well as the physician names listed beneath them. The second stage reads the page image and extracts the biographical fields defined by the directory key. A carry-forward mechanism allows location information to be filled in across page and column boundaries (Appendix § 12 describes the passes and matching rounds in detail). A deterministic linking algorithm then combines the list of towns and names from the first stage with the biographical information from the second into a single record-level dataset. The matching procedure blocks on page and name, while subsequent matching rounds allow for fuzzy matches on name and incorporate location information. A final stage uses the OCR text to fill missing biographical fields when the image-reading stage fails to extract them. This is the only stage in which OCR text contributes biographical information directly.

AI-assisted parser and combined workflow

As an alternative to prompting an MLLM via API calls, we prompted a coding agent to develop a deterministic script that parses the Azure OCR text. The AI-assisted parser follows a set of rule-based instructions to identify document sections, locate town headings, separate individual physician entries, and extract fields according to the conventions defined in the directory key. For every extracted field, it retains both the original OCR text and the cleaned value.

The combined workflow aligns the outputs of the MLLM and the AI-assisted parser at the record level to produce a combined data table. For each field, it retains both extracted values, records the rule used to determine the release value, and preserves the alternative value for reference. Reconciliation uses only the two readings and fixed conventions (the medical-school code list, year-typo tables and county rules), not the manual transcription.

Open-source GPU-based OCR and LLM workflows

As one of our objectives is to evaluate the quality of data produced by open-source models in comparison to frontier models, our final two workflows use open weights and run on a GPU server. We start with the same Azure OCR text as the MLLM workflow, and replace the API calls to the gemini-2.0-flash model with a local deployment of DeepSeek-V4-Flash on a GPU server with temperature 0.01 to minimize variation across runs. The same prompts are used for both editions. Unlike the MLLM, the multi-stage open LLM extraction pipeline operates only on OCR text rather than page images. The first stage identifies town headings and physician names on each page. Outputs are checked automatically for obvious failures, such as implausibly short name lists, and the model call is repeated when necessary. The OCR text is then divided into physician-specific snippets, which form the input for the second stage. A second model call extracts the biographical fields defined by the directory key.

Finally, we assess an end-to-end open-source pipeline. The open OCR-LLM workflow uses OCR text generated by the open-source Chandra OCR model run locally on the GPU server, rather than the Azure OCR text. Pages are segmented into their printed columns using a table parsing algorithm prior to processing by Chandra. These open LLM workflows produce two additional datasets that are evaluated against the same validation sample.

The public data release

The public data release through Zenodo (version 1.0.0, DOI 10.5281/zenodo.23945788) contains 138,800 rows for the 1912 edition and 148,040 rows for the 1921 edition, covering physicians in every state in the continental United States and the District of Columbia. Table 1 summarizes the public release tables. For each edition, the release includes a primary table, the combined workflow’s output, which also holds the cross-reference lines. Each observation is identified by edition and record identifier. The outputs of the open LLM and open OCR-LLM workflows are provided as separate tables (see § 5 and Appendix § 9 for details). They are linked to the primary table through edition and record identifiers and include match-status information, but are distributed separately rather than merged into it. The agreement figures for the combined table in § 5 describe the reconciled output with its county replaced by the cleaned county; Table G.3 reports the release table’s columns as read and after cleaning. The release holds two cross-sections; it contains no identifier that links a physician across editions.

19121921
Listed physicians137,303145,608
Cross-reference lines1,4972,432
States and DC with coded rows4949
Physicians by Census region (%)
Northeast39,150 (28.5)41,787 (28.7)
Midwest47,238 (34.4)48,185 (33.1)
South39,195 (28.5)41,292 (28.4)
West11,657 (8.5)14,284 (9.8)
State not resolved to a code63 (0.0)60 (0.0)
Share of physicians with a value (%)
Birth year73.889.6
Medical school96.697.6
Graduation year96.697.6
License year78.085.2
City100.0100.0
County99.899.5

Table 1. Public Data Release: Descriptive Statistics. Listed rows are printed entries, not distinct people; cross-reference lines are counted on their own row. Census regions are the Bureau's four regions over the 48 states and the District of Columbia; a row whose state is not resolved to a code is retained and counted on its own line. A field has a value when its cleaned column is non-missing; county and medical school are counted from the values as read. Source: descriptive statistics calculated from the public data release.

The first public release covers the state sections of the two volumes, corresponding to the material included in the manual transcription. The release tables include all fields evaluated in § 5, with as-read and cleaned values for the years, medical school and county, and the workflow that found each entry. Additional extracted fields, including specialty, addresses and office hours, military information, printed marks (membership, fellowship, sect, and col.), and town populations, will remain unpublished until their validation is complete. These fields are documented in the codebook.

The release package also contains documentation, a codebook, a manifest identifying the included editions, checksums, citation information, and a CC BY 4.0 license. Both editions are in the public domain in the United States, which provides the basis for releasing the data tables. Appendix § 10 lists the printed key elements and the release field each one feeds. The Zenodo record also includes a preferred citation.

Validation

Double-entry manual transcription

We validate our machine-extraction workflows against a hand-keyed double-entry full-count transcription of the 1912 and 1921 AMD volumes. Each record was independently transcribed by two data entry workers and reconciled by a quality assurance checker. The manual transcription was completed during 2022, before LLM-based transcription tools were available.9 The transcription includes physicians located in 48 US states and the District of Columbia, while excluding the addendum and address-unknown lists that appear at the back of the volumes. The resulting spreadsheets contain one row per printed entry and one column for each recorded field. We treat the compiled transcription as our reference dataset for evaluation.

Reference population for validation

Our reference population is the set of physician entries on printed volume pages for which we have a complete transcription. Table D.1 (Appendix § 11) summarizes the page coverage of the reference population in the 1912 and 1921 AMD volumes. For 1921, the manually transcribed reference dataset covers 1,233 pages in the printed directory and 145,064 cleaned rows. One complication with the manual transcription is that the data entry workers read from lower-quality scans of the 1912 and 1921 AMD volumes, rather than the high-resolution digital images. Consequently, some scanned pages or column segments did not contain any legible content; transcribers flagged such pages or columns in the output. After excluding pages with incomplete reference data, around 10 to 16 percent of the total, we evaluate all machine extraction methods on a sample consisting of 1,036 pages and 124,167 cleaned rows (§ 4.4). For 1912, the manual transcription covers 1,006 pages while the reference population consists of 903 pages containing 123,726 cleaned rows after excluding incomplete pages.

Record linkage to the validation sample

In the absence of a unique record identifier, we apply a record linkage procedure to merge the reference population from the manual transcription to the output of each machine-extraction workflow. The goal is to identify which extracted records correspond to which transcribed records. The resulting links form the basis for the measures of record-level agreement.

Prior to linking, we standardize the given names, medical school codes, two-digit years, and county names in the manual transcription and the extraction outputs (See Appendix § 9 for details). We match records using combinations of page number and physician name whenever these identifiers are unique in both datasets. Remaining ambiguous cases are resolved using state and city information, but only when these fields uniquely identify a match in both datasets. School codes and years are never used to establish a link. Records in the manual transcription that cannot be linked, and ties that cannot be resolved, remain unmatched and continue to count toward precision and recall denominators. Linkage is performed on the full set of extracted records before the analysis frame is applied. A field value can agree only on a linked record; values on unlinked records stay in the field denominators and count as disagreements.

Precision and recall measures

The central question in this paper is how closely each machine extraction method agrees with the reference population from the manual transcription. We validate the extraction methods by evaluating performance at both the record and field level. For a method mm, let OmO_m denote the number of extracted records, NN the number of reference records (on pages with complete transcription), and LmL_m the number of linked records produced by the linkage procedure described in § 4.3. Record-level precision is defined as Precision⁡m=Lm/Om\operatorname{Precision}_m = L_m/O_m, that is, the share of extracted records that can be linked to a reference record. Record recall is defined as Recall⁡m=Lm/N\operatorname{Recall}_m = L_m/N, or the share of reference records recovered by the extraction method. A reference record is recovered when it receives a one-to-one link in the first or second pass: a page-and-name key that is unique on both sides, or, within ambiguous groups, a unique state–city pair (§ 4.3). Field-level measures are calculated similarly. Precision divides the number of agreeing linked values by the total number of scorable output values produced by a method, while recall divides the number of agreeing linked values by the number of scorable values in the reference dataset. Cells containing multiple years or multiple school codes contribute each individual value to the comparison. Unmatched records count as disagreements, while cases where both sides are missing are excluded.

These measures quantify agreement with the reference population used for validation rather than accuracy relative to the printed directory. The distinction is important because while double-entry transcription reduces keying errors, the reference data may itself contain errors. To illustrate, consider a set of aligned records. Suppose the extraction method is incorrect with probability eme_m and the reference transcription is incorrect with probability ere_r. If errors are independent and never produce the same incorrect value, the probability of agreement is Am=(1−em)(1−er)A_m = (1 - e_m)(1 - e_r). Even in this highly simplified case, observed agreement reflects both error rates. Recovering the extraction error rate requires information about the accuracy of the reference transcription. Shared errors may inflate agreement, while correlated errors invalidate the independence assumption altogether. High agreement therefore does not necessarily imply high accuracy. This distinction between agreement and accuracy is central to the evaluation of machine-generated measurements ([14]). Throughout the paper, the reported precision and recall statistics should therefore be interpreted as measures of agreement with the validation sample – the share of transcribed entries recovered by an extraction method rather than the share of all printed physicians recovered from the original volumes. Estimating scan-verified accuracy would require an independent transcription of the same pages from which to adjudicate against the original page image and estimate the error rate of the manual transcription, which is planned for future work.

Results

This section reports the main results for the rate of agreement between the machine extraction methods and the cleaned reference population used for validation. The measures are reported for the set of pages in each edition with a complete manual transcription (903 pages in 1912 and 1,036 pages in 1921). They quantify agreement with the reference transcription rather than scan-verified accuracy. All headline comparisons are based on the common precision and recall measures defined in § 4.4. Record-level agreement is reported in Table 2, while field-level agreement is summarized in Figure 3.

MethodOutput rowsLinksPrec. (%)Recall (%)Output rowsLinksPrec. (%)Recall (%)
1912: 903 pages, 123,726 reference rows1921: 1,036 pages, 124,167 reference rows
Primary table and its component workflows
MLLM123,108116,81794.8994.42124,195116,88394.1194.13
Parser123,496120,46297.5497.36125,101120,99896.7297.45
Combined123,829121,01097.7297.80124,477122,03298.0498.28
Open model pipelines
Open LLM121,370118,61397.7395.87122,915118,07096.0695.09
Open OCR-LLM119,991115,78096.4993.58121,295115,13694.9292.73

Table 2. Record agreement by method, 1912 and 1921

Figure 3. Field agreement by method, 1912 and 1921. Notes: Agreement with the cleaned validation sample; scan-verified accuracy not assessed. Same frames, links and scorer as Table 2. Precision divides agreeing linked values by all scorable output values; recall divides them by all scorable reference values. Unmatched values count as failures; missing/missing is excluded. Both editions and measures use the same 0–100% color scale, with percentages rounded to two decimals. Name and, for second-pass links, state/city agreement are linkage-conditioned. Years are two-digit tokens, school codes syntactically normalized, and places normalized for formatting. The combined table’s county uses the checked county-cleaning rule on fixed links; its other fields use the reconciled reading. Table G.1 retains the exact displayed percentages; Table G.2 gives the integer counts for both editions.

Record-level agreement

Table 2 provides the main record-level comparison across extraction methods. Precision measures the share of output records that can be linked to a transcribed record, while recall measures the share of transcribed records recovered by a method. We focus on comparisons that show how agreement changes when individual components of the extraction process are altered.

This table reports record-level precision and recall for each of the five machine extraction workflows and each AMD volume (1912 and 1921) compared to the manual transcription. The reference population for validation is the set of pages with complete manual transcription coverage. Measures reflect agreement with the cleaned validation sample. Recall uses the cleaned reference-row count, while precision uses each method’s output-row count. All workflows count printed cross-reference lines as records, in both denominators and with the same linkage key.

Primary table and its component workflows: multimodal language model (MLLM), gemini-2.0-flash on the page image and the Azure OCR text; AI-assisted parser, a deterministic parser on the Azure OCR text; Combined, the parser and MLLM outputs merged field by field.

Open model pipelines: Open LLM, a multi-stage LLM pipeline (DeepSeek-V4-Flash) on the Azure OCR text; Open OCR-LLM, the same pipeline on OCR text from an open-source model (Chandra).

Overall performance: Three findings stand out. First, the AI-assisted parser consistently outperforms the MLLM workflow. Second, combining the MLLM and parser outputs produces the highest overall recall. Third, agreement is broadly similar across the 1912 and 1921 editions.

AI-assisted parser versus MLLM workflow: The clearest contrast is between the deterministic AI-assisted parser and the MLLM workflow. In both editions, the parser links more reference records and does so with higher precision. For 1921, the parser achieves precision and recall of 96.72/97.45%, compared with 94.11/94.13% for the MLLM workflow. The same pattern appears in 1912, where the parser reaches 97.54/97.36%, compared with 94.89/94.42%. The precision advantage is between two and three percentage points in both editions, while recall is 3.31 percentage points higher in 1921 and roughly three percentage points higher in 1912. For this source, the deterministic parser achieves both higher precision and higher recall than the multimodal extraction workflow.

Image-based and text-based extraction within the MLLM workflow: The MLLM workflow combines two stages implemented with gemini-2.0-flash. The location stage operates on OCR text, while the biography stage operates on the page image itself. Table D.4 shows that the two stages achieve remarkably similar linkage rates. In 1921, the text-based stage links 91.64% of the transcribed rows at 95.44% precision, while the image-based stage links 92.68% at 95.64% precision. The differences are less than half a percentage point in precision and about one percentage point in recall. These figures do not provide a pure comparison between text and image inputs. The two stages perform different tasks: the OCR-based stage identifies town headings and physician names, while the image-based stage extracts biographical information.

Combining extraction methods: The combined table achieves the highest recall in both editions. In 1921 it reaches 98.04/98.28%, compared with 96.72/97.45% for the parser. In 1912 it reaches 97.72/97.80%, compared with 97.54/97.36%. The gains are modest: recall rises by 0.83 percentage points in 1921 and 0.44 in 1912, and precision by 1.32 and 0.18. The two methods therefore fail on partly overlapping sets of records, although most of the reference records that the parser does not link remain unlinked in the combined table. The largest field-level change is in county (§ 5.2), where most of the gain comes from a county-cleaning rule that we selected by comparison with the transcription, not from reconciliation.

Open LLM versus AI-assisted parser: The comparison between the open LLM workflow and the AI-assisted parser contrasts two OCR-based extraction workflows: one based on a multi-stage LLM pipeline and one based on deterministic parsing rules. In 1921, the open LLM workflow achieves precision 0.66 percentage points below the parser and recall 2.36 percentage points lower. In 1912, precision is about 0.19 percentage points higher and recall is about one and a half percentage points below the parser (95.87% compared with 97.36%). This comparison should be interpreted descriptively rather than causally. The two workflows differ not only in their extraction models but also in prompting strategy and document handling.

Changing the OCR source: The open OCR-LLM workflow applies the same code, model, and prompts as the open LLM workflow, but its OCR comes from an open-source model run on pages that are first segmented into columns. The contrast therefore changes the OCR step, the engine and the column segmentation together, and nothing downstream. Recall falls by 2.4 percentage points in 1921 and 2.3 in 1912, and precision by 1.1 and 1.2 (Table 2).

Comparisons across editions: Precision and recall are broadly similar across the 1912 and 1921 editions, generally within about a point of the other edition for the released workflows. The largest exception is the open LLM workflow, whose precision is about 1.7 percentage points higher in 1912 (97.73%) than in 1921 (96.06%). The comparison does not allow us to determine whether this difference arises from characteristics of the page images themselves or from edition-specific processing decisions. The editions differ in their printed keys and, for the MLLM workflow, in the prompts. The open model pipelines use the same prompts for both editions, so their difference across editions is not a prompt effect.

Field-level agreement

Record-level agreement masks important differences across fields. Figure 3 therefore examines agreement separately for names, locations, medical school codes, and other biographical characteristics. Exact percentages are reported in Table G.1 (Appendix § 14).

The broad pattern mirrors the record-level results. Across most biographical fields, the AI-assisted parser and the combined table achieve the highest levels of agreement with the reference transcription. In 1921, agreement for these methods ranges between 95 to 98 percent across the evaluated biographical fields. The corresponding range is 90 to 96 percent for the MLLM workflow and 92 to 96 percent for the open LLM workflow. For the open LLM workflow, medical school is the weakest of the four biographical fields, with recall of 92.49%. The same pattern appears in

1912, although field-level agreement varies somewhat more across fields. Medical school remains one of the most challenging fields for all methods and is again the weakest of the four biographical fields for the open LLM workflow (91.26%).

The largest differences across methods occur in place information. City agreement reaches 95 to 97 percent for the AI-assisted parser and the combined table in 1921, compared with 86 to 89 percent for the MLLM workflow. For 1912, the corresponding ranges are 96 to 97 and 90 to 93 percent. County shows the largest gain for the combined table, but most of it comes from county cleaning. In 1921, county precision and recall are 95.14 / 89.78 for the parser and 96.65 / 96.97 for the combined table. In 1912, the corresponding values are 94.88 / 86.20 and 96.71 / 96.69. The combined table’s county is scored after the county-cleaning rule. As read, its 1921 figures are 90.90 / 90.96 (Table G.3). Reconciliation alone therefore raises county recall over the parser by about one point and lowers precision, and the cleaning rule accounts for the rest.

Sources of disagreement

Understanding why methods disagree is as important as measuring how often they disagree. In many cases, disagreements arise not because a field value differs, but because a record is not linked: no record on the other side has the same page and name key, or the match is ambiguous. We currently have the most detailed diagnostic information for the 1921 edition. For the combined table, recall losses fall into three main groups: 1,569 transcribed records whose page-and-name key has no corresponding extracted record, 107 records whose key could not be resolved during second-pass matching, and 459 records lacking a usable name or page identifier. Precision losses arise from 2,327 machine-generated records with no corresponding transcription record and 118 records that remain ambiguous after matching. The available diagnostics cannot determine whether these discrepancies originate in the extraction process or in the reference transcription itself. Resolving these cases requires direct inspection of the page images.

Table G.4 (Appendix § 14) counts the disagreements at the field level, and Table D.5 (Appendix § 11) reports record agreement separately for pages where the transcribers flagged many entries as uncertain, and recall separately for the entries they flagged. Among observations where only one of the two component methods behind the combined table agrees with the reference transcription, the parser agrees more frequently for all four evaluated fields. For birth year and graduation year, most of this advantage reflects missing records in the MLLM output. Missing entries account for 1,798 of 1,893 disagreements in birth year and 1,958 of 3,031 disagreements in graduation year. For license year and medical school, missing records explain a smaller share of disagreements. They account for only 1,594 of 5,676 license-year discrepancies and 1,955 of 7,106 school-code discrepancies. In the remaining cases, both outputs contain a record, but one workflow extracts a different field value from the reference transcription. In Table D.5, record agreement is also lower, in all five workflows, on pages where the transcribers flagged at least five percent of entries as uncertain.

Cost

The commercial OCR and MLLM workflow cost an estimated €0.03 per page in direct processing charges (Gemini plus Azure OCR), roughly 440 times less than manual double-entry transcription, which cost €14.70 per page. These figures cover processing charges only and leave out engineering, review, model development, coding-agent assistance and the local computing used to run the open model pipelines, which also serves other project work. The machine outputs and the manual transcription do not cover exactly the same pages and fields, so this is not an end-to-end productivity comparison (Appendix § 15).

Robustness

The ordering of the AI-assisted parser and the MLLM workflow is robust to the choice of reference population, links, and page restrictions. Using all manually transcribed pages, including those the transcribers flagged as having incomplete coverage, moves recall by about 0.3 points or less in both editions and 1912 precision by at most 0.16 points. It lowers 1921 precision by about 1.6 points for every workflow, because machine rows for entries the transcribers skipped count as errors (Appendix § 16). Record agreement is lower on pages where the transcribers flagged at least five percent of entries as uncertain, in all five workflows (Table D.5, Appendix § 11). Using first-pass links only shifts 1921 record agreement by about 0.2 points or less (Appendix § 11). Dropping the pages used in parser development changes precision and recall by at most 0.23 percentage points for every workflow in both editions, and the ordering of workflows by recall stays the same (Appendix § 16).

Conclusion

How closely does machine transcription agree with double-entered manual transcription at full-count scale? This paper compares five machine extraction workflows against a double-entered full-count transcription of the 1912 and 1921 volumes of the American Medical Directory, holding roughly 140,000 and 150,000 entries of physician biographies, respectively. Machine extraction agrees closely with the double-entered manual transcription.

In 1921 the AI-assisted parser recovers 97.45% of the manually transcribed entries and the MLLM workflow 94.13%; in 1912 the figures are 97.36% and 94.42%. Precision is similar, at 96.72% and 94.11% in 1921 and 97.54% and 94.89% in 1912. The open LLM workflow falls between the MLLM workflow and the parser in recall. The open LLM workflow reaches 95.09% recall in 1921 and 95.87% in 1912, and replacing the commercial OCR with open-source OCR in the open OCR-LLM workflow costs 2.4 percentage points of recall in 1921 and 2.3 in 1912. The practical lesson is to combine workflows. The MLLM workflow and the AI-assisted parser fail on partly overlapping sets of records, and the combined table reaches 98.28% recall in 1921 and 97.80% in 1912, above any single workflow. Direct processing charges for the commercial OCR and MLLM workflow are an estimated €0.03 per page, against €14.70 per page for manual transcription, roughly 440 times less before engineering, review and hardware.

As we have emphasized, the reported precision and recall measures quantify agreement with the reference population, and do not measure scan-verified accuracy. Entries missed by both the extraction methods and the manual transcription are unobserved, so recall is defined relative to the reference transcription rather than to the printed directories themselves. Several questions therefore remain beyond the scope of the present comparison, including the accuracy of the reference transcription itself. The extraction methods are also not fully independent. The MLLM workflow, the AI-assisted parser, the combined table and the open LLM workflow all rely on the same Azure OCR text of the underlying page images. Agreement among these methods therefore provides less evidence than would agreement among fully independent readings. The disagreements examined in § 5.3 provide useful diagnostic information, but they do not establish which source is correct. Resolving such cases requires adjudication against the original page images.

Several extensions are planned. First, we will construct a scan-verified evaluation sample based on direct adjudication against the original page images, allowing accuracy to be assessed rather than inferred from agreement. Second, we will classify disagreements by extraction stage and apply the construct-versus-observation framework of Dell and Rambachan [14] to selected variables. Third, we will evaluate additional extraction workflows designed to provide a more independent comparison.

Prior transcriptions of the AMD

Because successive editions repeat a recurring core of fields for the licensed profession, the AMD supports three kinds of study that few other historical sources allow: the supply and location of physicians in a place and its change over time [21, 22]; the training of physicians, through the medical school code and graduation year; and the careers of groups the directory marks, Black physicians through the col. label (Chrisinger 2024) and members and specialists through the membership marks and specialty codes [1].

Table A.1 lists prior AMD transcriptions, including hand-keyed subsets and national OCR extractions of single editions, 1918 for Smith (2017) and 1950 for Alsan and Neberai (2026). Moehling et al. (2020) used hand transcriptions of four editions for four states to study physician labor markets. Our own earlier work used a vendor double-entry transcription of the AMD for North Carolina from every edition between 1912 and 1942 [18]. Alsan and Neberai (2026) digitized and OCR’d the 1950 edition for membership and specialists and checked state totals against published aggregates. Other studies use a state, a city or sample [26, 8, 23, 20, 27] (Schafer 2014). Andrews (2023) uses AMD aggregates and a state directory, not AMD microdata.

The closest precedent to our validation samples is Chrisinger (2024), which used a double-entry manual transcription for eighteen Southern and adjacent states and the District of Columbia. The article analyzes 41,828 entries, including 632 physicians labeled col. and 113 unlabeled graduates of Black medical schools, and its dataset is publicly deposited [9]. Smith (2017) provides a machine-transcription precedent with reported record checks. The 1918 prototype uses commercial OCR and two parsing programs, keeps 70 to 90 percent of single-line entries, varying by state, and reports more than 92 percent of manually checked entries as perfect. It also compares capture with printed counts for the largest cities.

A series of articles in Public Health Reports by Mountin, Pennell and coauthors examines physician location and movement between 1923 and 1938 (for example Mountin et al. 1942). Their titles and records show the subject but not how physicians were linked across editions.

StudyEditionsCoverageMethodValidation reportedPublic
Moehling et al. [21]1909, 1914, 1918, 1921CA, MS, NY, NChand keying——
Hollingsworth et al. [18]1912–1942North Carolinavendor double entry10—yes, deposited
Alsan and Neberai [1]1950 by OCR; earlier editions citednational (membership, specialists)OCRstate totals vs published aggregates—
Smith [25]1918national pilotOCR, two parsersprinted per-city counts; 70 to 90% of single-line entries kept; >92% of checked entries perfect—
Chrisinger (2024)190618 Southern and adjacent states, DCvendor double entrylikely undercount reportedyes, deposited
Mountin et al. [22]1923, 1938nationalPublic Health Service tabulation—in print
This note1912, 192148 states and DCOCR; MLLM, AI-assisted parser, open LLM and open OCR-LLM workflowsagreement with a double-entered transcription on its completely transcribed pagesyes, Zenodo, CC BY

Table A.1. Prior transcriptions of the American Medical Directory

— : not examined for this note.

Scoring protocol

This appendix collects the rules under which every figure in § 5 was produced.

Record linkage. Each workflow is linked to the validation sample separately, with the same rule in both editions. The first pass matches printed-page and name keys that occur exactly once on each side. Within page-and-name groups that remain ambiguous, the second pass matches state–city pairs that are valid and unique on both sides. Birth, graduation and license years and the school code never select a link. Unmatched rows and unresolved ties remain in both denominators, and links are fixed over the whole edition before any page restriction, so name agreement is conditioned on the rule. The linkage key is the printed page and the name as a multiset of tokens: names are normalized (Unicode composition, case folding), split into alphanumeric tokens and sorted, with no fuzzy repair, and a blank name gives no key. Years and school codes are never keys. Table D.3 (Appendix § 11) gives the 1921 figures after the first pass alone.

Scorer. All headline tables use a common scorer. The scorer applies the same normalization to city and county on both sides before linking: it removes accents, the St./Mt./Ft. abbreviations, a trailing County, punctuation and spacing, and nothing else. Field precision divides agreeing linked values by all scorable output values and field recall by all scorable reference values; an unmatched row’s values count as failures; a blank on both sides never counts as agreement; years are compared as two-digit tokens and school codes after the same syntactic normalization on both sides.

Cleaned reference. The scorer reads a cleaned version of the transcription, which normalizes school codes and two-digit years to the forms the key defines and does not touch names or places. Its largest effect is on the school code, where the transcribers’ varied spellings of one code would otherwise count as disagreement. That difference reflects how the reference represents a value, not the extraction.

Frames. The analysis frame of an edition is the set of transcribed printed pages without known incomplete coverage: pages on which the compiled transcription records no ignored-row placeholder, including the pages used to tune the parser, the pages drawn into a sampling design and the diagnostic pages (Appendix § 16 reports the frame without them). For 1921 that is 1,036 pages and 124,167 cleaned rows out of 1,233 transcribed page keys, after dropping the 197 pages with an ignored-row placeholder. For 1912 it is 903 pages and 123,726 rows out of 1,006 page keys, after dropping the 103 pages with an ignored-row placeholder. Table D.1 (Appendix § 11) sets the two editions side by side; Table D.3 gives the 1921 figures on the broad frame.

Evaluated inputs. The MLLM workflow is the matched output of the stored biography and location passes; the 1921 model passes date from February 2026, and the 1912 output is the one used in the tables prepared for release. The AI-assisted parser is the parser's raw candidate list for each edition. The combined table is the reconciled output in each edition, with the checked cleaned county substituted for the as-read county. The paired county checks retain exactly the same links and all non-county fields. The open LLM workflow is open-model-pipeline output for 1921 (162,215 evaluated rows, of 162,947 output rows with 732 duplicate-page rows dropped) and 1912 (149,471 rows). The open OCR-LLM workflow is the output of the same open model pipeline, with the same model and prompts, on the open-source OCR text of each edition. The open LLM and open OCR-LLM outputs carry their own page identifiers, which a reviewed page bridge maps to printed pages in both editions. Table D.2 (Appendix § 11) lists the inputs of each workflow.

The printed key for 1921

What an entry holds. The body is arranged by state, then by town, with the town’s population and county on a heading line and the physicians beneath it in alphabetical order, in narrow columns. A physician listed in full under one town may be printed under another as a cross-reference line. Each entry packs its fields into a fixed order of abbreviations, decoded by an edition-specific printed key; the front-matter key is not in our scans, and we read it from an Internet Archive copy (American Medical Association 1921). Table C.1 lists the printed elements and the release field each feeds. The license year is the year of license in the state where the physician is listed, so a mover carries a new license year in the next edition; a missing year has several printed meanings, most with their own mark, and some states print no year by convention. The release tables carry the year only, not which reason the mark gave. The address element R.D., the physician count, the town line and the cross-reference are layout elements rather than key definitions; we read their meanings from the entries themselves.

Printed elementMeaning in 1921Release field
Name in capitalsmembership in the AMA through a constituent state association and component county societywithheld
AMA mark, letter-coded before OCR, after the license year11Fellowship in the AMA; commissioned medical officers on active service are Fellows10withheld
(E), (H), (P-M) after the nameeclectic, homeopath, physio-medicwithheld
b’75born 1875birth year, as read and cleaned
Md. 7, ’09school key number (state abbreviation and number) and year of graduationschool code (cleaned to the key’s form), graduation year
♢information received does not show graduation from a medical college10not captured
△graduate of an institution not listed as a medical college by the Council on Medical Education and Hospitals10not captured
l’20year of license in the state where locatedlicense year
l†licensed, year not knownnot captured
‡information received does not show license in the state where locatednot captured
circle under a crossin practice in the state where located before passage of the present law, and licensed by filing a diploma or by years of practicenot captured
Specialty code (S, Oph, Pd, . . . ), *self-reported interest; with the asterisk, practice claimed limited to that branch; printed on request onlywithheld
R.D., R.D.1rural-delivery address element, never a specialtywithheld with the addresses
(col.) after the namethe Association’s label for a Black physician12withheld
NUMBER OF PHYSICIANS, nn under a town linecount of physicians listed under that townnot released
Town line TOWN, population, COUNTYpopulation and county of the towncounty, as read and cleaned; population withheld
(See Chicago)cross-reference to the full entry under another towncross-reference line in the release table
ADDENDA; index symbolsphysicians received too late to classify; corrections; reports of deathsnot released

Table C.1. The 1921 printed key and release fields

Linkage sensitivity and stage decomposition

Appendix B states the linkage rule, scorer, frames and inputs. Table D.1 gives the coverage of the validation samples (§ 4.1) and Table D.2 the five extraction workflows (§ 3.2).

19121921
Printed-page keys transcribed (broad frame)1,0061,233
Cleaned rows, broad frame137,501145,064
Pages without known incomplete coverage (analysis frame)9031,036
Cleaned rows, analysis frame123,726124,167
Analysis pages seen while tuning the parser93116
Analysis pages in the frozen design sample1616
Analysis pages used in diagnostics02
Fields transcribedname, birth year, school, graduation year, license year, city, county, state, population
Not transcribedaddenda; address-unknown list; count lines (inconsistently)

Table D.1. Coverage of the validation samples, 1912 and 1921. Sources: comparison results and validation-sample documentation for each edition.

Table D.3 reports the record comparison of § 5.1 for both editions under the first pass alone and after the second pass, and on the broad frame of every transcribed page key (1,006 pages in 1912 and 1,233 in 1921), which includes the pages with known incomplete coverage. The broad frame lowers precision for every workflow: on a page the transcribers covered only in part, a machine row for a physician they skipped counts against the extraction. Table D.4 decomposes the MLLM workflow by stage on the analysis frames; Appendix E describes the stages. Table D.5 splits record agreement by the share of each page’s entries that the transcribers flagged as uncertain and, for recall, by each entry’s own flag; it is the evidence for § 5.3.

WorkflowInputModel or rulesFields producedEvaluated (1921)output
MLLMpage image (biography); OCR text (location)gemini-2.0-flash, prompts per edition; page matching; OCR enhancementname, birth year, school, graduation year, license year, specialty, city, county, statebiography and location runs of February 2026; matching run of October 2026
Parser (AI-assisted)OCR textdeterministic parser; same rules and dictionary in both editionsall of the above, plus population, membership and military flags, cross-reference linesraw candidate list (Appendix B)
CombinedMLLM and AI-assisted parserper-field reconciliation rules; release cleaning; 48-state rulethe parser's fields with chosen values and per-field provenancereconciled output; checked cleaned county on fixed links
Open LLMOCR textmulti-stage LLM extraction (open model pipeline)name, birth year, school, graduation year, license year, city, county, state1921: 162,215 evaluated rows (of 162,947 output rows, with 732 duplicate-page rows dropped)
Open OCR-LLMOCR text of the page images by an open-source modelopen model pipeline (same code, model and prompts)as Open LLMscores in Tables 2 and G.1

Table D.2. The five extraction workflows. The MLLM workflow, the AI-assisted parser, the combined table and the open LLM workflow consume the same stored OCR output; the MLLM workflow also reads page images for biographical fields. The open OCR-LLM workflow consumes a separate OCR of the same page images and shares no OCR text with the others. Specialty and symbol flags are neither scored here nor included in the public release.

WorkflowFirst-pass linksSecond pass addsOutput rowsFirst passSecond passSecond pass
Links and rows, analysis frameAnalysis frameBroad frame
1912 (903 analysis pages)
MLLM116,76552123,10894.85 / 94.3794.89 / 94.4294.76 / 94.21
Parser120,42042123,49697.51 / 97.3397.54 / 97.3697.48 / 97.27
Combined120,96545123,82997.69 / 97.7797.72 / 97.8097.65 / 97.72
Open LLM118,57142121,37097.69 / 95.8397.73 / 95.8797.57 / 95.60
1921 (1,036 analysis pages)
MLLM116,602281124,19593.89 / 93.9194.11 / 94.1392.55 / 93.96
Parser120,92375125,10196.66 / 97.3996.72 / 97.4595.11 / 97.44
Combined121,94983124,47797.97 / 98.2198.04 / 98.2896.43 / 98.32
Open LLM117,98783122,91595.99 / 95.0296.06 / 95.0994.44 / 95.22

Table D.3. Record agreement by link pass and page frame, 1912 and 1921 (precision / recall, %). Agreement with the cleaned validation sample; scan-verified accuracy not assessed. Broad frame: 1912, 1,006 page keys and 137,501 cleaned rows; 1921, 1,233 page keys and 145,064 cleaned rows. Table 2 reports the open OCR-LLM scores on the analysis frame. Source: the common scorer's comparison results.

StagePrecision (%)Recall (%)
1912
Location pass alone (OCR text)93.7790.34
Biography pass alone (image)96.4692.10
Matched, before OCR enhancement95.0992.13
Matched, after OCR enhancement (MLLM workflow)94.8994.42
1921
Location pass alone (OCR text)95.4491.64
Biography pass alone (image)95.6492.68
Matched, before OCR enhancement94.7692.77
Matched, after OCR enhancement (MLLM workflow)94.1194.13

Table D.4. The MLLM workflow by stage, 1912 and 1921. Agreement with the cleaned validation sample; scan-verified accuracy not assessed. Analysis frames; second-pass links. The biography pass carries no geography and links on the first pass only. Source: the stage comparison.

GroupReference rowsMLLMParserCombinedOpen LLMOpen OCR-LLM
1912
Pages grouped by the share of entries flagged (precision / recall)
≥5%\ge5\% of entries flagged (61 pages)7,88481.34 / 80.9183.58 / 83.4383.77 / 83.8383.97 / 82.9782.30 / 79.60
<5%<5\% of entries flagged (842 pages)115,84295.81 / 95.3498.49 / 98.3198.67 / 98.7698.67 / 96.7597.45 / 94.53
Entries grouped by their own flag (recall)
Flagged entries2,27145.1346.8147.0746.1544.65
Unflagged entries121,45595.3498.3198.7596.8094.49
1921
Pages grouped by the share of entries flagged (precision / recall)
≥5%\ge5\% of entries flagged (42 pages)5,08592.18 / 90.8493.34 / 94.7594.03 / 95.3692.43 / 93.3592.05 / 91.72
<5%<5\% of entries flagged (994 pages)119,08294.19 / 94.2796.87 / 97.5698.21 / 98.4196.22 / 95.1695.05 / 92.77
Entries grouped by their own flag (recall)
Flagged entries1,62770.3184.4586.1783.4182.42
Unflagged entries122,54094.4597.6298.4495.2492.86

Table D.5. Agreement by uncertainty flags, 1912 and 1921

Notes: Agreement with the cleaned validation sample; scan-verified accuracy not assessed. Second-pass links on each edition’s analysis frame (1912, 903 pages; 1921, 1,036 pages), with the same links and scorer as Table 2. The first panel of each edition restricts both measures to the pages in its group; pages are grouped by the share of their reference entries that the transcribers flagged as uncertain. The second panel splits the reference entries by their own flag and reports recall only: an output row that matches no reference entry carries no flag, so precision cannot be assigned to an entry group. The flags record the transcribers’ uncertainty; their absence does not verify an entry. The flagged entries include entries whose transcribed name contains a question mark. Matching uses the name, so no workflow can match these entries, and they count as unmatched for every workflow. They account for most of the low recall on flagged entries, especially in 1912, where about half of the flagged entries have such names.

The MLLM workflow in detail

This appendix describes the stored 1921 implementation of the MLLM workflow (§ 3.2.1): the two model passes, the checks on their output and the matching that joins them. Every call is to gemini-2.0-flash through the vendor’s API with a prompt written for the 1921 edition; the stored runs date from February 2026 (Appendix § 13), and no call has been repeated since the model was shut down on 1 June 2026.

Location pass. The first pass reads the OCR text of a page, not the image. The prompt asks for structured JSON: the state, each town heading with its county and population, and the physician names printed beneath it. A page whose output contains neither a town heading nor a physician name is skipped as presumed non-entry material; this rule can also exclude a physician page whose headings the pass fails to extract.

Biography pass. The second pass reads the page image. The model first transcribes the page with its layout into sections, then extracts from each section the physicians and the fields the key defines. A section that fails the schema check is sent back once for repair, and a section whose output overran the model’s limit is split and read again. Pages and entries with none of the four key fields (birth year, school, graduation year, license year) are dropped at collection.

Matching. The matcher joins the two passes in four greedy, one-to-one rounds: on the normalized name and its occurrence on the page; on the name alone by Jaro–Winkler similarity at or above 0.85; by carrying a location forward from the previous page to a biography that has none; and, last, by filling the fields of a name the location pass found but the image pass did not from the OCR text. That last step is the only one at which OCR text supplies biographical values to this workflow; it filled 3,654 rows in 1912 and 2,958 in 1921 (3,227 and 2,632 on the analysis frames). Table D.4 scores each stage on the analysis frames. In 1921, each pass alone links 92 to 93 percent of the transcribed rows at about 95 to 96 percent precision, the two matched without the OCR step reach 92.77% at 94.76%, and the full extraction 94.13% at 94.11%. The OCR step adds 2,824 links in 1912 and 1,691 in 1921 and about 1.4 points of recall in 1921, at a small cost in precision.

Reproducibility statement

The comparisons use stored outputs; no OCR call, model call or extraction was made for this note. Appendix B describes the linkage rule, scorer, frames and inputs. This appendix describes what is released and the limits to reproducing the results.

Extraction and release. The stored inputs include the OCR text, the MLLM workflow’s biography and location passes and their matched output, the parser candidates and the combined tables. For 1921, OCR was performed in January 2026 and the model passes in February 2026, using gemini-2.0-flash. For the combined table, county is taken from the separately checked county-cleaning rule on identical links; the other fields remain those of the reconciled output. The public dataset is identified by Zenodo version 1.0.0 and DOI 10.5281/zenodo.23945788. The descriptive statistics (Table 1) are calculated from the public release tables and can be reproduced from the public files.

Limits to reproduction. The model gemini-2.0-flash was shut down on 1 June 2026, so the MLLM extraction cannot be rerun through the vendor’s API. The manual transcriptions are excluded from the public release, which prevents full public recomputation of the agreement results.

Field tables by workflow

Table G.1 gives the percentages shown in Figure 3, for both editions.

FieldMLLMParserCombinedOpen LLMOpen OCR-LLM
Panel A: 1912
Name96.26 / 95.6097.54 / 98.5997.72 / 99.0497.73 / 97.0796.49 / 94.76
Birth year96.86 / 96.9697.91 / 97.7098.06 / 98.1798.01 / 95.8495.74 / 93.21
Medical school93.84 / 85.9795.80 / 95.2896.00 / 96.0995.68 / 91.2693.84 / 86.87
Graduation year95.46 / 95.1995.78 / 96.9295.73 / 97.8497.32 / 95.3895.62 / 92.40
License year93.55 / 90.7696.99 / 97.3495.70 / 97.8496.96 / 95.7192.66 / 90.84
State94.62 / 90.3997.54 / 97.3697.75 / 97.7897.25 / 95.2196.49 / 93.19
City93.26 / 89.9295.88 / 95.6196.53 / 96.5794.93 / 93.1393.31 / 90.50
County88.86 / 85.6794.88 / 86.2096.71 / 96.6991.48 / 89.7488.45 / 85.78
Panel B: 1921
Name95.06 / 94.4896.72 / 97.8198.04 / 98.6596.06 / 95.4494.92 / 93.07
Birth year95.00 / 96.1397.33 / 97.5798.00 / 98.3096.29 / 95.0394.84 / 92.58
Medical school94.65 / 89.7296.94 / 96.7497.45 / 98.0394.48 / 92.4992.99 / 90.48
Graduation year94.03 / 94.8595.44 / 96.8695.88 / 98.2495.90 / 94.4494.55 / 91.88
License year93.06 / 90.6696.78 / 97.2996.85 / 98.2195.73 / 94.5393.56 / 89.95
State94.24 / 90.5896.72 / 97.4598.06 / 98.2796.06 / 95.0894.50 / 92.21
City88.98 / 85.6895.25 / 96.1496.58 / 96.9591.76 / 91.0091.65 / 89.69
County83.36 / 80.2895.14 / 89.7896.65 / 96.9786.31 / 85.6187.31 / 85.42

Table G.1. Field agreement by workflow, 1912 and 1921 (precision / recall, %)

Agreement with the cleaned validation sample; scan-verified accuracy not assessed. Same frames, links and scorer as Table 2. Name is a link key, so its agreement is linkage-conditioned. Years as two-digit tokens; school codes after syntactic normalization; places after formatting normalization. Integer counts for both editions appear in Table G.2.

Table G.2 gives the integer counts behind Table G.1: agreeing linked values, scorable output values and scorable reference values, for each field and the five workflows, on the 1912 and 1921 analysis frames. Table G.4 gives the disagreement counts of § 5.3. Table G.3 shows what release cleaning does to the release table’s agreement, scored as read and after cleaning on links fixed from the as-read values. We selected the cleaning rules by comparing candidates against the transcription in the 1912 and 1921 editions before running this comparison.

FieldMLLMParserCombinedOpen LLMOpen OCR-LLM
1912
Name116,817 / 121,356 / 122,188120,462 / 123,496 / 122,188121,010 / 123,829 / 122,188118,613 / 121,370 / 122,188115,780 / 119,991 / 122,188
Birth year87,462 / 90,301 / 90,20888,130 / 90,014 / 90,20888,557 / 90,308 / 90,20886,459 / 88,212 / 90,20884,086 / 87,829 / 90,208
Medical school102,980 / 109,737 / 119,787114,134 / 119,140 / 119,787115,109 / 119,902 / 119,787109,317 / 114,258 / 119,787104,060 / 110,891 / 119,787
Graduation year112,071 / 117,404 / 117,731114,101 / 119,124 / 117,731115,193 / 120,326 / 117,731112,291 / 115,385 / 117,731108,787 / 113,771 / 117,731
License year85,040 / 90,908 / 93,69391,205 / 94,036 / 93,69391,669 / 95,786 / 93,69389,678 / 92,491 / 93,69385,107 / 91,852 / 93,693
State111,837 / 118,200 / 123,725120,461 / 123,496 / 123,725120,982 / 123,769 / 123,725117,799 / 121,129 / 123,725115,294 / 119,489 / 123,725
City111,237 / 119,280 / 123,713118,282 / 123,371 / 123,713119,470 / 123,768 / 123,713115,220 / 121,370 / 123,713111,960 / 119,991 / 123,713
County105,991 / 119,280 / 123,725106,655 / 112,414 / 123,725119,630 / 123,702 / 123,725111,035 / 121,370 / 123,725106,128 / 119,991 / 123,725
1921
Name116,883 / 122,953 / 123,708120,998 / 125,101 / 123,708122,032 / 124,477 / 123,708118,070 / 122,915 / 123,708115,136 / 121,295 / 123,708
Birth year105,239 / 110,774 / 109,480106,823 / 109,749 / 109,480107,624 / 109,825 / 109,480104,037 / 108,043 / 109,480101,355 / 106,873 / 109,480
Medical school108,682 / 114,831 / 121,137117,191 / 120,890 / 121,137118,746 / 121,851 / 121,137112,038 / 118,578 / 121,137109,605 / 117,862 / 121,137
Graduation year112,852 / 120,019 / 118,984115,252 / 120,764 / 118,984116,894 / 121,920 / 118,984112,374 / 117,178 / 118,984109,326 / 115,622 / 118,984
License year93,603 / 100,580 / 103,249100,454 / 103,791 / 103,249101,401 / 104,701 / 103,24997,604 / 101,961 / 103,24992,869 / 99,265 / 103,249
State112,472 / 119,344 / 124,165120,998 / 125,101 / 124,165122,013 / 124,424 / 124,165118,053 / 122,897 / 124,165114,488 / 121,146 / 124,165
City106,196 / 119,344 / 123,948119,158 / 125,101 / 123,948120,164 / 124,424 / 123,948112,788 / 122,915 / 123,948111,167 / 121,295 / 123,948
County99,482 / 119,344 / 123,925111,256 / 116,944 / 123,925120,170 / 124,334 / 123,925106,087 / 122,915 / 123,925105,855 / 121,241 / 123,925

Table G.2. Field counts by workflow, 1912 and 1921: agreeing / scorable output / scorable reference values. Notes: Agreement with the cleaned validation sample; scan-verified accuracy not assessed. Unmatched rows’ values stay in the denominators; a cell with several years or schools contributes each value. Source: the field-agreement comparison.

FieldAs readAfter cleaning
1912
Medical school96.00 / 96.0997.47 / 97.59
Birth year98.06 / 98.1798.07 / 98.16
City96.53 / 96.5796.53 / 96.57
County91.34 / 91.2696.71 / 96.69
1921
Medical school97.45 / 98.0397.48 / 98.11
Birth year98.00 / 98.3098.01 / 98.30
City96.58 / 96.9596.58 / 96.95
County90.90 / 90.9696.65 / 96.97

Table G.3. Release cleaning and agreement, combined table, 1912 and 1921 (precision / recall, %). Agreement with the cleaned validation sample; scan-verified accuracy not assessed. Links fixed from the as-read values. The as-read school, birth-year and city figures and the cleaned county figure are the combined table’s entries in Table G.1; the comparison is between the release table’s *as_read columns and its cleaned columns. Source: the release-cleaning comparison.

FieldBoth agreeOnly MLLM agreesOnly parser agreesNeither agrees
1912
Birth year86,1302312,21377
Graduation year110,6828473,703192
License year83,5242697,911160
Medical school101,1301,06711,5301,718
1921
Birth year105,3533811,89364
Graduation year112,6591,2283,031118
License year95,1965345,676120
Medical school108,4711,1407,106318

Table G.4. Disagreement between the MLLM workflow and the AI-assisted parser on rows linked through the combined table, 1912 and 1921. Agreement with the cleaned validation sample; scan-verified accuracy not assessed. Cells on the combined table’s linked rows: 1912, 121,010 rows on 903 pages; 1921, 122,032 rows on 1,034 pages. An absent extraction entry counts as non-agreement. Counts describe agreement with the transcription and do not establish which extraction reproduces the printed entry. Rows where the reference is blank are left out; a cell with several school codes counts once here and once per value in Table G.2. Source: the comparison of extraction disagreements.

Cost accounting

Table H.1 gives the cost of manual transcription and the direct processing charges for the machine workflows, in total and per page. Manual costs are the delivered workbook page counts, 2,242 pages across both editions, at the invoiced €14.70 per page. The machine cost per page divides the combined Gemini and Azure charges by the same page count. We allocate Gemini charges by whole billing days and assume one successful Azure OCR pass per page, so the machine figures are estimates, not invoice totals. Ratios use unrounded inputs and one exchange-rate date (ECB, 3 February 2026).

The comparison covers direct processing charges only. It leaves out engineering, review, model development, coding-agent assistance, energy and the local computing hardware. The hardware is shared with other project work and hosted-API workflows do not need it, so adding it would not make the comparison like for like. The manual transcription also covers a different set of pages and fields than the machine outputs, and we do not estimate a separate cost for the open model pipelines.

ComponentUnit19121921Both editions
Purchased transcriptionPages1,0071,2352,242
Manual transcriptionEUR14,802.9018,154.5032,957.40
Gemini, allocated billingDKK136.99241.71378.70
Azure, one-pass estimateUSD13.8814.8028.68
Gemini plus AzureEUR30.1044.9075.01
Manual cost per pageEUR14.7014.7014.70
Machine cost per pageEUR0.030.040.03
Manual / estimated API costRatio492404439

Table H.1. Manual transcription and estimated machine processing charges

Manual costs use delivered workbook page counts at €14.70 per page, reconciled to the vendor’s invoices; the 1918 batch and unassigned supplement are excluded. These are purchased-coverage counts, not the analysis frames. API figures cover Gemini and Azure, not a separate cost estimate for the open model pipelines.

Gemini 2.0 Flash billing is allocated by whole days: 28–31 January 2026 (1912), 23–25 January and 3–5 February (1921); other-model charges are excluded. Edition overlap on 25 and 31 January makes attribution approximate. Excluding 31 January gives DKK 114.12 for 1912; excluding 25 January gives DKK 185.45 for 1921. These are scenarios, not confidence bounds.

Azure assumes 1,388 and 1,480 successful one-page images at the conditional West Europe prebuilt-layout S0 list price of USD 10 per 1,000 pages (effective from August 2021). Actual Azure billing and repeat-call counts were not available; these are list-price estimates, not invoice totals. EUR equivalents and ratios use ECB reference rates on 3 February 2026: EUR 1 = DKK 7.4687 = USD 1.1801, a common-date presentation convention; totals and ratios use unrounded inputs.

Robustness to the page frame

The analysis frames keep every transcribed page without an ignored-row placeholder (§ 4.1). Two choices could move the results: leaving out the pages flagged for incomplete coverage, and keeping the pages seen during parser development, drawn into the frozen design sample or used in diagnostics. Table I.Table 1 scores every workflow on all transcribed pages and on the incomplete pages alone; Table I.Table 2 scores them on the analysis frame without the exposed pages and on the exposed pages alone. Links are fixed edition-wide before any page restriction, so each frame scores the same links.

Coverage19121921
All transcribedIncomplete pagesAll transcribedIncomplete pages
WorkflowPrec. (%)Recall (%)Prec. (%)Recall (%)Prec. (%)Recall (%)Prec. (%)Recall (%)
MLLM94.7694.2193.6192.4092.5593.9684.1192.90
Parser97.4897.2796.8996.4895.1197.4486.5497.42
Combined97.6597.7297.0396.9396.4398.3287.8998.53
Open LLM97.5795.6096.1493.1594.4495.2285.9096.00
Open OCR-LLM96.4293.5495.7793.2193.3592.7485.0292.85

Table I.1. Record agreement by coverage frame, 1912 and 1921

Agreement with the cleaned validation sample; scan-verified accuracy not assessed. All transcribed: 1,233 pages and 145,064 reference rows in 1921, 1,006 and 137,501 in 1912. Incomplete pages: the pages with an ignored-row placeholder, 197 pages and 20,897 rows in 1921, 103 and 13,775 in 1912. On these pages part of the printed entries were not transcribed, so precision understates agreement. Links after the second pass; open LLM and open OCR-LLM as in Table 2.

Exposure19121921
Without exposed pagesExposed pages onlyWithout exposed pagesExposed pages only
WorkflowPrec. (%)Recall (%)Prec. (%)Recall (%)Prec. (%)Recall (%)Prec. (%)Recall (%)
MLLM95.0194.6494.0592.8394.3394.2992.7493.12
Parser97.6297.4696.9696.6696.5997.4697.5997.39
Combined97.8097.8997.2097.2098.0698.3197.8798.10
Open LLM97.7895.9497.3695.3196.0995.3295.8493.57
Open OCR-LLM96.5593.6496.0793.1694.8792.6795.2893.10

Table I.2. Record agreement with and without exposed pages, 1912 and 1921

Agreement with the cleaned validation sample; scan-verified accuracy not assessed. Without exposed pages: the analysis frame less every page with a scan seen while tuning the parser, drawn into the frozen design sample or used in diagnostics, 902 pages and 107,565 reference rows in 1921, 794 and 108,531 in 1912. Exposed pages only: the 134 and 109 pages left out there (16,602 and 15,195 rows): in 1921, 16 design, 116 tuning and 2 diagnostic pages; in 1912, 16, 93 and 0. A page on several lists counts once, as design before tuning before diagnostic. The record of pages seen during parser development is incomplete for 1921. Links after the second pass.

References

  1. [1]Alsan, Marcella and Yousra Neberai (2026) “Why Doesn’t the United States Have National Health Insurance? The Political Role of the American Medical Association,” Quarterly Journal of Economics, Vol. 141, No. 3, pp. 2147–2204.
  2. [2]American Medical Association (1912) American Medical Directory: A Register of Legally Qualified Physicians of the United States, Alaska, Canal Zone, Hawaii, Porto Rico, Philippine Islands, Canada and Newfoundland, Chicago: American Medical Association, 3rd edition.
  3. [6]Bäcker-Peral, Verónica, Vitaly Meursault, and Christopher Severeen (2026) “Can LLMs Credibly Transform the Creation of Panel Data from Diverse Historical Tables?,” arXiv preprint arXiv:2505.11599v2.
  4. [7]Carlson, Jacob, Tom Bryan, and Melissa Dell (2024) “Efficient OCR for Building a Diverse Digital History,” arXiv preprint arXiv:2304.02737.arxiv.org/abs/2304.02737
  5. [8]Catillon, Maryaline, David Cutler, and Thomas Getzen (2018) “Two Hundred Years of Health and Medical Care: The Importance of Medical Care for Life Expectancy Gains,” Working Paper No. 25330, National Bureau of Economic Research.DOI
  6. [9]Chrisinger, Benjamin W. (2023) “Deep Roots of Racial Inequalities in US Healthcare: The 1906 American Medical Directory,” Dataset, version 2, University of Oxford Sustainable Digital Scholarship portal, https://doi.org/10.25446/oxford.24065709.v2.
  7. [11]Correia, Sergio and Stephan Luck (2023) “Digitizing Historical Balance Sheet Data: A Practitioner’s Guide,” Explorations in Economic History, Vol. 87, p. 101475.
  8. [12]Dahl, Christian M., Torben S. D. Johansen, Emil N. Sørensen, Christian E. Westermann, and Simon Wittrock (2023) “Applications of machine learning in tabular document digitisation,” Historical Methods: A Journal of Quantitative and Interdisciplinary History, Vol. 56, No. 1, pp. 34–48.
  9. [13]Dell, Melissa (2025) “Deep Learning for Economists,” Journal of Economic Literature, Vol. 63, No. 1, pp. 5–58.
  10. [14]Dell, Melissa and Ashesh Rambachan (2026) “The Measurement Revolution? Credible Measurement and Inference in the Age of AI,” Working Paper No. 35744, National Bureau of Economic Research.arxiv.org/abs/2608.23524
  11. [15]Ferrara, Andreas (2026) “A Practitioner’s Guide to Using Large Language Models and Generative AI in Economic History,” Discussion Paper No. 171/26, Rockwool Foundation Berlin (RFBerlin).
  12. [16]Fleischhacker, David, Roman Kern, and Wolfgang Göderle (2025) “Enhancing OCR in Historical Documents with Complex Layouts through Machine Learning,” International Journal on Digital Libraries, Vol. 26, p. 3.DOI
  13. [17]Gutmann, Myron P., Emily Klan­cher Merchant, and Evan Roberts (2018) ““Big Data” in Economic History,” The Journal of Economic History, Vol. 78, No. 1, pp. 268–299.DOI
  14. [18]Hollingsworth, Alex, Krzysztof Karbownik, Melissa A. Thomasson, and Anthony Wray (2024) “The Gift of a Lifetime: The Hospital, Modern Medicine, and Mortality,” American Economic Review, Vol. 114, No. 7, pp. 2201–2238.
  15. [20]McNamara, Robert L., Edward W. Hassinger, and John B. Mitchell (1955) “Supply of Physicians in Rural Missouri,” Bulletin 651, University of Missouri College of Agriculture, Agricultural Experiment Station.
  16. [21]Moehling, Carolyn M., Gregory T. Niemesh, Melissa A. Thomasson, and Jaret Treber (2020) “Medical Education Reforms and the Origins of the Rural Physician Shortage,” Cliometrica, Vol. 14, pp. 181–225.DOI
  17. [22]Mountin, Joseph W., Elliott H. Pennell, and Virginia Nicolay (1942) “Location and Movement of Physicians, 1923 and 1938: General Observations,” Public Health Reports (1896–1970), Vol. 57, No. 37, pp. 1363–1375.DOI
  18. [23]Mullally, Sasha and Katherine MacDonald (2017) “Call the Doctor? Understanding Health Service Trends in New Brunswick, Part I, 1918–1950,” Journal of New Brunswick Studies, Vol. 8.
  19. [25]Smith, Sean Morey (2017) “Digitizing Doctors: Methodologies for Creating a Database from Historical Directories of Physicians,” Medical History, Vol. 61, No. 4, pp. 611–614.DOI
  20. [26]Staines, Amber (2015) “The Effect of Medical Care on Infant Mortality in the United States in the Early 20th Century,” M.A. thesis, Miami University.
  21. [27]Weisz, George (2006) “Regulating Specialists in National Medical Directories,” in Divide and Conquer: A Comparative History of Medical Specialization, Oxford: Oxford University Press, pp. 87–104.DOI

Paper details

vlnc:W_af6qdk2r
Version
v1 · latest
Published
Fresh — added to the Hub within the last 30 days
Added
License
CC BY-SA 4.0
Usage

views · 0

downloads · 0

A view counts once per browser per day, a download each time the PDF is requested.

  • Computer Science

ChatGPT Desktop, Codex, Claude Code

View statement
Looking for related papers…
Contents