betaFeedback

WorksMathematics

FlexRL: an R package for Flexible Record Linkage and Linked Data Quality Assessment

Abstract

FlexRL is an R package for probabilistic record linkage and inference on linked data. It provides a method to link observations from the same entities across two data sources that lack a unique identifier but share partially identifying variables. This method fits a hierarchical latent variable model of the data generation process, rather than classifying pairs of records from the comparison of their values, and accommodates registration errors and variables changing over time. Since linkage may affect subsequent analyses, the package also provides tools to assess the linked data quality as functions of the linkage decision rule. Two false discovery proportion estimators are implemented: a probabilistic approach contingent on the record linkage model and an augmented approach relying on an auxiliary estimation using synthetic data. Several measures of divergence of the linked sample from the source population are provided. These tools are agnostic to the underlying record linkage model and can be applied to any method. Together, they connect the output of the linkage to the validity of the subsequent analyses. The implementation combines standard R functions and S3 methods with Rcpp computations, resulting in fast execution times and a low memory footprint.

Introduction

Researchers and statistical agencies increasingly combine data sources that do not share a unique identifier [8, 29, 27]. Record linkage connects records referring to the same entity across sources based on Partially Identifying Variables (PIVs, e.g. birth year, sex, postal code or product code, brand and category). Each PIV has limited discriminative power on its own and may be missing, incorrectly recorded or change over time. Record linkage underpins the production of official statistics, enables the reuse of administrative and registry data for research [60, 39, 10] and improves data quality through linkage and deduplication.

Linked data sets become the study sample for downstream analyses, where linkage errors may propagate [14, 58] (Vo et al. 2024; Robach et al. 2026). Errors arise when links between entities that appear in both sources are missed or when records from distinct entities are falsely linked. These errors cannot be reduced simultaneously and may bias the conclusions drawn from linked data. A stricter decision rule reduces falsely linked pairs but misses more links. Practitioners therefore need not only record linkage methods, but also tools to quantify linkage errors and appraise their consequences for statistical analysis.

Two main probabilistic approaches to record linkage have developed through times. First, the traditional mixture model [18, 24, 31, 19, 45] treats linkage as a classification problem between pairs of records that are similar and pairs that are not. Second, the hierarchical entity resolution framework [51, 48, 50, 33, 34] models the data generation process, considering observed values as potentially distorted versions of latent true values. The latter framework uses all the information contained in the records and is therefore more flexible but more computationally demanding. It has been implemented in R for entity resolution (clustering records from a single file into entities) rather than for linkage between two data sources.

We address this gap with the FlexRL R package, which makes two contributions. First, it implements the hierarchical record linkage model proposed by Robach et al. [2025b]. The model is fitted with a Stochastic Expectation Maximisation (StEM) algorithm that accounts for registration errors and temporal changes in PIVs. Its architecture and efficient Rcpp backend support scalability to large data sets and a wide range of applications: health monitoring studies, survey waves, census and administrative records or data from different retailers. Second, it provides tools to assess the quality of the linked data, which quantify the linkage error and sample selection. FlexRL implements two estimators of the False Discovery Proportion (FDP): one derived from linkage scores [3, 40, 17] and one based on synthetic data [42]. FlexRL measures the divergence of the linked sample from the original sources using standardised mean differences, overlap of supports, maximum mean discrepancy and agreement rates. These tools only need the linked pairs and, when available, their linkage scores. They therefore apply to the output of any linkage method and FlexRL provides wrappers for several existing R packages.

The FlexRL package is aimed at practitioners, statistical agencies and researchers who need to link data sources without a unique identifier or who want to measure the error and the sample selection induced by record linkage. It builds on three lines of work: the biases that linkage errors cause in inference [28, 52, 14, 58] (Robach et al. 2026); the need for linked data quality control in practice [22, 30, 29] (Harron K L 2020); advances in record linkage methodology [45, 34, 43]. By combining record linkage and diagnostics in a practical tool, it helps evaluating whether inference can be conducted on linked data.

Throughout this chapter, we motivate and illustrate the main components of FlexRL on the 2012 and 2014 waves of the Survey on Household Income and Wealth (SHIW), conducted and made available by the Bank of Italy for research purposes. As a secondary analysis of public data, no additional ethics approval or participant consent was required. § 2 introduces the notation, the modelling paradigms, the existing software and demonstrates the potential of FlexRL. § 4 presents the record linkage model implemented in FlexRL and its estimation on a subset of the SHIW. On the resulting linked data, § 5 describes the diagnostic tools FlexRL provides, and illustrates the impact of linkage errors on a socio-economic regression analysis. § 6 demonstrates the scalability of the FlexRL approach to record linkage on the SHIW. § 7 concludes.

The record linkage problem

Setup and notation

We consider two overlapping samples A\mathcal{A} and B\mathcal{B} from the same population with NAN_{\mathcal{A}} and NB≥NAN_{\mathcal{B}} \ge N_{\mathcal{A}} records. Both sources record the same KK categorical PIVs, indexed by k=1,…,Kk=1,\ldots,K with nkn_k possible values each. Index ii refers to a record in A\mathcal{A} and index jj to a record in B\mathcal{B}. For PIV kk, we denote by Gi,kAG^{\mathcal{A}}_{i,k} the registered (observed) value and by Hi,kAH^{\mathcal{A}}_{i,k} the latent true value in record i∈Ai \in\mathcal{A}. We define Gj,kBG^{\mathcal{B}}_{j,k} and Hj,kBH^{\mathcal{B}}_{j,k} analogously. Registered values differ from true values because of registration errors: missing values or mistakes (e.g. typographical errors). We write G\mathbf{G} and H\mathbf{H} for the collections of all registered and all true values. True values may also differ between the two data collections. This is the case for dynamic PIVs like postal code, whereas stable PIVs like birth year cannot change. For an entity recorded in both sources, the true values of stable PIVs are therefore identical in A\mathcal{A} and B\mathcal{B} and any disagreement between their registered values is due to registration errors. We elaborate on dynamic PIVs later.

The object of interest is the linkage matrix Δ∈{0,1}NA×NB\Delta\in\{0,1\}^{N_{\mathcal{A}} \times N_{\mathcal{B}}}, with Δi,j=1\Delta_{i,j}=1 if records i∈Ai \in\mathcal{A} and j∈Bj \in\mathcal{B} refer to the same entity. We assume that there are no duplicates within a data source. Thus, each record forms a link with at most one record in the other data source: ∑jΔi,j≤1\sum_j \Delta_{i,j} \le1 for all ii and ∑iΔi,j≤1\sum_i \Delta_{i,j} \le1 for all jj. This is the one-to-one assignment constraint, also referred to as bipartite matching [45]. We denote by γ∈(0,1)\gamma\in(0,1) the probability that a record from A\mathcal{A} forms a link with a record from B\mathcal{B}, that is, the overlap between the sources as a fraction of the smaller data set.

Since record linkage is imperfect, the linked data contain two types of error: false positives (pairs declared linked while referring to distinct entities) and false negatives (true links that are missed). The non-links in the Cartesian product A×B\mathcal{A} \times\mathcal{B} are far more numerous than the links and are rarely of interest. The useful summaries are therefore the False Discovery Proportion (FDP) among the linked data (complement of precision or positive predictive value) and the sensitivity (also called recall, complement of miss rate). The two errors affect inference differently: falsely linked pairs contaminate the inference and missed links induce sample selection [14] (Robach et al. 2026).

Modelling paradigms

Linkage methods can be grouped into four paradigms [16, 4]. Deterministic (rule based) linkage declares a link when records agree on a set of variables, exactly or up to a tolerance. Supervised methods train a classifier on pairs whose linkage status is known [35]; such labelled pairs are seldom available. The two remaining paradigms are unsupervised and probabilistic, and differ in what they model:

Paradigm 1: classification of comparison vectors

The traditional ‘Fellegi-Sunter’ approach [18, 15, 36, 54] reduces the data to comparison vectors. For each pair (i,j)∈A×B(i,j) \in\mathcal{A} \times\mathcal{B}, a vector ci,j=(ci,j,1,…,ci,j,K)\mathbf{c}_{i,j}=(c_{i,j,1},\ldots,c_{i,j,K}) records whether elements Gi,kAG^{\mathcal{A}}_{i,k} and Gj,kBG^{\mathcal{B}}_{j,k} for k=1,…,Kk=1,\ldots,K agree or how similar they are. The pairs of records are split into the set of matched MM and the set of unmatched UU. The comparison vectors follow a two components mixture:

P(ci,j)=γ m(ci,j)+(1−γ) u(ci,j),P(\mathbf{c}_{i,j})=\gamma\,m(\mathbf{c}_{i,j})+(1-\gamma)\,u(\mathbf{c}_{i,j}),
m(c)=P(c∣M),m(\mathbf{c})=P(\mathbf{c}\mid M),
u(c)=P(c∣U),(1)u(\mathbf{c})=P(\mathbf{c}\mid U), \tag*{(1)}

where γ\gamma in this paradigm is the proportion of links among all pairs. Under the assumption that the KK comparisons are independent given the linkage status, mm and uu factorise over k∈{1,…,K}k \in\{1,\ldots,K\}. The parameters are then estimated with an EM algorithm [24, 31]. Pairs are classified by thresholding the ratios m(ci,j)/u(ci,j)m(\mathbf{c}_{i,j})/u(\mathbf{c}_{i,j}) or the posterior linkage probabilities. Bayesian versions place a prior on the linkage matrix Δ\Delta to enforce the one-to-one assignment constraint [19, 45] or a prior on a partition of the records when there are more than two sources or duplicates [1]. The computations are simple and fast, but memory intensive because they operate on NA×NBN_{\mathcal{A}} \times N_{\mathcal{B}} comparison vectors. Scaling to large data sets may therefore require blocking, which compares only pairs that agree on some variables [49]. This paradigm treats pairs as independent although they share records, and the reduction to comparison vectors discards part of the information contained in the records.

Paradigm 2: hierarchical modelling

The second family models how the data were generated [51, 48, 50, 33, 34, 43]. Registered values G\mathbf{G} are seen as distorted versions of latent true values H\mathbf{H}, which define the linkage structure Δ\boldsymbol{\Delta}. The joint model reads

P(G,H,Δ)=P(G∣H)P(H∣Δ)P(Δ),(2)P(\mathbf{G}, \mathbf{H}, \boldsymbol{\Delta}) = P(\mathbf{G} \mid\mathbf{H})P(\mathbf{H} \mid\boldsymbol{\Delta})P(\boldsymbol{\Delta}), \tag*{(2)}

with a model for registration errors P(G∣H)P(\mathbf{G} \mid\mathbf{H}), a model for underlying true values P(H∣Δ)P(\mathbf{H} \mid\boldsymbol{\Delta}) and a model for the linkage structure P(Δ)P(\boldsymbol{\Delta}). Since records are never reduced to pairwise comparisons, the model can separate missing values, mistakes and changes over time, and it does not need blocking to fit in memory. However, it is more computationally demanding.

Existing software

In Python, recordlinkage [13], Splink [32] and NameMatch [26] implement Paradigm 1. We list the main R packages for the task considered here in Table 1. Paradigm 2 is only available in R through methods for entity resolution [47, 33, 34]. These methods cluster records from a single file, possibly containing duplicates, into entities.

PackageParadigmEstimationLinkage structureRecord deduplicationPIV dynamicsQuality tools
fedmatch 2.1.0 [20, 9]comparison vectorspush relabel maximum flowmany-to-manynonono
diyar 0.5.1 [38]comparison vectorsmultistage weights comparisons, fixed score level parametersclustersyesnono
reclin2 0.6.0 [56]comparison vectors, supervisedEM, machine learningone-to-one post hocnonono
fastLink 0.6.1 [17, 53]comparison vectorsEMone-to-one post hocnonono
BRL 0.1.0 [46, 45]comparison vectorsBayesian, bipartite prior, Gibbs samplerone-to-one in modelnonono
multilink 0.1.1 [1, 1]comparison vectorsBayesian, partition prior, Gibbs samplerclusters, one-to-one in modelyesnono
FlexRL 2.0.0 [41]hierarchicalstochastic EMone-to-one in modelnoyesyes

Table 1. Record linkage packages on CRAN compared with FlexRL. The FlexRL package diagnostic tools apply to all these packages.

To our knowledge, no R package implements Paradigm 2 for the linkage of two sources under the one-to-one assignment constraint or models the dynamics of PIVs over time. While the postlink package [Buk et al., 2026] develops statistical tools for post-linkage data analysis, no record linkage package provides tools to assess the quality of the linked data and their suitability for inference.

The FlexRL package is available from the Comprehensive R Archive Network (CRAN) or from the package GitHub repository alongside documentation and vignettes describing its functionality. FlexRL is also available as a module in JASP 1.0 [25]. The implementation of FlexRL relies on Rcpp (potentially parallelised with OpenMP) for the backend samplers, Matrix for the sparse linkage matrix, and other standard R packages. The diagnostic tools apply to any record linkage output in the required format, with built-in wrappers for BRL, diyar, fastLink, fedmatch, multilink and reclin2. Figure 1 shows the workflow, which follows the structure of § 4 and § 5: the data are prepared, linked with StEM() and the linked data quality is assessed with RL_diagnostics(). We elaborate on each component along this chapter.

Workflow of FlexRL

Figure 1. Workflow of FlexRL.

Demonstration

Simulation study

As a preliminary demonstration of the functionalities of FlexRL, we simulate data with the package simulate_data() function. This function synthesises two files with a given overlap, given rates of mistakes and missing values per PIV and file, and given hazard coefficients for structured PIVs, useful for exploring and comparing record linkage methods. Here, we generate two data sets of 250 and 300 records, with 200 entities common to both. Entities are represented by four independent discrete non-uniform PIVs. We distort the data in each file uniformly with 2% of mistakes and 0.5% of missing values; one of the PIVs undergoes temporal changes.

Note that this data generation process, on which all packages cited in Table 1 are evaluated, is in line with the hierarchical model FlexRL fits and represents a simple setting. The fourth PIV follows a dynamic process whose hazard is piecewise constant over time and modulated by a Gaussian covariate. We fit these changes with an exponential survival model in FlexRL, illustrating the benefits of modelling PIV dynamics, even under model misspecification, rather than treating them as registration errors. Later in this chapter, we evaluate all packages on real data for a fairer comparison.

In Table 2, we compare the methods presented in Table 1 to the FlexRL record linkage algorithm with and without modelling PIV dynamics in terms of sensitivity and FDP. When appropriate, modelling PIV dynamics improves record linkage by increasing sensitivity (more true links are recovered) while keeping the FDP under control. The original methodological article provides a more comprehensive study of the behaviour of FlexRL with and without modelling PIV dynamics [43]. Table 2 also shows the differences in computational time and in memory usage between the methods. More details on this demonstration showcasing the utility of accounting for PIV dynamics are given in the package vignette ‘Linkage and Diagnostics with FlexRL’ available through vignette('FlexRL-RL-vignette', 'FlexRL') or the online documentation.

NaiveBRLmultilinkfastLinkreclin2diyarfedmatchFlexRL (dynamics)FlexRL (no dynamics)
TP118.40 (5.13)80.40 (5.04)80.60 (5.95)101.70 (6.48)114.10 (6.38)34.90 (20.24)122.10 (6.33)114.00 (7.92)87.50 (8.30)
FP83.10 (11.02)12.30 (3.23)14.20 (4.13)43.10 (5.59)68.30 (8.82)3.30 (2.91)125.80 (6.32)25.90 (5.15)19.00 (4.67)
FN81.60 (5.13)119.60 (5.04)119.40 (5.95)98.30 (6.48)85.90 (6.38)165.10 (20.24)77.90 (6.33)86.00 (7.92)112.50 (8.30)
sensitivity0.59 (0.03)0.40 (0.03)0.40 (0.03)0.51 (0.03)0.57 (0.03)0.17 (0.10)0.61 (0.03)0.57 (0.04)0.44 (0.04)
FDP0.41 (0.03)0.13 (0.03)0.15 (0.04)0.30 (0.03)0.37 (0.03)0.08 (0.05)0.51 (0.03)0.18 (0.03)0.18 (0.04)
time (secs)0.27 (0.04)0.97 (0.05)18.57 (1.48)1.29 (0.24)0.29 (0.01)1.17 (0.06)1.44 (0.01)4.59 (0.02)3.72 (0.01)
memory (Mb)1.04 (0.05)78.60 (10.08)105.68 (10.18)24.85 (4.42)41.52 (0.18)104.31 (10.84)142.00 (1.64)102.11 (9.03)70.18 (6.42)

Table 2. Linkage performance of competing record linkage methods on simulated data for the default linkage decision rule; see § 4.3. We report average performance over 10 replications, with standard deviations in parentheses.

The results presented in this chapter are obtained on a MacBook Pro (Apple M1 Max ARM chip architecture, 32GB memory, 8 cores) with R version 4.6.1. The script is available on the GitHub repository of the package. The random seed is set to 1 in each code cell that produces output (some compared packages call set.seed() internally). The computations are robust to different operating systems (MacOS, Linux, Windows). We measure the memory usage using the R garbage collector. External allocations are therefore likely underestimated. All methods compared in this chapter rely on Rcpp or C, except for diyar which is fully implemented in R. Time and memory usage may vary depending on the operating system, hardware, and system configuration.

The hierarchical architecture

Data preparation and PIVs configuration

The record linkage model requires each PIV kk to be categorical, encoded by integers 1,…,nk1,\ldots,n_k with missing values coded as 00. The PIVs must cover a common support across the two sources. The smaller source should be labelled as AA.

As a first step in the workflow represented in Figure 1, the prepare_data() function which we will describe validates the PIVs configuration, labels the data sources and encodes the PIVs. When a unique identifier is available, like in simulation studies or in benchmark record linkage data such as the SHIW, it also returns the set of true links. This set may be used to evaluate linkage performance.

The function naive_linkage() links records that agree exactly on all PIVs (a missing value matching any value when na_match = TRUE), without enforcing the one-to-one constraint. When the true links are known, the performance of this deterministic linkage indicates the discriminative power of the PIVs and the difficulty of the task. The FDP of the ‘Naive’ linkage reflects how many records share the same combination of PIV values.

The package distinguishes different kinds of PIVs, which determine how the components of the complete data likelihood of Eq. (3) are used when fitting the model in § 4.2. Candidate pairs of records are formed from records whose latent true values agree on every PIV for which temporal changes are not modelled; whether a candidate pair is linked is then decided from its contribution to the complete likelihood. We elaborate on candidate pairs and linked pairs in § 4.2.

  • Stable: the latent true values do not change (e.g. birth year). Such PIV is used to form candidate pairs and its probability of mistake may be bounded (for example by 0.10) for algorithm stability [43].

  • Dynamic: the true values may change (e.g. postal code between two survey waves):

    • Flexible: changes cannot be modelled, typically because registration dates are not available. The PIV is still used to form candidate pairs, but its probability of mistake should not be bounded, so that the registration model absorbs both mistakes and changes.

    • Structured: changes are modelled by the survival function of Eq. (4). Such PIV is not used to form candidate pairs. Registration dates are needed in both sources and covariates for the hazard may be supplied. Mistakes cannot be distinguished from temporal changes, so the probability of mistake should be fixed (for example at 0.02), unless enough linked pairs have time gaps near zero, where changes are not plausible and mistakes can be estimated without confusion. Estimating the changes also requires variation in the time gaps among linked pairs.

To illustrate the usage of the FlexRL package throughout this chapter, we consider the Emilia-Romagna regional subset of the SHIW data. The record linkage task can be validated in this example where the linkage structure can be reconstructed. After preparation, the 2012 and 2014 waves cover 1,479 and 1,524 individuals, with 898 individuals observed in both waves. In § 5, we analyse a regression of the expected age of retirement in 2014 on the expected age of retirement in 2012.

To link the two survey waves, we use the following variables: year and region of birth, sex, marital status, working status and branch of activity. The strong association between working status and branch of activity violates the PIV independence assumption of the method and may degrade the record linkage performance [43]. We thus combine these variables into a single variable ‘occupation’. Although marital status and occupation are ‘dynamic’ PIVs, we do not have enough information to model their temporal changes and set them as ‘flexible’ in the configuration:

PIVs_config <- list(BirthYear     = list(dynamics       = "stable",
                                         bound_mistakes  = c(0.10, 0.10),
                                         fix_mistakes    = c(NA, NA)),
                    Sex           = list(dynamics       = "stable",
                                         bound_mistakes  = c(0.10, 0.10),
                                         fix_mistakes    = c(NA, NA)),
                    MaritalStatus = list(dynamics       = "flexible",
                                         bound_mistakes  = c(NA, NA),
                                         fix_mistakes    = c(NA, NA)),
                    BirthRegion   = list(dynamics       = "stable",
                                         bound_mistakes  = c(0.10, 0.10),
                                         fix_mistakes    = c(NA, NA)),
                    Occupation    = list(dynamics       = "flexible",
                                         bound_mistakes  = c(NA, NA),
                                         fix_mistakes    = c(NA, NA)))

PIVs <- names(PIVs_config)

We prepare the two data sources for the StEM() record linkage algorithm with the prepare_data() function, which takes as inputs the data sources and their labels df2012, df2014, ‘2012’, ‘2014’; the PIVs configuration PIVs_config; a logical same_mistakes describing whether the parameters fitting the probability of mistakes among PIVs should be the same between data sources (TRUE when the data are sampled from a same source population); a potential record identifier variable uniq_id when available; a logical restrict_support_intersection to harmonise the support of the PIVs in the observed data. In the application, records with birth years outside of the range shared by the sources are excluded.

prep_data <- prepare_data(df2012, df2014, "2012", "2014", PIVs_config,
                          same_mistakes = TRUE, uniq_id = "ID")

‘2014’ is the larger source, saved as B; ‘2012’ saved as A.

This function outputs a list suited for StEM(), where data sources have been encoded and labelled as encodedA and encodedB. It additionally includes the number of unique values per PIV n_values, the validated PIVs_config, the logical same_mistakes and a data set of indices of true links true_pairs when uniq_id was provided. It alerts the user to potential issues, including inconsistent configurations, strongly associated PIVs and values outside the common support. For instance, the following warnings appear on the Emilia-Romagna subsets of the 2012 and 2014 waves of the SHIW data:

PIV ‘BirthYear’: values out of the common support in data1 [1920, 1912]
in data2 [2013, 2014].
PIVs ‘Occupation’ and ‘Sex’ are associated (Cramer’s V = 0.32 in data1,
0.35 in data2); FlexRL assumes conditional independence between PIVs,
consider merging or dropping one.

Among the true links, known in this application, there are some registration errors or temporal changes in the PIVs. As a consequence the true links do not necessarily carry the exact same information in their records. With the RL_agreement function, the user can see the proportion of agreements among pairs of records (for example among linked pairs of records or, in the case presented below, among true links).

BirthYear   Sex   MaritalStatus   BirthRegion   Occupation
   0.99    1.00       0.98           0.99          0.82

In Table 3, we compare the record linkage methods mentioned in Table 1 on this regional subset of the SHIW data. Memory usage is of the same order of magnitude on these relatively small data sets for all methods. The figures and tables based on the SHIW data reflect a single random realisation. They depend on the seed we use in the script and on the seed some other packages use internally.

NaiveBRLmultilinkfastLinkreclin2diyarfedmatchFlexRL
TP710.00158.00160.00467.00721.0020.00510.00297.00
FP3,494.0024.0072.00850.00339,232.000.00969.00103.00
FN188.00740.00738.00431.00177.00878.00388.00601.00
sensitivity0.790.180.180.520.800.020.570.33
FDP0.830.130.310.651.000.000.660.26
time (secs)0.3717.781,118.0433.690.8411.048.8822.57
memory (Mb)8.30523.201,901.902,436.201,180.20699.50213.20263.20

Table 3. Comparison of record linkage methods on the SHIW in Emilia-Romagna (for the default linkage decision rule, see § 4.3).

StEM estimation

The FlexRL hierarchical model for record linkage adapts Eq. (2) to two sources under the one-to-one assignment constraint. For non-linked records, the true values are independent draws from the population distribution. For a linked pair (i,j)(i,j), Hj,kB\mathbf{H}^{B}_{j,k} is a copy of Hi,kA\mathbf{H}^{A}_{i,k} except when kk is a dynamic PIV, whose values may have changed between the two registrations. The registered values are versions of the true values distorted by registration errors.

With parameters θ=(γ,η,α,ϕ)\boldsymbol{\theta}=(\gamma,\eta,\alpha,\phi), the complete data likelihood factorises as

Lθ(GA,GB,HA,HB,Δ)=Lϕ,α(GA∣HA)Lϕ,β(GB∣HB)⏟registration errorsLα,η(HB∣HA,Δ)Lη(HA)⏟changes over time and true valuesLγ(Δ)⏟linkage(3)\mathcal{L}_{\theta}\left(\mathbf{G}^{A},\mathbf{G}^{B},\mathbf{H}^{A},\mathbf{H}^{B},\Delta\right) = \underbrace{\mathcal{L}_{\phi,\alpha}\left(\mathbf{G}^{A}\mid\mathbf{H}^{A}\right) \mathcal{L}_{\phi,\beta}\left(\mathbf{G}^{B}\mid\mathbf{H}^{B}\right)}_{\text{registration errors}} \underbrace{\mathcal{L}_{\alpha,\eta}\left(\mathbf{H}^{B}\mid\mathbf{H}^{A},\Delta\right) \mathcal{L}_{\eta}\left(\mathbf{H}^{A}\right)}_{\text{changes over time and true values}} \underbrace{\mathcal{L}_{\gamma}\left(\Delta\right)}_{\text{linkage}} \tag*{(3)}

The parameter ϕ\phi governs the possible distortion mechanisms (mistakes, missing values) at the source of the differences between registered and latent true values. The parameter α\alpha shapes the model for temporal changes between latent true values in data sources depending on the chosen ‘structure’ (see the different survival models for Eq. (4)). The parameter η\eta characterises the distributions of the PIVs. The parameter γ\gamma represents the probability that an observation in A\mathcal{A} forms a link with an observation in B\mathcal{B}. Each component of Eq. (3) and each parameter is developed in the original methodological paper [43].

Notably, the probability that true values have not changed between data collections is modelled with a survival function. By default, the package uses an exponential model with proportional hazards. For a dynamic structured PIV kk whose changes are modelled and a linked pair (i,j)(i,j) with time gap ti,j=∣tjB−tiA∣t_{i,j} = \lvert t^B_j - t^A_i \rvert:

P(Hj,kB=Hi,kA∣ti,j,xi,j,Δi,j=1)=Sαk(xi,j,ti,j)=exp⁡{−exp⁡(xi,jTαk)ti,j},(4)P(H^B_{j,k}=H^A_{i,k}\mid t_{i,j},\boldsymbol{x}_{i,j},\Delta_{i,j}=1)=S_{\alpha_k}(\boldsymbol{x}_{i,j},t_{i,j})=\exp\left\{-\exp(\boldsymbol{x}_{i,j}^{T}\alpha_k)t_{i,j}\right\}, \tag*{(4)}

where xi,j\boldsymbol{x}_{i,j} contains an intercept and, optionally, covariates from either data source that explain the hazard of change (for example age or income for a relocation) and αk\alpha_k contains the log hazard coefficients. The user can choose another survival model, such as a Weibull, a Gompertz, a piecewise constant hazard or any custom model.

The likelihood of the observed data is obtained by summing Eq. (3) over all values of the latent H\mathbf{H} and Δ\Delta. FlexRL maximises it with the StEM() function, a Stochastic Expectation Maximisation (StEM) algorithm [7]. In a classical EM, the E-step computes the conditional expectation of the complete likelihood given the observed data. In the stochastic version, this expectation is approximated by Monte Carlo: a Gibbs sampler simulates the latent variables given the current parameters. The averages of the summary statistics collected after burn-in estimate the conditional expectation from which the M-step derives the parameter updates. Algorithm 1 summarises the procedure, detailed hereafter.

Algorithm Stochastic EM for record linkage.

Input: encoded sources GA,GB\mathcal{G}^{A},\mathcal{G}^{B}; PIV configuration; NStEMN_{\mathrm{StEM}} iterations (burn-in NStEM0N^{0}_{\mathrm{StEM}}); NGibbsN_{\mathrm{Gibbs}} iterations (burn-in NGibbs0N^{0}_{\mathrm{Gibbs}})

Initialise θ=(γ,η,α,ϕ)\theta=(\gamma,\eta,\alpha,\phi)

for StEM iter =1,…,NStEM=1,\ldots,N_{\mathrm{StEM}} do

&nbsp;&nbsp;&nbsp;&nbsp;Δ←0\Delta\leftarrow0

&nbsp;&nbsp;&nbsp;&nbsp;for Gibbs iter =1,…,NGibbs=1,\ldots,N_{\mathrm{Gibbs}} do ▷ stochastic E-step

&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;draw HA,HB∣GA,GB,Δ,θ,x,t\mathbf{H}^{A},\mathbf{H}^{B}\mid\mathcal{G}^{A},\mathcal{G}^{B},\Delta,\theta,\mathbf{x},\mathbf{t}

&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;form candidate pairs (i,j)(i,j) such that Hi,kA=Hj,kBH^{A}_{i,k}=H^{B}_{j,k} for all stable or flexible kk

&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;for each non-linked candidate pair: propose to link

&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;for each linked candidate pair: propose to break the link

&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;accept each proposal with the likelihood of the model

&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;if Gibbs iter >NGibbs0>N^{0}_{\mathrm{Gibbs}} then

&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;accumulate summary statistics:

&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;number of linked pairs,

&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;latent true values distribution,

&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;temporal changes in linked (Hi,kA=Hj,kB,x,t)(H^{A}_{i,k}=H^{B}_{j,k},\mathbf{x},\mathbf{t}),

&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;registration errors in linked and non-linked (G=H)(\mathbf{G}=\mathbf{H})

&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;end if

&nbsp;&nbsp;&nbsp;&nbsp;end for

&nbsp;&nbsp;&nbsp;&nbsp;average the accumulated summary statistics ▷ estimate E[H,Δ∣G;θ]\mathbb{E}[\mathbf{H},\Delta\mid\mathbf{G};\theta]

&nbsp;&nbsp;&nbsp;&nbsp;update and store θ=(γ,η,α,ϕ)\theta=(\gamma,\eta,\alpha,\phi) ▷ M-step

end for

θ^←\hat{\theta}\leftarrow average of θ\theta over StEM iter >NStEM0>N^{0}_{\mathrm{StEM}}

run further Gibbs iterations at θ^\hat{\theta} to build D^\hat{D}:

D^i,j←\hat{D}_{i,j}\leftarrow proportion of iterations in which (i,j)(i,j) is linked

Initialisation

The probability γ\gamma is drawn uniformly in (0.2,0.8)(0.2, 0.8), the distribution of each PIV kk, ηk\eta_k, are uniform over the nkn_k values, the probabilities of mistakes are drawn in (0.03,0.20)(0.03, 0.20) and the probabilities of missing values are set to the observed proportions. For structured PIVs, the log baseline hazard is drawn between 0.01 and 1 per time unit and the covariate coefficients near 0. The user can supply other starting values.

Stochastic E-step

Given θ\theta, the Gibbs sampler alternates two blocks. First, the latent true values are drawn. For a non-linked record Hi,kAH^A_{i,k} or Hj,kBH^B_{j,k} is drawn from ηk\eta_k weighted by the registration model. For a linked pair (Hi,kA,Hj,kB)(H^A_{i,k}, H^B_{j,k}) is drawn jointly among the nk×nkn_k \times n_k possible values, from ηk\eta_k weighted by the registration model and the survival model.

Second, the linkage is updated. Candidate pairs are formed from matching sampled latent true values: records i∈Ai \in\mathcal{A} and j∈Bj \in\mathcal{B} are candidates when Hi,kA=Hj,kBH^A_{i,k} = H^B_{j,k} for every non-structured PIV kk (stable or flexible). For each candidate pair (i,j)(i,j), we evaluate the log likelihood contribution to Eq. (3) when the records are linked, log⁡Li,jlinked\log\mathcal{L}^{\mathrm{linked}}_{i,j}, and when they are not, log⁡LiA+log⁡LjB\log\mathcal{L}^A_i + \log\mathcal{L}^B_j. A sequential sweep over the candidate pairs then proposes to create or break links. A proposal is accepted by comparing the proposed ‘new’ state of Eq. (3) to the current ‘old’ state with probability Lnew/(Lold+Lnew)\mathcal{L}_{\mathrm{new}}/(\mathcal{L}_{\mathrm{old}}+\mathcal{L}_{\mathrm{new}}); the two states differ by log⁡Li,jlinked−log⁡LiA−log⁡LjB\log\mathcal{L}^{\mathrm{linked}}_{i,j} - \log\mathcal{L}^A_i - \log\mathcal{L}^B_j and by the contributions from changes in the linkage matrix.

Over the Gibbs iterations, the sampler stochastically explores the latent space of possible linkage matrices and PIVs’ true values. At each iteration after burn-in we store the number of pairs that were linked, the distribution of latent true values, the temporal changes (disagreements between the linked records latent true values) and the registration errors (disagreements between G and H)—summary statistics required for the M-step.

Restricting the sweep to candidate pairs keeps the algorithm light in memory: likelihood contributions are computed only for these pairs and for the remaining non-linked records in each data source, which together represent the estimated unique entities in the population.

M-step

The summary statistics averaged over the Gibbs draws after burn-in give the parameter updates. Thanks to the factorisation of Eq. (3), γ\gamma, η\eta and ϕ\phi have separate closed form updates [43], and α\alpha is obtained by numerical optimisation. Optional bounds on the probabilities of a mistake are applied at this stage. A warning is displayed when a bound is reached in many iterations, suggesting that the bound is too tight or that the PIV is dynamic.

Linkage scores and decision rule

Based on the StEM chains of estimates, we build an estimate of the set of parameters, θ^\hat{\theta}, by averaging the post burn-in elements in the sequence. These are used to sample further linkage matrices that are averaged to return a probabilistic estimate of Δ\Delta as a final output, namely D^\hat{D}, whose entries D^i,j\hat{D}_{i,j} are the posterior probabilities for the pair (i,j)(i,j) to be linked, which we call the linkage scores. A decision rule on each entry ξ∈[0.5,1]\xi\in[0.5, 1] then defines the final set of linked data D={(i,j):D^i,j>ξ}\mathcal{D} = \{(i,j) : \hat{D}_{i,j} > \xi\}. Since the scores of a record sum to at most one, a threshold of at least 0.5 ensures the one-to-one assignment [51] (Sadilne 2017). The choice of ξ\xi is the subject of the upcoming Section 4.

Note that probabilistic record linkage methods estimate Δ\Delta in comparable ways, by classifying pairs according to their linkage scores. Although these scores may be on different scales (e.g. likelihood ratios, log odds, posterior probabilities), they can be mapped to a probability scale. Hereafter, we take linkage scores to be posterior linkage probabilities in [0,1][0,1] and treat all the methods listed in Table 1 in the same way. Accordingly, the default decision rule is ξ=0.5\xi= 0.5. The ‘FDP versus linkage decision rule’ curves (see Figure 3) provide a more comprehensive assessment of record linkage performance.

Four-panel plot. Top panel: “FDP versus linkage decision rule”; y-axis “estimated FDP”; right y-axis “number of links”; x-axis “decision rule threshold”. Legend: “links (RL)”, “real links (augm. RL)”, “links (augm. RL)”, “FDP score (RL)”, “FDP synth (augm. RL)”, “FDP score (augm. RL)”, “true FDP”. Second panel: “Multivariate discrepancy”; y-axis “MMD”; x-axis “decision rule threshold”; legend: “linked A vs. A”, “linked B vs. B”. Bottom-left panel: “linked A vs. A”; y-axis “SMD”; x-axis “decision rule threshold”. Bottom-right panel: “linked B vs. B”; y-axis “SMD”; x-axis “decision rule threshold”; legend: “MaritalStatus_3”, “MaritalStatus_2”, “MaritalStatus_4”, “BirthRegion_7”, “BirthRegion_13”, “BirthRegion_15”, “BirthRegion_17”, “BirthRegion_18”, “Occupation_7”, “Occupation_25”, “BirthYear”.

Figure 3. Estimated FDP and discrepancy measures against the linkage decision rule threshold obtained with FlexRL RL_diagnostics() on the SHIW in Emilia-Romagna.

Record linkage with StEM()

StEM() runs Algorithm 1. Its main arguments are:

  • data: a list, typically the output of prepare_data(), with the encoded sources encodedA and encodedB, the number of distinct values of each PIV n_values (excluding missing values), a logical indicating whether both sources share the registration error parameters same_mistakes, and the PIVs configuration PIVs_config.

  • StEM_iter, StEM_burnin, gibbs_iter, gibbs_burnin: the numbers of iterations of the outer StEM algorithm and of the inner Gibbs sampler, and their burn-in periods. gibbs_burnin = 0 sets the burn-in automatically: the sampler runs until the number of links changes by fewer than 10 for 10 consecutive iterations, with a maximum of 500 iterations.

  • threads: the number of OpenMP threads used by the Rcpp samplers, default 4. The argument is ignored on systems where the package is built without OpenMP.

  • model_dynamics: the survival model for structured PIVs, by default the exponential model of Eq. (4). It is only used if a PIV is declared structured in PIVs_config.

  • n_post_sample: the number of posterior draws used to compute the linkage scores.

  • Starting values for γ\gamma and for the agreement probabilities 1−ϕ1 - \phi can be supplied.

  • For long runs, new_directory and save_info—where save the environment after each StEM iteration, and music_on = TRUE plays a short tune when the algorithm finishes running.

The function returns Delta, a data frame with columns i (indices in A\mathcal{A}), j (indices in B\mathcal{B}) and x (scores D^i,j\hat{D}_{i,j}) for every pair with a positive score, and the four chains: gamma (a vector of length StEM_iter), eta (one StEM_iter ×\times nkn_k matrix per PIV), alpha (one matrix per structured PIV, with StEM_iter rows and one column per parameter of the survival model) and phi (one StEM_iter ×\times 4 matrix per PIV: agreement in A\mathcal{A}, in B\mathcal{B}, missing in A\mathcal{A}, in B\mathcal{B}).

We run the FlexRL StEM algorithm for record linkage as follows:

fit <- StEM(data = prep_data)

Computational aspects

The two latent variable samplers are implemented with Rcpp. To draw the true values, sampleH() goes through the nk2n_k^2 joint values once per PIV kk for linked pairs and through the nkn_k values once per PIV kk for non-linked records. sampleD() performs the sequential sweep over candidate pairs, recording the current linkage in two vectors that give, for each record of AA and of BB, the index of the record it is linked to (or a negative value when not linked). Reading and updating the linkage takes a fixed number of operations, so the sweep is linear in the number of candidate pairs and the memory is linear in the number of records NA+NBN_A + N_B rather than in NA×NBN_A \times N_B.

Candidate pairs are found by hashing the pattern of sampled true values on the non-structured PIVs, an operation linear in NA+NBN_A + N_B. Their contributions to the complete data likelihood are computed in a single pass over the records and the candidate pairs, implemented in Rcpp. Built-in survival models are evaluated in Rcpp; custom survival functions keep their R implementation.

The single pass over the candidate pairs likelihood contributions can be parallelised over OpenMP threads, each processing a disjoint set of candidates. Without OpenMP the code runs sequentially. All random draws occur upstream, so the threads never call the R random number generator. Results are therefore identical regardless of the number of threads, and set.seed() ensures reproducibility.

Running time grows with the number of records; the number of candidate pairs, larger when PIVs have few levels; the number of PIVs and their numbers of levels; the number of structured PIVs, whose survival model is estimated at each iteration; the number of iterations; and decreases with the number of threads.

Inference on linked data

Impact of linkage errors on linear regression

Suppose we are interested in estimating the association between XX recorded in A\mathcal{A} and YY recorded in B\mathcal{B}, for instance through the slope β\beta of the linear regression Y=Xβ+εY = X\beta+ \varepsilon, with E[ε]=0\mathbb{E}[\varepsilon] = 0 and E[Xε]=0\mathbb{E}[X\varepsilon] = 0, so that

β^=Cov⁡(X,Y)/Var⁡(X).(5)\hat{\beta} = \operatorname{Cov}(X,Y) / \operatorname{Var}(X). \tag*{(5)}

The target is β\beta in the population of entities present in both sources. The analyst only observes the linked data D={(i,j);D^i,j>ξ}\mathcal{D} = \{(i,j) ; \hat{D}_{i,j} > \xi\} with XiX_i from A\mathcal{A} and YjY_j from B\mathcal{B}. YjY_j equals YiY_i for truly linked pairs but belongs to another entity for falsely linked records. Among the linked data, a proportion 1−FDP1 - \mathrm{FDP} (that may in reality depend on XX) carries the correct YY and a proportion FDP\mathrm{FDP} carries a spurious one. Below we denote by Δ^\hat{\Delta} the estimator of Δ\Delta, with Δ^i,j=1\hat{\Delta}_{i,j} = 1 for pairs (i,j)(i,j) in D\mathcal{D}. Truly linked pairs of records (true positives, TP) are thus denoted {Δ^=1,Δ=1}\{\hat{\Delta} = 1, \Delta= 1\} and falsely linked pairs of records (false positives, FP) {Δ^=1,Δ=0}\{\hat{\Delta} = 1, \Delta= 0\}. Heuristically,

E[Y∣X,Δ^=1]=(1−FDP)⋅E[Y∣X,Δ^=1,Δ=1]+FDP⋅E[Y∣X,Δ^=1,Δ=0]=(1−FDP)⋅ETP[Y∣X]+FDP⋅EFP[Y∣X],(6)\mathbb{E}[Y \mid X,\hat{\Delta} = 1] = (1-\mathrm{FDP}) \cdot\mathbb{E}[Y \mid X,\hat{\Delta} = 1,\Delta= 1] \\ {}+ \mathrm{FDP} \cdot\mathbb{E}[Y \mid X,\hat{\Delta} = 1,\Delta= 0] \\ {}= (1-\mathrm{FDP}) \cdot\mathbb{E}_{TP}[Y \mid X] + \mathrm{FDP} \cdot\mathbb{E}_{FP}[Y \mid X], \tag*{(6)}

where ETP\mathbb{E}_{TP} and EFP\mathbb{E}_{FP} denote expectations among truly and falsely linked pairs.

Let us recast β^\hat{\beta} accordingly. If XX has the same mean and variance among all data in A\mathcal{A}, then β^\hat{\beta} is a mixture of β^TP\hat{\beta}_{TP} and β^FP\hat{\beta}_{FP}, defined as in (5) for the subsets of TP and FP,

β^=(1−FDP)⋅β^TP+FDP⋅β^FP.(7)\hat{\beta} = (1-\mathrm{FDP}) \cdot\hat{\beta}_{TP} + \mathrm{FDP} \cdot\hat{\beta}_{FP}. \tag*{(7)}

Linkage errors are often assumed to occur at random, in which EFP[Y∣X]=EFP[Y]\mathbb{E}_{FP}[Y \mid X] = \mathbb{E}_{FP}[Y] and β^FP=0\hat{\beta}_{FP} = 0. However, the PIVs, often demographic, are likely to be associated with variables of interest in the analysis, so it is more realistic to consider an association between XX and YY among falsely linked pairs.

The estimation error of the analysis can be decomposed into two parts:

β^−β=(β^TP−β)⏟sampling bias+(β^−β^TP)⏟contamination bias.(8)\hat{\beta} - \beta= \underbrace{(\hat{\beta}_{TP} - \beta)}_{\text{sampling bias}} + \underbrace{(\hat{\beta} - \hat{\beta}_{TP})}_{\text{contamination bias}}. \tag*{(8)}

Missed links may cause sample selection and falsely linked data may cause measurement error in the linked variables [58, 14] (Robach et al. 2026).

Linkage decision rule trade-off

Raising the threshold ξ\xi on the linkage scores reduces the number of falsely linked pairs, which lowers the FDP and hence the contamination bias. Falsely linked data would distort the signal in proportion to the FDP. Indeed, from Eq. (7) we derive:

β^−β^TP⏟contamination bias=FDP(β^FP−β^TP).(9)\underbrace{\widehat{\beta}-\widehat{\beta}_{TP}}_{\text{contamination bias}}=\mathrm{FDP}\left(\widehat{\beta}_{FP}-\widehat{\beta}_{TP}\right). \tag*{(9)}

However, raising the threshold ξ\xi also selects fewer and more atypical records, which may increase the sampling bias due to the implicit record linkage selection. Indeed, the chance of being linked may be related to XX; in the context of linear regression, this yields:

β^TP−β⏟sampling bias=Cov⁡TP(X,βX+ε)Var⁡TP(X)−β=Cov⁡TP(X,ε)Var⁡TP(X).(10)\underbrace{\widehat{\beta}_{TP}-\beta}_{\text{sampling bias}}=\frac{\operatorname{Cov}_{TP}(X,\beta X+\varepsilon)}{\operatorname{Var}_{TP}(X)}-\beta=\frac{\operatorname{Cov}_{TP}(X,\varepsilon)}{\operatorname{Var}_{TP}(X)}. \tag*{(10)}

These aspects motivate the complementary tools we develop in FlexRL to quantify these two biases. Estimating the FDP (see § 5.2) and evaluating the divergence of the linked sample from the data sources (see § 5.3) over a range of values of ξ\xi may help users appraise this trade-off.

False discovery proportion

For the decision rule ξ∈[0.5,1)\xi\in[0.5,1), the FDP is FDP(ξ)=FP(ξ)/{TP(ξ)+FP(ξ)}\mathrm{FDP}(\xi)=\mathrm{FP}(\xi)/\{\mathrm{TP}(\xi)+\mathrm{FP}(\xi)\} where TP(ξ)\mathrm{TP}(\xi) and FP(ξ)\mathrm{FP}(\xi) are the numbers of truly and falsely linked pairs. In real data applications this quantity is unknown. Linkage errors can be estimated from the scores of the linkage model [3, 40, 17], from synthetic data [30, 42] or from a separate model under conditions on the distribution of the number of records per entity [5, 11, 12]. FlexRL implements the first two estimation procedures. We focus on the FDP because it drives the contamination bias of Eq. (9); the sampling bias is addressed by the diagnostics of § 5.3. The estimators are provided as functions of ξ\xi, together with the linked sample size available for inference, so that the user can choose a threshold controlling the estimated FDP (and hopefully, the true underlying FDP).

Score based estimator

If the linkage model is well calibrated, D^i,j\widehat{D}_{i,j} should represent the probability that the pair (i,j)(i,j) is a true link, and the expected number of truly linked pairs among linked data is ∑(i,j)D^i,j1{D^i,j>ξ}\sum_{(i,j)}\widehat{D}_{i,j}1\{\widehat{D}_{i,j}>\xi\}. Therefore

FDP^(ξ)=1−∑(i,j)D^i,j1{D^i,j>ξ}∑(i,j)1{D^i,j>ξ}.(11)\widehat{\mathrm{FDP}}(\xi)=1-\frac{\sum_{(i,j)}\widehat{D}_{i,j}1\{\widehat{D}_{i,j}>\xi\}}{\sum_{(i,j)}1\{\widehat{D}_{i,j}>\xi\}}. \tag*{(11)}

This estimator comes at no extra cost for any method that returns linkage scores, but it is only as good as the linkage model.

Notably, the calibration of the scores and the record linkage performance are affected by missing values and mistakes, the size of the overlap between the sources, the number of entities in the population, the discriminative power of the PIVs, their changes over time, and their marginal and joint distributions. The literature discusses the strengths and limitations of different record linkage paradigms in detail, including comparisons of FlexRL and BRL across a range of configurations [43 a].

Synthetic data estimator

An estimator that does not rely on linkage scores can be computed with synthetic data [42]. Source B\mathcal{B} is augmented with NsynthN_{\mathrm{synth}} synthetic records, drawn from a generative model fitted on the PIVs of B\mathcal{B}—for example arf [59], synthpop [37] or mice [55]. Synthetic records by definition do not form a link in A\mathcal{A}, so synthetic links can represent false positives when links and non-links have similar distributions [42]. Then FPsynth(ξ)\mathrm{FP}_{\mathrm{synth}}(\xi) is rescaled by NB/NsynthN_{\mathcal{B}}/N_{\mathrm{synth}}, which follows from equating FP(ξ)/(NA⋅NB)\mathrm{FP}(\xi)/(N_{\mathcal{A}}\cdot N_{\mathcal{B}}) with FPsynth(ξ)/(NA⋅Nsynth)\mathrm{FP}_{\mathrm{synth}}(\xi)/(N_{\mathcal{A}}\cdot N_{\mathrm{synth}}). As for the score based estimator, TP(ξ)+FP(ξ)\mathrm{TP}(\xi)+\mathrm{FP}(\xi) corresponds to the number of (real) linked pairs Nreal(ξ)=∑(i,j)1{Di,j>ξ}N_{\mathrm{real}}(\xi)=\sum_{(i,j)}1\{D_{i,j}>\xi\}:

FDP^synth(ξ)=FPsynth(ξ)⋅NB/NsynthNreal(ξ).(12)\widehat{\mathrm{FDP}}_{\mathrm{synth}}(\xi)=\frac{\mathrm{FP}_{\mathrm{synth}}(\xi)\cdot N_{\mathcal{B}}/N_{\mathrm{synth}}}{N_{\mathrm{real}}(\xi)}. \tag*{(12)}

The procedure is repeated over several synthetic draws and the results are averaged. The estimator does not depend on the linkage model, but it depends on the quality of the data synthesis.

When the two estimators disagree, a practitioner should rely on the one whose assumptions are more plausible. The score based estimator relies on the correct specification of the linkage model, including its specific assumptions. It reflects the performance the model would achieve under correct specification. The synthetic data estimator relies on the fidelity of the synthesiser and on the assumption that links and non-links have similar distributions. Earlier work assessed the robustness of both estimators to the degree of overlap between the data sources, the discriminative power of the PIVs and differences between the PIV distributions of true links and non-links [42], but not to deviations from PIV independence. In the SHIW in Emilia-Romagna application, the PIV independence is violated (occupation and sex are associated; see the warning in § 4.1). For models assuming independent PIVs, such as FlexRL and Fellegi-Sunter mixture models (see Eq. (1)), we expect the score based estimator to be biased, as its validity rests on the violated assumption. The synthetic data estimator though, should remain valid provided the synthesiser captures the dependencies among PIVs. This is in line with what we observe in the output of Figure 4.

Example of output from the print() method on FlexRL RL_diagnostics class

Figure 4. Example of output from the print() method on FlexRL RL_diagnostics class, at threshold 0.86, on the SHIW in Emilia-Romagna.

Representativeness of the linked sample

As explained in § 5.1, the linked sample may differ from the data sources in ways the analyst cannot see [14, 58] (Robach et al. 2026). In particular, we appraise the sampling bias of Eq. (10) by comparing the distributions of variables of interest between the linked data (‘selected’ through record linkage) and the original data sources. For any decision rule ξ∈[0.5,1)\xi\in[0.5, 1), FlexRL compares each source with its linked subset using the following measures, computed on sets of variables VA\mathcal{V}_{\mathcal{A}} and VB\mathcal{V}_{\mathcal{B}} available in sources A\mathcal{A} and B\mathcal{B} respectively:

  • Agreement rates: the proportion of linked pairs whose records agree on vv, for each variable v∈VA∩VBv \in\mathcal{V}_{\mathcal{A}} \cap\mathcal{V}_{\mathcal{B}} (such as the PIVs).

  • Standardised Mean Differences (SMD): (mean⁡(vlinked)−mean⁡(vsource))/sd⁡(vsource)(\operatorname{mean}(v_{\mathrm{linked}}) - \operatorname{mean}(v_{\mathrm{source}}))/\operatorname{sd}(v_{\mathrm{source}}), for a continuous variable vv in a given set; the same quantity is computed for the indicator of each level when vv is categorical.

  • Overlap of supports (support IoU): the Intersection over Union (IoU) of the sets of values taken in the linked sample and in the source, for a variable vv in a given set.

  • Maximum Mean Discrepancy (MMD): a multivariate measure of the difference between the linked sample and the source, computed jointly over a given set of variables, with a Gaussian kernel and the median heuristic for the bandwidth [21].

These quantities are computed over a grid of values of ξ\xi. They are built on the declared pairs and, when available, their linkage scores; they therefore apply to any linkage method.

In Table 4 we proceed with the demonstration of Section 2.4 showcasing the functionalities of FlexRL on a simulation study. The synthetic data FDP estimates (‘est. FDP* synth’) are not reliable for fedmatch and FlexRL (dynamics), possibly because of the specific linkage procedures these packages implement. For FlexRL (dynamics), the cause is likely the misspecified PIV dynamics model, or the added difficulty of estimating the FDP under modelling of PIV dynamics, since the estimate for FlexRL (no dynamics) is reliable.

NaïveBRLmultilinkfastLinkreclin2d y i a rfedmatchFlexRL (dynamics)FlexRL (no dynamics)
TP118.40 (5.13)80.40 (5.04)80.60 (5.95)101.70 (6.48)114.10 (6.38)34.90 (20.24)122.10 (6.33)114.00 (7.92)87.50 (8.30)
FP83.10 (11.02)12.30 (3.23)14.20 (4.13)43.10 (5.59)68.30 (8.82)3.30 (2.91)125.80 (6.32)25.90 (5.15)19.00 (4.67)
FN81.60 (5.13)119.60 (5.04)119.40 (5.95)98.30 (6.48)85.90 (6.38)165.10 (20.24)77.90 (6.33)86.00 (7.92)112.50 (8.30)
sensitivity0.59 (0.03)0.40 (0.03)0.40 (0.03)0.51 (0.03)0.57 (0.03)0.17 (0.10)0.61 (0.03)0.57 (0.04)0.44 (0.04)
FDP0.41 (0.03)0.13 (0.03)0.15 (0.03)0.30 (0.03)0.37 (0.03)0.08 (0.05)0.51 (0.03)0.18 (0.03)0.18 (0.04)
est. FDP scoreNA0.19 (0.02)NA0.37 (0.02)0.44 (0.03)NA0.10 (0.01)0.20 (0.01)0.18 (0.02)
est. FDP* scoreNA0.21 (0.02)NA0.39 (0.02)0.43 (0.07)NA0.10 (0.01)0.21 (0.01)0.19 (0.02)
est. FDP* synthNA0.18 (0.10)0.20 (0.05)0.33 (0.07)0.45 (0.14)0.14 (0.07)0.08 (0.03)0.37 (0.11)0.27 (0.08)
max SMDNA0.22 (0.05)0.22 (0.06)0.17 (0.03)0.24 (0.04)0.68 (0.31)0.15 (0.03)0.17 (0.04)0.22 (0.04)
min support IoUNA0.80 (0.01)0.80 (0.01)0.80 (0.01)0.80 (0.01)0.76 (0.10)0.82 (0.03)0.82 (0.03)0.81 (0.02)
max MMDNA0.01 (0.00)0.01 (0.00)0.01 (0.00)0.01 (0.00)0.08 (0.05)0.01 (0.00)0.01 (0.00)0.01 (0.00)
time (secs)0.27 (0.04)0.97 (0.05)18.57 (1.48)1.29 (0.24)0.29 (0.01)1.17 (0.06)1.44 (0.01)4.59 (0.02)3.72 (0.01)
memory (Mb)1.04 (0.05)78.60 (10.08)105.68 (10.18)24.85 (4.42)41.52 (0.18)104.31 (10.84)42.14 (2.59)102.11 (9.03)70.18 (6.42)

Table 4. Linkage performance of competing record linkage methods on simulated data for the default decision rule ξ=0.5\xi= 0.5, with linked data quality assessed through FlexRL. The * symbol denotes the record linkage augmented task, see § 5.2. We report average performance over 10 replications, with standard deviations in parentheses.

This short simulation study demonstrates the potential of the FlexRL package. Since FlexRL provides linked data quality diagnostics for any record linkage method, we report three estimates of the FDP along with measures of the divergence of the linked sample from the data sources. § 5.2 presented the FDP estimators implemented in the package and § 5.3 discussed the measures of divergence. The FDP estimators are averaged over 5 procedures in each of the 10 replications.

Diagnostics

RL_diagnostics() gathers the tools of § 5.2 and § 5.3, summarised in Table 5, and returns an S3 object of class RL_diagnostics, with print and plot methods. All measures are computed over a grid of thresholds ξ∈[0.5,1)\xi\in[0.5, 1) (when the linkage method allows it by returning scores) or for a fixed threshold when specified. Each diagnostic can also be computed on its own. We present all commands hereafter but show only some outputs. Further details on these diagnostic tools are given in the package vignette ‘Linkage and Diagnostics with FlexRL’ available through vignette('FlexRL-RL-vignette', 'FlexRL') or the online documentation.

DiagnosticComputationDisplay
Convergenceplot.StEM_convergence() on the chains returned by StEM()plot(x, ‘convergence’)
Linkage scoresplot_linkage_scores()plot(x, ‘scores’)
FDP estimationcompute_RL_FDP_score() (Eq. (11)), compute_augmRL_FDP_synth() (Eq. (12))plot(x, ‘FDP’)
Discrepancy measuressmd(), support_iou(), mmd(), RL_agreement()plot(x, ‘discrepancy’)
Data comparisonplot.distributions() on each source and its linked subsetplot(x, ‘distributions’)

Table 5. Diagnostics performed by RL_diagnostics().

Diagnostics

The function takes several arguments as input. The record linkage output fit, the data sources encodedA and encodedB, the set of variables of interest to compare compare_vars, their type vars_type_cont (TRUE if continuous) and the PIVs are passed to RL_diagnostics(), which computes FDP estimates and discrepancy measures that characterise the linked sample across a grid of thresholds ξ\xi. The true_pairs enables reporting the true FDP and sensitivity when known.

compare_vars <- c("BirthYear", "Sex", "MaritalStatus",
                  "BirthRegion", "Occupation")

vars_type_cont <- list(BirthYear = TRUE, Sex = FALSE, MaritalStatus = FALSE,
                       BirthRegion = FALSE, Occupation = FALSE)

diag <- RL_diagnostics(fit, prep_data$encodedA, prep_data$encodedB,
                       compare_vars, vars_type_cont, PIVs,
                       true_pairs = prep_data$true_pairs, FDP_estimation = TRUE,
                       RL_method = "FlexRL", data = prep_data)

The FDP estimation is optional (FDP_estimation = TRUE) since it can be time consuming. For a given RL_method, it is repeated n_repeats times and the estimates are averaged. compute_RL_FDP_score() returns the score based estimate on the original task. compute_augmRL_FDP_synth() adds n_synth records generated by synth_method to BB, links the augmented sources as an auxiliary task and computes both the score based and the synthetic data estimates on it. Similar score based estimates on the original and augmented tasks support the use of the synthetic data estimator. Table 4 reported these three estimates as ‘est. FDP score’, ‘est. FDP* score’, ‘est. FDP* synth’.

The plot(x, type) method for an object of class RL_diagnostics draws the plots listed in Table 5, with type being one of ‘convergence’, ‘scores’, ‘FDP’, ‘discrepancy’ and ‘distributions’. With the ‘convergence’ plots, we could verify that the StEM chains have stabilised.

plot(diag, ‘convergence’)

With the ‘scores’ plot, we can analyse the distribution of the linkage scores of the fit as shown in Figure 2.

Linkage score distribution

Figure 2. Distribution of the linkage scores returned by FlexRL StEM() on the SHIW in Emilia-Romagna.

plot(diag, ‘scores’)

Albeit bimodal, the distribution of linkage scores produced by the record linkage algorithm does not indicate where to set the threshold ξ\xi. The FDP estimation results and the divergence plots may help further. In Figure 3 we show the ‘FDP’ plot and some of the ‘discrepancy’ plots for the MMD and the SMD. Similar plots for the overlap of supports and the agreement rate are usually displayed by plot(diag, ‘discrepancy’).

plot(diag, ‘FDP’)
plot(diag, ‘discrepancy’)

The graphics from Figure 3 show how each linkage decision rule affects linked data quality. They illustrate the trade-off of § 5.1: as the threshold ξ\xi increases, the estimated FDP diminishes while the divergence of the linked sample from the sources grows. Ideally, we choose an appropriate threshold for inference at the ‘elbow’ of the divergence curves, provided that the estimated FDP at that level is reasonable. This decision rule is a heuristic motivated by the potential impact of linkage errors on inference, as discussed in § 5.1, and its reliability has not been assessed. Note that the divergence measures are indicators of the sampling bias but do not estimate Eq. (10). For our example, we choose ξ=0.86\xi= 0.86 following this rule, as indicated by the vertical line in Figure 3.

With the ‘distributions’ plots, we could compare the distribution of the linked data and of the data sources at the selected threshold—for example ξ=0.86\xi= 0.86.

plot(diag, ‘distributions’, threshold = 0.86)

The print() method for an S3 object of class RL_diagnostics reports the linked sample size, the FDP estimates, the true FDP and sensitivity when available, and the discrepancy measures. We report the output in Figure 4. As discussed in § 5.2, the FDP estimators have different strengths and weaknesses. In the SHIW in Emilia-Romagna, the synthetic data estimator yields an estimate of 0.10 and the score based estimator yields an estimate of 0.02, compared with the true value of 0.16.

Throughout the diagnostics, the record linkage is run by wrappers with a common form, link_with_<pkg>(dataA, dataB, ...), returning the indices of linked pairs and their linkage score. Wrappers exist for FlexRL, fastLink, BRL, reclin2, multilink, diyar and fedmatch. That way, the diagnostic tools apply to any of these methods in a similar fashion.

On the illustrative example developed along this chapter, we regress the expected age of retirement in 2014 on the expected age of retirement in 2012 (originally named ‘etapen’ in each wave). After preparation of the SHIW in Emilia-Romagna data for record linkage (see prepare_data() in § 4.1), we work with 1,479 and 1,524 records in the two data sources, which share 898 true links. The solid red line in Figure 5 is the reference estimate computed on these true links. The tools presented so far help identify a suitable linkage threshold by assessing the trade-off between contamination and sampling bias. In Figure 5, a low threshold ξ=0.55\xi= 0.55 yields a linked data estimate (dotted orange) affected by contamination bias, due to noisy falsely linked data (short dashed yellow estimate); see Eq. (9). Conversely, a high threshold ξ=0.99\xi= 0.99 yields a linked data estimate (dotted orange) affected by sampling bias, due to atypical truly linked data (long dashed yellow estimate); see Eq. (10). Guided by the diagnostic tools, we choose ξ=0.86\xi= 0.86, which yields a linked data estimate (dotted orange) closer to the reference. At this threshold, we retain 282 linked observations, corresponding to an FDP of 0.16 and a sensitivity of 0.26.

Regression plots for the SHIW in Emilia-Romagna linked through FlexRL

Figure 5. Regression analysis on the SHIW in Emilia-Romagna linked through FlexRL.

Scalability

In Table 3 of § 4, we compared record linkage methods on the SHIW in Emilia-Romagna. Despite heterogeneous linkage performance, running time and memory usage, all methods ran at that scale. To complete our investigation, we demonstrate the scalability of FlexRL and examine that of other record linkage methods on the complete SHIW data sets. After preparation, the 2012 and 2014 waves over the 20 Italian regions cover 19,215 and 20,002 individuals, with 10,568 individuals observed in both waves. We consider two settings, without and with regional blocking. Note that blocking on the region assumes that this variable is stable and free of registration errors.

In § 4.1, we showed the proportion of agreements between the information carried by records among true links of the SHIW in Emilia-Romagna. Below we show the proportion of agreements among true links of the complete SHIW data. In this application, the region is indeed free of registration errors and stable.

BirthYear    Sex   MaritalStatus   BirthRegion   Occupation   Region
   0.99     1.00        0.98           1.00          0.82      1.00

Record linkage becomes harder on large data sets: the discriminative power of the PIVs is unlikely to grow with the size of the sources, whereas the number of candidate records in B\mathcal{B} for each record in A\mathcal{A} does. Without blocking, we may therefore expect higher FDPs and also higher sensitivities, since true links that blocking excludes (due to registration errors or changes between waves) become recoverable. However, in the complete SHIW the region is stable and error free so the expected gain does not apply. FDPs and sensitivities are similar without and with blocking for the comparable methods, except for FlexRL which has both a higher FDP and a higher sensitivity without blocking. Note that prepare_data() eliminates out-of-support records at regional levels, which explains the differences in the partial confusion matrix of the ‘Naïve’ method between Table 6 and Table 7.

NaiveBRLmultilinkfastLinkreclin2diyarfedmatchFlexRL
TP8,289.00NANA5,061.00NANA5,057.004,105.00
FP49,350.00NANA11,175.00NANA14,154.005,417.00
FN2,279.00NANA5,507.00NANA5,511.006,463.00
sensitivity0.78NANA0.48NANA0.480.39
FDP0.86NANA0.69NANA0.740.57
time (mins)0.00NANA9.00NANA5.006.00
memory (Mb)123.20NANA20,070.10NANA1,877.004,030.30

Table 6. Comparison of record linkage methods on the complete SHIW data application for the default decision rule ξ=0.5\xi= 0.5.

Below, we report the number of unique values of each PIV in the SHIW in Emilia-Romagna and in the complete SHIW. These counts indicate the discriminative power of the PIVs in the regional and national applications. In addition, we appraise the difficulty of the tasks,

without and with regional blocking, through the performance of the ‘Naive’ approach in Table 6 and Table 7, which reflects how many records share the same combination of PIV values.

BirthYear Sex MaritalStatus BirthRegion Occupation 94 2 4 18 52

BirthYear Sex MaritalStatus BirthRegion Occupation Region 99 2 4 20 84 20

Table 6 presents the comparative study without blocking; ‘Not Available’ results indicate memory errors. When supported, methods run with four threads (threads for FlexRL, n.cores for fastLink, nthread for fedmatch). Table 7 shows the results with blocking on the Italian regions. More details on this data application showcasing the scalability of FlexRL are given in the package vignette ‘Scalable Record Linkage with FlexRL’ available through vignette(‘FlexRL-RL-block-vignette’, ‘FlexRL’) or the online documentation.

Discussion

This chapter presented FlexRL: an R package for probabilistic record linkage of two overlapping data sources using a hierarchical model that accounts for registration errors and accommodates temporal changes in Partially Identifying Variables (PIVs). FlexRL provides tools

NaiveBRLmultilinkfastLinkreclin2diyarfedmatchFlexRL
TP8,289.001,609.001,771.004,911.008,993.00316.005,284.002,916.00
FP49,346.00449.001,092.0011,295.004,092,774.0012.0013,123.001308.00
FN2,059.008,739.008,577.005,437.001,355.0010,032.005,064.007,432.00
sensitivity0.800.160.170.480.870.030.510.28
FDP0.860.220.380.701.000.040.710.31
time (mins)0.333.63206.825.820.422.672.255.03
memory (Mb)16.001,150.003,837.004,716.002,223.001,530.00766.00862.00

caption: Comparison of record linkage methods on the complete SHIW data application with regional blocking, for the default decision rule ξ=0.5\xi= 0.5.

:::

for assessing the quality of the resulting linked data. These diagnostics are independent of the linkage method and thus apply broadly. These tools are essential for using record linkage in practice, where potential linkage errors and induced sample selection cannot be observed directly.

To draw attention to the implications of linkage quality for statistical analysis, FlexRL provides two complementary tools as functions of the linkage threshold. First, the estimated False Discovery Proportion (FDP), which quantifies contamination among the linked pairs. Second, the divergence of the linked sample from the source populations, which captures potential sampling bias. Examining these quantities jointly makes the trade-off between contamination and sampling bias explicit and provides users with information to select a linkage decision rule in light of its potential consequences for subsequent inference.

Several directions follow naturally from this work. On the computational side, the memory footprint is linear in the number of records and the running time scales with the number of records, candidate pairs, PIVs, their levels and configurations. Memory and running time allow record linkage between large data sets on a standard computer. On the modelling side though, the linkage model currently assumes independence of PIVs given the latent true values, no duplicates within a source, categorical PIVs; extensions would broaden the range of applications. Moreover, the diagnostic estimators and the proposed quality measures, whose validity rests on the assumptions underlying their estimation, would benefit from further theoretical and empirical investigation.

Notwithstanding these considerations, FlexRL provides a framework that connects record linkage to linked data quality assessment. It offers a solution to record linkage that is scalable in terms of memory usage and running time, and flexible in accommodating diverse scenarios. Linkage performance depends on the difficulty of the task and the chosen decision rule, controlling the FDP and recovering a fraction of the true links accordingly. It brings record linkage into practice, helping researchers and practitioners not only to link data, but also to quantify and communicate the extent to which linkage decisions may affect downstream inference.

Author contributions statement

KR developed the record linkage algorithm pioneered by MH and implemented tools for inference conceptualized by MW. All authors drew up the simulation framework. KR conducted the analysis, prepared the first manuscript draft, reviewed and edited by MH and MW. All authors read and approved this manuscript.

Acknowledgments

The authors thank the anonymous reviewers for their valuable suggestions and for conducting a thorough reproducibility check.

Data and Code Availability

The FlexRL package is available from the Comprehensive R Archive Network (CRAN) or from the package GitHub repository alongside documentation and vignettes describing its functionality. FlexRL is also available as a module in JASP 1.0 [25]. The script is available as supplementary material and on the GitHub repository of the package.

Financial disclosure

This research received no specific grant from any funding agency in the public, commercial, or not-for-profit sectors.

Conflict of interest

The authors are the developers of FlexRL.

References

  1. [1]S. Aleshin-Guendel. Multifile Record Linkage and Duplicate Detection, 2023. URL https://CRAN.R-project.org/package=multilink. R package version 0.1.1.
  2. [3]T. R. Belin and D. B. Rubin. A method for calibrating false-match rates in record linkage. Journal of the American Statistical Association, 90(430):694–707, 1995. ISSN 01621459, 1537274X. URL http://www.jstor.org/stable/2291082.DOI
  3. [4]O. Binette and R. C. Steorts. (almost) all of entity resolution. Science Advances, 8(12):eabi8021, 2022. doi: 10.1126/sciadv.abi8021.DOI
  4. [5]T. Blakely and C. Salmond. Probabilistic record linkage and a method to calculate the positive predictive value. International Journal of Epidemiology, 31(6):1246–1252, 12 2002. ISSN 0300-5771. doi: 10.1093/ije/31.6.1246. URL https://doi.org/10.1093/ije/31.6.1246.
  5. [7]G. Celeux and J. Diebolt. L’algorithme SEM : un algorithme d’apprentissage probabiliste pour la reconnaissance de mélange de densités. Revue de Statistiques Appliquées, 34(2):35–52, 1986. URL http://www.numdam.org/item/RSA_1986__34_2_35_0/.
  6. [8]M. Cleaton, J. Hall, R. Shipsey, Z. White, and K. Xhaferaj. A case study of using Splink: Census duplicate matching. In Proceedings of Statistics Canada Symposium 2022. Data Disaggregation: Building a More Representative Data Portrait of Society. Statistics Canada, 2024.
  7. [9]G. J. Cohen, J. Dice, M. Friedrichs, K. Gupta, W. Hayes, I. Kitschelt, S. J. Lee, W. B. Marsh, N. Mislang, M. Shaton, M. Sicilian, and C. Webster. The U.S. syndicated loan market: Matching data. Journal of Financial Research, 44(4):695–723, 2021. doi: https://doi.org/10.1111/jfir.12261. URL https://onlinelibrary.wiley.com/doi/abs/10.1111/jfir.12261.
  8. [10]T. Cooney, S. Yu, S. Daral, M. Field, B. Krebs, M. Kieran, P. Manley, S. Govinda Raju, and S. Zelt. Healthcare resource use for pediatric low-grade glioma care: A cohort of linked electronic health records and claims data. Journal of Clinical Oncology, 41(16_suppl):10038–10038, 2023. doi: 10.1200/JCO.2023.41.16_suppl.10038. URL https://ascopubs.org/doi/abs/10.1200/JCO.2023.41.16_suppl.10038.
  9. [11]A. Dasylva and A. Goussanou. Estimating linkage errors under regularity conditions. In Proceedings of the Survey Research Methods Section. American Statistical Association, 2020.
  10. [12]A. Dasylva and A. Goussanou. On the consistent estimation of linkage errors without training data. Japanese Journal of Statistics and Data Science, 5, 2022. doi: 10.1007/s42081-022-00153-3.DOI
  11. [13]J. de Bruin. Python record linkage toolkit: A toolkit for record linkage and duplicate detection in python, 2019. URL https://doi.org/10.5281/zenodo.3559043. Python package.
  12. [14]J. C. Doidge and K. L. Harron. Reflections on modern methods: linkage error bias. International Journal of Epidemiology, 04 2020. doi: 10.1093/ije/dyz203.DOI
  13. [15]H. L. Dunn. Record linkage. American Journal of Public Health and the Nations Health, 36(12):1412–1416, 1946.
  14. [16]S. B. Dusetzina, S. Tyree, A. M. Meyer, A. Meyer, L. Green, and W. R. Carpenter. Linking data for health services research: A framework and instructional guide [internet]. https://www.ncbi.nlm.nih.gov/books/NBK253312/, 2014. URL https://www.ncbi.nlm.nih.gov/books/NBK253312/. An Overview of Record Linkage Methods.
  15. [17]T. Enamorado, B. Fifield, and K. Imai. Using a probabilistic model to assist merging of large-scale administrative records. American Political Science Review, 113(2):353–371, 2019. doi: 10.1017/S0003055418000783.
  16. [18]I. P. Fellegi and A. B. Sunter. A theory for record linkage. Journal of the American Statistical Association, 64(328):1183–1210, 1969.DOI
  17. [19]M. Fortini, B. Liseo, A. Nuccitelli, and M. Scanu. On Bayesian record linkage. Research in Official Statistics, 4(6):185–198, 2001.
  18. [20]M. Friedrichs, C. Webster, B. Marsh, J. Dice, and S. Lee. fedmatch: Fast, Flexible, and User-Friendly Record Linkage Methods, 2023. URL https://CRAN.R-project.org/package=fedmatch. R package version 2.0.6.
  19. [21]A. Gretton, K. M. Borgwardt, M. J. Rasch, B. Schölkopf, and A. Smola. A kernel two-sample test. Journal of Machine Learning Research, 13:723–773, 2012.DOI
  20. [22]K. L. Harron, J. C. Doidge, H. E. Knight, R. E. Gilbert, H. Goldstein, D. A. Cromwell, and J. H. van der Meulen. A guide to evaluating linkage quality for the analysis of linked data. International Journal of Epidemiology, 46(5):1699–1710, 09 2017. ISSN 0300-5771. doi: 10.1093/ije/dyx177. URL https://doi.org/10.1093/ije/dyx177.
  21. [24]M. A. Jaro. Advances in record-linkage methodology as applied to matching the 1985 census of tampa, florida. Journal of the American Statistical Association, 84(406):414–420, 1989.DOI
  22. [25]JASP Team. JASP (Version 1.0.0)[Computer software], 2026. URL https://jasp-stats.org/.
  23. [26]Z. Jelveh, M. McNeill, and T.-Y. Lin. Namematch, 2022. URL https://github.com/urban-labs/namematch. Python package.
  24. [27]R. Kramer. Towards scalable probabilistic linkage of administrative population data in Germany: Insights from a method test with Splink. International Journal of Population Data Science, 11(4), 2026. doi: 10.23889/ijpds.v11i4.3814.
  25. [28]P. Lahiri and M. D. Larsen. Regression analysis with linked data. Journal of the American Statistical Association, 100(469):222–230, 2005. ISSN 01621459. URL http://www.jstor.org/stable/27590532.DOI
  26. [29]J. Laidler, A. Imaz Blanco, and D. Balasubramanian. Probabilistic linkage pipeline improving linkage quality and explainability in healthcare. International Journal of Population Data Science, 10(4), 2025. doi: 10.23889/ijpds.v10i4.3271.
  27. [30]J. Lam, A. Boyd, R. Linacre, R. Blackburn, and K. Harron. Generating synthetic identifiers to support development and evaluation of data linkage methods. International Journal of Population Data Science, 9(1), July 2024. doi: 10.23889/ijpds.v9i1.2389. URL https://ijpds.org/index.php/ijpds/article/view/2389.
  28. [31]M. D. Larsen. Multiple imputation analysis of records linked using mixture models. Proceedings of the Survey Methods Section, Statistical Society of Canada Annual Meeting, 63:65–71, 1999.
  29. [32]R. Linacre, S. Lindsay, T. Manassis, Z. Slade, T. Hepworth, R. Kennedy, and A. Bond. Splink: Free software for probabilistic record linkage at scale. International Journal of Population Data Science, 7(3), Aug. 2022. doi: 10.23889/ijpds.v7i3.1794. URL https://ijpds.org/article/view/1794.
  30. [33]N. G. Marchant, A. Kaplan, D. N. Elazar, B. I. P. Rubinstein, and R. C. Steorts. d-blink: Distributed end-to-end Bayesian entity resolution. Journal of Computational and Graphical Statistics, 30(2):406–421, 2021.DOI
  31. [34]N. G. Marchant, B. I. P. Rubinstein, and R. C. Steorts. Bayesian graphical entity resolution using exchangeable random partition priors. Journal of Survey Statistics and Methodology, 11(3):569–596, 2023.
  32. [35]S. Mudgal, H. Li, T. Rekatsinas, A. Doan, Y. Park, G. Krishnan, R. Deep, E. Arcaute, and V. Raghavendra. Deep learning for entity matching: A design space exploration. In Proceedings of the 2018 International Conference on Management of Data, SIGMOD ’18, page 19–34, New York, NY, USA, 2018. SIGMOD 18: Proceedings of the 2018 International Conference on Management of Data, Association for Computing Machinery. ISBN 9781450347037. doi: 10.1145/3183713.3196926. Code available at https://pypi.org/project/deeppart/.DOI
  33. [36]H. B. Newcombe, J. M. Kennedy, S. J. Axford, and A. P. James. Automatic linkage of vital records. Science, 130(3381):954–959, 1959. doi: 10.1126/science.130.3381.954.DOI
  34. [37]B. Nowok, G. M. Raab, and C. Dibben. synthpop: Bespoke creation of synthetic data in r. Journal of Statistical Software, 74(11):1–26, 2016. doi: 10.18637/jss.v074.i11. URL https://CRAN.R-project.org/package=synthpop.
  35. [38]O. Nsonwu. diyar: Record Linkage and Epidemiological Case Definitions in ’R’, 2023. URL https://olisansonwu.github.io/diyar/index.html. R package version 0.5.1.
  36. [39]M. Pontesilli, M. H. Hof, A. C. J. Ravelli, A. J. van Altena, A. T. Soufan, B. W. Mol, E. H. Kostelijk, E. Slappendel, D. Consten, A. E. P. Cantineau, L. A. J. van der Westerlaken, W. van Inzen, J. C. M. Dumoulin, L. Ramos, E. B. Baart, F. J. M. Broekmans, P. M. Rijnders, M. H. J. M. Curfs, S. Mastenbroek, S. Repping, T. J. Roseboom, and R. C. Painter. Effect of parental and ART treatment characteristics on perinatal outcomes. Human Reproduction, 36(6):1640–1665, 04 2021. ISSN 0268-1161. doi: 10.1093/humrep/deab008. URL https://doi.org/10.1093/humrep/deab008.
  37. [40]D. Resnick and J. Asher. Measurement of type i and type ii record linkage error. In Using and Linking Administrative and Auxiliary Data, Government Statistics Section. Joint Statistical Meetings, 09 2019. URL https://ww2.amstat.org/meetings/proceedings/2019/data/assets/pdf/1199462.pdf.
  38. [41]K. Robach and M. H. Hof. FlexRL: Flexible Record Linkage and Inference on Linked Data, 2026. URL https://CRAN.R-project.org/package=flexrl. R package version 2.0.0.
  39. [42]K. Robach, M. H. Hof, and M. A. van de Wiel. False discovery estimation in record linkage. Statistics in Medicine, 44(23-24):e70292, 2025a. doi: https://doi.org/10.1002/sim.70292. URL https://onlinelibrary.wiley.com/doi/abs/10.1002/sim.70292.
  40. [43]K. Robach, S. L. van der Pas, M. A. van de Wiel, and M. H. Hof. A flexible model for record linkage. Journal of the Royal Statistical Society C: Applied Statistics, page qlaf016, Feb. 2025b. ISSN 0035-9254. doi: 10.1093/jrsssc/qlaf016. URL https://doi.org/10.1093/jrsssc/qlaf016.
  41. [45]M. Sadinle. Bayesian estimation of bipartite matchings for record linkage. Journal of the American Statistical Association, 112(518): 600–612, 2017.arxiv.org/abs/1601.06630
  42. [46]M. Sadinle. Beta Record Linkage, 2020. URL https://CRAN.R-project.org/package=BRL. R package version 0.1.0.DOI
  43. [47]R. Steorts, R. Hall, and S. Fienberg. SMERED: A Bayesian Approach to Graphical Record Linkage and De-duplication. In S. Kaski and J. Corander, editors, Proceedings of the Seventeenth International Conference on Artificial Intelligence and Statistics, volume 33 of Proceedings of Machine Learning Research, pages 922–930, Reykjavik, Iceland, 22–25 Apr 2014a. PMLR. URL https://proceedings.mlr.press/v33/steorts14.html.
  44. [48]R. C. Steorts. Entity resolution with empirically motivated priors. Bayesian Analysis, 10(4):849–875, 2015.DOI
  45. [49]R. C. Steorts, S. L. Ventura, M. Sadinle, and S. E. Fienberg. A Comparison of Blocking Methods for Record Linkage. Springer International Publishing, 2014b.arxiv.org/abs/1407.3191
  46. [50]R. C. Steorts, R. Hall, and S. E. Fienberg. A Bayesian approach to graphical record linkage and deduplication. Journal of the American Statistical Association, 111(516):1660–1672, 2016.DOI
  47. [51]A. Tancredi and B. Liseo. A hierarchical Bayesian approach to record linkage and population size problems. The Annals of Applied Statistics, 5(2B), 2011.
  48. [52]A. Tancredi and B. Liseo. Regression analysis with linked data: problems and possible solutions. Statistica, 75(1):19–35, Jan. 2015. doi: 10.6092/issn.1973-2201/5821. URL https://rivista-statistica.unibo.it/article/view/5821.DOI
  49. [53]K. I. Ted Enamorado, Ben Fifield. fastLink: Fast Probabilistic Record Linkage with Missing Data, 2023. URL https://CRAN.R-project.org/package=fastLink. R package version 0.6.1.
  50. [54]B. J. Tepping. A model for optimum linkage of records. Journal of the American Statistical Association, 63:1321–1332, 1968.DOI
  51. [55]S. van Buuren and K. Groothuis-Oudshoorn. mice: Multivariate imputation by chained equations in r. Journal of Statistical Software, 45(3):1–67, 2011. doi: 10.18637/jss.v045.i03. URL https://CRAN.R-project.org/package=mice.DOI
  52. [56]J. van der Laan. reclin2: Record Linkage Toolkit, 2025. URL https://github.com/djvanderlaan/reclin2. R package version 0.6.0.
  53. [58]Z. Wang, E. Ben-David, G. Diao, and M. Slawski. Regression with linked datasets subject to linkage error. WIREs Computational Statistics, 14(4):e1570, 2022. doi: https://doi.org/10.1002/wics.1570. URL https://wires.onlinelibrary.wiley.com/doi/abs/10.1002/wics.1570.
  54. [59]D. S. Watson, K. Blesch, J. Kapar, and M. N. Wright. Adversarial random forests for density estimation and generative modeling. https://arxiv.org/abs/2205.09435, 2023. URL https://CRAN.R-project.org/package=arf.
  55. [60]E. J. Williamson, A. J. Walker, K. Bhaskaran, S. Bacon, C. Bates, C. E. Morton, H. J. Curtis, A. Mehrkar, D. Evans, P. Inglesby, J. Cockburn, H. I. McDonald, B. MacKenna, L. Tomlinson, I. J. Douglas, C. T. Rentsch, R. Mathur, A. Y. S. Wong, R. Grieve, D. Harrison, H. Forbes, A. Schultze, R. Croker, J. Parry, F. Hester, S. Harper, R. Perera, S. J. W. Evans, L. Smeeth, and B. Goldacre. Opensafely: factors associated with covid-19-related hospital death in the linked electronic health records of 17 million adult nhs patients. MedRxiv, pages 430–436, 2020.DOI

Paper details

vlnc:W_sj7xp9bt
Version
v1 · latest
Published
Fresh — added to the Hub within the last 30 days
Added
License
CC BY 4.0
Usage

views · 0

downloads · 0

A view counts once per browser per day, a download each time the PDF is requested.

  • Mathematics

Not recorded for this paper

View statement
Looking for related papers…
Contents