StEM()Computational aspectsInference on linked dataImpact of linkage errors on linear regressionLinkage decision rule trade-offFalse discovery proportionScore based estimatorSynthetic data estimatorRepresentativeness of the linked sampleDiagnosticsDiagnosticsScalabilityDiscussionAuthor contributions statementAcknowledgmentsData and Code AvailabilityFinancial disclosureConflict of interestReferencesWorksMathematics
FlexRL: an R package for Flexible Record Linkage and Linked Data Quality Assessment
Abstract
FlexRL is an R package for probabilistic record linkage and inference on linked data. It provides a method to link observations from the same entities across two data sources that lack a unique identifier but share partially identifying variables. This method fits a hierarchical latent variable model of the data generation process, rather than classifying pairs of records from the comparison of their values, and accommodates registration errors and variables changing over time. Since linkage may affect subsequent analyses, the package also provides tools to assess the linked data quality as functions of the linkage decision rule. Two false discovery proportion estimators are implemented: a probabilistic approach contingent on the record linkage model and an augmented approach relying on an auxiliary estimation using synthetic data. Several measures of divergence of the linked sample from the source population are provided. These tools are agnostic to the underlying record linkage model and can be applied to any method. Together, they connect the output of the linkage to the validity of the subsequent analyses. The implementation combines standard R functions and S3 methods with Rcpp computations, resulting in fast execution times and a low memory footprint.
Introduction
Researchers and statistical agencies increasingly combine data sources that do not share a unique identifier [8, 29, 27]. Record linkage connects records referring to the same entity across sources based on Partially Identifying Variables (PIVs, e.g. birth year, sex, postal code or product code, brand and category). Each PIV has limited discriminative power on its own and may be missing, incorrectly recorded or change over time. Record linkage underpins the production of official statistics, enables the reuse of administrative and registry data for research [60, 39, 10] and improves data quality through linkage and deduplication.
Linked data sets become the study sample for downstream analyses, where linkage errors may propagate [14, 58] (Vo et al. 2024; Robach et al. 2026). Errors arise when links between entities that appear in both sources are missed or when records from distinct entities are falsely linked. These errors cannot be reduced simultaneously and may bias the conclusions drawn from linked data. A stricter decision rule reduces falsely linked pairs but misses more links. Practitioners therefore need not only record linkage methods, but also tools to quantify linkage errors and appraise their consequences for statistical analysis.
Two main probabilistic approaches to record linkage have developed through times. First, the traditional mixture model [18, 24, 31, 19, 45] treats linkage as a classification problem between pairs of records that are similar and pairs that are not. Second, the hierarchical entity resolution framework [51, 48, 50, 33, 34] models the data generation process, considering observed values as potentially distorted versions of latent true values. The latter framework uses all the information contained in the records and is therefore more flexible but more computationally demanding. It has been implemented in R for entity resolution (clustering records from a single file into entities) rather than for linkage between two data sources.
We address this gap with the FlexRL R package, which makes two contributions. First, it implements the hierarchical record linkage model proposed by Robach et al. [2025b]. The model is fitted with a Stochastic Expectation Maximisation (StEM) algorithm that accounts for registration errors and temporal changes in PIVs. Its architecture and efficient Rcpp backend support scalability to large data sets and a wide range of applications: health monitoring studies, survey waves, census and administrative records or data from different retailers. Second, it provides tools to assess the quality of the linked data, which quantify the linkage error and sample selection. FlexRL implements two estimators of the False Discovery Proportion (FDP): one derived from linkage scores [3, 40, 17] and one based on synthetic data [42]. FlexRL measures the divergence of the linked sample from the original sources using standardised mean differences, overlap of supports, maximum mean discrepancy and agreement rates. These tools only need the linked pairs and, when available, their linkage scores. They therefore apply to the output of any linkage method and FlexRL provides wrappers for several existing R packages.
The FlexRL package is aimed at practitioners, statistical agencies and researchers who need to link data sources without a unique identifier or who want to measure the error and the sample selection induced by record linkage. It builds on three lines of work: the biases that linkage errors cause in inference [28, 52, 14, 58] (Robach et al. 2026); the need for linked data quality control in practice [22, 30, 29] (Harron K L 2020); advances in record linkage methodology [45, 34, 43]. By combining record linkage and diagnostics in a practical tool, it helps evaluating whether inference can be conducted on linked data.
Throughout this chapter, we motivate and illustrate the main components of FlexRL on the 2012 and 2014 waves of the Survey on Household Income and Wealth (SHIW), conducted and made available by the Bank of Italy for research purposes. As a secondary analysis of public data, no additional ethics approval or participant consent was required. § 2 introduces the notation, the modelling paradigms, the existing software and demonstrates the potential of FlexRL. § 4 presents the record linkage model implemented in FlexRL and its estimation on a subset of the SHIW. On the resulting linked data, § 5 describes the diagnostic tools FlexRL provides, and illustrates the impact of linkage errors on a socio-economic regression analysis. § 6 demonstrates the scalability of the FlexRL approach to record linkage on the SHIW. § 7 concludes.
The record linkage problem
Setup and notation
We consider two overlapping samples and from the same population with and records. Both sources record the same categorical PIVs, indexed by with possible values each. Index refers to a record in and index to a record in . For PIV , we denote by the registered (observed) value and by the latent true value in record . We define and analogously. Registered values differ from true values because of registration errors: missing values or mistakes (e.g. typographical errors). We write and for the collections of all registered and all true values. True values may also differ between the two data collections. This is the case for dynamic PIVs like postal code, whereas stable PIVs like birth year cannot change. For an entity recorded in both sources, the true values of stable PIVs are therefore identical in and and any disagreement between their registered values is due to registration errors. We elaborate on dynamic PIVs later.
The object of interest is the linkage matrix , with if records and refer to the same entity. We assume that there are no duplicates within a data source. Thus, each record forms a link with at most one record in the other data source: for all and for all . This is the one-to-one assignment constraint, also referred to as bipartite matching [45]. We denote by the probability that a record from forms a link with a record from , that is, the overlap between the sources as a fraction of the smaller data set.
Since record linkage is imperfect, the linked data contain two types of error: false positives (pairs declared linked while referring to distinct entities) and false negatives (true links that are missed). The non-links in the Cartesian product are far more numerous than the links and are rarely of interest. The useful summaries are therefore the False Discovery Proportion (FDP) among the linked data (complement of precision or positive predictive value) and the sensitivity (also called recall, complement of miss rate). The two errors affect inference differently: falsely linked pairs contaminate the inference and missed links induce sample selection [14] (Robach et al. 2026).
Modelling paradigms
Linkage methods can be grouped into four paradigms [16, 4]. Deterministic (rule based) linkage declares a link when records agree on a set of variables, exactly or up to a tolerance. Supervised methods train a classifier on pairs whose linkage status is known [35]; such labelled pairs are seldom available. The two remaining paradigms are unsupervised and probabilistic, and differ in what they model:
Paradigm 1: classification of comparison vectors
The traditional ‘Fellegi-Sunter’ approach [18, 15, 36, 54] reduces the data to comparison vectors. For each pair , a vector records whether elements and for agree or how similar they are. The pairs of records are split into the set of matched and the set of unmatched . The comparison vectors follow a two components mixture:
where in this paradigm is the proportion of links among all pairs. Under the assumption that the comparisons are independent given the linkage status, and factorise over . The parameters are then estimated with an EM algorithm [24, 31]. Pairs are classified by thresholding the ratios or the posterior linkage probabilities. Bayesian versions place a prior on the linkage matrix to enforce the one-to-one assignment constraint [19, 45] or a prior on a partition of the records when there are more than two sources or duplicates [1]. The computations are simple and fast, but memory intensive because they operate on comparison vectors. Scaling to large data sets may therefore require blocking, which compares only pairs that agree on some variables [49]. This paradigm treats pairs as independent although they share records, and the reduction to comparison vectors discards part of the information contained in the records.
Paradigm 2: hierarchical modelling
The second family models how the data were generated [51, 48, 50, 33, 34, 43]. Registered values are seen as distorted versions of latent true values , which define the linkage structure . The joint model reads
with a model for registration errors , a model for underlying true values and a model for the linkage structure . Since records are never reduced to pairwise comparisons, the model can separate missing values, mistakes and changes over time, and it does not need blocking to fit in memory. However, it is more computationally demanding.
Existing software
In Python, recordlinkage [13], Splink [32] and NameMatch [26] implement Paradigm 1. We list the main R packages for the task considered here in Table 1. Paradigm 2 is only available in R through methods for entity resolution [47, 33, 34]. These methods cluster records from a single file, possibly containing duplicates, into entities.
| Package | Paradigm | Estimation | Linkage structure | Record deduplication | PIV dynamics | Quality tools |
| fedmatch 2.1.0 [20, 9] | comparison vectors | push relabel maximum flow | many-to-many | no | no | no |
| diyar 0.5.1 [38] | comparison vectors | multistage weights comparisons, fixed score level parameters | clusters | yes | no | no |
| reclin2 0.6.0 [56] | comparison vectors, supervised | EM, machine learning | one-to-one post hoc | no | no | no |
| fastLink 0.6.1 [17, 53] | comparison vectors | EM | one-to-one post hoc | no | no | no |
| BRL 0.1.0 [46, 45] | comparison vectors | Bayesian, bipartite prior, Gibbs sampler | one-to-one in model | no | no | no |
| multilink 0.1.1 [1, 1] | comparison vectors | Bayesian, partition prior, Gibbs sampler | clusters, one-to-one in model | yes | no | no |
| FlexRL 2.0.0 [41] | hierarchical | stochastic EM | one-to-one in model | no | yes | yes |
Table 1. Record linkage packages on CRAN compared with FlexRL. The FlexRL package diagnostic tools apply to all these packages.
To our knowledge, no R package implements Paradigm 2 for the linkage of two sources under the one-to-one assignment constraint or models the dynamics of PIVs over time. While the postlink package [Buk et al., 2026] develops statistical tools for post-linkage data analysis, no record linkage package provides tools to assess the quality of the linked data and their suitability for inference.
The FlexRL package is available from the Comprehensive R Archive Network (CRAN) or from the package GitHub repository alongside documentation and vignettes describing its functionality. FlexRL is also available as a module in JASP 1.0 [25]. The implementation of FlexRL relies on Rcpp (potentially parallelised with OpenMP) for the backend samplers, Matrix for the sparse linkage matrix, and other standard R packages. The diagnostic tools apply to any record linkage output in the required format, with built-in wrappers for BRL, diyar, fastLink, fedmatch, multilink and reclin2. Figure 1 shows the workflow, which follows the structure of § 4 and § 5: the data are prepared, linked with StEM() and the linked data quality is assessed with RL_diagnostics(). We elaborate on each component along this chapter.

Figure 1. Workflow of FlexRL.
Demonstration
Simulation study
As a preliminary demonstration of the functionalities of FlexRL, we simulate data with the package simulate_data() function. This function synthesises two files with a given overlap, given rates of mistakes and missing values per PIV and file, and given hazard coefficients for structured PIVs, useful for exploring and comparing record linkage methods. Here, we generate two data sets of 250 and 300 records, with 200 entities common to both. Entities are represented by four independent discrete non-uniform PIVs. We distort the data in each file uniformly with 2% of mistakes and 0.5% of missing values; one of the PIVs undergoes temporal changes.
Note that this data generation process, on which all packages cited in Table 1 are evaluated, is in line with the hierarchical model FlexRL fits and represents a simple setting. The fourth PIV follows a dynamic process whose hazard is piecewise constant over time and modulated by a Gaussian covariate. We fit these changes with an exponential survival model in FlexRL, illustrating the benefits of modelling PIV dynamics, even under model misspecification, rather than treating them as registration errors. Later in this chapter, we evaluate all packages on real data for a fairer comparison.
In Table 2, we compare the methods presented in Table 1 to the FlexRL record linkage algorithm with and without modelling PIV dynamics in terms of sensitivity and FDP. When appropriate, modelling PIV dynamics improves record linkage by increasing sensitivity (more true links are recovered) while keeping the FDP under control. The original methodological article provides a more comprehensive study of the behaviour of FlexRL with and without modelling PIV dynamics [43]. Table 2 also shows the differences in computational time and in memory usage between the methods. More details on this demonstration showcasing the utility of accounting for PIV dynamics are given in the package vignette ‘Linkage and Diagnostics with FlexRL’ available through vignette('FlexRL-RL-vignette', 'FlexRL') or the online documentation.
| Naive | BRL | multilink | fastLink | reclin2 | diyar | fedmatch | FlexRL (dynamics) | FlexRL (no dynamics) | |
| TP | 118.40 (5.13) | 80.40 (5.04) | 80.60 (5.95) | 101.70 (6.48) | 114.10 (6.38) | 34.90 (20.24) | 122.10 (6.33) | 114.00 (7.92) | 87.50 (8.30) |
| FP | 83.10 (11.02) | 12.30 (3.23) | 14.20 (4.13) | 43.10 (5.59) | 68.30 (8.82) | 3.30 (2.91) | 125.80 (6.32) | 25.90 (5.15) | 19.00 (4.67) |
| FN | 81.60 (5.13) | 119.60 (5.04) | 119.40 (5.95) | 98.30 (6.48) | 85.90 (6.38) | 165.10 (20.24) | 77.90 (6.33) | 86.00 (7.92) | 112.50 (8.30) |
| sensitivity | 0.59 (0.03) | 0.40 (0.03) | 0.40 (0.03) | 0.51 (0.03) | 0.57 (0.03) | 0.17 (0.10) | 0.61 (0.03) | 0.57 (0.04) | 0.44 (0.04) |
| FDP | 0.41 (0.03) | 0.13 (0.03) | 0.15 (0.04) | 0.30 (0.03) | 0.37 (0.03) | 0.08 (0.05) | 0.51 (0.03) | 0.18 (0.03) | 0.18 (0.04) |
| time (secs) | 0.27 (0.04) | 0.97 (0.05) | 18.57 (1.48) | 1.29 (0.24) | 0.29 (0.01) | 1.17 (0.06) | 1.44 (0.01) | 4.59 (0.02) | 3.72 (0.01) |
| memory (Mb) | 1.04 (0.05) | 78.60 (10.08) | 105.68 (10.18) | 24.85 (4.42) | 41.52 (0.18) | 104.31 (10.84) | 142.00 (1.64) | 102.11 (9.03) | 70.18 (6.42) |
Table 2. Linkage performance of competing record linkage methods on simulated data for the default linkage decision rule; see § 4.3. We report average performance over 10 replications, with standard deviations in parentheses.
The results presented in this chapter are obtained on a MacBook Pro (Apple M1 Max ARM chip architecture, 32GB memory, 8 cores) with R version 4.6.1. The script is available on the GitHub repository of the package. The random seed is set to 1 in each code cell that produces output (some compared packages call set.seed() internally). The computations are robust to different operating systems (MacOS, Linux, Windows). We measure the memory usage using the R garbage collector. External allocations are therefore likely underestimated. All methods compared in this chapter rely on Rcpp or C, except for diyar which is fully implemented in R. Time and memory usage may vary depending on the operating system, hardware, and system configuration.
The hierarchical architecture
Data preparation and PIVs configuration
The record linkage model requires each PIV to be categorical, encoded by integers with missing values coded as . The PIVs must cover a common support across the two sources. The smaller source should be labelled as .
As a first step in the workflow represented in Figure 1, the prepare_data() function which we will describe validates the PIVs configuration, labels the data sources and encodes the PIVs. When a unique identifier is available, like in simulation studies or in benchmark record linkage data such as the SHIW, it also returns the set of true links. This set may be used to evaluate linkage performance.
The function naive_linkage() links records that agree exactly on all PIVs (a missing value matching any value when na_match = TRUE), without enforcing the one-to-one constraint. When the true links are known, the performance of this deterministic linkage indicates the discriminative power of the PIVs and the difficulty of the task. The FDP of the ‘Naive’ linkage reflects how many records share the same combination of PIV values.
The package distinguishes different kinds of PIVs, which determine how the components of the complete data likelihood of Eq. (3) are used when fitting the model in § 4.2. Candidate pairs of records are formed from records whose latent true values agree on every PIV for which temporal changes are not modelled; whether a candidate pair is linked is then decided from its contribution to the complete likelihood. We elaborate on candidate pairs and linked pairs in § 4.2.
Stable: the latent true values do not change (e.g. birth year). Such PIV is used to form candidate pairs and its probability of mistake may be bounded (for example by 0.10) for algorithm stability [43].
Dynamic: the true values may change (e.g. postal code between two survey waves):
Flexible: changes cannot be modelled, typically because registration dates are not available. The PIV is still used to form candidate pairs, but its probability of mistake should not be bounded, so that the registration model absorbs both mistakes and changes.
Structured: changes are modelled by the survival function of Eq. (4). Such PIV is not used to form candidate pairs. Registration dates are needed in both sources and covariates for the hazard may be supplied. Mistakes cannot be distinguished from temporal changes, so the probability of mistake should be fixed (for example at 0.02), unless enough linked pairs have time gaps near zero, where changes are not plausible and mistakes can be estimated without confusion. Estimating the changes also requires variation in the time gaps among linked pairs.
To illustrate the usage of the FlexRL package throughout this chapter, we consider the Emilia-Romagna regional subset of the SHIW data. The record linkage task can be validated in this example where the linkage structure can be reconstructed. After preparation, the 2012 and 2014 waves cover 1,479 and 1,524 individuals, with 898 individuals observed in both waves. In § 5, we analyse a regression of the expected age of retirement in 2014 on the expected age of retirement in 2012.
To link the two survey waves, we use the following variables: year and region of birth, sex, marital status, working status and branch of activity. The strong association between working status and branch of activity violates the PIV independence assumption of the method and may degrade the record linkage performance [43]. We thus combine these variables into a single variable ‘occupation’. Although marital status and occupation are ‘dynamic’ PIVs, we do not have enough information to model their temporal changes and set them as ‘flexible’ in the configuration:
PIVs_config <- list(BirthYear = list(dynamics = "stable",
bound_mistakes = c(0.10, 0.10),
fix_mistakes = c(NA, NA)),
Sex = list(dynamics = "stable",
bound_mistakes = c(0.10, 0.10),
fix_mistakes = c(NA, NA)),
MaritalStatus = list(dynamics = "flexible",
bound_mistakes = c(NA, NA),
fix_mistakes = c(NA, NA)),
BirthRegion = list(dynamics = "stable",
bound_mistakes = c(0.10, 0.10),
fix_mistakes = c(NA, NA)),
Occupation = list(dynamics = "flexible",
bound_mistakes = c(NA, NA),
fix_mistakes = c(NA, NA)))
PIVs <- names(PIVs_config)
We prepare the two data sources for the StEM() record linkage algorithm with the prepare_data() function, which takes as inputs the data sources and their labels df2012, df2014, ‘2012’, ‘2014’; the PIVs configuration PIVs_config; a logical same_mistakes describing whether the parameters fitting the probability of mistakes among PIVs should be the same between data sources (TRUE when the data are sampled from a same source population); a potential record identifier variable uniq_id when available; a logical restrict_support_intersection to harmonise the support of the PIVs in the observed data. In the application, records with birth years outside of the range shared by the sources are excluded.
prep_data <- prepare_data(df2012, df2014, "2012", "2014", PIVs_config,
same_mistakes = TRUE, uniq_id = "ID")
‘2014’ is the larger source, saved as B; ‘2012’ saved as A.
This function outputs a list suited for StEM(), where data sources have been encoded and labelled as encodedA and encodedB. It additionally includes the number of unique values per PIV n_values, the validated PIVs_config, the logical same_mistakes and a data set of indices of true links true_pairs when uniq_id was provided. It alerts the user to potential issues, including inconsistent configurations, strongly associated PIVs and values outside the common support. For instance, the following warnings appear on the Emilia-Romagna subsets of the 2012 and 2014 waves of the SHIW data:
PIV ‘BirthYear’: values out of the common support in data1 [1920, 1912]
in data2 [2013, 2014].
PIVs ‘Occupation’ and ‘Sex’ are associated (Cramer’s V = 0.32 in data1,
0.35 in data2); FlexRL assumes conditional independence between PIVs,
consider merging or dropping one.
Among the true links, known in this application, there are some registration errors or temporal changes in the PIVs. As a consequence the true links do not necessarily carry the exact same information in their records. With the RL_agreement function, the user can see the proportion of agreements among pairs of records (for example among linked pairs of records or, in the case presented below, among true links).
BirthYear Sex MaritalStatus BirthRegion Occupation
0.99 1.00 0.98 0.99 0.82
In Table 3, we compare the record linkage methods mentioned in Table 1 on this regional subset of the SHIW data. Memory usage is of the same order of magnitude on these relatively small data sets for all methods. The figures and tables based on the SHIW data reflect a single random realisation. They depend on the seed we use in the script and on the seed some other packages use internally.
| Naive | BRL | multilink | fastLink | reclin2 | diyar | fedmatch | FlexRL | |
| TP | 710.00 | 158.00 | 160.00 | 467.00 | 721.00 | 20.00 | 510.00 | 297.00 |
| FP | 3,494.00 | 24.00 | 72.00 | 850.00 | 339,232.00 | 0.00 | 969.00 | 103.00 |
| FN | 188.00 | 740.00 | 738.00 | 431.00 | 177.00 | 878.00 | 388.00 | 601.00 |
| sensitivity | 0.79 | 0.18 | 0.18 | 0.52 | 0.80 | 0.02 | 0.57 | 0.33 |
| FDP | 0.83 | 0.13 | 0.31 | 0.65 | 1.00 | 0.00 | 0.66 | 0.26 |
| time (secs) | 0.37 | 17.78 | 1,118.04 | 33.69 | 0.84 | 11.04 | 8.88 | 22.57 |
| memory (Mb) | 8.30 | 523.20 | 1,901.90 | 2,436.20 | 1,180.20 | 699.50 | 213.20 | 263.20 |
Table 3. Comparison of record linkage methods on the SHIW in Emilia-Romagna (for the default linkage decision rule, see § 4.3).
StEM estimation
The FlexRL hierarchical model for record linkage adapts Eq. (2) to two sources under the one-to-one assignment constraint. For non-linked records, the true values are independent draws from the population distribution. For a linked pair , is a copy of except when is a dynamic PIV, whose values may have changed between the two registrations. The registered values are versions of the true values distorted by registration errors.
With parameters , the complete data likelihood factorises as
The parameter governs the possible distortion mechanisms (mistakes, missing values) at the source of the differences between registered and latent true values. The parameter shapes the model for temporal changes between latent true values in data sources depending on the chosen ‘structure’ (see the different survival models for Eq. (4)). The parameter characterises the distributions of the PIVs. The parameter represents the probability that an observation in forms a link with an observation in . Each component of Eq. (3) and each parameter is developed in the original methodological paper [43].
Notably, the probability that true values have not changed between data collections is modelled with a survival function. By default, the package uses an exponential model with proportional hazards. For a dynamic structured PIV whose changes are modelled and a linked pair with time gap :
where contains an intercept and, optionally, covariates from either data source that explain the hazard of change (for example age or income for a relocation) and contains the log hazard coefficients. The user can choose another survival model, such as a Weibull, a Gompertz, a piecewise constant hazard or any custom model.
The likelihood of the observed data is obtained by summing Eq. (3) over all values of the latent and . FlexRL maximises it with the StEM() function, a Stochastic Expectation Maximisation (StEM) algorithm [7]. In a classical EM, the E-step computes the conditional expectation of the complete likelihood given the observed data. In the stochastic version, this expectation is approximated by Monte Carlo: a Gibbs sampler simulates the latent variables given the current parameters. The averages of the summary statistics collected after burn-in estimate the conditional expectation from which the M-step derives the parameter updates. Algorithm 1 summarises the procedure, detailed hereafter.
Algorithm Stochastic EM for record linkage.
Input: encoded sources ; PIV configuration; iterations (burn-in ); iterations (burn-in )
Initialise
for StEM iter do
for Gibbs iter do ▷ stochastic E-step
draw
form candidate pairs such that for all stable or flexible
for each non-linked candidate pair: propose to link
for each linked candidate pair: propose to break the link
accept each proposal with the likelihood of the model
if Gibbs iter then
accumulate summary statistics:
number of linked pairs,
latent true values distribution,
temporal changes in linked ,
registration errors in linked and non-linked
end if
end for
average the accumulated summary statistics ▷ estimate
update and store ▷ M-step
end for
average of over StEM iter
run further Gibbs iterations at to build :
proportion of iterations in which is linked
Initialisation
The probability is drawn uniformly in , the distribution of each PIV , , are uniform over the values, the probabilities of mistakes are drawn in and the probabilities of missing values are set to the observed proportions. For structured PIVs, the log baseline hazard is drawn between 0.01 and 1 per time unit and the covariate coefficients near 0. The user can supply other starting values.
Stochastic E-step
Given , the Gibbs sampler alternates two blocks. First, the latent true values are drawn. For a non-linked record or is drawn from weighted by the registration model. For a linked pair is drawn jointly among the possible values, from weighted by the registration model and the survival model.
Second, the linkage is updated. Candidate pairs are formed from matching sampled latent true values: records and are candidates when for every non-structured PIV (stable or flexible). For each candidate pair , we evaluate the log likelihood contribution to Eq. (3) when the records are linked, , and when they are not, . A sequential sweep over the candidate pairs then proposes to create or break links. A proposal is accepted by comparing the proposed ‘new’ state of Eq. (3) to the current ‘old’ state with probability ; the two states differ by and by the contributions from changes in the linkage matrix.
Over the Gibbs iterations, the sampler stochastically explores the latent space of possible linkage matrices and PIVs’ true values. At each iteration after burn-in we store the number of pairs that were linked, the distribution of latent true values, the temporal changes (disagreements between the linked records latent true values) and the registration errors (disagreements between G and H)—summary statistics required for the M-step.
Restricting the sweep to candidate pairs keeps the algorithm light in memory: likelihood contributions are computed only for these pairs and for the remaining non-linked records in each data source, which together represent the estimated unique entities in the population.
M-step
The summary statistics averaged over the Gibbs draws after burn-in give the parameter updates. Thanks to the factorisation of Eq. (3), , and have separate closed form updates [43], and is obtained by numerical optimisation. Optional bounds on the probabilities of a mistake are applied at this stage. A warning is displayed when a bound is reached in many iterations, suggesting that the bound is too tight or that the PIV is dynamic.
Linkage scores and decision rule
Based on the StEM chains of estimates, we build an estimate of the set of parameters, , by averaging the post burn-in elements in the sequence. These are used to sample further linkage matrices that are averaged to return a probabilistic estimate of as a final output, namely , whose entries are the posterior probabilities for the pair to be linked, which we call the linkage scores. A decision rule on each entry then defines the final set of linked data . Since the scores of a record sum to at most one, a threshold of at least 0.5 ensures the one-to-one assignment [51] (Sadilne 2017). The choice of is the subject of the upcoming Section 4.
Note that probabilistic record linkage methods estimate in comparable ways, by classifying pairs according to their linkage scores. Although these scores may be on different scales (e.g. likelihood ratios, log odds, posterior probabilities), they can be mapped to a probability scale. Hereafter, we take linkage scores to be posterior linkage probabilities in and treat all the methods listed in Table 1 in the same way. Accordingly, the default decision rule is . The ‘FDP versus linkage decision rule’ curves (see Figure 3) provide a more comprehensive assessment of record linkage performance.

Figure 3. Estimated FDP and discrepancy measures against the linkage decision rule threshold obtained with FlexRL RL_diagnostics() on the SHIW in Emilia-Romagna.
Record linkage with StEM()
StEM() runs Algorithm 1. Its main arguments are:
data: a list, typically the output ofprepare_data(), with the encoded sourcesencodedAandencodedB, the number of distinct values of each PIVn_values(excluding missing values), a logical indicating whether both sources share the registration error parameterssame_mistakes, and the PIVs configurationPIVs_config.StEM_iter,StEM_burnin,gibbs_iter,gibbs_burnin: the numbers of iterations of the outer StEM algorithm and of the inner Gibbs sampler, and their burn-in periods.gibbs_burnin = 0sets the burn-in automatically: the sampler runs until the number of links changes by fewer than 10 for 10 consecutive iterations, with a maximum of 500 iterations.threads: the number of OpenMP threads used by the Rcpp samplers, default 4. The argument is ignored on systems where the package is built without OpenMP.model_dynamics: the survival model for structured PIVs, by default the exponential model of Eq. (4). It is only used if a PIV is declared structured inPIVs_config.n_post_sample: the number of posterior draws used to compute the linkage scores.Starting values for and for the agreement probabilities can be supplied.
For long runs,
new_directoryandsave_info—wheresavethe environment after each StEM iteration, andmusic_on = TRUEplays a short tune when the algorithm finishes running.
The function returns Delta, a data frame with columns i (indices in ), j (indices in ) and x (scores ) for every pair with a positive score, and the four chains: gamma (a vector of length StEM_iter), eta (one StEM_iter matrix per PIV), alpha (one matrix per structured PIV, with StEM_iter rows and one column per parameter of the survival model) and phi (one StEM_iter 4 matrix per PIV: agreement in , in , missing in , in ).
We run the FlexRL StEM algorithm for record linkage as follows:
fit <- StEM(data = prep_data)
Computational aspects
The two latent variable samplers are implemented with Rcpp. To draw the true values, sampleH() goes through the joint values once per PIV for linked pairs and through the values once per PIV for non-linked records. sampleD() performs the sequential sweep over candidate pairs, recording the current linkage in two vectors that give, for each record of and of , the index of the record it is linked to (or a negative value when not linked). Reading and updating the linkage takes a fixed number of operations, so the sweep is linear in the number of candidate pairs and the memory is linear in the number of records rather than in .
Candidate pairs are found by hashing the pattern of sampled true values on the non-structured PIVs, an operation linear in . Their contributions to the complete data likelihood are computed in a single pass over the records and the candidate pairs, implemented in Rcpp. Built-in survival models are evaluated in Rcpp; custom survival functions keep their R implementation.
The single pass over the candidate pairs likelihood contributions can be parallelised over OpenMP threads, each processing a disjoint set of candidates. Without OpenMP the code runs sequentially. All random draws occur upstream, so the threads never call the R random number generator. Results are therefore identical regardless of the number of threads, and set.seed() ensures reproducibility.
Running time grows with the number of records; the number of candidate pairs, larger when PIVs have few levels; the number of PIVs and their numbers of levels; the number of structured PIVs, whose survival model is estimated at each iteration; the number of iterations; and decreases with the number of threads.
Inference on linked data
Impact of linkage errors on linear regression
Suppose we are interested in estimating the association between recorded in and recorded in , for instance through the slope of the linear regression , with and , so that
The target is in the population of entities present in both sources. The analyst only observes the linked data with from and from . equals for truly linked pairs but belongs to another entity for falsely linked records. Among the linked data, a proportion (that may in reality depend on ) carries the correct and a proportion carries a spurious one. Below we denote by the estimator of , with for pairs in . Truly linked pairs of records (true positives, TP) are thus denoted and falsely linked pairs of records (false positives, FP) . Heuristically,
where and denote expectations among truly and falsely linked pairs.
Let us recast accordingly. If has the same mean and variance among all data in , then is a mixture of and , defined as in (5) for the subsets of TP and FP,
Linkage errors are often assumed to occur at random, in which and . However, the PIVs, often demographic, are likely to be associated with variables of interest in the analysis, so it is more realistic to consider an association between and among falsely linked pairs.
The estimation error of the analysis can be decomposed into two parts:
Missed links may cause sample selection and falsely linked data may cause measurement error in the linked variables [58, 14] (Robach et al. 2026).
Linkage decision rule trade-off
Raising the threshold on the linkage scores reduces the number of falsely linked pairs, which lowers the FDP and hence the contamination bias. Falsely linked data would distort the signal in proportion to the FDP. Indeed, from Eq. (7) we derive:
However, raising the threshold also selects fewer and more atypical records, which may increase the sampling bias due to the implicit record linkage selection. Indeed, the chance of being linked may be related to ; in the context of linear regression, this yields:
These aspects motivate the complementary tools we develop in FlexRL to quantify these two biases. Estimating the FDP (see § 5.2) and evaluating the divergence of the linked sample from the data sources (see § 5.3) over a range of values of may help users appraise this trade-off.
False discovery proportion
For the decision rule , the FDP is where and are the numbers of truly and falsely linked pairs. In real data applications this quantity is unknown. Linkage errors can be estimated from the scores of the linkage model [3, 40, 17], from synthetic data [30, 42] or from a separate model under conditions on the distribution of the number of records per entity [5, 11, 12]. FlexRL implements the first two estimation procedures. We focus on the FDP because it drives the contamination bias of Eq. (9); the sampling bias is addressed by the diagnostics of § 5.3. The estimators are provided as functions of , together with the linked sample size available for inference, so that the user can choose a threshold controlling the estimated FDP (and hopefully, the true underlying FDP).
Score based estimator
If the linkage model is well calibrated, should represent the probability that the pair is a true link, and the expected number of truly linked pairs among linked data is . Therefore
This estimator comes at no extra cost for any method that returns linkage scores, but it is only as good as the linkage model.
Notably, the calibration of the scores and the record linkage performance are affected by missing values and mistakes, the size of the overlap between the sources, the number of entities in the population, the discriminative power of the PIVs, their changes over time, and their marginal and joint distributions. The literature discusses the strengths and limitations of different record linkage paradigms in detail, including comparisons of FlexRL and BRL across a range of configurations [43 a].
Synthetic data estimator
An estimator that does not rely on linkage scores can be computed with synthetic data [42]. Source is augmented with synthetic records, drawn from a generative model fitted on the PIVs of —for example arf [59], synthpop [37] or mice [55]. Synthetic records by definition do not form a link in , so synthetic links can represent false positives when links and non-links have similar distributions [42]. Then is rescaled by , which follows from equating with . As for the score based estimator, corresponds to the number of (real) linked pairs :
The procedure is repeated over several synthetic draws and the results are averaged. The estimator does not depend on the linkage model, but it depends on the quality of the data synthesis.
When the two estimators disagree, a practitioner should rely on the one whose assumptions are more plausible. The score based estimator relies on the correct specification of the linkage model, including its specific assumptions. It reflects the performance the model would achieve under correct specification. The synthetic data estimator relies on the fidelity of the synthesiser and on the assumption that links and non-links have similar distributions. Earlier work assessed the robustness of both estimators to the degree of overlap between the data sources, the discriminative power of the PIVs and differences between the PIV distributions of true links and non-links [42], but not to deviations from PIV independence. In the SHIW in Emilia-Romagna application, the PIV independence is violated (occupation and sex are associated; see the warning in § 4.1). For models assuming independent PIVs, such as FlexRL and Fellegi-Sunter mixture models (see Eq. (1)), we expect the score based estimator to be biased, as its validity rests on the violated assumption. The synthetic data estimator though, should remain valid provided the synthesiser captures the dependencies among PIVs. This is in line with what we observe in the output of Figure 4.

Figure 4. Example of output from the print() method on FlexRL RL_diagnostics class, at threshold 0.86, on the SHIW in Emilia-Romagna.
Representativeness of the linked sample
As explained in § 5.1, the linked sample may differ from the data sources in ways the analyst cannot see [14, 58] (Robach et al. 2026). In particular, we appraise the sampling bias of Eq. (10) by comparing the distributions of variables of interest between the linked data (‘selected’ through record linkage) and the original data sources. For any decision rule , FlexRL compares each source with its linked subset using the following measures, computed on sets of variables and available in sources and respectively:
Agreement rates: the proportion of linked pairs whose records agree on , for each variable (such as the PIVs).
Standardised Mean Differences (SMD): , for a continuous variable in a given set; the same quantity is computed for the indicator of each level when is categorical.
Overlap of supports (support IoU): the Intersection over Union (IoU) of the sets of values taken in the linked sample and in the source, for a variable in a given set.
Maximum Mean Discrepancy (MMD): a multivariate measure of the difference between the linked sample and the source, computed jointly over a given set of variables, with a Gaussian kernel and the median heuristic for the bandwidth [21].
These quantities are computed over a grid of values of . They are built on the declared pairs and, when available, their linkage scores; they therefore apply to any linkage method.
In Table 4 we proceed with the demonstration of Section 2.4 showcasing the functionalities of FlexRL on a simulation study. The synthetic data FDP estimates (‘est. FDP* synth’) are not reliable for fedmatch and FlexRL (dynamics), possibly because of the specific linkage procedures these packages implement. For FlexRL (dynamics), the cause is likely the misspecified PIV dynamics model, or the added difficulty of estimating the FDP under modelling of PIV dynamics, since the estimate for FlexRL (no dynamics) is reliable.
| Naïve | BRL | multilink | fastLink | reclin2 | d y i a r | fedmatch | FlexRL (dynamics) | FlexRL (no dynamics) | |
| TP | 118.40 (5.13) | 80.40 (5.04) | 80.60 (5.95) | 101.70 (6.48) | 114.10 (6.38) | 34.90 (20.24) | 122.10 (6.33) | 114.00 (7.92) | 87.50 (8.30) |
| FP | 83.10 (11.02) | 12.30 (3.23) | 14.20 (4.13) | 43.10 (5.59) | 68.30 (8.82) | 3.30 (2.91) | 125.80 (6.32) | 25.90 (5.15) | 19.00 (4.67) |
| FN | 81.60 (5.13) | 119.60 (5.04) | 119.40 (5.95) | 98.30 (6.48) | 85.90 (6.38) | 165.10 (20.24) | 77.90 (6.33) | 86.00 (7.92) | 112.50 (8.30) |
| sensitivity | 0.59 (0.03) | 0.40 (0.03) | 0.40 (0.03) | 0.51 (0.03) | 0.57 (0.03) | 0.17 (0.10) | 0.61 (0.03) | 0.57 (0.04) | 0.44 (0.04) |
| FDP | 0.41 (0.03) | 0.13 (0.03) | 0.15 (0.03) | 0.30 (0.03) | 0.37 (0.03) | 0.08 (0.05) | 0.51 (0.03) | 0.18 (0.03) | 0.18 (0.04) |
| est. FDP score | NA | 0.19 (0.02) | NA | 0.37 (0.02) | 0.44 (0.03) | NA | 0.10 (0.01) | 0.20 (0.01) | 0.18 (0.02) |
| est. FDP* score | NA | 0.21 (0.02) | NA | 0.39 (0.02) | 0.43 (0.07) | NA | 0.10 (0.01) | 0.21 (0.01) | 0.19 (0.02) |
| est. FDP* synth | NA | 0.18 (0.10) | 0.20 (0.05) | 0.33 (0.07) | 0.45 (0.14) | 0.14 (0.07) | 0.08 (0.03) | 0.37 (0.11) | 0.27 (0.08) |
| max SMD | NA | 0.22 (0.05) | 0.22 (0.06) | 0.17 (0.03) | 0.24 (0.04) | 0.68 (0.31) | 0.15 (0.03) | 0.17 (0.04) | 0.22 (0.04) |
| min support IoU | NA | 0.80 (0.01) | 0.80 (0.01) | 0.80 (0.01) | 0.80 (0.01) | 0.76 (0.10) | 0.82 (0.03) | 0.82 (0.03) | 0.81 (0.02) |
| max MMD | NA | 0.01 (0.00) | 0.01 (0.00) | 0.01 (0.00) | 0.01 (0.00) | 0.08 (0.05) | 0.01 (0.00) | 0.01 (0.00) | 0.01 (0.00) |
| time (secs) | 0.27 (0.04) | 0.97 (0.05) | 18.57 (1.48) | 1.29 (0.24) | 0.29 (0.01) | 1.17 (0.06) | 1.44 (0.01) | 4.59 (0.02) | 3.72 (0.01) |
| memory (Mb) | 1.04 (0.05) | 78.60 (10.08) | 105.68 (10.18) | 24.85 (4.42) | 41.52 (0.18) | 104.31 (10.84) | 42.14 (2.59) | 102.11 (9.03) | 70.18 (6.42) |
Table 4. Linkage performance of competing record linkage methods on simulated data for the default decision rule , with linked data quality assessed through FlexRL. The * symbol denotes the record linkage augmented task, see § 5.2. We report average performance over 10 replications, with standard deviations in parentheses.
This short simulation study demonstrates the potential of the FlexRL package. Since FlexRL provides linked data quality diagnostics for any record linkage method, we report three estimates of the FDP along with measures of the divergence of the linked sample from the data sources. § 5.2 presented the FDP estimators implemented in the package and § 5.3 discussed the measures of divergence. The FDP estimators are averaged over 5 procedures in each of the 10 replications.
Diagnostics
RL_diagnostics() gathers the tools of § 5.2 and § 5.3, summarised in Table 5, and returns an S3 object of class RL_diagnostics, with print and plot methods. All measures are computed over a grid of thresholds (when the linkage method allows it by returning scores) or for a fixed threshold when specified. Each diagnostic can also be computed on its own. We present all commands hereafter but show only some outputs. Further details on these diagnostic tools are given in the package vignette ‘Linkage and Diagnostics with FlexRL’ available through vignette('FlexRL-RL-vignette', 'FlexRL') or the online documentation.
| Diagnostic | Computation | Display |
| Convergence | plot.StEM_convergence() on the chains returned by StEM() | plot(x, ‘convergence’) |
| Linkage scores | plot_linkage_scores() | plot(x, ‘scores’) |
| FDP estimation | compute_RL_FDP_score() (Eq. (11)), compute_augmRL_FDP_synth() (Eq. (12)) | plot(x, ‘FDP’) |
| Discrepancy measures | smd(), support_iou(), mmd(), RL_agreement() | plot(x, ‘discrepancy’) |
| Data comparison | plot.distributions() on each source and its linked subset | plot(x, ‘distributions’) |
Table 5. Diagnostics performed by RL_diagnostics().
Diagnostics
The function takes several arguments as input. The record linkage output fit, the data sources encodedA and encodedB, the set of variables of interest to compare compare_vars, their type vars_type_cont (TRUE if continuous) and the PIVs are passed to RL_diagnostics(), which computes FDP estimates and discrepancy measures that characterise the linked sample across a grid of thresholds . The true_pairs enables reporting the true FDP and sensitivity when known.
compare_vars <- c("BirthYear", "Sex", "MaritalStatus",
"BirthRegion", "Occupation")
vars_type_cont <- list(BirthYear = TRUE, Sex = FALSE, MaritalStatus = FALSE,
BirthRegion = FALSE, Occupation = FALSE)
diag <- RL_diagnostics(fit, prep_data$encodedA, prep_data$encodedB,
compare_vars, vars_type_cont, PIVs,
true_pairs = prep_data$true_pairs, FDP_estimation = TRUE,
RL_method = "FlexRL", data = prep_data)
The FDP estimation is optional (FDP_estimation = TRUE) since it can be time consuming. For a given RL_method, it is repeated n_repeats times and the estimates are averaged. compute_RL_FDP_score() returns the score based estimate on the original task. compute_augmRL_FDP_synth() adds n_synth records generated by synth_method to , links the augmented sources as an auxiliary task and computes both the score based and the synthetic data estimates on it. Similar score based estimates on the original and augmented tasks support the use of the synthetic data estimator. Table 4 reported these three estimates as ‘est. FDP score’, ‘est. FDP* score’, ‘est. FDP* synth’.
The plot(x, type) method for an object of class RL_diagnostics draws the plots listed in Table 5, with type being one of ‘convergence’, ‘scores’, ‘FDP’, ‘discrepancy’ and ‘distributions’. With the ‘convergence’ plots, we could verify that the StEM chains have stabilised.
plot(diag, ‘convergence’)
With the ‘scores’ plot, we can analyse the distribution of the linkage scores of the fit as shown in Figure 2.

Figure 2. Distribution of the linkage scores returned by FlexRL StEM() on the SHIW in Emilia-Romagna.
plot(diag, ‘scores’)
Albeit bimodal, the distribution of linkage scores produced by the record linkage algorithm does not indicate where to set the threshold . The FDP estimation results and the divergence plots may help further. In Figure 3 we show the ‘FDP’ plot and some of the ‘discrepancy’ plots for the MMD and the SMD. Similar plots for the overlap of supports and the agreement rate are usually displayed by plot(diag, ‘discrepancy’).
plot(diag, ‘FDP’)
plot(diag, ‘discrepancy’)
The graphics from Figure 3 show how each linkage decision rule affects linked data quality. They illustrate the trade-off of § 5.1: as the threshold increases, the estimated FDP diminishes while the divergence of the linked sample from the sources grows. Ideally, we choose an appropriate threshold for inference at the ‘elbow’ of the divergence curves, provided that the estimated FDP at that level is reasonable. This decision rule is a heuristic motivated by the potential impact of linkage errors on inference, as discussed in § 5.1, and its reliability has not been assessed. Note that the divergence measures are indicators of the sampling bias but do not estimate Eq. (10). For our example, we choose following this rule, as indicated by the vertical line in Figure 3.
With the ‘distributions’ plots, we could compare the distribution of the linked data and of the data sources at the selected threshold—for example .
plot(diag, ‘distributions’, threshold = 0.86)
The print() method for an S3 object of class RL_diagnostics reports the linked sample size, the FDP estimates, the true FDP and sensitivity when available, and the discrepancy measures. We report the output in Figure 4. As discussed in § 5.2, the FDP estimators have different strengths and weaknesses. In the SHIW in Emilia-Romagna, the synthetic data estimator yields an estimate of 0.10 and the score based estimator yields an estimate of 0.02, compared with the true value of 0.16.
Throughout the diagnostics, the record linkage is run by wrappers with a common form, link_with_<pkg>(dataA, dataB, ...), returning the indices of linked pairs and their linkage score. Wrappers exist for FlexRL, fastLink, BRL, reclin2, multilink, diyar and fedmatch. That way, the diagnostic tools apply to any of these methods in a similar fashion.
On the illustrative example developed along this chapter, we regress the expected age of retirement in 2014 on the expected age of retirement in 2012 (originally named ‘etapen’ in each wave). After preparation of the SHIW in Emilia-Romagna data for record linkage (see prepare_data() in § 4.1), we work with 1,479 and 1,524 records in the two data sources, which share 898 true links. The solid red line in Figure 5 is the reference estimate computed on these true links. The tools presented so far help identify a suitable linkage threshold by assessing the trade-off between contamination and sampling bias. In Figure 5, a low threshold yields a linked data estimate (dotted orange) affected by contamination bias, due to noisy falsely linked data (short dashed yellow estimate); see Eq. (9). Conversely, a high threshold yields a linked data estimate (dotted orange) affected by sampling bias, due to atypical truly linked data (long dashed yellow estimate); see Eq. (10). Guided by the diagnostic tools, we choose , which yields a linked data estimate (dotted orange) closer to the reference. At this threshold, we retain 282 linked observations, corresponding to an FDP of 0.16 and a sensitivity of 0.26.

Figure 5. Regression analysis on the SHIW in Emilia-Romagna linked through FlexRL.
Scalability
In Table 3 of § 4, we compared record linkage methods on the SHIW in Emilia-Romagna. Despite heterogeneous linkage performance, running time and memory usage, all methods ran at that scale. To complete our investigation, we demonstrate the scalability of FlexRL and examine that of other record linkage methods on the complete SHIW data sets. After preparation, the 2012 and 2014 waves over the 20 Italian regions cover 19,215 and 20,002 individuals, with 10,568 individuals observed in both waves. We consider two settings, without and with regional blocking. Note that blocking on the region assumes that this variable is stable and free of registration errors.
In § 4.1, we showed the proportion of agreements between the information carried by records among true links of the SHIW in Emilia-Romagna. Below we show the proportion of agreements among true links of the complete SHIW data. In this application, the region is indeed free of registration errors and stable.
BirthYear Sex MaritalStatus BirthRegion Occupation Region
0.99 1.00 0.98 1.00 0.82 1.00
Record linkage becomes harder on large data sets: the discriminative power of the PIVs is unlikely to grow with the size of the sources, whereas the number of candidate records in for each record in does. Without blocking, we may therefore expect higher FDPs and also higher sensitivities, since true links that blocking excludes (due to registration errors or changes between waves) become recoverable. However, in the complete SHIW the region is stable and error free so the expected gain does not apply. FDPs and sensitivities are similar without and with blocking for the comparable methods, except for FlexRL which has both a higher FDP and a higher sensitivity without blocking. Note that prepare_data() eliminates out-of-support records at regional levels, which explains the differences in the partial confusion matrix of the ‘Naïve’ method between Table 6 and Table 7.
| Naive | BRL | multilink | fastLink | reclin2 | diyar | fedmatch | FlexRL | |
| TP | 8,289.00 | NA | NA | 5,061.00 | NA | NA | 5,057.00 | 4,105.00 |
| FP | 49,350.00 | NA | NA | 11,175.00 | NA | NA | 14,154.00 | 5,417.00 |
| FN | 2,279.00 | NA | NA | 5,507.00 | NA | NA | 5,511.00 | 6,463.00 |
| sensitivity | 0.78 | NA | NA | 0.48 | NA | NA | 0.48 | 0.39 |
| FDP | 0.86 | NA | NA | 0.69 | NA | NA | 0.74 | 0.57 |
| time (mins) | 0.00 | NA | NA | 9.00 | NA | NA | 5.00 | 6.00 |
| memory (Mb) | 123.20 | NA | NA | 20,070.10 | NA | NA | 1,877.00 | 4,030.30 |
Table 6. Comparison of record linkage methods on the complete SHIW data application for the default decision rule .
Below, we report the number of unique values of each PIV in the SHIW in Emilia-Romagna and in the complete SHIW. These counts indicate the discriminative power of the PIVs in the regional and national applications. In addition, we appraise the difficulty of the tasks,
without and with regional blocking, through the performance of the ‘Naive’ approach in Table 6 and Table 7, which reflects how many records share the same combination of PIV values.
BirthYear Sex MaritalStatus BirthRegion Occupation 94 2 4 18 52
BirthYear Sex MaritalStatus BirthRegion Occupation Region 99 2 4 20 84 20
Table 6 presents the comparative study without blocking; ‘Not Available’ results indicate memory errors. When supported, methods run with four threads (threads for FlexRL, n.cores for fastLink, nthread for fedmatch). Table 7 shows the results with blocking on the Italian regions. More details on this data application showcasing the scalability of FlexRL are given in the package vignette ‘Scalable Record Linkage with FlexRL’ available through vignette(‘FlexRL-RL-block-vignette’, ‘FlexRL’) or the online documentation.
Discussion
This chapter presented FlexRL: an R package for probabilistic record linkage of two overlapping data sources using a hierarchical model that accounts for registration errors and accommodates temporal changes in Partially Identifying Variables (PIVs). FlexRL provides tools
| Naive | BRL | multilink | fastLink | reclin2 | diyar | fedmatch | FlexRL | |
| TP | 8,289.00 | 1,609.00 | 1,771.00 | 4,911.00 | 8,993.00 | 316.00 | 5,284.00 | 2,916.00 |
| FP | 49,346.00 | 449.00 | 1,092.00 | 11,295.00 | 4,092,774.00 | 12.00 | 13,123.00 | 1308.00 |
| FN | 2,059.00 | 8,739.00 | 8,577.00 | 5,437.00 | 1,355.00 | 10,032.00 | 5,064.00 | 7,432.00 |
| sensitivity | 0.80 | 0.16 | 0.17 | 0.48 | 0.87 | 0.03 | 0.51 | 0.28 |
| FDP | 0.86 | 0.22 | 0.38 | 0.70 | 1.00 | 0.04 | 0.71 | 0.31 |
| time (mins) | 0.33 | 3.63 | 206.82 | 5.82 | 0.42 | 2.67 | 2.25 | 5.03 |
| memory (Mb) | 16.00 | 1,150.00 | 3,837.00 | 4,716.00 | 2,223.00 | 1,530.00 | 766.00 | 862.00 |
caption: Comparison of record linkage methods on the complete SHIW data application with regional blocking, for the default decision rule .
:::
for assessing the quality of the resulting linked data. These diagnostics are independent of the linkage method and thus apply broadly. These tools are essential for using record linkage in practice, where potential linkage errors and induced sample selection cannot be observed directly.
To draw attention to the implications of linkage quality for statistical analysis, FlexRL provides two complementary tools as functions of the linkage threshold. First, the estimated False Discovery Proportion (FDP), which quantifies contamination among the linked pairs. Second, the divergence of the linked sample from the source populations, which captures potential sampling bias. Examining these quantities jointly makes the trade-off between contamination and sampling bias explicit and provides users with information to select a linkage decision rule in light of its potential consequences for subsequent inference.
Several directions follow naturally from this work. On the computational side, the memory footprint is linear in the number of records and the running time scales with the number of records, candidate pairs, PIVs, their levels and configurations. Memory and running time allow record linkage between large data sets on a standard computer. On the modelling side though, the linkage model currently assumes independence of PIVs given the latent true values, no duplicates within a source, categorical PIVs; extensions would broaden the range of applications. Moreover, the diagnostic estimators and the proposed quality measures, whose validity rests on the assumptions underlying their estimation, would benefit from further theoretical and empirical investigation.
Notwithstanding these considerations, FlexRL provides a framework that connects record linkage to linked data quality assessment. It offers a solution to record linkage that is scalable in terms of memory usage and running time, and flexible in accommodating diverse scenarios. Linkage performance depends on the difficulty of the task and the chosen decision rule, controlling the FDP and recovering a fraction of the true links accordingly. It brings record linkage into practice, helping researchers and practitioners not only to link data, but also to quantify and communicate the extent to which linkage decisions may affect downstream inference.
Author contributions statement
KR developed the record linkage algorithm pioneered by MH and implemented tools for inference conceptualized by MW. All authors drew up the simulation framework. KR conducted the analysis, prepared the first manuscript draft, reviewed and edited by MH and MW. All authors read and approved this manuscript.
Acknowledgments
The authors thank the anonymous reviewers for their valuable suggestions and for conducting a thorough reproducibility check.
Data and Code Availability
The FlexRL package is available from the Comprehensive R Archive Network (CRAN) or from the package GitHub repository alongside documentation and vignettes describing its functionality. FlexRL is also available as a module in JASP 1.0 [25]. The script is available as supplementary material and on the GitHub repository of the package.
Financial disclosure
This research received no specific grant from any funding agency in the public, commercial, or not-for-profit sectors.
Conflict of interest
The authors are the developers of FlexRL.
References
- [1]S. Aleshin-Guendel. Multifile Record Linkage and Duplicate Detection, 2023. URL https://CRAN.R-project.org/package=multilink. R package version 0.1.1.
- [3]T. R. Belin and D. B. Rubin. A method for calibrating false-match rates in record linkage. Journal of the American Statistical Association, 90(430):694–707, 1995. ISSN 01621459, 1537274X. URL http://www.jstor.org/stable/2291082.DOI
- [4]O. Binette and R. C. Steorts. (almost) all of entity resolution. Science Advances, 8(12):eabi8021, 2022. doi: 10.1126/sciadv.abi8021.DOI
- [5]T. Blakely and C. Salmond. Probabilistic record linkage and a method to calculate the positive predictive value. International Journal of Epidemiology, 31(6):1246–1252, 12 2002. ISSN 0300-5771. doi: 10.1093/ije/31.6.1246. URL https://doi.org/10.1093/ije/31.6.1246.
- [7]G. Celeux and J. Diebolt. L’algorithme SEM : un algorithme d’apprentissage probabiliste pour la reconnaissance de mélange de densités. Revue de Statistiques Appliquées, 34(2):35–52, 1986. URL http://www.numdam.org/item/RSA_1986__34_2_35_0/.
- [8]M. Cleaton, J. Hall, R. Shipsey, Z. White, and K. Xhaferaj. A case study of using Splink: Census duplicate matching. In Proceedings of Statistics Canada Symposium 2022. Data Disaggregation: Building a More Representative Data Portrait of Society. Statistics Canada, 2024.
- [9]G. J. Cohen, J. Dice, M. Friedrichs, K. Gupta, W. Hayes, I. Kitschelt, S. J. Lee, W. B. Marsh, N. Mislang, M. Shaton, M. Sicilian, and C. Webster. The U.S. syndicated loan market: Matching data. Journal of Financial Research, 44(4):695–723, 2021. doi: https://doi.org/10.1111/jfir.12261. URL https://onlinelibrary.wiley.com/doi/abs/10.1111/jfir.12261.
- [10]T. Cooney, S. Yu, S. Daral, M. Field, B. Krebs, M. Kieran, P. Manley, S. Govinda Raju, and S. Zelt. Healthcare resource use for pediatric low-grade glioma care: A cohort of linked electronic health records and claims data. Journal of Clinical Oncology, 41(16_suppl):10038–10038, 2023. doi: 10.1200/JCO.2023.41.16_suppl.10038. URL https://ascopubs.org/doi/abs/10.1200/JCO.2023.41.16_suppl.10038.
- [11]A. Dasylva and A. Goussanou. Estimating linkage errors under regularity conditions. In Proceedings of the Survey Research Methods Section. American Statistical Association, 2020.
- [12]A. Dasylva and A. Goussanou. On the consistent estimation of linkage errors without training data. Japanese Journal of Statistics and Data Science, 5, 2022. doi: 10.1007/s42081-022-00153-3.DOI
- [13]J. de Bruin. Python record linkage toolkit: A toolkit for record linkage and duplicate detection in python, 2019. URL https://doi.org/10.5281/zenodo.3559043. Python package.
- [14]J. C. Doidge and K. L. Harron. Reflections on modern methods: linkage error bias. International Journal of Epidemiology, 04 2020. doi: 10.1093/ije/dyz203.DOI
- [15]H. L. Dunn. Record linkage. American Journal of Public Health and the Nations Health, 36(12):1412–1416, 1946.
- [16]S. B. Dusetzina, S. Tyree, A. M. Meyer, A. Meyer, L. Green, and W. R. Carpenter. Linking data for health services research: A framework and instructional guide [internet]. https://www.ncbi.nlm.nih.gov/books/NBK253312/, 2014. URL https://www.ncbi.nlm.nih.gov/books/NBK253312/. An Overview of Record Linkage Methods.
- [17]T. Enamorado, B. Fifield, and K. Imai. Using a probabilistic model to assist merging of large-scale administrative records. American Political Science Review, 113(2):353–371, 2019. doi: 10.1017/S0003055418000783.
- [18]I. P. Fellegi and A. B. Sunter. A theory for record linkage. Journal of the American Statistical Association, 64(328):1183–1210, 1969.DOI
- [19]M. Fortini, B. Liseo, A. Nuccitelli, and M. Scanu. On Bayesian record linkage. Research in Official Statistics, 4(6):185–198, 2001.
- [20]M. Friedrichs, C. Webster, B. Marsh, J. Dice, and S. Lee. fedmatch: Fast, Flexible, and User-Friendly Record Linkage Methods, 2023. URL https://CRAN.R-project.org/package=fedmatch. R package version 2.0.6.
- [21]A. Gretton, K. M. Borgwardt, M. J. Rasch, B. Schölkopf, and A. Smola. A kernel two-sample test. Journal of Machine Learning Research, 13:723–773, 2012.DOI
- [22]K. L. Harron, J. C. Doidge, H. E. Knight, R. E. Gilbert, H. Goldstein, D. A. Cromwell, and J. H. van der Meulen. A guide to evaluating linkage quality for the analysis of linked data. International Journal of Epidemiology, 46(5):1699–1710, 09 2017. ISSN 0300-5771. doi: 10.1093/ije/dyx177. URL https://doi.org/10.1093/ije/dyx177.
- [24]M. A. Jaro. Advances in record-linkage methodology as applied to matching the 1985 census of tampa, florida. Journal of the American Statistical Association, 84(406):414–420, 1989.DOI
- [25]JASP Team. JASP (Version 1.0.0)[Computer software], 2026. URL https://jasp-stats.org/.
- [26]Z. Jelveh, M. McNeill, and T.-Y. Lin. Namematch, 2022. URL https://github.com/urban-labs/namematch. Python package.
- [27]R. Kramer. Towards scalable probabilistic linkage of administrative population data in Germany: Insights from a method test with Splink. International Journal of Population Data Science, 11(4), 2026. doi: 10.23889/ijpds.v11i4.3814.
- [28]P. Lahiri and M. D. Larsen. Regression analysis with linked data. Journal of the American Statistical Association, 100(469):222–230, 2005. ISSN 01621459. URL http://www.jstor.org/stable/27590532.DOI
- [29]J. Laidler, A. Imaz Blanco, and D. Balasubramanian. Probabilistic linkage pipeline improving linkage quality and explainability in healthcare. International Journal of Population Data Science, 10(4), 2025. doi: 10.23889/ijpds.v10i4.3271.
- [30]J. Lam, A. Boyd, R. Linacre, R. Blackburn, and K. Harron. Generating synthetic identifiers to support development and evaluation of data linkage methods. International Journal of Population Data Science, 9(1), July 2024. doi: 10.23889/ijpds.v9i1.2389. URL https://ijpds.org/index.php/ijpds/article/view/2389.
- [31]M. D. Larsen. Multiple imputation analysis of records linked using mixture models. Proceedings of the Survey Methods Section, Statistical Society of Canada Annual Meeting, 63:65–71, 1999.
- [32]R. Linacre, S. Lindsay, T. Manassis, Z. Slade, T. Hepworth, R. Kennedy, and A. Bond. Splink: Free software for probabilistic record linkage at scale. International Journal of Population Data Science, 7(3), Aug. 2022. doi: 10.23889/ijpds.v7i3.1794. URL https://ijpds.org/article/view/1794.
- [33]N. G. Marchant, A. Kaplan, D. N. Elazar, B. I. P. Rubinstein, and R. C. Steorts. d-blink: Distributed end-to-end Bayesian entity resolution. Journal of Computational and Graphical Statistics, 30(2):406–421, 2021.DOI
- [34]N. G. Marchant, B. I. P. Rubinstein, and R. C. Steorts. Bayesian graphical entity resolution using exchangeable random partition priors. Journal of Survey Statistics and Methodology, 11(3):569–596, 2023.
- [35]S. Mudgal, H. Li, T. Rekatsinas, A. Doan, Y. Park, G. Krishnan, R. Deep, E. Arcaute, and V. Raghavendra. Deep learning for entity matching: A design space exploration. In Proceedings of the 2018 International Conference on Management of Data, SIGMOD ’18, page 19–34, New York, NY, USA, 2018. SIGMOD 18: Proceedings of the 2018 International Conference on Management of Data, Association for Computing Machinery. ISBN 9781450347037. doi: 10.1145/3183713.3196926. Code available at https://pypi.org/project/deeppart/.DOI
- [36]H. B. Newcombe, J. M. Kennedy, S. J. Axford, and A. P. James. Automatic linkage of vital records. Science, 130(3381):954–959, 1959. doi: 10.1126/science.130.3381.954.DOI
- [37]B. Nowok, G. M. Raab, and C. Dibben. synthpop: Bespoke creation of synthetic data in r. Journal of Statistical Software, 74(11):1–26, 2016. doi: 10.18637/jss.v074.i11. URL https://CRAN.R-project.org/package=synthpop.
- [38]O. Nsonwu. diyar: Record Linkage and Epidemiological Case Definitions in ’R’, 2023. URL https://olisansonwu.github.io/diyar/index.html. R package version 0.5.1.
- [39]M. Pontesilli, M. H. Hof, A. C. J. Ravelli, A. J. van Altena, A. T. Soufan, B. W. Mol, E. H. Kostelijk, E. Slappendel, D. Consten, A. E. P. Cantineau, L. A. J. van der Westerlaken, W. van Inzen, J. C. M. Dumoulin, L. Ramos, E. B. Baart, F. J. M. Broekmans, P. M. Rijnders, M. H. J. M. Curfs, S. Mastenbroek, S. Repping, T. J. Roseboom, and R. C. Painter. Effect of parental and ART treatment characteristics on perinatal outcomes. Human Reproduction, 36(6):1640–1665, 04 2021. ISSN 0268-1161. doi: 10.1093/humrep/deab008. URL https://doi.org/10.1093/humrep/deab008.
- [40]D. Resnick and J. Asher. Measurement of type i and type ii record linkage error. In Using and Linking Administrative and Auxiliary Data, Government Statistics Section. Joint Statistical Meetings, 09 2019. URL https://ww2.amstat.org/meetings/proceedings/2019/data/assets/pdf/1199462.pdf.
- [41]K. Robach and M. H. Hof. FlexRL: Flexible Record Linkage and Inference on Linked Data, 2026. URL https://CRAN.R-project.org/package=flexrl. R package version 2.0.0.
- [42]K. Robach, M. H. Hof, and M. A. van de Wiel. False discovery estimation in record linkage. Statistics in Medicine, 44(23-24):e70292, 2025a. doi: https://doi.org/10.1002/sim.70292. URL https://onlinelibrary.wiley.com/doi/abs/10.1002/sim.70292.
- [43]K. Robach, S. L. van der Pas, M. A. van de Wiel, and M. H. Hof. A flexible model for record linkage. Journal of the Royal Statistical Society C: Applied Statistics, page qlaf016, Feb. 2025b. ISSN 0035-9254. doi: 10.1093/jrsssc/qlaf016. URL https://doi.org/10.1093/jrsssc/qlaf016.
- [45]M. Sadinle. Bayesian estimation of bipartite matchings for record linkage. Journal of the American Statistical Association, 112(518): 600–612, 2017.arxiv.org/abs/1601.06630
- [46]M. Sadinle. Beta Record Linkage, 2020. URL https://CRAN.R-project.org/package=BRL. R package version 0.1.0.DOI
- [47]R. Steorts, R. Hall, and S. Fienberg. SMERED: A Bayesian Approach to Graphical Record Linkage and De-duplication. In S. Kaski and J. Corander, editors, Proceedings of the Seventeenth International Conference on Artificial Intelligence and Statistics, volume 33 of Proceedings of Machine Learning Research, pages 922–930, Reykjavik, Iceland, 22–25 Apr 2014a. PMLR. URL https://proceedings.mlr.press/v33/steorts14.html.
- [48]R. C. Steorts. Entity resolution with empirically motivated priors. Bayesian Analysis, 10(4):849–875, 2015.DOI
- [49]R. C. Steorts, S. L. Ventura, M. Sadinle, and S. E. Fienberg. A Comparison of Blocking Methods for Record Linkage. Springer International Publishing, 2014b.arxiv.org/abs/1407.3191
- [50]R. C. Steorts, R. Hall, and S. E. Fienberg. A Bayesian approach to graphical record linkage and deduplication. Journal of the American Statistical Association, 111(516):1660–1672, 2016.DOI
- [51]A. Tancredi and B. Liseo. A hierarchical Bayesian approach to record linkage and population size problems. The Annals of Applied Statistics, 5(2B), 2011.
- [52]A. Tancredi and B. Liseo. Regression analysis with linked data: problems and possible solutions. Statistica, 75(1):19–35, Jan. 2015. doi: 10.6092/issn.1973-2201/5821. URL https://rivista-statistica.unibo.it/article/view/5821.DOI
- [53]K. I. Ted Enamorado, Ben Fifield. fastLink: Fast Probabilistic Record Linkage with Missing Data, 2023. URL https://CRAN.R-project.org/package=fastLink. R package version 0.6.1.
- [54]B. J. Tepping. A model for optimum linkage of records. Journal of the American Statistical Association, 63:1321–1332, 1968.DOI
- [55]S. van Buuren and K. Groothuis-Oudshoorn. mice: Multivariate imputation by chained equations in r. Journal of Statistical Software, 45(3):1–67, 2011. doi: 10.18637/jss.v045.i03. URL https://CRAN.R-project.org/package=mice.DOI
- [56]J. van der Laan. reclin2: Record Linkage Toolkit, 2025. URL https://github.com/djvanderlaan/reclin2. R package version 0.6.0.
- [58]Z. Wang, E. Ben-David, G. Diao, and M. Slawski. Regression with linked datasets subject to linkage error. WIREs Computational Statistics, 14(4):e1570, 2022. doi: https://doi.org/10.1002/wics.1570. URL https://wires.onlinelibrary.wiley.com/doi/abs/10.1002/wics.1570.
- [59]D. S. Watson, K. Blesch, J. Kapar, and M. N. Wright. Adversarial random forests for density estimation and generative modeling. https://arxiv.org/abs/2205.09435, 2023. URL https://CRAN.R-project.org/package=arf.
- [60]E. J. Williamson, A. J. Walker, K. Bhaskaran, S. Bacon, C. Bates, C. E. Morton, H. J. Curtis, A. Mehrkar, D. Evans, P. Inglesby, J. Cockburn, H. I. McDonald, B. MacKenna, L. Tomlinson, I. J. Douglas, C. T. Rentsch, R. Mathur, A. Y. S. Wong, R. Grieve, D. Harrison, H. Forbes, A. Schultze, R. Croker, J. Parry, F. Hester, S. Harper, R. Perera, S. J. W. Evans, L. Smeeth, and B. Goldacre. Opensafely: factors associated with covid-19-related hospital death in the linked electronic health records of 17 million adult nhs patients. MedRxiv, pages 430–436, 2020.DOI