WorksComputer Science
ARIDA - Active railway infrastructure dataset annotation using neural network predictions
Abstract
Modern railway perception requires large comprehensive image datasets, but creating them is slowed by manual labeling, especially for track polylines and other complex infrastructure geometry. This paper presents the Active Railway Infrastructure Dataset Annotation framework, a railway-specific semi-automated pipeline that converts instance predictions from a mask scoring instance segmentation network into bounding-boxes, polygons, polylines, and paired polylines. To keep automatic track geometry annotation reliable without ground truth, it applies confidence thresholds and an unique contour-agreement quality score as an acceptance gate. An active learning loop ranks remaining images by uncertainty and adapts the number of manual queries per cycle. This is achieved through a novel quantity control mechanism, in which easy images are auto-accepted and human effort targets hard cases. Experiments on disjoint, diverse imagery and time-sequential, redundant imagery quantify effort reduction and accuracy trade-offs, achieving about two thirds reduction in the disjoint regime and up to 93% in the sequential regime. The study also provides guidance for initial label budgets and stopping to avoid over-annotation.
Introduction
Computational perception through Artificial Intelligence (AI) is a safety-critical enabler for Advanced Driver-Assistance Systems (ADAS) and higher levels of automation in railway transportation. Reliable scene understanding supports functions such as obstacle detection, track availability assessment, and operational decision-making, where errors can have direct safety and economic consequences. [12][22][17]
While deep learning has substantially advanced visual perception in other domains, progress in autonomous railway perception remains comparatively limited, and one of the primary bottlenecks is the availability of large, high-quality annotated datasets [23][15].
A central challenge is, that public railway datasets are not only scarce [11], but the available datasets, most notably Open Sensor Data for Rail 2023 (OSDaR23) [15] and the currently most comprehensive RailSem19 [23], are often insufficient for training modern computer vision systems at scale. Such existing public datasets are limited in size and do not provide the data variety or annotation primitives required to learn robust models for real-world operation. At the same time, creating improved or more advanced railway datasets in an own endeavour is challenging with current tooling. [4][15]
Railway perception commonly requires instance-based geometric annotations beyond standard bounding-boxes or masks (see Figure 1), including coordinate-based polylines and polyline-pairs that represent track geometry [23].

Figure 1. Cropped example images of the annotation types contained in the RailSem19 dataset. Only a single annotation type with an arbitrarily chosen color is shown per image for demonstration purposes. Images source: [23]
These primitives are essential for track understanding yet are particularly labor-intensive to produce [4][23]. Moreover, many general-purpose annotation tools like LabelMe [6] or the Computer Vision Annotation Tool (CVAT) [9] (s. related work in section II) are not tailored to railway requirements and do not support the full set of required primitives. In addition, when annotation budgets are constrained, it is not sufficient to merely assist labeling. One must also prioritize which samples should be labeled to maximize performance gains per human annotation. This creates a reinforcing limitation: datasets remain suboptimal for training, but producing better datasets is prohibitively expensive and technically cumbersome. [4][13][7] These constraints motivate the following research question RQ:
RQ: How can manual annotation effort be significantly reduced when creating high-quality railway image datasets with multi-primitive annotations suitable for training modern perception models?
In this work, we use annotation effort to denote the human workload required to transform raw railway imagery into a usable training set. This includes the frequency with which human annotators must intervene to correct or complete labels, and the overall share of images that require manual labeling rather than simple verification. Reducing annotation effort therefore means reducing the amount of manual geometric drawing and limiting human intervention to genuinely difficult cases, while still producing accurate, application-relevant annotations.
To address the RQ, we propose the Active Railway Infrastructure Dataset Annotation (ARIDA) framework. ARIDA is a semi-automated image annotation framework that combines standalone model-based annotation capabilities with data-efficient sample selection. ARIDA is centered around an instance-segmentation model (Mask Scoring region-based convolutional neural network (MS R-CNN) [5]), that generates per-instance predictions. An annotation module then converts these predictions into fully featured RailSem-compatible primitives with a dedicated procedure to derive polyline-pair track geometries from predicted track masks. We choose RailSem-style primitives because they represent the most complete publicly established railway annotation scheme and include the track-critical polyline-pair primitives that many pipelines do not support [23][4]. To ensure that automatically generated track annotations are reliable without requiring ground truth, ARIDA includes a no-ground-truth quality gate based on our Contour F Score (CFS). Finally, ARIDA integrates an Active Learning (AL) loop that prioritizes informative and difficult samples for human labeling, so that the annotator effort is concentrated where it yields the highest benefit [13][7]. To our knowledge, no prior railway-specific framework combines semi-automatic multi-primitive annotation generation, track-polyline-pair quality gating without ground truth, and an integrated AL-loop.
We hypothesize firstly, that ARIDA reduces manual annotation effort compared to fully manual labeling by automatically producing a substantial fraction of annotations and routing only low-confidence or low-quality cases to human annotators. Secondly, we hypothesize, that AL improves sample efficiency for railway instance segmentation, achieving comparable model performance with fewer human-labeled images than non-selective supervised training [13].
The main contributions of this paper are:
ARIDA framework: A railway-specific semi-automatic pipeline with focus on automated annotation designed to reduce manual dataset-creation effort with an instance segmentation and iterative AL core.
Multi-primitive annotation module: Conversion of predictions into RailSem-style primitives, with a dedicated focus on polyline-pair track annotation.
CFS quality control: A no-ground-truth metric used as an acceptance gate for automatically generated track polyline-pairs.
AL evaluation baseline: First time evaluation of AL for railway instance segmentation, providing a baseline for future research.
Empirical guidance: Guidance for parameterization of ARIDA on commonly occuring disjoint versus time-sequential datasets in the railway domain.
The remainder of this paper is organized as follows: § 1 reviews related work, Section III presents ARIDA and its components, while Section IV describes the experimental setup and evaluation results. In § 4, findings and limitations are discussed with § 5 concluding the paper.
Related Work
The railway domain continues to face a pronounced data bottleneck. Public datasets for railway perception are comparatively rare and often limited in size, scope, or annotation richness. Creating new datasets remains costly because high-quality railway annotations are labor-intensive to produce. [11][4] Prior work addresses this problem from two complementary directions. Upstream approaches aim to reduce human effort during annotation, whereas downstream approaches aim to reduce the amount of labeled data needed to reach a target model performance. To keep the discussion directly comparable to ARIDA, we structure the related work along the following dimensions:
Domain specificity: Is the method railway-focused or general-purpose?
Data modality: Is the approach designed for image data and/or other data types (e.g. Light Detection and Ranging (LiDAR))?
Annotation primitives: Which primitives are supported, and are they instance-level applicable?
Automation level: To what extent does the approach reduce manual annotation effort?
Sample selection: Does the approach include a method to prioritize samples for annotation?
Upstream optimization: Annotation procedures
General-purpose semi-automated annotation tools follow an assistive paradigm. Models provide suggestions that humans verify and correct, improving suggestions over time. CVAT exemplifies this approach and supports image and video annotation with suggestion workflows for bounding-boxes and polygons [9]. However, such tools are not tailored to railway-specific topology, semantics, or workflows, which limits their efficiency for railway image annotation despite their broad primitive support.
The strongest recent railway-specific upstream baseline is Labels4Rails [4], a railway image annotation tool for track geometry, switches, and scene tags with automation for ego-track annotation and switch derivation. Compared with CVAT, it reports weighted-average per-image annotation time reductions of 42 % in manual mode and 58 % with automated annotation and manual amendment. However, it remains centered on semi-automated labeling under continuous human supervision and its reported automation is limited in scope. [4] In contrast, ARIDA targets automatic generation of all railway primitives considered here and uses quality gating to restrict human effort to uncertain or low-quality cases.
Downstream optimization: Data-efficient Active Learning
AL addresses the data bottleneck from the opposite direction. By selecting the most informative unlabeled samples in a dataset for human annotation, model improvement per labeled sample can be maximized. [13]
In the railway domain, Denzler et al. demonstrate this for multi-sensor, sequence-based object detection. Using uncertainty estimation to select low-confidence sequences leveraging temporal multi-sensor context and label interpolation, they report reaching fully-annotated-dataset performance with 25 % fewer annotations and outperforming a random baseline by 2 percentage points. While this validates AL’s downstream value, the approach is sequence-dependent, focused on bounding-box detection, and not designed as an image-dataset annotation pipeline supporting richer railway primitives. [2]
Wulff et al. [19] illustrate the synergy of upstream bootstrapping and downstream AL, but in a LiDAR-centric setting. They use model-generated labels and an AL-loop to focus manual effort on the most informative LiDAR scans. 110 of 602 scans required manual correction after five cycles achieving 71.48 % performance, yet their pipeline is not designed for image annotation while also not supporting an image-centered multi-primitive output. [19]
Railway track detection procedures
Even with progress in assisted annotation and AL, fine-grained track geometry remains one of the most labor-intensive parts of railway dataset creation. Track detection models can in principle provide reusable geometry for annotation automation, but they differ substantially in how directly their outputs align with annotation needs.
Segmentation approaches
1) Segmentation approaches: Segmentation approaches like RailNet (Wang et al. [18]) output pixel-wise track-region masks [18]. From ARIDA’s viewpoint, they are image-based and railway-specific, but their outputs are class masks rather than coordinate primitives. Producing polylines and polyline-pairs requires additional post-processing, whereby instance separation is not inherent.
Hybrid approaches
2) Hybrid approaches: Hybrid approaches like the method of Tao et al. [16] combine learned predictions with deterministic geometry extraction to yield coordinate-based outputs that are closer to annotation needs. Such approaches recover rail coordinates from intermediate representations and output polylines, sometimes with instance separation. [16] However, these methods are typically dedicated perception pipelines, focusing on operational use (e.g. ego-drivepath extraction) and a single primitive type, rather than being designed as a multi-primitive annotation pipeline with sample selection.
End-to-end deep learning approaches
3) End-to-end deep learning approaches: End-to-end approaches directly regress coordinate-based rail representations and can produce instance-separated polylines (and sometimes polyline-pairs), but they still typically output only a single primitive family and do not address dataset annotation holistically. DALNet (Yu et al. [21]) for instance, yields instance-separated polylines without post-processing via an anchor-line formulation [21]. In principle, these outputs could be converted into polyline-pairs and other primitives, making the representation compatible with annotation needs.
Taken together, recent railway-specific upstream work has reduced the gap to generic tools, with Labels4Rails demonstrating substantial time savings for railway image annotation [4]. Downstream studies further show that AL can lower labeling demand in railway perception [2]``[19]. Track-detection methods provide increasingly annotation-compatible geometry, but remain perception pipelines rather than dataset-construction frameworks [16]``[21]. ARIDA addresses the remaining gap by combining railway-specific multi-primitive annotation, quality-gated automation, and model-driven sample selection.
Materials and Methods
To address the data scarcity and tooling limitations identified in the Introduction and Related Work, we propose the ARIDA framework. ARIDA is a comprehensive semi-automated pipeline for annotating railway image datasets efficiently. This section describes the framework modules and their internal procedures in detail. For the automatic annotation capability, we focus on rail-track annotation as it is the most challenging task and remains insufficiently studied in this principle.
ARIDA implements a semi-automatic workflow that reduces manual labeling effort using an AI-model. The model generates instance-level predictions that are converted into structured annotations. Figure 2 provides a simplified overview of the system architecture.
Figure 2. Simplified overview of the ARIDA-framework architecture. All framework modules and data flows comprising AL are shown in red, while raw MS R-CNN predictions are shown in black or grey (after sorting). Automatic annotations obtained by the annotation module and manual human annotations are shown in light-blue. Numerical procedure control data for and from the TBAL-module is shown in violet. The scalar control signal used for the Query Quantity Parameterization is shown in dark blue.
The core of ARIDA combines MS R-CNN with an annotation module. This base system enables automatic generation of instance-based annotations from model predictions. It produces bounding-boxes, polygons, polylines, and polyline-pairs, as illustrated in Figure 1, using the hybrid approach.
We adopt a hybrid design because most related approaches from the previous Section generate either semantic segmentation or a single annotation primitive. Producing all required primitives with separate specialized models would substantially increase training time, memory consumption, and system complexity. Such a multi-model setup would also require additional aggregation and post-processing for quality assurance and AL. Moreover, many existing approaches do not provide instance-based annotations, which our annotation policy requires. Our hybrid design therefore combines one instance-based model with deterministic conversion to deliver the full set of annotation types.
Mask Scoring R-CNN
MS R-CNN [5] is a descendant of the Mask region-based convolutional neural network (Mask R-CNN) [3] network architecture principle designed for efficient instance segmentation of up to 100 image objects via a concatenated two-stage approach (object detection and instance segmentation) [5][3].
Firstly, Regions of Interest (RoI), proposed by the Region Proposal Network (RPN), are projected on image feature maps, which are then resized into a square shape through the RoI-align component. These square feature maps are then evaluated by a regression network to predict class scores and bounding-box coordinates for up to 100 object instances. The second network stage uses the object detection results to create semantic segmentation masks for each detected image object instance. [3] MS R-CNN adds a "maskIoU" network extension additionally, which regresses an Intersection over Union (IoU) quality score of the predicted masks [5], giving ARIDA a more reliable indicator on the mask predictions quality compared to the normally referred object classification score. This is especially important, as the mask quality directly influences the annotation quality in the subsequent annotation module.
MS R-CNN provides qualitative instance-level predictions, including bounding-boxes and segmentation masks, which our annotation module converts into the required railway primitives. Its dedicated maskIoU head explicitly estimates mask quality, enabling reliable filtering of low-quality predictions and supporting uncertainty and informativeness scoring for AL. MS R-CNN is also available in established toolchains, can be fine-tuned with standard hyperparameters, and supports specialized AL strategies [20]. Finally, it remains a strong-performing instance-segmentation baseline, making it a suitable backbone for semi-automated annotation.
Annotation module and the Contour F Score
The annotation module is part of the base system and converts raw MS R-CNN predictions to the RailSem19 data format using four annotation type specific coordinate generation algorithms. Because railway tracks are among the most important objects to annotate [23], we focus mainly on the polyline-pair annotation type generation algorithm in this section. Other primitives are generated using substeps of this algorithm (polygons through contour coordinates, polylines through the centerline) or directly from the model predictions (bounding-boxes). The scheme for annotating railway tracks (label: "rail") through the polyline-pair annotation algorithm is exemplarily shown for a single object instance in the following Figure 3.
First, MS R-CNN predicts an instance segmentation mask for each detected rail object. We convert the soft mask into a binary mask using Otsu thresholding [10] and RoI-based histogram analysis. Spurious regions within the RoI are removed using a blob artifact filtering step.
Second, we apply a distance transform [14] to the binary mask to obtain a distance map. The rail centerline is extracted by sliding a window along the RoI and selecting local maxima. Specifically, for each image row, we choose the pixel with the highest pixel value, representing the center of the rail track at that row. Using this approach, we obtain the centerline coordinates as a set of points, regardless of track curvature.
Third, we fit the extracted centerline coordinates with a fourth-degree polynomial to smooth outliers without excessive oscillation. The fitted points are then sorted by ascending -coordinate to enable sequential processing (3a). In parallel, we extract and aggregate contour coordinates from the binary mask to represent both rail boundaries (3b).
Because a RailSem19 polyline-pair consists of two polylines [23], contour points must be assigned to one track side. We therefore classify each contour coordinate relative to the fitted centerline from step 3a. The assignment is performed using a 2D cross-product side-membership test, illustrated in Figure 4.
For each contour point , we select the nearest centerline point as reference. We define from to and from to its successor. A preliminary indicator is computed as the cross product of the L2-normalized vectors:
The side membership is then obtained from the sign of .
Points with are assigned to the right polyline, and to the left polyline. If , the point is treated as invalid and excluded from both polylines. We apply this procedure to all contour points to construct the complete RailSem19 polyline-pair annotation. The resulting annotation is exemplarily shown in Figure 3 with both polylines drawn as red lines on top of the input image. In the last step of the annotation module (step 5 in Figure 3), the generated annotations are filtered by the network quality indicators and or our own CFS-score.
Figure 3. Schematic of the polyline-pair annotation algorithm for railway tracks with label "rail". All images have been cropped to fit only one rail track instance exemplarily. For the sake of clarity, the first step of the pipeline shows both the input image to MS R-CNN and the predicted instance segmentation mask overlayed on top of each other.
The CFS-score is a custom quality indicator for automatically generated rail-track polyline-pair annotations. It adapts the score [1] to assess geometric consistency using contour agreement. The score is defined as:
where is the weighted harmonic mean of precision (P) and recall (R) [1]. Precision and recall are computed from pointwise matches between two contour sets:
Here, TP counts contour points consistent with the reference contour, FP counts erroneous assignments, and FN counts missing rail parts. We obtain these counts by overlaying the predicted rail contour prior to conversion with the contour reconstructed after the annotation pipeline. False positives typically arise from side-membership or centerline extraction errors, whereas false negatives reflect incomplete coverage. We set empirically to prioritize precision and penalize incorrect rail assignments.
CFS serves as a no-ground-truth quality gate that rejects low-quality polyline-pair annotations using a threshold. Combined with cls and maskIoU filtering, it enables largely autonomous generation of high-quality railway annotations. While Figure 3 and Figure 4 illustrate the conversion procedure schematically, Figure 5 presents a qualitative output example on a real, automatically annotated RailSem19 image, showing per-instance rail splitting and the curvature-following capability of the proposed polyline-pair generation algorithm.
Figure 4. Functionality behind the cross-product point side membership decisioning algorithm. The contour coordinates are represented as large light-blue dots, whereas the middle line coordinates are represented by small yellow dots. The contour points for which the side membership has to be exemplarily decided are shown as large violet dots.

Figure 5. Qualitative ARIDA output on "rs00057.jpg" from the RailSem19 dataset [23], shown here only for the polyline-pair rail annotation type as a showcase. The source image was cropped to emphasize the central content. Native ARIDA annotation information markers were deactivated. The maskIoU scores for the detected three rail instances are 0.98, 0.95 and 0.93, while the CFS scores are 0.96, 0.94 and 0.97 respectively (in colored order of red, orange and yellow), also indicating the observed high annotation quality.
Active Learning and Triplets based Active Learning
AL is an iterative training paradigm that reduces labeling effort by selecting informative samples for annotation. The underlying hypothesis is that a model reaches a target performance using fewer labels when it queries the most relevant unlabeled examples. To operationalize this idea, AL alternates between training the model and requesting new annotations from a human oracle. [13][7]
To enable systematic querying, the available dataset is partitioned into a labeled subset and an unlabeled subset . The model is initialized on and then applied to candidate images sampled from . A query strategy is required to assign each candidate a relevancy score and to select the next images for manual annotation. After annotation, the queried samples are added to and the model is retrained, forming an AL-cycle. [13][7] Within ARIDA, this cyclic optimization is essential to maintain reliable automatic annotations as image difficulty increases. Query strategies are commonly grouped into representativeness-based and informativeness-based methods [13][7].
Representativeness strategies seek diverse samples using clustering or density estimation [13][7], which is impractical for high-dimensional images. We therefore use informativeness-based querying, which selects samples where the model is uncertain. However, standard uncertainty measures such as class-probability entropy [7] are not directly suited for two-stage instance segmentation. MS R-CNN outputs a variable number of instances and distinct confidence indicators for classification and mask quality. Consequently, a single probability-based uncertainty score cannot capture stage disagreement or low mask fidelity. [20]
We therefore adopt the Triplets based Active Learning (TBAL) strategy by Xu et al. [20] and specifically tailor it to MS R-CNN, by omitting the box-quality term, which is not needed for the instance segmentation-focused annotation task in ARIDA. For each image and detected instance , we form the indicator tuple [20]:
where denotes the classification score and the predicted maskIoU score. These indicators quantify semantic confidence and estimated mask quality per object instance respectively. [20] Instance informativeness is then computed from their mean and standard deviation [20]:
where low average confidence and high indicator disagreement increase [20]. Image informativeness is then obtained by aggregating over all instances [20]:
This definition prioritizes images containing many uncertain instances and thus high expected labeling benefit. If no instances are detected, we propose to heuristically set to prioritize likely failure cases. This value is chosen empirically to exceed typical informativeness scores, obtainable from a maximum of 100 detectable objects using the general MS R-CNN architecture [3]. The highest-ranked images are sent to the oracle for annotation and incorporated into . The procedure repeats until a predefined stopping criterion, such as an annotation budget, is reached. [20][13] In case of ARIDA, this budget is dynamically controlled by the sorting module through the Query Quantity Parameterization (QQP) mechanism described in the next Section.
Sorting module and Query Quantity Parameterization
AL in ARIDA improves MS R-CNN cyclically by querying only high-value samples for manual annotation. This mechanism is particularly important for images that cannot be annotated automatically under set quality constraints. To further reduce effort, oracle queries should be issued only when automatic annotation is insufficient due to missing model capability. ARIDA therefore separates "easy" and "hard" images within each annotation cycle.
All images are first processed by the automatic annotation module using fixed acceptance criteria. Images that satisfy these criteria are removed from the oracle queue. Unsuccessful cases remain candidates for manual annotation and are prioritized by informativeness.
The sorting module receives input images and corresponding MS R-CNN predictions (s. Figure 2). It computes the TBAL informativeness score and ranks images accordingly. Low-informativeness images are processed first for automatic annotation, whereas high-informativeness images are reserved for the oracle. The number of successful automatic annotations in a cycle is denoted by .
The oracle query budget is adapted using QQP. It decreases monotonically with increasing according to our proposed formula:
Here, and define lower and upper bounds for oracle queries. Parameter specifies the maximum number of automatic annotations per cycle. When is small, increases to provide additional labeled data for difficult cases. When is large, is reduced because little oracle intervention is required, thus current model knowledge suffices.
Results
This section presents the results of evaluating the ARIDA-framework on the disjoint RailSem19 and sequential OSDaR23 datasets. Also, the contribution of the TBAL-based query strategy to the overall performance and effort reduction is analyzed in a dedicated evaluation step.
In the first part (A) of this section, the overall parameterization and the individual evaluation steps of the ARIDA-framework are explained. Subsequently, the results for each evaluation step are presented in the second part (B).
Parameterization and evaluation procedure
We evaluate ARIDA under two complementary railway data regimes. First, we study disjoint and heterogeneous imagery, where viewpoints and capture settings vary strongly. This setting stresses model generalization and makes sample selection harder. We use RailSem19 to represent this regime.
Second, we study sequential and highly redundant imagery from a fixed capture setup. This setting reflects practical data acquisition along a route and highlights the risk of over-annotation. We use OSDaR23 to represent this regime, though all scenery will be used for both training and evaluation. Based on these regimes, we perform three evaluation steps.
ARIDA on disjoint RailSem19 data to test robustness.
A TBAL study on RailSem19 to isolate query ranking effects.
ARIDA on sequential OSDaR23 data to test redundancy handling.
All experiments train MS R-CNN on a single-GPU server with limited memory. We therefore fix the per-iteration batch size to 1 and apply gradient accumulation to obtain an effective batch size of 2. This choice keeps training stable without substantially increasing operational time. We use AdamW with a learning rate of throughout. The complete training configuration is summarized in Table I.
| Parameter | Setting |
| Optimizer | AdamW |
| Learning rate | |
| Batch size | 1 |
| Gradient accumulation | 2 (effective batch size) |
| Pretraining epochs | 500 |
| Retraining epochs per AL cycle | 100 |
Table I. General training parameters (constant across experiments).
We deliberately evaluate the hardest track-annotation setting by restricting the task to one class and one primitive. Concretely, we only use polyline-pair annotations for training and for ARIDA’s annotation actions. To ensure that the learning signal is rail-relevant, we filter samples by their polyline-pair label "rail" for RailSem19 and "track" for OSDaR23. For each run, we construct different labeled pool sizes, the unlabeled pool and the evaluation set by random sampling with seed 1234 without overlap. This disjoint construction avoids leakage that would underestimate the oracle workload. Pool sizes, dataset-specific adjustments, and the fixed sampling seed are listed in Table II.
| Parameter | Setting |
| Default unlabeled pool size | |
| RailSem19 evaluation set size | |
| OSDaR23 evaluation set size | |
| OSDaR23 largest-run unlabeled pool | (remaining data) |
Table II. AL parameterization and pool construction.
For both RailSem19 and OSDaR23, we fix the unlabeled pool size to 1000 images. As OSDaR23 contains fewer total images, the evaluation set is reduced to 200 images in comparison to RailSem19’s 500 images. This allows for a sufficiently large unlabeled pool even when the initial labeled set is large. Each evaluation step begins with a pretraining phase on for 500 epochs. After every AL-cycle, the model is retrained for 100 epochs, since only a small number of images is added per cycle. Performance is measured after each cycle on using torchmetrics segmentation Mean Average Precision (mAP) [8] with its standard IoU sweep from 0.5 to 0.95 [8]. This yields learning curves as a function of the labeled set size .
ARIDA’s annotation module accepts automatic annotations only if their quality is sufficiently high. We gate predictions using three indicators: the classification confidence, the predicted maskIoU score, and the CFS score. The thresholds (Table III) were set empirically to reject low-quality automatic labels early and to route uncertain cases to the oracle instead. This trades some automation for higher label reliability and cleaner feedback during AL.
| Parameter | Setting |
| Classification score threshold | |
| Predicted maskIoU threshold | |
| CFS threshold |
Table III. Annotation module acceptance thresholds.
Oracle interaction is controlled by QQP, which bounds manual intervention per cycle while still allowing correction when needed. In each cycle, QQP limits the number of oracle-provided annotations to the interval and caps the number of automatically generated annotations. We simulate the oracle by revealing hidden ground truth labels from , because the goal is to isolate the contribution of ARIDA components without human variability. The QQP settings used in all experiments are given in Table IV.
| Parameter | Setting |
| Oracle-query lower bound | |
| Oracle-query upper bound | |
| Max. automatic annotations per cycle |
Table IV. QQP settings for oracle querying.
We run three evaluation steps to validate ARIDA and its components.
Disjoint RailSem19: compared to supervised baselines trained on labeled images.
TBAL study: RailSem19 is used again, comparing TBAL-high, random, and TBAL-low score selection under the best-performing configuration from step (1).
Sequential OSDaR23: compared to supervised baselines trained on labeled images.
For steps (1) and (3), we consider AL beneficial if the AL learning curve reaches a supervised baseline mAP while using fewer labeled images. For step (2), we consider TBAL effective if the highest-TBAL strategy yields higher mAP at a given labeled-set size than random or lowest-TBAL selection. In addition to accuracy, we quantify manual workload reduction by tracking the total number of oracle annotations . We compute the aggregate effort reduction relative to the unlabeled pool size as:
This measure accounts for both the initial bootstrapping effort and subsequent oracle interventions across cycles.
Results on effort reduction
Evaluation of arida on disjoint RailSem19 data
1) Evaluation of ARIDA on disjoint RailSem19 data: This experiment evaluates ARIDA on disjoint RailSem19 imagery to address RQ on reducing manual annotation effort while maintaining model utility. Also, both hypotheses regarding overall effort reduction through ARIDA (hypothesis 1) and AL reaching supervised accuracy with fewer manual labels (hypothesis 2) are tested in this experiment. Figure 6 reports the segmentation mAP progression over the current labeled-set size.
Figure 6. Segmentation mAP progression on disjoint RailSem19 data for three initial label budgets (50, 100 and 200). The x-axis shows the current labeled-set size . The horizontal markers indicate supervised baselines trained on 100, 200, and 500 labeled images.
Across all three runs, mAP increases as additional oracle-labeled samples enter . The run initialized with 50 labeled images shows the steepest improvement, but converges at the lowest final accuracy (mAP ). Initializing with 100 or 200 labeled images yields higher mAP throughout most of the process. Notably, the 100-initial configuration reaches comparable performance to the 200-initial run after roughly 200 – 315 labeled images. This behavior reduces over-annotation risk by avoiding a large up-front labeling commitment for small gains.
Comparing the AL curves to supervised baselines trained on fixed budgets (100, 200, and 500 images), the 50-initial run does not exceed any supervised baseline while utilizing less labeled data. Thus, hypothesis 2 is not supported for this scarce initialization. In contrast, the 100-initial run matches the supervised 200-image baseline at exactly 200 images and also surpasses the 200-image baseline mostly thereafter, confirming hypothesis 2 under moderate prior knowledge. None of the runs reach the 500-image baseline because the available unlabeled pool is exhausted before that budget. Importantly, for the early plateauing 50- and 100-initial runs, stopping after 325 labeled images in comparison to 390 labeled images for the 200-initial run results in a modest 5 percentage point drop in mAP, while saving 65 manual annotations. This illustrates how AL can act as a safeguard against over-annotation when returns diminish.
Table 5 summarizes the resulting effort reduction per run. All configurations reduce manual effort substantially, with between 61.6 % and 67.2 %. The 100-initial run achieves the highest effort reduction (67.2 %) while also providing the most favorable accuracy–cost trade-off. Starting from 200 labeled images reduces additional oracle queries, but lowers due to the doubled initial manual investment.
| 50 | 280 | 720 | 67.0% |
| 100 | 228 | 772 | 67.2% |
| 200 | 184 | 816 | 61.6% |
Table 5. Manual and automatic annotation counts, and resulting effort reduction, for disjoint RailSem19 runs.
In disjoint, diverse scenes, this pattern highlights the practical tension between small accuracy improvements and unnecessary up-front labeling.
On disjoint RailSem19 data, ARIDA consistently reduces manual labeling by about two thirds. The AL sample-efficiency hypothesis is supported for the 100-initial configuration, but not for 50 initial labels. Overall, 100 initial labels best balances accuracy, effort reduction, and over-annotation risk, thus making it the optimal parameterization for subsequent evaluations and practical use on disjoint data.
Evaluation of the TBAL-component
This experiment isolates the TBAL-based query strategy within ARIDA to assess its contribution to the paper’s RQ. In line with hypothesis 1, we measure how TBAL affects overall manual effort through and . In line with hypothesis 2, we analyze whether TBAL improves sample efficiency by reaching a given mAP with fewer labeled images. Because RailSem19 is highly diverse, avoiding over-annotation is especially relevant once gains plateau.
We reuse the best-performing disjoint-data configuration from Evaluation 1 with and compare three sampling strategies: highest TBAL score (most informative), random, and lowest TBAL score (least informative). Figure 7 shows the segmentation mAP progression over the current labeled-set size .
Figure 7. Segmentation mAP progression on disjoint RailSem19 data with fixed and three AL sampling strategies. The x-axis shows the current labeled-set size .
In the early phase (about 100 – 125 labeled images), selecting highest-TBAL samples yields a small mAP advantage over lowest-TBAL selection (up to 2 percentage points). Compared to random querying, the difference is minor, but the highest-TBAL curve tends to improve more steadily. Around , all three curves meet near mAP , indicating that query choice has limited influence at this stage. As labeling proceeds, lowest-TBAL selection shows the weakest growth, which translates into lower accuracy gain per additional annotation.
Random querying briefly outperforms highest-TBAL by up to 2 percentage points near , but this advantage does not persist. Beyond roughly , the highest-TBAL strategy becomes the most efficient again and ends with the best final mAP.
At the end of the process, highest-TBAL reaches approximately 50 % mAP, compared to 49 % for random and 48 % for lowest-TBAL. This difference is small in absolute terms, yet it is practically relevant when annotation should stop at a target quality to avoid over-annotation. For example, to reach 49 % mAP, highest-TBAL needs about 290 labeled images, whereas random querying requires about 330. In other words, informativeness-guided selection can save roughly 40 manual labels at a fixed target performance, directly supporting the goal of reducing unnecessary annotation work. Table VI confirms that this performance advantage coincides with higher end-to-end efficiency.
| Sampling strategy | |||
| Highest TBAL | 272 | 728 | 62.8% |
| Random | 293 | 707 | 60.7% |
| Lowest TBAL | 294 | 706 | 60.6% |
Table VI. Oracle and automatic annotation counts, and resulting effort reduction, for the TBAL comparison on RailSem19.
With the same initial pool of 100 labeled images, highest-TBAL queries the oracle least () and enables the most automatic annotations (), yielding the best effort reduction (%). Random querying and lowest-TBAL selection require about 21 – 22 more oracle annotations, reducing efficiency to %. These results align with hypothesis 1 by demonstrating measurable workload differences attributable to the query strategy. They also support hypothesis 2 in the sense that highest-TBAL reaches target mAP values with fewer labeled images than random selection.
Evaluation of ARIDA on sequential OSDaR23 data
This experiment evaluates ARIDA on time-sequential, potentially redundant railway imagery from OSDaR23. In this regime, many frames share similar content, so over-annotation is a primary practical risk.
Figure 8 shows segmentation mAP over the current labeled-set size for initial budgets from 10 to 500 images.
Figure 8. Segmentation mAP progression on sequential OSDaR23 data for different initial label budgets. The x-axis shows the current labeled-set size . Horizontal markers indicate supervised baselines trained on fixed budgets (20 - 1000 images).
The key pattern is that small initial budgets learn fastest, because early AL cycles still expose new information despite redundancy. In particular, the 50-initial run reaches about 79 % mAP (the supervised 100-label level) after roughly 60 labeled images, confirming hypothesis 2 at a practically relevant operating point and avoiding about 40 potentially redundant labels. The 20-initial run tracks the 50-initial curve closely, offering a safer starting point when over-annotation must be minimized.
By contrast, 10 initial labels can be the least risky upfront investment, but it reaches the supervised 20-label baseline only after roughly 40 labels and therefore does not satisfy hypothesis 2 early on.
For larger initial pools, redundancy dominates and marginal gains per new label shrink. The 100-initial run does not reach the 200-label baseline, yet AL still prevents unnecessary labeling by stopping near 125 images, saving about 75 labels for only a small accuracy gain. The same effect strengthens at 200 and 500 initial labels, where runs fail to catch the next larger baselines and mainly demonstrate AL as a safeguard against over-annotation. This behavior matches the intended role of AL in sequential streams, where performance plateaus well before the remaining frames are exhausted.
Table VII supports hypothesis 1 by showing very large effort reductions for small initial budgets.
| 10 | 87 | 913 | 90.3% |
| 20 | 52 | 948 | 92.8% |
| 50 | 45 | 955 | 90.5% |
| 100 | 21 | 979 | 87.9% |
| 200 | 22 | 978 | 77.8% |
| 500 | 9 | 239 | 31.9% |
Table VII. Oracle and automatic annotation counts, and resulting effort reduction, for sequential OSDaR23 runs.
The best overall reduction occurs at 20 initial labels with % while 10 and 50 initial labels remain above 90 %. Efficiency deteriorates as the initial pool grows, dropping to 77.8 % at 200 and to 31.9 % at 500 initial labels, because upfront labeling dominates the budget.
For time-sequential railway data, we recommend starting with 20 – 50 labeled images. The 20-initial setting is a strong default when over-annotation is a priority, while 50-initial provides a robust "fast ramp" to high mAP. It is recommended to avoid large initial pools with 200 or more labeled images on sequential streams, because redundancy limits returns and increases over-annotation risk.
To conclude, on OSDaR23 ARIDA achieves up to 92.8 % effort reduction, supporting hypothesis 1. Hypothesis 2 is best supported for 20 – 50 initial labels, where AL reaches strong accuracy with fewer manual annotations. Over-annotation is mitigated most effectively by conservative initial budgets and AL-based early stopping.
Discussion
Modern railway perception requires large, high-quality datasets, yet manual annotation remains costly and operationally constrained. To address this problem, ARIDA combines automatic annotation, domain-specific quality gating, and AL so that human effort is concentrated on uncertain or informative samples. Across all experiments, ARIDA substantially reduced manual effort, but its benefit depended strongly on dataset structure. The results suggest that ARIDA is most effective either when the model must first cross a competence threshold before automatic acceptance and uncertainty-based querying become reliable, or when sequential redundancy makes additional manual labeling increasingly inefficient. For disjoint and heterogeneous data, the main finding is the existence of such a competence threshold.
In the present RailSem19 setting, an initial pool of about 100 labeled images provided the best trade-off between early model stability and upfront labeling cost. Below this level, strict acceptance gates reject many predictions and labeled-set growth depends mainly on oracle corrections, so uncertainty estimates are not yet sufficiently informative. Once this threshold is reached, both automatic acceptance and active selection become more effective. The TBAL study supports this interpretation: informativeness-based selection improves sample efficiency only after the detector has learned reasonably stable instance representations. For sequential and highly redundant data, the interpretation differs.
Here, ARIDA is less important for discovering diverse supervision than for preventing over-annotation once the model has already captured most of the available information. On OSDaR23, small initial pools of about 20 to 50 images reached strong performance quickly, while larger initial pools mainly increased upfront cost and reduced achievable effort savings. In this regime, AL acts primarily as a stopping and prioritization mechanism: easy samples are accepted early, and the remaining pool becomes progressively harder and less informative. The resulting performance plateau should therefore be interpreted as diminishing return rather than simple underperformance.
Direct comparison with earlier related work remains limited because existing studies differ in evaluation setup, annotation workflow, and effort metric. This is especially true for the closest railway-specific upstream reference, Labels4Rails, which reports wall-clock per-image annotation-time reductions in a semi-automated workflow with continuous human supervision and manual amendment [4]. By contrast, ARIDA targets reduction of manual intervention demand through quality-controlled automatic acceptance and AL, and measures effort through initial manual labels and oracle interventions relative to the unlabeled pool. Comparison with Labels4Rails is therefore informative at a qualitative level, but not directly quantitative.
These conclusions should be interpreted within the limits of the present study. All experiments used a single random seed, the oracle was simulated from hidden ground truth rather than implemented with human annotators, only two datasets were evaluated, and the experiments were restricted to one class and one complex primitive. In addition, acceptance thresholds and QQP settings were chosen empirically. The reported operating points should therefore be viewed as practically useful guidelines for this setup rather than universally optimal settings. Within these limits, the experiments suggest starting with a modest but nontrivial initial pool for disjoint data (roughly 5 % to 10 % of the unlabeled set) and a smaller initial pool for sequential data (about 20 to 50 images), while monitoring saturation and stopping when further gains become marginal.
Conclusion
This paper answers the research question RQ on how to reduce manual effort while still producing high-quality, railway-specific multi-primitive image annotations. It introduces ARIDA, a semi-automated framework that converts MS R-CNN instance predictions into RailSem-compatible primitives, including track polyline-pairs, and controls automatic track geometry annotation via the no-ground-truth CFS acceptance gate. ARIDA further integrates an AL-loop with TBAL-based informativeness ranking and a QQP, so that "easy" images are auto-accepted first and oracle effort is concentrated on genuinely hard, high-value samples.
Empirically, ARIDA reduces manual labeling by roughly two thirds on disjoint RailSem19 data (), with 100 initial labels providing the best accuracy–over-effort trade-off. On sequential, redundant OSDaR23 data, ARIDA achieves up to 92.8 % effort reduction, and small initial pools (20 – 50 labels) learn fastest while mitigating over-annotation through early saturation and AL-driven stopping. For future work, some limitations should be addressed to test the generality of these conclusions.
The most immediate next steps are multi-seed experiments, real human-in-the-loop evaluation with timing measurements, and extension to multi-class and multi-primitive workloads. It would also be useful to study the sensitivity of ARIDA to its acceptance thresholds and QQP settings more systematically, since these parameters directly shape both effort and accuracy. Finally, richer selection strategies, including temporal cues for sequential data, may further improve the balance between automatic annotation, oracle workload, and stopping decisions.
References
- [1]P. Christen, D. J. Hand, and N. Kirielle, “A review of the f-measure: Its history, properties, criticism, and alternatives,” ACM Comput. Surv., vol. 56, no. 3, Oct. 2023.DOI
- [2]P. Denzler, M. Ziegler, A. Jacobs, V. Eiselein, P. Neumaier, and M. Köppel, “Multi-sensor data annotation using sequence-based active learning,” in 2022 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), Kyoto, Japan: IEEE, Oct. 23, 2022, pp. 258–263.DOI
- [3]K. He, G. Gkioxari, P. Dollár, and R. Girshick, Mask r-CNN, Jan. 24, 2018. arXiv: 1703.06870[cs].arxiv.org/abs/1703.06870
- [4]T. Hiebert, F. Hofstetter, C. Thomas, S. Mushtaq, E. Kaan, and B. Parameswaran, “Labels4rails: A railway image annotation tool and associated reference dataset,” Data, vol. 10, no. 12, pp. 1–18, 2025.DOI
- [5]Z. Huang, L. Huang, Y. Gong, C. Huang, and X. Wang, Mask scoring r-CNN, Mar. 1, 2019. arXiv: 1903. 00241[cs].arxiv.org/abs/1903.00241
- [6]“Labelme: Open image annotation tool,” LabelMe. [Online]. Available: http://labelme.csail.mit.edu/Release3.0/
- [7]D. Li, Z. Wang, Y. Chen, R. Jiang, W. Ding, and M. Okumura, “A survey on deep active learning: Recent advances and new frontiers,” IEEE Transactions on Neural Networks and Learning Systems, vol. 36, no. 4, pp. 5879–5899, Apr. 2025.arxiv.org/abs/2405.00334
- [8]“Mean-average-precision (mAP) — PyTorch-metrics 1.9.0 documentation,” Accessed: Mar. 20, 2026. [Online]. Available: https://lightning.ai/docs/torchmetrics/stable/detection/mean_average_precision.html
- [9]“Open data annotation platform,” CVAT. [Online]. Available: https://www.cvat.ai/DOI
- [10]N. Otsu, “A threshold selection method from gray-level histograms,” IEEE Transactions on Systems, Man, and Cybernetics, vol. 9, no. 1, pp. 62–66, 1979.DOI
- [11]M. J. Papattera, F. Flammini, V. Vittorini, and N. Bešinović, “A systematic review of artificial intelligence public datasets for railway applications,” Infrastructures, vol. 6, no. 10, p. 136, 2021.
- [12]J. Peleska, A. E. Haxthausen, and T. Lecomte, “Standardisation considerations for autonomous train control,” in Leveraging Applications of Formal Methods, Verification and Validation: Practice, ser. Lecture Notes in Computer Science, T. Margaria and B. Steffen, Eds., vol. 13704, Cham: Springer, 2022, pp. 286–307.
- [13]B. Settles, “Active learning literature survey,” University of Wisconsin–Madison, Computer Sciences Technical Report 1648, 2009.
- [14]T. Strutz, The distance transform and its computation, Feb. 24, 2023. arXiv: 2106.03503[cs].arxiv.org/abs/2106.03503
- [15]R. Tagiew et al., “Osdar23: Open sensor data for rail 2023,” in 2023 8th International Conference on Robotics and Automation Engineering (ICRAE), Singapore, Singapore: IEEE, 2023, pp. 270–276.DOI
- [16]Z. Tao, S. Ren, Y. Shi, X. Wang, and W. Wang, “Accurate and lightweight RailNet for real-time rail line detection,” Electronics, vol. 10, no. 16, p. 2038, Aug. 23, 2021.DOI
- [17]Verband Deutscher Verkehrsunternehmen (VDV), “Operation of autonomous tramways,” Verband Deutscher Verkehrsunternehmen (VDV), Cologne, Germany, Position Paper, Aug. 2019.
- [18]Y. Wang, L. Wang, Y. H. Hu, and J. Qiu, “Railnet: A segmentation network for railroad detection,” IEEE Access, vol. 7, pp. 143 772–143 779, 2019.DOI
- [19]F. Wulff, B. Schäufele, J. Pfeifer, and I. Radusch, “Railway LiDAR semantic segmentation based on intelligent semi-automated data annotation,” in 2024 IEEE 100th Vehicular Technology Conference (VTC2024-Fall), IEEE, 2024, pp. 1–7.DOI
- [20]F. Xu, J. Huang, J. Wu, and L. Jiang, “Active mask-box scoring r-CNN for sonar image instance segmentation,” Electronics, vol. 11, no. 13, p. 2048, Jun. 29, 2022.DOI
- [21]Z. Yu, Q. Liu, W. Wang, L. Zhang, and X. Zhao, DALNet: A rail detection network based on dynamic anchor line, Aug. 24, 2023. arXiv: 2308.11381[cs].arxiv.org/abs/2308.11381
- [22]M. Zeller, M. Rothfelder, and C. Klein, “Safe.trAIn – engineering and assurance of a driverless regional train,” in 2023 IEEE/ACM 2nd International Conference on AI Engineering – Software Engineering for AI (CAIN), Melbourne, Australia: IEEE, May 2023, pp. 197–197.DOI
- [23]O. Zendel, M. Murschitz, M. Zeilinger, D. Steininger, S. Abbasi, and C. Beleznai, “Railsem19: A dataset for semantic rail scene understanding,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops, 2019.DOI





