WorksComputer Science
Silent Reward and Environment Defects in EV Charging Simulation Benchmarks: A Diagnostic Checklist
Abstract
Reinforcement-learning (RL) benchmarks built on microscopic traffic simulators such as SUMO are increasingly used to compare agent architectures for electric vehicle (EV) charging assignment. Unlike crashes, defects in the reward function, the environment's stochastic model, or the state encoding often fail silently: the simulation completes, produces plausible numbers, and the comparison is biased. We report five such defects, found by auditing a five-agent benchmark on OpenStreetMap road networks, stated generically: a shared random generator that desynchronises seed-paired comparisons, an ablation-specific reward clip applied unconditionally, a normalisation mismatch that zeroes a reward term, a per-timestep penalty that should fire once per event, and a state dimensionality documented but never wired into the network. We quantify three: a rejection counter inflated up to 125x across agents/cities, a reward term bounded at about of its intended scale, and a 16.6%/16.3% greedy-action divergence from a state-encoding mismatch. Re-analysing a historical ablation, no A1-A2 effect survives family-wise correction; a 56-run campaign confirms the RNG defect's mechanism directly while showing its correction does not change that conclusion. We provide an eight-item checklist as a low-cost first-pass audit; in a later regeneration, three checks caught two new defects while three further process/artefact defects fell outside its boundary.
This is an open access article under the CC BY-SA license.
Corresponding Author:
Nour-Eddine Moumni
Laboratory of Research in Computer Science and Telecommunications (LRIT), Mohammed V University
Rabat, Morocco
Email: [email protected]
Introduction
In EV-charging assignment studies, software-integrity defects are not merely a reporting concern. They can alter the simulated station availability seen by each policy, the number of rejected or served vehicles, queue waiting time, completion rate, and delivered energy, while the traffic simulation still terminates normally. Consequently, an apparently plausible algorithm ranking may reflect inconsistent electrical-service and transportation conditions rather than a policy difference. This paper therefore treats implementation integrity as a prerequisite for interpreting EV-charging performance, not as a post-processing reproducibility check. In smart-city EV-charging deployments, such a biased comparison would misinform exactly the decisions these KPIs are meant to support — charging-station coordination, demand-responsive scheduling, and energy-management policy — even though no grid-level effect is measured in this paper.
While potential-based reward shaping guarantees policy invariance under precise theoretical conditions [1], poorly designed reward shaping is known to induce reward hacking and unintended policies [2], formalised more recently as a general property of proxy reward functions [3]. We examine two concrete mechanisms in the audited EV-charging pipeline — silent normalisation mismatches and unconditional clipping outside their intended ablation scope — without claiming that these mechanisms are absent from prior implementations or reports. Prior EV-charging RL studies [4, 5] report single-seed or small-sample results, motivating the multi-seed guidance on adequately powered comparisons [6], which this paper’s protocol adopts as its non-parametric, seed-paired analysis unit (§ 2.4.) without itself claiming adequate power. In a related EV-charging setting, Sadeghianpourhamami et al. [7] formulate model-free reinforcement learning for coordination across multiple charging stations. Qian et al. [8] couple grid and transportation data in a single DRL charging-navigation state vector — exactly the kind of interface where the unit-consistency and dimensionality checks we propose (D3, D5) apply — and, beyond RL, Tadi et al. [9] forecast demand and schedule charging slots with supervised models, aggregating station-level capacity over predicted arrivals and states of charge, exactly the kind of layer where the counting and unit-consistency checks proposed here (D3, D4) would apply; in neither case do we claim that the implementation contains these defects. Where the field has invested in shared, openly documented artefacts, ACN-Data [10] made real workplace charging sessions inspectable and reusable on the data side; this paper’s checklist targets an analogous inspectability on the simulator/reward/state side.
Comparative RL studies increasingly rely on stochastic microscopic traffic simulators such as SUMO [11] — coupled to external controllers via interfaces such as TraCI [12] — to evaluate charging-assignment, routing, and scheduling policies for electric vehicles, following established SUMO-based RL benchmark suites for traffic signal control and mixed-autonomy traffic [13, 14]. These pipelines combine several independently-seeded sources of randomness — vehicle departure times, driver routes, environment failure models, and the agents’ own exploration policies — glued together through subprocess orchestration and file-based configuration. This complexity creates a class of defects qualitatively different from ordinary software bugs: they do not crash the simulation, do not produce invalid numerical output, and the resulting KPI tables look entirely plausible while simply biasing the comparison between agents, often in a direction that happens to support (or undermine) the paper’s own hypothesis. Sculley et al. [15] described a closely analogous failure mode in production machine-learning systems, where configuration drift, undeclared data dependencies and entangled components accumulate as “hidden technical debt” that degrades system behaviour without raising an error, and Breck et al. [16] proposed testing rubrics to audit production readiness against exactly this risk; the setting differs from a research benchmark’s comparative bias, but both show defects remaining operationally silent. The problem is systemic, not anecdotal: Semmelrock et al. [17] map reproducibility barriers across the method, code, data and experiment layers, and Gundersen and Kjensmo [18]’s survey of 400 AAAI and IJCAI papers found none documenting all variables needed to reproduce its results (only 20–30% reported per factor) — precisely the documentation gap in which the defects described below survive undetected. We show that even a rigorous multi-seed, Wilcoxon-validated protocol [19] is vulnerable to defects that undermine the very seed-pairing the statistical test relies on.
Henderson et al. [20] documented the broader reproducibility crisis in deep RL, showing that unreported implementation details can reverse published conclusions. Engstrom et al. [21] independently showed, for policy-gradient methods, that undocumented code-level choices — rather than the headline algorithmic contribution — account for most of the reported PPO/TRPO performance gap, a code-level attribution problem structurally analogous to the silent defects reported here. Agarwal et al. [22] and Andrychowicz et al. [23] further argue that small-sample point estimates routinely used to rank RL algorithms are themselves statistically unreliable, reinforcing the case for the seed-paired, non-parametric protocol we adopt ([19], § 2.4.). Our contribution is complementary and more operational: rather than surveying reproducibility in general, we document five specific, recurring defect patterns in the reward function, environment model, and state encoding of RL-traffic-simulation pipelines, each illustrated with a real, quantified instance from our five-agent EV-charging benchmark. What is new is not every check taken alone — several restate ordinary hygiene — but their instantiation at interfaces where a comparison can be silently biased, and the measured cost of omitting them, generic enough to be plausible in any SUMO-based (or comparable) multi-agent RL benchmark pairing simulation stochasticity with per-agent exploration noise.
Research questions and audit objective
We frame this study around two research questions: RQ1: Which silent implementation defects can invalidate cross-agent or cross-configuration comparisons in a SUMO-based EV-charging benchmark, and through which mechanisms? RQ2: Which low-cost, machine-checkable invariants can detect these defects before KPI tables are interpreted? We do not ask, and this single-project audit cannot answer, how frequently these defects occur across the broader literature, or how well the resulting checklist transfers to an independent codebase; both remain future work (§ 3.7.).
The contributions of this paper are: (i) a taxonomy of five silent defect patterns (D1–D5), derived from one audited pipeline, affecting reward functions, environment stochasticity, and state encoding in RL-traffic-simulation benchmarks; (ii) quantified evidence of impact from a real benchmark, namely an at-least-125x inflation of a rejection counter (D4) and a reward term neutralised to about of its intended scale (D3); (iii) a re-analysis of a controlled ablation showing that a partial reward fix can yield non-robust directional signals on secondary KPIs — none of which survives family-wise correction — while leaving the primary KPI unchanged everywhere; and (iv) an eight-item, actionable audit checklist for researchers building similar pipelines.
Related work and positioning
Prior work relevant to this audit spans three strands. EV-charging assignment, routing, and reward shaping (Sadeghianpourhamami et al. [7]; Qian et al. [8]; Tadi et al. [9]; ACN-Data [10]) is developed above in the Introduction; SUMO-based traffic-simulation benchmarks and reproducibility and statistical rigour in RL supply the remaining substrate and methodological backdrop, likewise introduced above. All three strands are compared work-by-work in Table 1 below.
| Study | Setting | Method | Stat. rep. | Integrity checks (RNG / reward / state) | Pipeline scope | Gap vs. present study |
| Patterson et al. [24] | RL design, gridworlds/MuJoCo | empirical E-Sarsa, EQRC, DQN, SAC, DDPG | R | R / R / R | General RL (grid-worlds/MuJoCo) | no traffic/EV interface |
| Ault & Sharon; Vinitsky et al. [13, 14] | Traffic-signal and mixed-autonomy traffic, SUMO/Flow | IDQN, IPPO, MPLight, FMA2C; ARS, ES, TRPO, PPO | R | N / R / R | SUMO traffic (non-EV) | no EV-charging arm |
| Qian et al. [8] | EV charging navigation | DQN + DSCRM | N | N / U / R | EV-charging (non-SUMO) | reward check unclear, not reported in text |
| Tadi et al. [9] | EV smart charging, energy harvesting | Random Forest, K-means, LSTM | N | N / N / U | EV-charging (non-SUMO) | RNG/reward not reported in text |
| Lee et al. (ACN-Data); Sadeghianpourhamami et al. [7, 10] | Workplace EV charging prediction; EV charging coordination | Gaussian mixture models; fitted Q-iteration | N | N / N / R | EV-charging (non-SUMO) | RNG/reward not reported in text |
| Agarwal et al.; Pineau et al. [22, 25] | RL methodology/evaluation; ML reproducibility policy | N/A (survey); N/A (policy report) | R / N | N / N / N | General RL/ML methodology | no integrity checks reported in text |
| Henderson et al. [20] | RL methodology, reproducibility crisis | TRPO, DDPG, PPO, ACKTR | R | N / U / N | General RL | RNG/state not reported in text |
| Present study | EV charging, silent-defect audit, SUMO | DQN, DDQN, Tabular Q-L, Heuristic, Random | R | R / R / R | SUMO EV-charging pipeline | audits RNG/reward/state jointly |
Table 1. Related-work comparison across all ten audited works, merging works that share an identical RNG/reward/state combination into one row. R = reported, N = not reported, U = unclear/not verified.
Each strand, however, risks studying its own object without auditing the software interfaces where all three meet. We quantify that gap over a representative, not exhaustive, sample of ten works: those cited above that describe their own artefacts or process — an experiment, a report, or a checklist programme — in enough textual detail to be checked against the five criteria. Table 1 below reports the full ten-work comparison directly, merging works that share an identical RNG/reward/state combination into one row for compactness (per-cell source citations for every entry are archived; see Data Availability). Cited foundational/theory works, external surveys [17, 18], and non-selected empirical studies [21, 23] are excluded without prejudice to their protocols. Each cell is individually verified against the cited work’s full text and traceable either to a direct citation or to an explicit negative search record. Among these ten works, seed/uncertainty reporting is present in half (5/10), environment–agent RNG parity in one (1/10, Patterson et al. [24] ), reward-unit/ablation-integrity checks in three (3/10), and state-schema checks in six (6/10); among the ten prior works, only Patterson et al. report all three integrity checks jointly, and none does so within a traffic- or EV-charging-simulation pipeline. Beyond the contributions already stated (§ 1.1.), we claim no novelty in RL reproducibility methodology in general, in the SUMO/TraCI substrate, or in the specific EV-charging algorithms used; and beyond the ten works audited individually here, we do not claim, and have not individually verified, whether any specific study does or does not audit these interfaces.
Method
Benchmark and EV-charging environment
The benchmark simulates EV charging assignment with SUMO on real OpenStreetMap road networks of two Moroccan cities, Rabat and Tangier, under distinct traffic periods per city (morning, midday, evening peaks), giving six (city, period) scenarios in total. Each scenario couples vehicle demand, road-network stochasticity, and a fixed set of charging stations to a policy under evaluation; the queue-wait time, completion rate, delivered energy, and vehicles-served KPIs reported throughout this paper (§ 3.) are computed per episode from this simulated environment. The audit uses a 13-dimensional implemented state, four station actions, the declared reward terms, and a 40-episode training plus 15-episode evaluation schedule; the complete parameter set (simulator version, network geometry, optimiser/dependency detail) is Table S3, referenced explicitly rather than restated here. Finer network geometry (individual lane/junction coordinates, per-vehicle routes) lives in the archived SUMO scenario configuration files, and the campaign manifests separately record the full agent/scenario/seed grid.
Agent selection and training protocol
We audited five agents: a random baseline and an adaptive heuristic baseline as non-learning controls, a tabular Q-learning agent [26], a Deep Q-Network (DQN) agent [27] with a uniformly-sampled experience-replay buffer, and a Double DQN (DDQN) agent [28] addressing DQN’s overestimation bias. This diversity is methodologically useful for the audit because the agents consume different numbers of random draws and use different state and reward representations, thereby exposing parity failures such as D1 and D3 (§ 2.5.). It is not intended as an exhaustive sample of contemporary RL algorithms, and no claim of algorithm-independence is inferred from these five agents alone. Seed counts and the scenario/agent protocol are recorded in the archived campaign manifests.
Audit procedure and defect-confirmation rule
The audit compared, for every agent, (i) the source of randomness driving environment stochasticity against the source driving agent-internal decisions, (ii) every ablation feature flag against every operation it was expected to gate, (iii) every reward-function input against its corresponding state-vector representation to check for normalisation consistency, (iv) every counter and penalty tied to a discrete simulation event against the control-flow condition governing it, and (v) the state-vector dimensionality actually constructed at runtime against the dimensionality documented in the project’s companion papers. A defect was confirmed only once all four of the following were established: the expected invariant, the observed violation traced to a specific code path, a minimal reproducing condition, and a fix locked in by a regression test (see § 3., § 3.4.). Each finding was, where feasible, additionally quantified using production run data recorded before the corresponding fix was applied. We additionally assign each defect an evidence-strength tag on a three-level scale — deliberately not a severity or real-world-impact ranking, which this single-project audit cannot establish: quantified (a direct KPI-level or counter-level effect is measured, e.g. D4), propagating (the defect corrupts a shared resource — the RNG stream or the learning signal — so every downstream KPI is plausibly affected without an isolated causal estimate, e.g. D1, D2, D3), and structural (a comparability or architecture mismatch is established without a measured KPI effect, e.g. D5’s documented-vs-implemented dimensionality). A propagating tag does not imply a larger real-world consequence than a quantified one; it reports that the causal chain to KPIs is plausible but unmeasured, which is a statement about the evidence available, not about impact. A severity scale proper — ranking defects by consequence rather than by evidence strength — would need at least two ingredients this single-project audit does not have: an established KPI-causal magnitude for every defect (available only for D4 and, at the decision level, D5) and a stated deployment or safety threshold past which a given KPI shift becomes consequential for EV-charging operations, neither of which this paper measures or assumes.
Statistical analysis
The paired analysis unit throughout this paper is the random seed. Seven seeds per configuration were the campaign’s fixed compute budget, set before either the ablation or the audit reported here; we do not claim this budget was chosen for adequate power, and § 3. discusses its consequence for the resolution of the test explicitly. The declared A1–A2 comparison family (§ 3., Table S2) is six scenarios four KPIs tests; the D5 secondary-endpoint family (§ 2.5.) is four KPIs two agents tests. Within each family, -values are computed by exact Wilcoxon signed-rank enumeration over the sign assignments for matched seed pairs – a conditional, distribution-free test of the null hypothesis that the sign of each pair’s difference is equally likely, which does not require the paired differences themselves to be normally distributed, only exchangeable in sign under the null. Zero differences are excluded from the ranked set before enumeration, following the standard Wilcoxon convention (none occur in this paper’s declared families); where non-zero differences tie in absolute value, ranks are averaged (mid-rank) before enumeration, which this paper’s declared 24-comparison family requires for exactly one cell (vehicles-served, Tangier midday, where two seeds’ paired differences are exactly equal in absolute value); every other cell, including a near-equal pair at vehicles-served/Tangier morning that agrees only to 15 significant figures, has strictly distinct, non-zero absolute paired differences, as verified directly on the released seed-level data and applied identically by the released recomputation script. Holm–Bonferroni correction is applied across the full declared family, never mixed across families. Where reported, paired mean differences receive a percentile bootstrap 95% confidence interval (10,000 resamples, fixed RNG seed), and Cohen’s (mean paired difference over its standard deviation) is reported separately alongside it (Figure 2); the full 24-comparison Table S2 reports direction and -values only, without bootstrap CIs or for every cell. We treat a corrected non-significant result as insufficient evidence of an effect at this seed resolution, not as evidence of no effect (§ 3.).
Figure 2. Seed-paired secondary-KPI differences between A1 and A2 at Tangier morning (). Lines connect each seed’s A1 and A2 values; is the mean paired difference with a percentile-bootstrap 95% CI (RNG seed 20260817), and is paired Cohen’s . The differences are exploratory: none survives Holm–Bonferroni correction across the 24-comparison family (Table S2).
Taxonomy of defects
Each defect is stated generically below, then illustrated with the concrete instance found in our pipeline. Figure 1 locates each defect at the pipeline stage it affects.

Figure 1. RL-traffic-simulation pipeline from seed/configuration to KPI output, showing silent-defect points D1–D5. Solid arrows indicate nominal flow; dashed arrows indicate defect mechanisms. All paths yield plausible KPI tables without exceptions.
D1 — Shared RNG invalidates seed pairing (evidence: propagating). When one global generator is seeded per episode and shared by environment stochasticity and agent randomness, agents that consume different numbers of draws observe different environment realisations under the same nominal seed. In our pipeline, the two-state station-failure Markov chain shared the stream with exploration and replay sampling; DQN draws up to 64 values per learning step versus one for the random-policy baseline, so failure sequences diverged from the first learning step.
D2 — Reward clip outside its ablation condition (evidence: propagating). A feature-flagged reward fix can leak a clipping or rescaling side effect into the nominally pre-fix arm. In our pipeline, the bound [, ] was applied unconditionally in the DQN/DDQN caller immediately after the shared reward function returned, while the rest of the fix was flag-gated.
D3 — Normalisation mismatch silently zeroes a reward term (evidence: propagating). When a state observation is normalised for neural-network input and the same normalised value is later reused as an input to the reward function — which applies its own, independent normalisation — the result is double normalisation: a term intended to range over collapses to a negligible magnitude, becoming numerically inert without raising any error. In our pipeline, a route-distance feature already normalised to for the state vector was passed unchanged into the reward function, which itself divided by a detour-normalisation constant of the same order of magnitude, scaling the resulting term by about relative to its intended weight for the two neural-network agents sharing this representation, while a tabular agent receiving the distance in unnormalised metres retained a fully active detour term. Concretely, the intended detour weight is and the reward-side normalisation constant is DETOUR_MAX = 10,000 (config.py); the defect affects specifically the charging-completion (terminal) transition, whose detour term is drawn from a distance already normalised once by the state-side MAX_DIST (6,500 m Rabat, 2,000 m Tangier), then divided by DETOUR_MAX a second time, bounding it at regardless of which city-specific constant produced the state-side value — a strict upper bound, not a distribution, since the archived evidence preserves the code and its constants but not a transition-level reward log at that granularity. By contrast, the periodic (every-10-step) reward computed while a vehicle is still en route queries road distance directly in metres, bypassing the state-normalised value entirely; the defect therefore stays confined to the terminal transition. Four other reward terms are active in the same configuration — , (quadratic), , — each three to five orders of magnitude larger than the bound on the detour term, so on the terminal transition the latter’s maximum possible contribution is negligible beside them. No new simulation was needed to verify this — only re-reading the normalisation constants already present in the archived pre-audit code. Fix: see Table 3, check 3.
| # | Invariant | Executable assertion / pass criterion / evidence | Mapped defect(s) | Corrective action |
| 1 | RNG separation | Two agents, same seed identical environment-event hash. Evidence: covered by the project’s regression suite; the 56-run validation campaign (Sec. 3.3.) confirms directly that violating this invariant desynchronises the trace (0/28 identical sequences between regimes). | D1 | Dedicated, independently-seeded environment RNG; traces identical across agents for every matched seed |
| 2 | Ablation flag coverage | Every downstream operation of a conditional fix gated by the same flag as the primary computation; “off” path bit-identical to a frozen pre-fix fixture. Evidence: not yet included. | D2, D6, D7 | Gate clipping and every side effect under the same flag; off path bit-identical to the pre-fix fixture |
| 3 | Unit consistency at reward boundaries | Known raw input reward term falls within a declared numeric interval. Evidence: not yet included. | D3 | Separate raw reward-facing and normalised state-facing values; term magnitude in the declared interval for representative inputs |
| 4 | Per-event vs. per-step counters | One persistent rejected/failed entity across multiple timesteps counter increments exactly once. Evidence: not yet included. | D4 | Per-episode already counted guard; exactly one counter increment/penalty per entity event |
| 5 | Documented vs. implemented state/action space | len(state) == network_input_dim == documented_dim for every agent and state mode. Evidence: locked by regression tests (Sec. 2.5., D5). | D5 | Single derived dimension constant; state length, network input, and documented dimension are equal |
| 6 | Cross-agent objective parity | On one fixed transition, every agent’s realised reward has the same active terms; magnitude comparability has no numeric threshold fixed yet (oracle not yet specified). Evidence: active-terms comparison alone caught D6 in the prospective cycle (Sec. 3.4.), since a crash needs no numeric threshold; the magnitude criterion is not yet included. | D3, D6 | See check 3 for D3; D6 required no numeric fix, only the active-terms comparison itself |
| 7 | Ablation-condition purity | “Off” configuration hash/trace identical to a known pre-fix baseline on a declared fixture. Evidence: not yet included as an assertion; D7 was caught in the prospective cycle (Sec. 3.4.) via direct simulator instrumentation. | D2, D7 | See check 2 for D2; D7 fixed via direct simulator instrumentation, no generic fixture yet |
| 8 | Re-verify after each fix | Each individual defect fix ships with its own smoke test before being folded into a full campaign. Evidence: exercised for D6–D10 (Sec. 3.4.). | D6–D10 | Individual fix plus smoke test per defect (Fig. S1); no single generic corrective action |
Table 3. Eight-item diagnostic checklist as executable assertions. No full campaign rerun is required.
D4 — Per-timestep penalty instead of per-event penalty (evidence: quantified). If a persistent rejection or failure condition is updated inside the per-step loop without a once-per-entity guard, one logical event is counted and penalised repeatedly. In our pipeline, a vehicle unable to secure a station was re-evaluated every 1-second step; each step incremented the rejection counter and added a negative-reward transition to replay.
D5 — State dimensionality documented but never wired (evidence: structural). A paper’s Methods section and a project’s code can drift apart silently when a described input feature is defined in configuration, imported into the relevant module, but never actually concatenated into the state vector passed to the network, instantiating the formalism–implementation gap framed by Castro [29]. In our pipeline, an earlier documented specification and the implemented network input diverged: the specification described a 17-dimensional state vector including four station-power features, while the corresponding neural-network implementation used a 13-dimensional input, so the trained networks never used the specified architecture. D5’s fix is implemented in the audited pipeline and locked by regression tests; the corresponding implementation checks are summarized in Table 3.
D5 decision-level quantification. To quantify D5’s decision-level impact, we generated a frozen decision corpus: a provenance-locked, deterministic post-hoc replay of the control-arm DQN (seed 45, rabat_evening, 15 greedy evaluation episodes; not an extraction of a log from the original campaign, since none was produced — see Data Availability). This yields a single shared sequence of 499 states and action masks on which the matched 13D and 17D checkpoints from all 7 trained seeds are scored; reported confidence intervals therefore capture checkpoint-to-checkpoint variability conditional on this fixed decision sequence, not variability across independent evaluation trajectories. The greedy actions disagree at 16.6% (DQN, 95% CI [13.9%, 19.7%]) and 16.3% (DDQN, 95% CI [13.0%, 18.8%]), reported descriptively. None of the 4 downstream KPIs (queue-wait time, completion rate, delivered energy, vehicles served) shows a Holm-corrected significant difference across the 8-test family (smallest ), though DQN completion rate and delivered energy show the largest paired effects (, ), not interpreted here as an established causal effect.
D1 affects the comparability of the failure conditions each agent is compared under — and therefore, indirectly, every KPI that depends on station availability — without being isolated as a standalone causal effect on any one of them in the historical archive; § 3.3 tests this directly and finds no measurable change to the historical ablation’s conclusion — a robustness finding for that one ablation, not a general claim that D1 has no effect anywhere it could occur. D2 and D3 alter the learning signal and can plausibly displace queue-wait time, completion rate, delivered energy, or vehicles served, without their causal effect being quantified here; because both bias the reward an agent optimises for congestion and detour, the engineering decision most at risk is charging-station assignment under load, not a single KPI in isolation. D4 directly corrupts the rejection counter and the replay-buffer transitions it feeds, and its magnitude is quantified in full episode-level detail for one representative configuration and as a mean across the four non-random agents and both cities (§ 3). D5 changes the DQN/DDQN state encoding, omitting four station-power features a real power-aware charging-coordination decision would need; its decision-level effect is quantified, but its downstream KPI effect is not statistically distinguishable from zero at (D5 decision-level quantification paragraph above). We do not convert D1, D2, D3, or D5’s plausible mechanisms into established KPI-level effects beyond what is directly measured, nor any grid-level or station-utilisation-rate effect beyond the KPIs already reported.
Consolidated defect-impact and detection table
Table 2 summarises, for each defect D1–D5, the affected component, mechanism, expected versus observed behaviour, measurable impact, and affected agents, answering the mechanism-and-evidence framing requested in review. The corresponding detection-and-remediation framing is answered by Table 3 below, whose Corrective action column gives the same fixes alongside the checklist that encodes them as executable assertions, rather than restating them in a separate panel.
| D | Component / mechanism | Expected vs. observed | Measurable impact | Affected agents |
| D1 | Environment/agent RNG boundary; shared stream consumed at agent-dependent rates | Expected: same environment trace for a matched seed. Observed: episode-level station-failure count profile diverges although demand matches. | Demand (n_ev_launched) matches in 42/42 pairs; 42/42 pairs have different 15-episode profiles; cumulative totals differ in 34/42 (81.0%; mean abs. diff. 3.90, max 16); performance effect not isolated. | Any agent sharing the RNG stream |
| D2 | Reward ablation flag; clip stays active when the fix flag is off | Expected: off path reproduces the pre-fix reward. Observed: off path inherits clipping. | Code-path contamination demonstrated; KPI magnitude not isolated. | Agents using the shared clipped reward path |
| D3 | State-to-reward interface; already-normalised distance divided again | Expected: detour term has declared non-trivial scale. Observed: term collapses to of intended scale. | Reward-term magnitude quantified; KPI effect not isolated. | DQN, DDQN |
| D4 | Event counter/replay insertion; persistent condition fires once per step | Expected: one rejection/penalty per vehicle event. Observed: repeated count/transition over consecutive steps. | At least 125.4× inflation in one DQN configuration (full detail); 93.8–103.1× mean (Rabat), 8.9–12.5× mean (Tangier), recalculated across the four non-random agents from the pre-audit archive. | Quantified for the four non-random agents (Rabat/Tangier means); full detail for DQN only |
| D5 | State builder/network input; four documented power features omitted | Expected: 17D documented state reaches the network. Observed: implemented network receives 13D. | Architectural mismatch established; on a fixed decision corpus (7 seeds, 499 states), greedy actions disagree at 16.6% (DQN)/16.3% (DDQN), descriptive; no KPI effect distinguishable from zero after Holm correction (). | DQN, DDQN |
Table 2. Defect mechanism and evidence.
Results and Discussion
This section addresses RQ1 (§ 1.1) by presenting empirical evidence on confirmed defect mechanisms, their measured magnitudes, and the affected agent architectures. In turn, RQ2 is answered through the diagnostic checklist (Table 3) and its prospective validation (§ 3.4–§ 3.5).
Quantified impact of D4
In one representative configuration — DQN agent, Rabat evening-peak scenario, single seed, 55 episodes, pre-fix production data — defect D4 drove the mean recorded rejection count to 9,402 against 75 vehicles launched per episode: a 125.4x inflation, with observed counts ranging from 7,888 to 10,712. Why a single seed suffices here, deliberately: what is being established is a deterministic counting defect, a counter incremented once per simulation step rather than once per event, not a statistical estimate of performance — the multi-seed requirements discussed in [6] apply to the latter, not the former. The factor is also a conservative lower bound, since the denominator counts all launched vehicles while only vehicles that actually requested charging could be rejected; the true per-event inflation therefore exceeds the value reported. Extending the same recalculation across the four non-random agents and both cities (Table 2, row D4) yields a substantially lower mean inflation at Tangier (8.9–12.5×) than at Rabat (93.8–103.1×). Both remain a severe, qualitatively identical counting defect; what changes is that the factor is not scenario-invariant, and we stop short of extrapolating the single-configuration value past what this multi-city, multi-agent recalculation shows. Beyond corrupting the reported KPI itself, the defect floods the replay buffer with near-duplicate terminal transitions during congestion — plausibly biasing policy learning away from the true congestion signal.
Case study: partial fixes can still mislead ablations
The defects described above were identified in a later audit of this benchmark, separate from an earlier round of reward-shaping refinements applied to the same codebase. That earlier ablation — comparing three configurations of the DQN agent’s reward function, A0 (an earlier queue-wait redefinition only), A1 (A0 plus a rejection penalty), and A2 (A1 plus a quadratic congestion penalty, a raised congestion weight, and a longer discount horizon) — was originally reported, on its own performance-oriented terms, as part of our companion deployment study [30]. We revisit it here under a different lens: not to report the reward fix’s performance, but to illustrate, with the benefit of the later D1–D5 audit, a related and independent reproducibility risk using a fix set unrelated to D1–D5 — that a seemingly clean ablation study can still mislead. The ablation spans six (city, traffic-period) scenarios, Wilcoxon signed-rank test, seeds, comparing queue-wait time (QW, the metric the A2 changes specifically targeted) against three secondary KPIs (completion rate CR, energy delivered ED, vehicles served) between configurations A1 and A2.
Start with the primary KPI, QW. No scenario reached statistical significance: ranges 0.109–0.938 across the six scenarios, not statistically distinguishable from zero at the available seven-seed resolution. Absence of significance is not evidence of absence, though. Seven matched pairs give exact Wilcoxon inference little room to move, power stays limited, and D1 adds environment variation inside nominal pairs (see below); given all that, we read the corrected non-significance as insufficient evidence of an A2 effect, nothing stronger.
Figure 2 tells a different-looking story on three secondary KPIs at Tangier morning, and every one of them moved the wrong way: completion rate fell from 81.2% to 78.1% (unadjusted ), energy delivered from 820.5 to 785.8 kWh (), vehicles served from 23.9 to 23.0 (). Exploratory, unadjusted signals — that is how we report them, not as evidence that A2 degraded performance. And here correction genuinely decides the outcome rather than trimming at the margins: across a family of 24 tests the first Holm threshold is , and even the smallest unadjusted -value on record, 0.0312, sits well above it. (Table S2 lists all 24 comparisons with both their unadjusted and Holm-adjusted -values, so this is checkable directly.) One might reasonably ask whether the four-KPI case-study family — the narrower, more permissive choice, since a smaller family softens the Holm correction — was picked to produce this outcome. It was not. Widen the family to every KPI in the underlying synthesis table instead (, first Holm threshold ) and the smallest adjusted -value still lands at 1.000, with zero comparisons surviving. However the family is drawn, the conclusion does not move.
Every -value quoted above was recomputed independently from the released seed-level measurements, via the distributed script, as an exact conditional enumeration over all sign assignments: all 24 declared-family comparisons reproduce the values reported. One comparison sits outside that family — the Rabat-evening first-step assignment rate — and it differs (0.9375 versus 0.8125); we report it for transparency, though it does not enter the declared inference.
The re-analysis does not establish degradation; it illustrates how directional secondary signals can mislead when multiplicity and later code-audit findings are ignored, because a defect discovered after an ablation can invalidate its premise without retroactive checking.
D1 can be seen in these runs: the A1 and A2 campaigns used the shared-generator code before D1 was found. Across 42 matched pairs (6 scenarios 7 seeds), vehicle demand matches in 42/42 pairs, but every 15-episode station-failure count profile differs in at least one episode; cumulative totals differ in 34/42 (81.0%; mean absolute difference 3.90, maximum 16). This is an episode-level count profile, not a timestamped event trace: the archived CSVs contain neither event timing nor station identity, so no finer-grained divergence metric is computable. The independently seeded demand component matched; station failures sharing the agents’ generator did not, degrading the seed pairing and lowering test power.
D1 revisited: does the RNG fix change the ablation conclusion?
Sec. 3.2. reports that the historical A1–A2 ablation’s seed pairing was already degraded by D1, which lowers the signed-rank test’s power without establishing what a corrected pairing would have concluded. Per reviewer feedback on the 2026-08-21 decision, we quantify this directly rather than leaving it as a plausible mechanism: a dedicated 56-run validation campaign (2 scenarios [Rabat evening, Tangier morning] 2 reward configurations [A1, A2] 2 RNG regimes [independent (the corrected D1 fix) and shared_compat, a controlled, fail-closed reintroduction of the pre-fix shared stream gated behind an explicit environment flag and a locked experiment manifest] 7 seeds; 40 training plus 15 evaluation episodes sharing one checkpoint per run) isolates the RNG regime as the only varying factor, everything else — code, reward configuration, scenario, seed — held fixed.
Primary estimand: for each (scenario, KPI) cell and each seed, the interaction contrast . This is not a repeat of the historical A1–A2 test; it tests whether the D1 fix changes the A1–A2 effect that test measured. Across the resulting family of 8 exact Wilcoxon signed-rank tests ( seeds; 2 scenarios 4 KPIs: queue-wait time, completion rate, energy delivered, vehicles served), no cell survives Holm–Bonferroni correction (smallest , all ; paired effect sizes small to moderate, ; 10,000-resample percentile bootstrap CI, RNG seed 20260817). This is evidence of robustness at the available resolution, not proof of exact equivalence: the D1 fix does not detectably alter the non-significant A1–A2 conclusion already reported in Sec. 3.2.
Secondary analysis: using event-level logging enabled specifically for this campaign, we compare the station-failure/station-recovery trace between the independent and shared_compat arms for each of the 28 matched (scenario, reward configuration, seed) triples. None of the 28 pairs produce an identical train-phase event sequence (0/28); in the earliest case examined, the two arms’ first station failure already disagrees within the opening minutes of episode 1. This is direct, event-level confirmation — unavailable from the historical archive, whose episode-level counters cannot resolve event timing (Sec. 3.2.) — that D1 is a demonstrated mechanism rather than a plausible one: sharing the RNG stream measurably desynchronises the environment’s Markov process from a matched seed alone.
Together, the two analyses answer the reviewer’s request precisely: D1 demonstrably alters the environment trajectory (secondary analysis), while the A1–A2 ablation conclusion it could have biased is not measurably changed by correcting it (primary analysis). We report both rather than only the reassuring one, because a null interaction result at seeds is evidence of no detected effect, not evidence of no effect, and does not retroactively certify every comparison run under the historical shared-RNG protocol outside this specific 56-run campaign.
Prospective checklist detections: D6–D7
The checklist was subsequently applied prospectively, live, during a pilot regeneration of the same benchmark under a corrected reward pipeline. Two further silent defects distinct from D1–D5 were exposed directly by checklist checks during that cycle:
D6 — an action mask built as a list of feasible-action indices instead of a binary vector, crashing the masked-target computation on the first full station (checks 2, 6).
D7 — a vehicle-release guard deferring
resume()whenever a vehicle was still inside the charging-station interval, true of every vehicle that had just finished charging: physical occupancy pinned at capacity across 585 consecutive episode-rows, zero variance, caught only by comparing the “off” condition against direct simulator instrumentation rather than the agent’s own bookkeeping (check 7).
Both were fixed and locked in with regression tests before the next campaign (check 8). We state the evidential value of this cycle precisely, because it is the only prospective evidence available: D1–D5 cannot count, the checklist having been derived from them. The prospective cycle directly exercised checklist checks 2, 6 and 7 through D6–D7.
Boundary defects found outside the checklist: D8–D10
D8–D10 are not validation successes of the eight-item checklist. They were discovered by broader engineering inspection during the same regeneration cycle and are reported to delimit the checklist’s current scope and motivate candidate future checks, not to demonstrate its coverage.
D8 — a multiprocess pool silently re-importing an unfiltered scope configuration under
spawn, so a manifest declaring one traffic period ran all three.D9 — an unconditional
--appendflag producing a headerless results file whenever an evaluation-only run skipped the training phase.D10 — an unguarded stdout rewrap that only manifested under a test harness, silently corrupting its output capture.
All three were fixed and locked in with regression tests before the next campaign (check 8). To avoid retroactive validation, two candidate checks follow from them but are presented as future extensions and are not credited with detecting D8–D10: process-boundary configuration parity — every spawned worker must receive and log the resolved canonical configuration, and the expected task cardinality must be checked; and execution-interface and artefact contracts — every execution mode must be exercised end to end, with the CSV schema, header, parseability, stdout/stderr separation and expected artefacts asserted. That five defects of the same silent, comparison-biasing character appeared in a single regeneration cycle, months after D1–D5 were fixed, supports treating this as a recurring hazard class rather than a closed, one-time list.
Diagnostic checklist
We distil the above into eight checks, ordered roughly by ease of application; none requires re-running a full experimental campaign. The format is deliberate: Pineau et al. [25], reporting on the NeurIPS 2019 reproducibility programme, showed the value of short, actionable reproducibility checklists adopted at conference scale. Ours is narrower and complementary — it audits the environment and reward implementation rather than submission reporting — operationalising, for a single pipeline, the empirical-design principle Patterson et al. [24] argue should govern RL experiments: the apparatus itself must be validated before any comparison it produces can be interpreted. At least four checks restate known practice: RNG separation (check 1) restates Patterson et al.’s [24] env–agent seed guidance; unit consistency at reward boundaries (check 3) and cross-agent objective parity (check 6) are the code-level scrutiny Engstrom et al. [21] showed matters more than headline algorithmic choices; re-verification after each fix (check 8) is ordinary regression-testing hygiene. We claim no antecedent for checks 2, 4, 5, 7 individually; what is not already established practice, to our knowledge, is auditing the three RNG/reward/state interfaces jointly, as one ordered pass, on a SUMO-based EV-charging pipeline specifically — of the ten works compared in § 1.2, only Patterson et al. [24] reports all three jointly, and not within a traffic- or EV-charging-simulation pipeline, a claim bounded to that comparison, not the wider literature. Each check also targets a specific audit stage: checks 1 and 5 are pre-training assertions, checks 2, 3 and 7 are reward-function unit tests independent of training, check 6 is a fixed-transition sanity check across agents, and checks 4 and 8 apply during and after a campaign respectively. Table 3 states each check as an executable assertion rather than a verb without an oracle, with checks 1–5, 7, and 8 giving a concrete pass/fail criterion; check 6 has no numeric pass threshold yet for “comparable magnitude” in the audited pipeline and is marked oracle not yet specified rather than machine-checkable, so as not to present an unenforced criterion as executable — a project adopting it must first fix its own tolerance (e.g. relative to the largest active reward term). An evidence artefact marked not yet included is a check we run in the audited codebase’s own test suite but do not ship in the reproducibility archive; it is reported as such rather than presented as an already-distributed test.
This checklist runs as a loop, not a one-pass list: the eight checks execute in order, and a failure at any step triggers a minimal fix followed by a smoke test, after which the workflow restarts from check 1 — since a fix can invalidate a check that had already passed — until all eight pass in sequence. This mirrors how the defects reported above were actually found and corrected in the audited codebase. The full diagram, as a UML activity workflow looping over the check index, is provided as Figure S1.
Limitations
The checklist targets silent defects that bias agent or configuration comparisons, not general software-engineering hygiene; it excludes training instability, architecture search, and simulator artefacts outside the RL pipeline. Training-environment reproducibility is partial: the supplementary parameter table reports the SUMO version, reward function, and principal hyperparameters, but historical NumPy, TraCI, and optional MQTT versions were never pinned and cannot be recovered without fabricating a record; this gap is reported explicitly. The taxonomy is project-specific and non-exhaustive. D4’s 125.4x rejection-counter inflation (the representative calculation; Table 2) recurs across agents and cities but remains scenario-dependent: it does not establish a per-episode distribution or a causal policy-level KPI effect, and we do not extrapolate beyond the two reported summary statistics. The 56-run D1 interaction-contrast campaign (Sec. 3.3) re-evaluated one A1–A2 ablation across two scenarios, seven seeds, and two RNG regimes; it does not certify every historical shared-RNG comparison outside that campaign, and its null result at this seed count cannot exclude smaller effects. Independent-codebase validation and a causal, multi-agent D4 extension remain future work (§ 4.). The checklist is a low-cost first pass (each check is executable in well under a day), not a substitute for independent re-implementation or replication, nor for the broader empirical-design guidance of Jordan et al. [31, 32].
Conclusion
We documented five silent defect patterns affecting reward functions, environment stochasticity, and state encoding in RL-traffic-simulation benchmarks, quantified their impact on a real five-agent EV-charging benchmark — including an at-least-125.4× rejection-counter inflation in the representative Rabat-evening configuration (8.9–103.1× across agents and cities, D4), a reward term neutralised to about of its intended scale (D3), and a degraded, shared-RNG seed pairing affecting the very comparisons a companion ablation relied on (D1) — and re-analysed that historical reward ablation, in which no effect survived family-wise correction — evidence that even careful, multi-seed benchmarks can yield directional but non-robust signals when multiplicity control and a subsequent code audit are absent. This five-defect taxonomy and the checklist derived from it are a starting inventory from a single audited pipeline, not an exhaustive or universally applicable classification; whether they recur, and at what rate, in other RL-traffic-simulation codebases remains to be established by future work; the same non-exhaustive, single-pipeline status applies to the eight-item checklist we offer as a practical, low-cost starting point for researchers building similar pipelines. We report partial prospective evidence for its value (§ 3.4.): applied live to the same codebase during a subsequent pilot regeneration, checks 2, 6 and 7 exposed two further silent defects, while three additional process and artefact defects surfaced through broader engineering inspection and delimit the checklist’s present coverage. All five were fixed and locked in with regression tests before being folded into the next campaign. Future work will extend the checklist to these uncovered classes and apply it to independent codebases, to assess whether the D1–D10 patterns, and their apparent recurrence rate, generalise beyond this one project’s benchmark. For researchers and developers building similar EV-charging or traffic-simulation RL pipelines, the practical implication is procedural: run the eight checks before trusting any KPI comparison the pipeline produces, and re-run the checks a fixed number of times after every subsequent change to the pipeline (Fig. S1), since each is inexpensive relative to a full campaign and D1–D10 show that a plausible-looking KPI table is not, by itself, evidence that no such defect is present.
Acknowledgments
The authors acknowledge support from the Laboratory of Research in Computer Science and Telecommunications (LRIT), Mohammed V University, Rabat, and the Laboratory of Probability, Statistics and Intelligent Systems (LPAIS), Sidi Mohamed Ben Abdellah University, Fez, Morocco.
Funding Information
This research received no specific grant from funding agencies in the public, commercial, or not-for-profit sectors.
Author Contributions Statement
| Name of Author | C | M | So | Va | Fo | I | R | D | O | E | Vi | Su | P | Fu |
| Nour-Eddine Moumni | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ||||
| Rachid Alaoui | ✓ | ✓ | ✓ | |||||||||||
| Driss Kiouach | ✓ | ✓ |
Table 12.
C : Conceptualization M : Methodology So : Software Va : Validation Fo : Formal Analysis I : Investigation R : Resources D : Data Curation O : Writing - Original Draft E : Writing - Review & Editing Vi : Visualization Su : Supervision P : Project Administration Fu : Funding Acquisition
CONFLICT OF INTEREST STATEMENT
The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.
DATA AVAILABILITY
A supplementary reproducibility archive — the raw campaign data, seed-level measurements, figures, SHA-256 manifests, and supplementary Tables S1–S3, including the frozen D5 corpus and the D1 validation-campaign data — is permanently archived on Zenodo, DOI: 10.5281/zenodo.22166118. The remainder of the archive is available from the corresponding author on request, and is planned for a later version of the same Zenodo record.
Biographies of Authors

Figure 3 (PDF p. 14).
Nour-Eddine Moumni is a PhD candidate at the Laboratory of Research in Computer Science and Telecommunications (LRIT), Mohammed V University, Rabat, Morocco. His research addresses reinforcement learning for EV charging management in smart cities — reward-function design, multi-agent coordination, and reproducibility of microscopic-traffic-simulation benchmarks — and develops an agent-based charging-assignment framework coupling SUMO with IoEV communication on real Moroccan OpenStreetMap networks; validating a simulation pipeline before trusting its comparisons motivated the diagnostic checklist presented here. Current interests include prospectively specified evaluation protocols for applied RL. Contact: [email protected].

Figure 4 (PDF p. 14).
Rachid Alaoui is a Full Professor of Higher Education at the Laboratory of Research in Computer Science and Telecommunications (LRIT), Mohammed V University, Rabat, and at the Higher School of Technology of Salé (EST Salé), Morocco. His research spans biometric systems, intelligent systems, and smart electric vehicle charging: deep-learning multimodal biometric identification (convolutional fusion of fingerprint, finger-vein, and face images; score-level fusion for human recognition), smart coordination systems minimising EV trip duration and peak charging demand, and simulation platforms for optimising EV routes on Moroccan urban networks. That experience in simulation-based mobility studies underpins his role in the present work, where he supervised the audit and the framing of its findings as a reusable checklist. He is indexed on Scopus with 25 documents, an h-index of 8, and more than 250 citations (August 2026). Contact: [email protected].

Figure 5.
Driss Kiouach is a Full Professor at the Laboratory of Probability, Statistics and Intelligent Systems (LPAIS), Faculty of Sciences Dhar El Mahraz, Sidi Mohamed Ben Abdellah University, Fez, Morocco. His research lies in stochastic processes, applied mathematics, and mathematical epidemiology, with a sustained focus on the stability of nonlinear dynamical systems; he has published widely on stochastic epidemic models, including global stability analyses of SIRS models with generalised nonlinear incidence and vaccination, threshold behaviour under vertical transmission, and the influence of Lévy noise on contagion dynamics. This background motivates his contribution here: the paper’s environment stochasticity, seed-paired comparison, and family-wise error control rest on the same foundations, and a stochastic model whose randomness is not controlled cannot support the inference drawn from it. He has also co-authored smart coordination and highway charging-plan management systems for electric vehicles on Moroccan networks. He is indexed on Scopus with 43 documents, an h-index of 14, and more than 880 citations (August 2026). Contact: [email protected].
References
- [1]A. Y. Ng, D. Harada, and S. Russell, “Policy invariance under reward transformations: Theory and application to reward shaping,” in Proc 16th Int Conf Mach. Learn. ICML, 1999, pp. 278–287.
- [2]D. Amodei, C. Olah, J. Steinhardt, P. Christiano, J. Schulman, and D. Mané, “Concrete problems in ai safety,” 2016, doi: 10.48550/arXiv.1606.06565.
- [3]J. Skalse, N. D. K. Krasheninnikov, and D. Krueger, “Defining and characterizing reward hacking,” in Adv. Neural Inf. Process. Syst. 35. New Orleans, Louisiana, USA: Neural Information Processing Systems Foundation, Inc. (NeurIPS), 2022, pp. 9460–9471, doi: 10.52202/068431-0687.DOI
- [4]Z. Wan, H. Li, H. He, and D. Prokhorov, “Model-free real-time ev charging scheduling based on deep reinforcement learning,” IEEE Trans. Smart Grid, vol. 10, no. 5, pp. 5246–5257, Sep. 2019, doi: 10.1109/TSG.2018.2879572.DOI
- [5]H.-M. Chung, W.-T. Li, C. Y. Yuen, C.-K. Wen, and N. Crespi, “Electric vehicle charge scheduling mechanism to maximize cost efficiency and user convenience,” IEEE Trans. Smart Grid, vol. 10, no. 3, pp. 3020–3030, May 2019, doi: 10.1109/TSG.2018.2817067.DOI
- [6]C. Colas, O. Sigaud, and P.-Y. Oudeyer, “How many random seeds? statistical power analysis in deep reinforcement learning experiments,” 2018, doi: 10.48550/arXiv.1806.08295.DOI
- [7]N. Sadeghianpourhamami, J. Deleu, and C. Develder, “Definition and evaluation of model-free coordination of electric vehicle charging with reinforcement learning,” IEEE Trans. Smart Grid, vol. 11, no. 1, pp. 203–214, Jan. 2020, doi: 10.1109/TSG.2019.2920320.DOI
- [8]T. Qian, C. Shao, X. Wang, and M. Shahidehpour, “Deep reinforcement learning for ev charging navigation by coordinating smart grid and intelligent transportation system,” IEEE Trans. Smart Grid, vol. 11, no. 2, pp. 1714–1723, Mar. 2020, doi: 10.1109/TSG.2019.2942593.DOI
- [9]K. S. Tadi, G. Swapna, K. V. G. Rao, M. K. Kumar, and S. R. Teja, “Smart charging of electric vehicles at a charging station using machine learning and pressure pad energy harvesting,” Bulletin EEI, vol. 15, no. 2, pp. 958–969, Apr. 2026, doi: 10.11591/eei.v15i2.10635.
- [10]Z. J. Lee, T. Li, and S. H. Low, “Acn-data: Analysis and applications of an open ev charging dataset,” in Proc. Tenth ACM Int. Conf. Future Energy Syst. Phoenix AZ USA: ACM, Jun. 2019, pp. 139–149, doi: 10.1145/3307772.3328313.DOI
- [11]P. A. Lopez, E. Wiessner, M. Behrisch, L. Bieker-Walz, J. Erdmann, Y.-P. Flotterod, R. Hilbrich, L. Lucken, J. Rummel, and P. Wagner, “Microscopic traffic simulation using sumo,” in 2018 21st Int. Conf. Intell. Transp. Syst. ITSC. Maui, HI: IEEE, Nov. 2018, pp. 2575–2582, doi: 10.1109/ITSC.2018.8569938.
- [12]A. Wegener, M. Piórkowski, M. Raya, H. Hellbrück, S. Fischer, and J.-P. Hubaux, “Traci: An interface for coupling road traffic and network simulators,” in Proc. 11th Commun. Netw. Simul. Symp. Ottawa Canada: ACM, Apr. 2008, pp. 155–163, doi: 10.1145/1400713.1400740.DOI
- [13]J. Ault and G. Sharon, “Reinforcement learning benchmarks for traffic signal control,” in Proc NeurIPS Datasets Benchmarks Track, vol. 1, 2021.
- [14]E. Vinitsky, A. Kreidieh, L. L. Flem, N. Kheterpal, K. Jang, C. Wu, F. Wu, R. Liao, E. Liang, and A. M. Bayen, “Benchmarks for reinforcement learning in mixed-autonomy traffic,” in Proc 2nd Conf Robot Learn. CoRL, vol. 87, 2018, pp. 399–409.
- [15]D. Sculley, G. Holt, D. Golovin, E. Davydov, T. Phillips, D. Ebner, V. Chaurdhary, M. Young, J.-F. Crespo, and D. Dennison, “Hidden technical debt in machine learning systems,” in Adv. Neural Inf. Process. Syst. NIPS, vol. 28, 2015, pp. 2503–2511.
- [16]E. Breck, S. Cai, E. Nielsen, M. Salib, and D. Sculley, “The ml test score: A rubric for ml production readiness and technical debt reduction,” in 2017 IEEE Int. Conf. Big Data Big Data. Boston, MA: IEEE, Dec. 2017, pp. 1123–1132, doi: 10.1109/BigData.2017.8258038.DOI
- [17]H. Semmelrock, S. Kopeinik, D. Theiler, T. Ross-Hellauer, and D. Kowald, “Reproducibility in machine-learning-based research: Overview, barriers, and drivers,” AI Mag., vol. 46, no. 2, 2025, art. no. e70002, doi: 10.1002/aaai.70002.DOI
- [18]O. E. Gundersen and S. Kjensmo, “State of the art: Reproducibility in artificial intelligence,” in Proc. AAAI Conf. Artif. Intell., vol. 32, 2018, pp. 1644–1651, doi: 10.1609/aaai.v32i1.11503.DOI
- [19]F. Wilcoxon, “Individual comparisons by ranking methods,” Biometrics Bulletin, vol. 1, no. 6, pp. 80–83, Dec. 1945, doi: 10.2307/3001968.
- [20]P. Henderson, R. Islam, P. Bachman, J. Pineau, D. Precup, and D. Meger, “Deep reinforcement learning that matters,” in Proc. AAAI Conf. Artif. Intell., vol. 32, 2018, pp. 3207–3214.
- [21]L. Engstrom, A. Ilyas, S. Santurkar, D. Tsipras, F. Janoos, L. Rudolph, and A. Madry, “Implementation matters in deep rl: A case study on ppo and trpo,” in Proc Int Conf Learn. Represent. ICLR, 2020.arxiv.org/abs/2005.12729
- [22]R. Agarwal, M. Schwarzer, P. S. Castro, A. Courville, and M. G. Bellemare, “Deep reinforcement learning at the edge of the statistical precipice,” in Adv. Neural Inf. Process. Syst. NeurIPS, vol. 34, 2021, pp. 29 304–29 320.arxiv.org/abs/2108.13264
- [23]M. Andrychowicz, A. Raichuk, P. Stańczyk, M. Orsini, S. Girgin, R. Marinier, L. Hussenot, M. Geist, O. Pietquin, M. Michalski, S. Gelly, and O. Bachem, “What matters for on-policy deep actor-critic methods? a large-scale study,” in Proc Int Conf Learn. Represent. ICLR, 2021.
- [24]A. Patterson, S. Neumann, M. White, and A. White, “Empirical design in reinforcement learning,” J. Mach. Learn. Res., vol. 25, no. 318, pp. 1–63, 2024.
- [25]J. Pineau, P. Vincent-Lamare, K. Sinha, V. Larivière, A. Beygelzimer, F. d’Alché-Buc, E. Fox, and H. Larochelle, “Improving reproducibility in machine learning research (a report from the neurips 2019 reproducibility program),” J. Mach. Learn. Res., vol. 22, no. 164, pp. 1–20, 2021.
- [26]C. J. C. H. Watkins and P. Dayan, “Q-learning,” Mach. Learn., vol. 8, no. 3–4, pp. 279–292, 1992.
- [27]V. Mnih, K. Kavukcuoglu, D. Silver, A. A. Rusu, J. Veness, M. G. Bellemare, A. Graves, M. Riedmiller, A. K. Fidjeland, G. Ostrovski, S. Petersen, C. Beattie, A. Sadik, I. Antonoglou, H. King, D. Kumaran, D. Wierst ra, S. Legg, and D. Hassabis, “Human-level control through deep reinforcement learning,” Nature, vol. 518, no. 7540, pp. 529–533, Feb. 2015, doi: 10.1038/nature14236.DOI
- [28]H. Van Hasselt, A. Guez, and D. Silver, “Deep reinforcement learning with double q-learning,” in Proc. AAAI Conf. Artif. Intell., vol. 30, 2016, pp. 2094–2100, doi: 10.1609/aaai.v30i1.10295.DOI
- [29]P. S. Castro, “The formalism-implementation gap in reinforcement learning research,” 2025, doi: 10.48550/arXiv.2510.16175.DOI
- [30]N. Moumni, R. Alaoui, and D. Kiouach, “A deep reinforcement learning framework for real-time electric vehicle charging management in ioev-enabled smart cities,” in Proc. Int. Conf. Advanced Intelligent Systems for Sustainable Development (AI2SD’2026), ser. Lecture Notes in Networks and Systems. Springer, 2026, accepted, in press.arxiv.org/abs/2611.01751
- [31]S. M. Jordan, Y. Chandak, D. Cohen, M. Zhang, and P. S. Thomas, “Evaluating the performance of reinforcement learning algorithms,” in Proc Int Conf Mach. Learn. ICML, vol. 119, 2020, pp. 4962–4973.
- [32]S. M. Jordan, A. White, B. Castro da Silva, M. White, and P. S. Thomas, “Position: Benchmarking is limited in reinforcement learning research,” in Proc 41st Int Conf Mach. Learn. ICML, ser. Proc. Mach. Learn. Res., vol. 235, 2024, pp. 22 551–22 569.arxiv.org/abs/2406.16241
