WorksEngineering
When Does Inter-BS Interference Hurt Sensing in ISAC? An Exact Fisher Information Analysis under Finite-Alphabet Interference
Abstract
In networked integrated sensing and communication (ISAC), a base station (BS) performing sensing using its own downlink signals is subject to interference from the downlink transmissions of neighboring BSs. This paper investigates the impact of the signal constellation and waveform on the sensing performance in the presence of such inter-BS interference. For any linear receiver, we show that the mean-squared cross-ambiguity between two random ISAC signals is invariant to the modulation bases and constellations. We then derive the exact Fisher information under finite-alphabet interference with a flat-fading link, which equals the interference-free Fisher information scaled by 1-γ\,\mmse_{\C}(γ), where denotes the interference-to-noise ratio (INR) and \mmse_{\C} is the minimum mean-square error (MMSE) in estimating an interfering data symbol from its noise-corrupted observation at the sensing receiver. Consequently, Gaussian interference is the worst case and corresponds to the CRB loss of the matched filtering, whereas finite-alphabet interference induces a non-monotonic CRB loss that peaks at intermediate INR and vanishes at high INR. The loss is waveform-independent for the flat-fading BS-BS link, and an expectation-maximization (EM) interference-cancellation receiver approaches the theoretical bound for delay estimation.
Introduction
Integrated sensing and communication (ISAC) has attracted significant attention as a key enabler of next-generation wireless networks, in which the downlink (DL) data signals of cellular base stations (BSs) are reused for radar sensing [1], [2]. In a dense, TDD-synchronized network, a BS that performs full-duplex monostatic sensing inevitably receives the DL transmissions of its neighbors on the same time-frequency resources [3], [4]. Since BS-to-BS links are typically line-of-sight (LoS), this inter-BS interference can exceed the thermal noise by tens of decibels, and it fundamentally limits the sensing performance of network-level ISAC [5].
Since communication payloads are random, the sensing performance of a data-bearing signal depends on its modulation constellation and basis. OFDM achieves the lowest average ranging sidelobe under sub-Gaussian constellations [6], while the sidelobes induced by high-order QAM can be suppressed by mismatched filtering receiver [7]. Constellation shaping or selection balances this randomness against the communication rate [8]–[10], and the delay-estimation accuracy of OFDM-ISAC under matched filtering (MF) and reciprocal filtering (RF) has been characterized via the fourth and inverse second moments of the constellation [11]. However, all of these results concern the auto-ambiguity of a BS’s own signal.
For inter-BS (inter-cell) interference, constellation selection and power allocation in multi-cell OFDM-ISAC have been studied under MF and RF processing [12], and blind estimation under co-channel OFDM sources has also been investigated [13], which characterize the interference by its constellation moments and bound the performance with a Gaussian surrogate, respectively. On the communication side, it is well known that discrete interference causes less performance degradation than Gaussian interference [14]. Despite these advances, an exact estimation-theoretic characterization of sensing under finite-alphabet inter-BS interference remains unexplored, particularly regarding how interference strength and modulation determine the achievable sensing accuracy.
To address this gap, we develop an estimation-theoretic framework for sensing under finite-alphabet inter-BS interference. First, we first show that the mean-squared cross-ambiguity is invariant to the modulation bases and constellations, so that no constellation or waveform design benefits a linear receiver. Then, we derive the exact Fisher information matrix (FIM) under finite-alphabet interference via a complex Brown identity, revealing that Gaussian interference is the worst case and that finite-alphabet interference yields a non-monotonic loss that vanishes at high INR. We further prove that the loss is waveform-independent for a flat-fading and synchronized BS-BS link, and show that an expectation-maximization (EM)-based soft interference-cancellation receiver approaches the bound.
System Model
Transmit Signal Model
As illustrated in Figure 1, BS performs full-duplex (FD) monostatic sensing with its DL data signal, while a neighboring BS transmits independent DL data over the same time-frequency resources. To cover a broad class of waveforms, we consider a generic block transmission, in which each BS maps data symbols onto a length- block through a unitary modulation basis and appends a cyclic prefix (CP), i.e.,

Figure 1. ISAC under Inter-BS interference: BS senses with its own DL data while receiving the DL data of BS on the same time-frequency resources.
where and are the symbol vectors of BS and BS, and are unitary. This model includes single carrier (SC, ), CP-OFDM ( with the unitary DFT matrix ), and OTFS ( with Doppler bins), as well as DFT-spread OFDM and AFDM. The symbols are i.i.d., zero-mean, and unit-power, and the interfering symbols are drawn from the constellation . We define the fourth moment (kurtosis) and the inverse second moment of a constellation as and .
Sensing Received Signal Model
After self-interference suppression and CP removal, the received signal of BS is
where is the target echo of BS’s own signal , e.g., for a single target with complex amplitude , delay , Doppler , and the delay-Doppler shift operator , and collects the real-valued sensing parameters. is the circulant matrix of the BS-to-BS channel , and we define the per-sample INR as . For a CP-OFDM interferer, , which yields the conventional per-subcarrier model after the DFT.
In this study, we adopt the following assumptions:
(A1) has i.i.d. entries uniform on and is independent of , , and .
(A2) The propagation delay of the BS signal is within the CP, and the residual carrier frequency offset (CFO) is negligible.
(A3) BS knows its own data, the static (estimated from the reference signals of BS), , and .
(A4) is invariant to rotations, and thus .
Main Results
Linear Receivers: Second-Order Invariance
With the unitary delay-Doppler shift , where is the cyclic shift matrix, the periodic cross-ambiguity function between the signals in (1) is defined as
With this in hand, we first show that the second moment of is invariant to the modulation bases and constellations for any , and so is its fourth moment for large .
Proposition 1. For all , all unitary bases , and all unit-power constellations with , . Moreover, with the kurtosis and of and ,
where .
Proof. Please refer to Appendix A.
The upper and lower bounds of are attained iff is monomial (e.g., OFDM-OFDM) and iff (e.g., OFDM-SC), respectively. Since for all , the interference after MF forms a flat floor over the delay-Doppler map approximately relative to the main lobe regardless of the basis and constellation of the interferer unlike the auto-ambiguity sidelobes [6]. Moreover, (6) yields with bounded in , implying that the floor fluctuates as Gaussian noise for large .
Remark 1. Linear filters such as MF, RF, and LMMSE depend on the interference only through its covariance , which is independent of and , and also of by (A1). Hence, this holds even when the linear filter is designed for each realization of the known . Therefore, the choice of the waveform and constellation of the interferer cannot affect linear receivers, and only the own constellation enters through for RF processing [11, 12]. Exploiting the interference structure thus requires non-linear processing.
Fisher Information under Finite-Alphabet Interference
Since enters (2) only through , the FIM is governed by the Fisher information of with respect to a location shift. For finite-alphabet , is a Gaussian mixture with components whose Fisher information admits no closed form, while a Gaussian surrogate yields only the worst case [15]. Instead, we use Brown’s identity [16, 17], which expresses this Fisher information with the minimum mean-square error (MMSE) in estimating an interfering data symbol from its noise-corrupted observation at the sensing receiver, which is efficiently computable. Its complex vector form is stated as follows.
Lemma 1. Let denote the Wirtinger score of . Then, and
Under (A1) and (A4), .
Proof. Please refer to Appendix B.
Theorem 1. The FIM of in (2) is given by
Moreover, and the equality holds for Gaussian .
Proof. Please refer to Appendix C.
To evaluate (7), consider the scalar channel with uniform on and . Then, we define
When the interference decouples into such scalar channels, Theorem 1 reduces to the following closed form.
Corollary 1. Let us define . If has orthogonal columns, i.e., with unitary and , then
If for all , then , where is the interference-free FIM. Hence, the entire CRB matrix is inflated by the same factor, which we define as the CRB loss
Proof. Please refer to Appendix D.
Note that is conditioned on the known and marginalized over . Since holds for every realization of , is identical whether or not the CRB is further averaged over .
Proposition 2. The CRB loss satisfies the following properties:
(i) , where the upper bound holds with equality for Gaussian and equals the loss of MF, i.e., treating interference as noise (TIN).
(ii) . Hence, at low INR, finite-alphabet interference is as harmful as Gaussian interference.
(iii) as for any finite .
(iv) is maximized at a finite INR .
Proof. Please refer to Appendix E.
Remark 2: Notably, Proposition 2 reveals that, unlike the TIN loss growing without bound, the loss under finite-alphabet interference is non-monotonic: it follows the Gaussian loss at low INR, peaks at , and vanishes at high INR, where the interfering symbols can be reliably detected and canceled as shown in Figure 2.
Figure 2. CRB loss in (11) versus INR. The dotted line corresponds to Gaussian interference, which equals the loss of MF (TIN).
Dependence of the FIM on the Interferer Waveform
By Theorem 1, the waveform of the interferer affects the FIM only through , and Corollary 1 yields a closed form when has orthogonal columns. The following proposition gives a sufficient condition for waveform independence.
Proposition 3. If the BS-BS channel is flat-fading, i.e., , then for every unitary . Hence, is identical for every interferer waveform (SC, CP-OFDM, OTFS, DFT-s-OFDM, AFDM, etc.), and holds for any waveform .
Proof. Since , Corollary 1 applies with and , i.e., for all . Thus, and does not depend on , and follows for any , i.e., any .
Under a frequency-selective channel, , so that has orthogonal columns for CP-OFDM (, ), whose loss averages the scalar losses over the channel-induced INR spread. In contrast, is not diagonal for SC and OTFS, so that (7) involves the joint MMSE attainable only by sequence-level processing. Since BS-BS links are typically LoS-dominated with a small delay spread, the flat-channel assumption of Proposition 3 is a reasonable approximation in practice.
EM-Based Soft Interference Cancellation
The exact FIM in Theorem 1 provides an asymptotic benchmark for maximum-likelihood (ML) estimation under standard regularity conditions. To approach this benchmark, we employ an expectation-maximization (EM)-based receiver that iteratively estimates and suppresses the finite-alphabet interference [18]. Given the current estimate , the E-step reduces to computing the posterior mean . In the separable case of Corollary 1, it decouples into per-symbol computations on as
whereas the non-separable case requires the joint posterior over all candidates of . Since the expected complete-data log-likelihood equals up to a constant, the M-step is the conventional interference-free estimator applied to the soft-cancelled signal , similar to soft interference cancellation as in iterative multiuser detection [19]. In the separable case, each iteration requires only distance evaluations, without backhaul data sharing or decoding. As summarized in Algorithm 1, the receiver is initialized interference-first with in the first E-step, so that it remains robust when strong interference makes the initial MF estimate unreliable.
Algorithm 1 EM-Based Soft Interference Cancellation
Require: , own data , , , , and number of iterations
Initialize (interference-first)
for do
E-step: compute , , via (12)
Soft cancellation:
M-step: (global search at , local refinement for )
end for
Ensure:
Numerical Results
Unless otherwise stated, both BSs employ CP-OFDM over a flat-fading BS-BS channel, for which Corollary 1 holds with . The MMSE in (11) has no closed form but is an expectation over the Gaussian noise, with given in closed form by (12) with and . We evaluate it deterministically by Gauss-Hermite quadrature instead of Monte Carlo simulation.
CRB Loss versus INR
Figure 2 evaluates the CRB loss in (11) for various constellations. All constellations follow the Gaussian curve at low INR, reach a peak, and then collapse to 0 dB. The peak losses of QPSK, 16QAM, 64QAM, 256QAM, and 1024QAM are 2.8, 5.7, 8.7, 11.6, and 14.5 dB at , 8, 14, 19.5, and 25.5 dB, respectively. That is, each fourfold increase of the QAM order raises the peak by approximately 2.9 dB and shifts it by approximately 6 dB. PSK exhibits a lower peak of approximately 3.5 dB owing to its constant amplitude. For 64QAM at dB, MF suffers a 20 dB CRB loss, whereas the optimal receiver can lose only 2 dB.
Receiver Performance
Figure 3 evaluates delay estimation with subcarriers and 8 OFDM symbols, where BS transmits QPSK data with
Figure 3. Delay estimation RMSE versus INR under 16QAM and 64QAM.
a per-sample echo SNR of dB and BS transmits 16QAM or 64QAM data. The EM receiver runs 10 iterations with a FFT-based delay estimator in the M-step, and the RMSE is averaged over 1000 trials. Since Corollary 1 scales the entire FIM, the delay CRB is inflated by the same factor . The RMSE of MF follows the TIN bound, which equals times the interference-free bound, and breaks down at dB, where the integrated SINR falls below the detection threshold. In contrast, the EM receiver follows the finite-alphabet bound and enables sensing at INRs where TIN fails completely.
Waveform Comparison
Figure 4 compares SC, CP-OFDM, and OTFS () interferers with QPSK over a flat-fading channel and two frequency-selective channels (a two-tap channel with and a three-tap channel with ), where denotes the mean INR. We evaluate (7) with 1000 Monte Carlo samples of per point. For a non-separable , this requires the joint posterior of over hypotheses, whose number grows exponentially with . Therefore, we set with hypotheses. Moreover, since is no longer a scaled identity, the CRB loss depends on the echo derivative in (8). We thus provide the average Fisher-information (FI) loss with , which reduces to under Proposition 3. Over the flat-fading channel in Figure 4(a), all waveforms coincide, confirming Proposition 3. Over the frequency-selective channels, SC and OTFS with joint processing remain close to the flat curve, since each symbol spreads over all subcarriers and thus experiences the full channel energy. In contrast, CP-OFDM exhibits a lower peak but a residual tail, since deep-fade subcarriers remain in the harmful INR band as shown in Figure 4(b) and Figure 4(c).
Figure 4. Average Fisher-information loss for a QPSK interferer over (a) a flat-fading channel, (b) a two-tap channel, and (c) a three-tap channel.
Conclusion
In this paper, we have presented an estimation-theoretic framework for ISAC sensing under finite-alphabet inter-BS interference. For linear receivers, the impact of the inter-BS interference is invariant to the constellation and waveform. The exact FIM is scaled by , so that the loss peaks at intermediate INR, vanishes at high INR, is waveform-independent for flat-fading and synchronized BS-BS links, and is approached by EM-based soft cancellation. Hence, inter-BS coordination should avoid intermediate INRs, where increasing the interference power may even improve sensing.
Appendix A: Proof of Proposition 1
Since and is unitary, the second moment is
For the fourth moment, let so that . For i.i.d. proper symbols with unit power and kurtosis , the only non-zero fourth-order moments are for and 1 for with , since . Conditioning on yields
Since , the first term gives
Applying (14) again to and using for each row of the unitary ,
Substituting (15) and (16) into (14) yields (6). Finally, since each row of the unitary has unit norm, with equality iff and iff the row has a single non-zero entry, respectively, which gives , completing the proof.
Appendix B: Proof of Lemma 1
The density of is with . Since , Bayes’ rule gives
By the law of total covariance with , , and hence
Since and is given, , and substituting this into (18) yields (7).
Under (A1) and (A4), the circularity of implies that has the same distribution as , satisfying . The Wirtinger chain rule with yields , and therefore
which implies . This completes the proof.
Appendix C: Proof of Theorem 1
The log-likelihood is given by . Let . Since is real-valued, , and the chain rule gives
Using with and ,
where the second term vanishes by Lemma 1, and since is Hermitian, which gives (8). For the bound, the MMSE error covariance is dominated by that of the LMMSE estimator, i.e., with , with equality for Gaussian . Substituting this into (7) yields
Appendix D: Proof of Corollary 1
If , the sufficient statistic consists of independent scalar channels , since and the entries of are i.i.d. Normalizing by and noting that the MMSE is invariant to the known phase of , we have , and thus
Substituting this into (7) yields (10). If , then , so that (8) becomes times the interference-free FIM [20], and , completing the proof.
Appendix E: Proof of Proposition 2
(i) Consider the linear estimator , whose mean-square error is . Since the MMSE estimator minimizes the mean-square error, we have . Substituting these bounds into (11) yields . The upper bound is attained for Gaussian , for which the conditional mean estimator is linear.
(ii) From (11), we have
As , the observation becomes independent of . Consequently, converges to , implying . Since , it follows that .
(iii) Let denote the minimum-distance detector, and let and be the minimum and maximum distances between distinct constellation points, respectively. Since the MMSE estimator minimizes the mean-square error, we have
where . For an -ary constellation in complex AWGN, the union bound and give
Combining these inequalities yields
Since , the right-hand side converges to zero as . Thus, , and (11) gives .
(iv) is continuous and approaches unity as and , as established in (ii) and (iii), and exceeds unity for since . Therefore, there exist such that the global maximum is attained within . By continuity, attains its maximum at some finite , completing the proof.
References
- [1]F. Liu, Y.-F. Liu, Y. Cui, C. Masouros, J. Xu, T. X. Han, S. Buzzi, Y. C. Eldar, and S. Jin, “Sensing with communication signals: From information theory to signal processing,” IEEE Journal on Selected Areas in Communications, 2025.arxiv.org/abs/2502.10819
- [2]C. Sturm and W. Wiesbeck, “Waveform design and signal processing aspects for fusion of wireless communications and radar sensing,” Proceedings of the IEEE, vol. 99, no. 7, pp. 1236–1259, 2011.DOI
- [3]K. Han, K. Meng, X.-Y. Wang, and C. Masouros, “Network-level ISAC design: State-of-the-art, challenges, and opportunities,” IEEE Journal of Selected Topics in Electromagnetics, Antennas and Propagation, 2025.arxiv.org/abs/2505.01295
- [4]K. Meng, K. Han, C. Masouros, and L. Hanzo, “Network-level ISAC: An analytical study of antenna topologies ranging from massive to cell-free MIMO,” IEEE Transactions on Wireless Communications, vol. 24, no. 12, pp. 10 003–10 018, 2025.DOI
- [5]K. Meng, C. Masouros, G. Chen, and F. Liu, “Network-level integrated sensing and communication: Interference management and BS coordination using stochastic geometry,” IEEE Transactions on Wireless Communications, vol. 23, no. 12, pp. 19 365–19 381, 2024.arxiv.org/abs/2311.09052
- [6]F. Liu, Y. Zhang, X. Xiong, S. Li, W. Yuan, F. Gao, S. Jin, and G. Caire, “CP-OFDM achieves the lowest average ranging sidelobe under QAM/PSK constellations,” IEEE Transactions on Information Theory, 2025.arxiv.org/abs/2407.06691
- [7]M. F. Keskin, M. M. Mojabedian, J. O. Lacruz, C. Marcus, M. Eriksson, A. Giorgetti, J. Widmer, and H. Wymeersch, “Fundamental trade-offs in monostatic ISAC: A holistic investigation toward 6G,” IEEE Transactions on Wireless Communications, vol. 24, no. 9, pp. 7856–7873, 2025.DOI
- [8]Z. Du, F. Liu, Y. Xiong, T. X. Han, Y. C. Eldar, and S. Jin, “Reshaping the ISAC tradeoff under OFDM signaling: A probabilistic constellation shaping approach,” IEEE Transactions on Signal Processing, vol. 72, pp. 4782–4797, 2024.arxiv.org/abs/2312.15941
- [10]K. Meng, K. Han, C. Masouros, and F. Liu, “Constellation selection and power control for OFDM-based ISAC: From theory to prototype,” IEEE Transactions on Signal Processing, 2026.arxiv.org/abs/2603.03895
- [11]K. Han, K. Meng, A. Chatzicharistou, and C. Masouros, “OFDM-ISAC over Data Payloads: MSE Analysis, Constellation Design, and Experimentation,” arXiv preprint arXiv:2608.15564, 2026.arxiv.org/abs/2608.15564
- [12]K. Meng, K. Han, C. Masouros, and L. Hanzo, “Constellation Selection and Power Allocation for Multi-Cell OFDM-ISAC: Managing Inter-Cell Interference and Sensing Sidelobes,” arXiv preprint arXiv:2607.21418, 2026.arxiv.org/abs/2607.21418
- [13]H. Pu, A. Musallam, H. Li, and L. Hanzo, “Blind OFDM-ISAC relying on asymmetric modem constellations,” arXiv:2604.26200, 2026.arxiv.org/abs/2604.26200
- [14]A. Dytso, D. Tuninetti, and N. Devroye, “Interference or noise: Friend or foe?” IEEE Transactions on Information Theory, vol. 62, no. 6, pp. 3561–3596, 2016.arxiv.org/abs/1506.02597
- [15]P. Stoica and P. Babu, “The Gaussian data assumption leads to the largest Cramér–Rao bound,” IEEE Signal Processing Magazine, vol. 28, no. 3, pp. 132–133, 2011.DOI
- [16]L. D. Brown, “A proof of the central limit theorem motivated by the Cramér–Rao inequality,” in Statistics and Probability: Essays in Honor of C. R. Rao. North-Holland, 1982, pp. 141–148.
- [17]M. Payaró and D. Palomar, “Hessian and concavity of mutual information, differential entropy, and entropy power in linear vector Gaussian channels,” IEEE Transactions on Information Theory, vol. 55, no. 8, pp. 3613–3628, 2009.arxiv.org/abs/0903.1945
- [18]A. P. Dempster, N. M. Laird, and D. B. Rubin, “Maximum likelihood from incomplete data via the EM algorithm,” Journal of the Royal Statistical Society Series B (Methodological), vol. 39, no. 1, pp. 1–38, 1977.DOI
- [19]X. Wang and H. V. Poor, “Iterative (turbo) soft interference cancellation and decoding for coded CDMA,” IEEE Transactions on Communications, vol. 47, no. 7, pp. 1046–1061, 1999.DOI
- [20]S. M. Kay, Fundamentals of Statistical Signal Processing: Estimation Theory. Upper Saddle River, NJ: Prentice Hall, 1993.


