betaFeedback

WorksEngineering

Synergizing Wireless Foundation Models and LLMs: A Multi-Task Framework for MA Systems

Abstract

While movable antennas (MAs) provide enhanced spatial flexibility through dynamic repositioning, they confront critical challenges in acquiring complete channel state information (CSI) and efficiently optimizing antenna positions. Encouragingly, wireless foundation models (WFMs) capture generalizable channel priors for high-quality CSI extrapolation, while large language models (LLMs) enable parameter-efficient, token-level optimization for wireless tasks. In this paper, we synergistically integrate a WFM and an LLM into a multi-task framework for MA systems. First, to address the mismatch between existing WFM pre-training and MA channel characteristics, we develop an MA-oriented ``ExtLWM'' based on the large wireless model (LWM) architecture. Second, we design a unified ExtLWM-Connector-LLM architecture that flexibly adapts to different MA configurations without structural redesign. Third, we reformulate discrete MA positioning as a sequential decision-making problem, where candidate positions are represented by numerical tokens and optimized using group relative policy optimization (GRPO) to exploit the serial generation capability of the LLM. Finally, a multi-stage training strategy is developed to maintain multi-task performance under limited training samples while mitigating catastrophic forgetting. Simulation results show that ExtLWM can produce high-quality channel embeddings while outperforming various benchmarks in CSI extrapolation. Meanwhile, the proposed LLM-based approach achieves a favorable trade-off between performance and computational complexity.

Introduction

Movable antenna (MA) is an emerging technology for next-generation wireless communications, providing a new approach to further enhance wireless transmission capability [?, ?]. Specifically, unlike conventional fixed-position antenna (FPA) arrays, whose elements are fabricated at predetermined locations, MAs are connected to radio-frequency chains through flexible cabling and can be programmably moved via precise motor control [?, ?]. This position reconfigurability introduces additional spatial degrees of freedom (DoFs), allowing MA systems to exploit geometry gain and spatial diversity to improve communication performance without the need for substantial array-size expansion [?, ?], [?, ?].

Existing MA systems typically adopt either continuous or discrete positioning mechanisms. Although continuous-position adjustment provides greater spatial flexibility and potentially higher performance, it poses significant implementation challenges in practice [?, ?]. First, the achievable positioning step size is fundamentally restricted by the physical resolution of mechanical actuators, necessitating sophisticated, high-precision hardware to approximate continuous movement [?, ?], [?, ?]. Second, continuous position adjustment typically entails frequent repositioning, resulting in elevated actuator power consumption, prolonged movement latency, and substantial control overhead [?, ?]. Consequently, discrete-position adjustment offers a more favorable trade-off between system performance and implementation complexity.

Despite its practical advantages, discrete-position MA system will inevitably confront two issues in deployment: The difficulty of acquiring complete CSI from limited channel observations and the complexity of discrete MA position optimization under implementation constraints. Firstly, for CSI acquisition, due to the limited movement speed of MAs and the constrained pilot overhead, only a small subset of candidate positions can be explored within a given channel coherence interval, resulting in incomplete CSI observations [?, ?]. To address this issue, a feasible approach is to extrapolate the CSI of all unobserved candidate positions from incomplete CSI. In [?, ?], the authors design an extrapolation algorithm by learning a low-dimensional channel basis and estimating the corresponding latent coefficients. Nevertheless, such model-based methods heavily rely on predefined channel priors. In contrast, data-driven deep neural network (DNN)-based methods can directly learn spatial channel correlations among different candidate positions from large-scale CSI samples, thereby enabling more flexible CSI extrapolation. For instance, the authors of [?, ?] design an asymmetric architecture consisting of a self-attention-based encoder and a graph-attention-based decoder. In [?, ?], the authors augment the encoder with a mixture-of-experts module to enhance feature extraction and noise resilience. Besides, similar studies [?, ?], [?, ?] have demonstrated that masked autoencoder-based architectures can achieve superior CSI extrapolation performance compared with conventional network architectures. Although the above dedicated CSI extrapolation models can recover complete CSI to directly support downstream tasks [?, ?], [?, ?], their training typically relies on sufficient complete CSI samples and their transferability to unseen scenarios remains limited.

Wireless foundation models (WFMs), which learn generalizable channel representations through large-scale pre-training, offer a promising avenue to mitigate these challenges by delivering superior sample efficiency across diverse downstream tasks [?, ?]–[?, ?]. The large wireless model (LWM) proposed in [?, ?] achieves near-saturated line-of-sight (LoS)/non-LoS (NLoS) identification performance with less than 1% labeled samples, while attaining comparable beam prediction performance using only 40%-50% of the training data. Regarding cross-scenario transferability, the ContraWiMAE proposed in [?, ?] maintains competitive performance when transferred to unseen propagation scenarios and carrier frequencies, further demonstrating the generalizability of its pre-trained channel representations. These advantages are particularly valuable for MA systems, where acquiring sufficient complete CSI samples across candidate positions is often impractical, while channel characteristics can vary substantially across different propagation environments and position configurations. Nevertheless, the use of WFMs for CSI extrapolation in MA systems, particularly under highly incomplete observations and diverse scenarios, remains insufficiently explored.

Secondly, discrete MA position optimization requires selecting the location of each MA from a large number of candidate points while satisfying inter-MA distance constraints, giving rise to a computationally challenging combinatorial optimization problem. Conventional branch-and-bound (BnB) methods can obtain high-quality solutions but suffer from prohibitive computational complexity as the search space grows [?, ?], whereas penalty-based relaxation (PBR) methods reduce the optimization burden at the cost of non-negligible performance loss [?, ?]. Besides, network-based approaches also face inherent limitations in handling such discrete combinatorial structures, since directly formulating the problem as a classification task leads to an excessively large output space, whereas generating continuous positions followed by discretization may result in suboptimal or even infeasible solutions [?, ?], [?, ?]. Overall, existing approaches still suffer from certain limitations in terms of computational efficiency and solution quality.

To strike a more favorable balance, the combinatorial MA position optimization problem can be reformulated as a sequential optimization problem. By decomposing the high-dimensional joint decision into a sequence of lower-dimensional decisions, sequential optimization can significantly alleviate the curse of dimensionality and substantially reduce optimization complexity [?, ?]. Meanwhile, compared with parallel feedforward computation, sequential optimization can provide greater effective computational depth, and thus has the potential to achieve better performance on problems with strong sequential dependencies under comparable model sizes and training data [?, ?], [?, ?]. Notably, large language model (LLM) inherently follows an autoregressive generation mechanism, which is well aligned with the sequential decision structure of the reformulated MA positioning problem. In addition to this structural compatibility, LLMs can be adapted to wireless communication tasks through lightweight task-specific training with only a limited number of trainable parameters. Existing studies have demonstrated such parameter-efficient adaptation [?, ?]-[?, ?]. For example, the authors of [?, ?] fine-tune only about 5.95M parameters of GPT-2 and achieve sum-rate performance comparable to that of a 13.15M-parameter transformer trained from scratch. Moreover, the authors of [?, ?] propose a prompt-as-prefix strategy that converts historical optimal beam indices and angle information into discrete numerical tokens, thereby improving the average normalized beamforming gain of GPT-2 by approximately 62.4% for the beam prediction task. However, existing studies mainly focus on parameter-efficient adaptation of pre-trained LLMs to individual wireless tasks, while their intrinsic autoregressive capability for sequential optimization in wireless communications remains largely unexplored. This gap motivates the use of pre-trained LLMs as sequential decision models for discrete MA position optimization.

More importantly, CSI extrapolation and MA position optimization should not be treated as isolated tasks, but rather jointly constitute an integrated wireless communication process. Developing separate task-specific models would lead to duplicated representation learning, inefficient use of limited training samples, and limited scalability across different MA configurations. To enable unified and efficient operation throughout this entire process, synergizing WFMs and LLMs offers a promising multi-task paradigm. Specifically, WFMs can perform reliable CSI extrapolation and extract informative feature embeddings from limited training samples, while LLMs can leverage these representations to support both classification and sequential optimization. However, effectively integrating these two types of foundation models into a unified architecture remains nontrivial, and systematic studies on multi-task MA systems are still limited.

Motivated by the above analysis, this paper innovatively integrates the WFM and LLM to support multiple downstream tasks in MA systems. Specifically, we consider three representative tasks, including CSI extrapolation as a regression task, LoS/NLoS identification as a classification task, and sum-rate maximization as a sequential optimization task. Building upon this framework, this paper aims to explore the under-investigated network architectures and training paradigms for synergizing WFMs and LLMs in MA systems, and seeks to answer the following pivotal questions: Q1) Can existing WFMs developed for complete CSI in FPA systems and primarily designed for feature extraction be effectively transferred to MA systems? In other words, is it necessary to develop a dedicated WFM for MA systems under incomplete CSI? Q2) How can WFMs and LLMs be seamlessly integrated at the network architecture level to construct a unified framework with a fixed backbone? Q3) How can the potential of LLMs for sequential optimization of discrete MA positions be effectively exploited? Q4) How can a unified training paradigm be designed under limited training samples while maintaining satisfactory performance across different tasks? Correspondingly, the main contributions of this paper can be summarized as follows:

  • For Q1), we develop a WFM for MA systems under incomplete CSI, termed “ExtLWM”. ExtLWM can simultaneously provide high-quality extrapolated CSI and informative feature embeddings. Simulation results demonstrate that ExtLWM exhibits strong generalization across different scenarios, CSI incompleteness ratios, orthogonal frequency division multiplexing (OFDM) configurations, and array structures. Moreover, its CSI extrapolation performance outperforms the FPA-oriented LWM-1.1 and models trained from scratch, with more pronounced advantages under limited training data.

  • For Q2), we design a lightweight Q-Former-based connector to seamlessly integrate the WFM and LLM. Meanwhile, representative tasks in MA systems are unified within the proposed framework: CSI extrapolation is handled by the WFM, while LoS/NLoS identification and sum-rate maximization are further formulated as token-level optimization tasks for the LLM. Simulation results demonstrate that the proposed framework can effectively accommodate different MA system configurations, and also exhibits good compatibility with different LLMs.

  • For Q3), we formulate the sequential MA positioning problem within a Markov decision process (MDP) framework and represent MA positions using numerical tokens. Building on this formulation, we employ a group relative policy optimization (GRPO) algorithm for optimization. Simulation results demonstrate that the proposed method can effectively exploit the capability of LLMs while alleviating the credit-assignment challenge. Moreover, it achieves higher sum-rate performance than conventional PBR methods with low execution overhead.

  • For Q4), we propose a multi-stage training algorithm for data-limited scenarios. In particular, to preserve the performance of the classification task while optimizing the sum-rate, we incorporate experience replay (ER) into GRPO training process. Simulation results demonstrate that the proposed multi-stage training strategy achieves better performance across multiple tasks than joint training, while the incorporation of ER more effectively mitigates catastrophic forgetting.

System Model and Problem Formulation

Figure 1. Overview of the MA system under incomplete CSI, where a multimodal model simultaneously enables multiple tasks.

We consider an MA-enabled OFDM downlink communication system, where a BS is equipped with NN MAs to serve KK single-antenna users. As shown in Figure 1, the BS is equipped with a one-dimensional array featuring MM candidate points spaced by distance DD that are available for the deployment of NN antenna elements (M≫NM \gg N). Let M≜{1,2,...,M}\mathcal{M}\triangleq\{1,2,...,M\}, N≜{1,2,...,N}\mathcal{N}\triangleq\{1,2,...,N\} and K≜{1,2,...,K}\mathcal{K}\triangleq\{1,2,...,K\} denote the sets of all candidate positions, MAs and users, respectively. Define position vector p≜[p1,p2,...,pN]T\mathbf{p}\triangleq[p_1,p_2,...,p_N]^T to indicate the position indices of NN MAs with the entries pn∈Mp_n\in\mathcal{M} arranged in ascending order, i.e., pi>pjp_i>p_j for i>ji>j. Besides, to mitigate mutual coupling effect, a minimum separation distance should be maintained between MAs, i.e.,

pi−pj≥dminD,∀i,j∈N,i>j,\begin{equation} p_i - p_j \geq \frac{d_\mathrm{min}}{D}, \forall i, j \in \mathcal{N}, i > j, \tag*{(1)} \end{equation}

where dmind_\mathrm{min} is the “safety spacing” with dmin≥Dd_\mathrm{min} \geq D. The BS employs OFDM modulation, where S≜{1,2,...,S}\mathcal{S}\triangleq\{1,2,...,S\}, λc\lambda_{\mathrm{c}}, fc=c0λcf_{\mathrm{c}}=\frac{c_0}{\lambda_{\mathrm{c}}}, and Δf\Delta f represent the subcarrier set, center carrier wavelength, center carrier frequency, and subcarrier spacing, respectively. The frequency of the ss-th subcarrier is fs=fc+(s−S+12)Δff_s =f_{\mathrm{c}}+(s-\frac{S+1}{2}) \Delta f.

Channel Model and Incomplete Observation

The downlink channel is described by a multi-carrier geometry-based channel model to characterize multipath propagation, which is consistent with the modeling principles adopted by state-of-the-art ray-tracing-based channel simulators/datasets, such as DeepMIMO [?, ?] and Sionna [?, ?]. Specifically, define hk,s∈CM\mathbf{h}_{k,s} \in \mathbb{C}^{M} as the channel vector between the kk-th user and all MM candidate locations at subcarrier ss, which is modeled as

hk,s=∑g=1Gk1ρk,gαk,ge−j2πfsτk,ga(φk,g),\begin{equation} \mathbf{h}_{k,s} =\sum_{g=1}^{G_k} \sqrt{\frac{1}{\rho_{k, g}}} \alpha_{k, g} e^{-j 2 \pi f_s \tau_{k, g}} \mathbf{a}(\varphi_{k, g}), \tag*{(2)} \end{equation}

where GkG_k denotes the number of effective propagation paths for user kk, while ρk,g\rho_{k,g}, αk,g\alpha_{k,g}, τk,g\tau_{k,g}, and φk,g\varphi_{k,g} denote the path loss, frequency-independent complex path coefficient, propagation delay, and angle of departure (AoD) of the gg-th path of user kk, respectively. Moreover, a(φk,g)\mathbf{a}(\varphi_{k,g}) denotes the array steering vector corresponding to the MM candidate MA positions, which is further expressed as

a(φ)=[1,e−j2πfcDsin⁡φc0,...,e−j2πfc(M−1)Dsin⁡φc0]T.\begin{equation} \mathbf{a}(\varphi) = [1, e^{-\frac{j 2 \pi f_{\mathrm{c}} D \sin \varphi}{c_0}}, ..., e^{ -\frac{ j 2 \pi f_{\mathrm{c}} (M-1)D \sin \varphi}{c_0}}]^T. \tag*{(3)} \end{equation}

where c0c_0 denotes the speed of light. Therefore, the complete channel matrix of the kk-th user is denoted as Hk≜[hk,1,hk,2,...,hk,S]∈CM×S\mathbf{H}_k \triangleq [\mathbf{h}_{k,1}, \mathbf{h}_{k,2}, ..., \mathbf{h}_{k,S} ] \in \mathbb{C}^{M \times S}. Subsequently, denote [hk,s]m[\mathbf{h}_{k,s}]_m as the channel coefficient associated with the mm-th candidate position, i.e., the mm-th element of hk,s\mathbf{h}_{k, s}. Then, the downlink channel vector between NN MAs with positions p\mathbf{p} and the kk-th user at the ss-th subcarrier can be expressed as

hˉk,s≜[hk,s]{p}∈CN.\begin{equation} \bar{\mathbf{h}}_{k, s} \triangleq \big[\mathbf{h}_{k,s}\big]_{\{\mathbf{p}\}} \in \mathbb{C}^{N}. \tag*{(4)} \end{equation}

Consequently, the channel matrix spanning NN MAs across all subcarriers is denoted by Hˉk≜[hˉk,1,hˉk,2,...,hˉk,S]\bar{\mathbf{H}}_k\triangleq [\mathbf{\bar{h}}_{k,1}, \mathbf{\bar{h}}_{k,2}, ..., \mathbf{\bar{h}}_{k,S} ], which is indeed a submatrix of the complete CSI Hk\mathbf{H}_k. Conventional studies [?, ?], [?, ?] typically assume perfect knowledge of complete CSI across all candidate positions, often acquired by concatenating channel measurements across multiple time slots. However, obtaining comprehensive CSI remains practically daunting due to prohibitive pilot overhead and stringent demands on antenna mechanics for instantaneous physical relocation of antenna elements. To be practical, we define the more accessible incomplete CSI for the kk-th user as Hinc,k≜[hinc,k,1,hinc,k,2,…,hinc,k,S]∈CM×S\mathbf{H}_{\mathrm{inc},k} \triangleq [\mathbf{h}_{\mathrm{inc},k,1}, \mathbf{h}_{\mathrm{inc},k,2}, \dots, \mathbf{h}_{\mathrm{inc},k,S}] \in \mathbb{C}^{M \times S}, where each subcarrier vector is formulated as

hinc,k,s=hk,s⊙o,\begin{equation} \mathbf{h}_{\mathrm{inc}, k, s} = \mathbf{h}_{k, s} \odot \mathbf{o}, \tag*{(5)} \end{equation}

where o≜[o1,o2,…,oM]T∈RM\mathbf{o} \triangleq [o_1, o_2, \dots, o_M]^T \in \mathbb{R}^{M} represents an indicator vector with om∈{0,1}o_m \in \{0, 1\} indicating whether the channel coefficient of position mm is observable. Furthermore, we define the CSI incompleteness ratio as ηmsk≜1M∑m=1M(1−om)\eta_{\mathrm{msk}}\triangleq \frac{1}{M}\sum_{m=1}^{M}(1-o_m), which represents the proportion of candidate positions with unobservable CSI.

CSI Extrapolation

Given that complete CSI is indispensable for MA configuration, it is highly desirable to reconstruct complete CSI matrix. Specifically, we aim to extrapolate the CSI of unobserved antenna positions based on the sparse measurements through inference, instead of conventional physical scan [?, ?]–[?, ?]. Extrapolation of the channel coefficients of unseen positions (i.e., those with oj=0o_j = 0) reads

hext,k,s=fext(hinc,k,s),\begin{equation} \mathbf{h}_{\mathrm{ext}, k, s} = \bm{f}_{\mathrm{ext}} (\mathbf{h}_{\mathrm{inc},k, s}), \tag*{(6)} \end{equation}

where fext(⋅)\bm{f}_{\mathrm{ext}}(\cdot) is the extrapolation function and hext,k,s\mathbf{h}_{\mathrm{ext}, k, s} is a prediction of the true channel hk,s\mathbf{h}_{k,s}. Accordingly, let Hext,k≜[hext,k,1,hext,k,2,...,hext,k,S]\mathbf{H}_{\mathrm{ext}, k}\triangleq [\mathbf{h}_{\mathrm{ext}, k, 1}, \mathbf{h}_{\mathrm{ext}, k, 2},..., \mathbf{h}_{\mathrm{ext}, k, S} ] represent the extrapolated CSI matrix for user kk, whose accuracy can be evaluated by normalized mean square error (NMSE) defined as

NMSEk=Es∈S[∥hext,k,s−hk,s∥2∥hk,s∥2].\begin{equation} \mathrm{NMSE}_{k} = \mathbb{E}_{s \in \mathcal{S}} \bigg[ \frac{\|\mathbf{h}_{\mathrm{ext}, k, s}-\mathbf{h}_{k, s} \|^2}{ \|\mathbf{h}_{k, s} \|^2} \bigg]. \tag*{(7)} \end{equation}

LoS/NLoS Identification

LoS/NLoS identification is an important component of wireless environment awareness and facilitates context-aware tasks such as parameter estimation and positioning [?, ?]. Leveraging the acquired incomplete CSI, the identification process for the kk-th user is formulated as

δ^k=fid(Hinc,k),\begin{equation} \widehat{\delta}_k = \bm{f}_{\mathrm{id}} (\mathbf{H}_{\mathrm{inc}, k}), \tag*{(8)} \end{equation}

where fid(⋅)\bm{f}_{\mathrm{id}}(\cdot) represents the identification algorithm/network, while δ^k\widehat{\delta}_k and δk∈{0,1}\delta_k \in \{0, 1\} signify the classified and genuine CSI category for user kk, with 11 and 00 indicating LoS and NLoS conditions, respectively. To assess the identification capability across KK users, we adopt the weighted F1-score, denoted as F1-score(K)∈[0,1]\mathrm{F1}\text{-}\mathrm{score}(\mathcal{K}) \in [0, 1], as the performance metric, which jointly accounts for class-wise precision and recall while weighting each class according to its sample proportion, making it suitable for evaluating performance under imbalanced channel-type distributions.

Sum-Rate Maximization

The sum-rate across all users and subcarriers is a key metric for MA-enabled multi-user communication systems. Let wk,s∈CN\mathbf{w}_{k, s} \in \mathbb{C}^{N} denote the precoder of the kk-th user on subcarrier ss. The received signal by the kk-th user on subcarrier ss can be written as

yk,s=h‾k,sHWsxs+ns,\begin{equation} y_{k,s}=\mathbf{\overline{h}}_{k, s}^H \mathbf{W}_{s} \mathbf{x}_s +n_s, \tag*{(9)} \end{equation}

where Ws≜[w1,s,w2,s,...,wK,s]∈CN×K\mathbf{W}_s \triangleq [\mathbf{w}_{1,s}, \mathbf{w}_{2,s}, ..., \mathbf{w}_{K,s} ] \in \mathbb{C}^{N \times K} denotes the precoding matrix for subcarrier ss, vector xs\mathbf{x}_s denotes uncorrelated information symbols following complex Gaussian distribution with zero-mean and unit variance, i.e., E{xsxsH}=IK\mathbb{E}\{\mathbf{x}_s\mathbf{x}_s^H\}=\mathbf{I}_K, and ns∼CN(0,σc2)n_s \sim\mathcal{C}\mathcal{N}(0, \sigma_{\mathrm{c}}^2) represents the receiving noise. Consequently, the signal-to-interference-plus-noise ratio (SINR) of the kk-th user on subcarrier ss is given as

SINRk,s=∣h‾k,sHwk,s∣2∑j≠k∣h‾k,sHwj,s∣2+σc2.\begin{equation} \mathrm{SINR}_{k, s} =\frac{|\mathbf{\overline{h}}_{k,s}^H \mathbf{w}_{k, s} |^2}{\sum_{j \neq k} |\mathbf{\overline{h}}_{k, s}^H \mathbf{w}_{j, s} |^2+\sigma_{\mathrm{c}}^2}. \tag*{(10)} \end{equation}

Subsequently, the achievable communication rate of the kk-th user on subcarrier ss is given by

Rk,s=log⁡2(1+SINRk,s).\begin{equation} R_{k,s} =\log _2 (1+\mathrm{SINR}_{k,s} ). \tag*{(11)} \end{equation}

Recalling (4), the sum-rate of all KK users at all SS subcarriers, i.e.,∑s=1S∑k=1KRk,s\sum_{s=1}^{S} \sum_{k=1}^{K} R_{k, s}, is jointly determined by the positions of MAs and beamforming. Therefore, the sum-rate maximization task under incomplete CSI can be expressed as {p,W1,...,WS}=fmax({Hinc,1,...,Hinc,K})\{ \mathbf{p}, \mathbf{W}_1, ..., \mathbf{W}_S \} = \bm{f}_{\mathrm{max}} (\{\mathbf{H}_{\mathrm{inc}, 1}, ..., \mathbf{H}_{\mathrm{inc}, K}\}).

Problem Formulation

In this paper, instead of relying on traditional task-specific architectures that design fext(⋅)\bm{f}_{\mathrm{ext}}(\cdot), fid(⋅)\bm{f}_{\mathrm{id}}(\cdot), and fmax(⋅)\bm{f}_{\mathrm{max}}(\cdot) separately, we propose a unified multimodal network with parameter θ\bm{\theta} to perform all tasks concurrently. Therefore, the multi-task optimization problem, centered on sum-rate maximization, is formulated as follows

max⁡θ  ∑s=1S∑k=1KRk,s,s.t.   pi−pj≥dminD, ∀i,j∈N,i>j,   pn∈M, ∀n∈N,   NMSEk≤ϵNMSE, ∀k∈K,   F1-score(K)≥ϵF1, ∑s=1S∑k=1K∥wk,s∥2≤Ptx,\begin{align} \max _{\bm{\theta}} & \, \, \sum_{s=1}^{S} \sum_{k=1}^{K} R_{k, s}, \\ \mathrm{ s.t. } & \,\,\, p_i - p_j \geq \frac{d_\mathrm{min}}{D}, ~\forall i, j \in \mathcal{N}, i > j, \\ & \,\,\, p_n \in \mathcal{M}, ~\forall n \in\mathcal{N}, \\ & \,\,\, \mathrm{NMSE}_{k} \leq \epsilon_{\mathrm{NMSE}}, ~\forall k \in\mathcal{K}, \\ & \,\,\, \mathrm{F1}\text{-}\mathrm{score}(\mathcal{K}) \geq \epsilon_{\mathrm{F1}}, \\ & \, \sum_{s=1}^{S}\sum_{k=1}^{K} \|\mathbf{w}_{k,s}\|^2 \leq P_{\mathrm{tx}}, \end{align}

where eq_constraint_pos and eq_constraint_p specify the constraints on the MA positions, eq_constraint_nmse and eq_constraint_f1 mandate performance thresholds for CSI extrapolation and LoS/NLoS identification, respectively, and eq_constraint_power imposes the power constraint with fixed transmit power PtxP_{\mathrm{tx}}.

In this paper, we synergistically integrate the WFM and LLM to construct a unified network parameterized by θ\bm{\theta}, which flexibly accommodates varying numbers of MAs and candidate positions. Specifically, the WFM supports the effective use of the LLM in sample-limited environments by providing high-quality channel embeddings and accurate CSI extrapolation. The LLM then leverages the wireless domain knowledge provided by the WFM to formulate different wireless tasks as token-level optimization problems. Besides supporting conventional classification tasks, the LLM can fully exploit its sequential generation capability for the MA positioning.

Unified network architecture

In this section, we introduce the unified framework that synergistically integrates the WFM and LLM via a connector, i.e., θ≜{θWFM,θCon,θLLM}\bm{\theta} \triangleq \{ \bm{\theta}_{\mathrm{WFM}}, \bm{\theta}_{\mathrm{Con}}, \bm{\theta}_{\mathrm{LLM}}\}. The detailed architectures of the individual network components are introduced in the following subsections.

ExtLWM-based WFM

LWM has been demonstrated to be effective in extracting channel features for OFDM systems and capable of facilitating diverse downstream physical-layer applications. Considering the mismatch between existing WFMs and MA systems, we develop an MA-oriented WFM based on the LWM architecture, termed “Extended LWM (ExtLWM)”. ExtLWM enables high-quality CSI extrapolation from incomplete CSI while simultaneously producing informative channel embeddings to support subsequent downstream tasks. Defining θWFM\bm{\theta}_{\mathrm{WFM}} as the parameters of ExtLWM, the designs of its components (as illustrated in Figure 2) are specified in the following.

Input Layer

The input to ExtLWM consists of incomplete CSI measurements Hinc,k\mathbf{H}_{\mathrm{inc},k} collected at candidate positions whose locations are known. These measurements are organized into a formatted data matrix Xin,k∈RM×2S\mathbf{X}_{\mathrm{in},k} \in \mathbb{R}^{M\times 2S}, which comprises MM position-wise patches. Specifically, for n∈Nn\in\mathcal{N}, the pnp_n-th patch xpn,kT\mathbf{x}_{p_n,k}^T of Xin,k\mathbf{X}_{\mathrm{in},k} is given as

xpn,kT=αp[ℜ{hinc,pn,k,1},...,ℜ{hinc,pn,k,S},ℑ{hinc,pn,k,1},...,ℑ{hinc,pn,k,S}],\begin{equation} \begin{aligned} \mathbf{x}_{p_n,k}^T=\alpha_\mathrm{p} \big[&\Re\{h_{\mathrm{inc},p_n,k,1}\},...,\Re\{h_{\mathrm{inc},p_n,k,S}\}, \\ &\Im\{h_{\mathrm{inc},p_n,k,1}\},...,\Im\{h_{\mathrm{inc},p_n,k,S}\} \big], \end{aligned} \tag*{(12)} \end{equation}

where hinc,pn,k,sh_{\mathrm{inc}, p_n,k,s} is the channel coefficient of the nn-th MA over the ss-th subcarrier, i.e., {hinc,k,s}pn\{\mathbf{h}_{\mathrm{inc},k,s}\}_{p_n}, and αp\alpha_\mathrm{p} is a scaling parameter. Moreover, the CSI patches corresponding to vacant candidate positions are replaced with a predefined [MASK][\mathrm{MASK}] value1, i.e.,

xi,k=[MASK]⋅12ST,  s.t.  [o]i=0,∀i∈M,\begin{equation} \mathbf{x}_{i,k}=[\mathrm{MASK}]\cdot \mathbf{1}_{2S}^T, \; \mathrm{s.t.} \; [\mathbf{o}]_i=0, \forall i \in \mathcal{M}, \tag*{(13)} \end{equation}

where o\mathbf{o} is the indication vector (recall (5)).

Embedding Layer

The embedding layer projects the masked CSI data into a high-dimensional space, facilitating the encoder in discovering hidden features related to spatial correlation among antenna positions as well as multipath and scattering characteristics. The signal flow of the embedding layer can be expressed as follows

Xemb,k=LN⁡(Linear⁡(Xin,k)+Xpos),\begin{equation} \mathbf{X}_{\mathrm{emb},k} = \operatorname{LN}\big(\operatorname{Linear}(\mathbf{X}_{\mathrm{in},k}) + \mathbf{X}_\mathrm{pos}\big), \tag*{(14)} \end{equation}

where the linear mapping Linear⁡(⋅)\operatorname{Linear}(\cdot) elevates the signal dimension from M×2SM\times 2S to M×DLWMM\times D_{\mathrm{LWM}} and LN⁡(⋅)\operatorname{LN}(\cdot) denotes the layer normalization operation, Xpos∈RM×DLWM\mathbf{X}_{\mathrm{pos}}\in\mathbb{R}^{M\times D_{\mathrm{LWM}}} denotes the position embedding of MM patches that encodes position information into signal.

Encoder Layer

The encoder layer aims to extract high-dimensional features from incomplete CSI embeddings, thereby establishing a robust representation capable of capturing intricate intrinsic correlations. Inspired by the architecture of LWM-1.1 [?, ?], the encoder layer consists of LencL_{\mathrm{enc}} cascaded encoder blocks with identical structures. The jj-th encoder block takes Xenc,k(j)\mathbf{X}_{\mathrm{enc},k}^{(j)} as the input and outputs the enhanced representation Xenc,k(j+1)\mathbf{X}_{\mathrm{enc},k}^{(j+1)}, where the initial input is given by Xenc,k(1)=Xemb,k\mathbf{X}_{\mathrm{enc},k}^{(1)}=\mathbf{X}_{\mathrm{emb},k}.

Each block integrates a multi-head attention module followed by a feed-forward network (FFN) module. Specifically, within the jj-th block, the input Xenc,k(j)\mathbf{X}_{\mathrm{enc},k}^{(j)} is first projected onto three independent linear subspaces to generate the query, key, and value matrices, denoted as Qenc,k(j)\mathbf{Q}_{\mathrm{enc},k}^{(j)}, Kenc,k(j)\mathbf{K}_{\mathrm{enc},k}^{(j)}, and Venc,k(j)∈RM×DLWM\mathbf{V}_{\mathrm{enc},k}^{(j)} \in \mathbb{R}^{M \times D_{\mathrm{LWM}}}, respectively. To enable parallel processing of distinct feature subspaces, each matrix is evenly partitioned into HencH_{\mathrm{enc}} heads. The scaled dot-product attention for the hh-th head is computed as

Xenc,k,h(j)=Softmax⁡(Qenc,k,h(j)(Kenc,k,h(j))TDLWMHenc)Venc,k,h(j),\begin{equation} \hspace{-0.03cm} \mathbf{X}_{\mathrm{enc}, k, h}^{(j)} = \operatorname{Softmax}\Bigg(\frac{\mathbf{Q}_{\mathrm{enc}, k, h}^{(j)}\big( \mathbf{K}_{\mathrm{enc}, k, h}^{(j)} \big)^T}{\sqrt{ \frac{D_{\mathrm{LWM}}}{H_{\mathrm{enc}}}}}\Bigg)\mathbf{V}_{\mathrm{enc}, k, h}^{(j)}, \tag*{(15)} \end{equation}

where Xenc,k,h(j)∈RM×DLWMHenc\mathbf{X}_{\mathrm{enc}, k, h}^{(j)} \in \mathbb{R}^{M \times \frac{D_{\mathrm{LWM}}}{H_{\mathrm{enc}}}}. Subsequently, the outputs of all attention heads are concatenated, linearly transformed, and further processed by the FFN, yielding the output of the jj-th encoder block, Xenc,k(j+1)∈RM×DLWM\mathbf{X}_{\mathrm{enc},k}^{(j+1)}\in\mathbb{R}^{M\times D_{\mathrm{LWM}}}. After passing through LencL_{\mathrm{enc}} encoder blocks, ExtLWM produces the final CSI representation as Xout,k=Xenc,k(Lenc+1)\mathbf{X}_{\mathrm{out},k}=\mathbf{X}_{\mathrm{enc},k}^{(L_{\mathrm{enc}}+1)}.

Figure 2. Proposed unified network architecture, with the multi-stage training strategy for supporting multiple tasks.

Masked Modeling Head

ExtLWM retains a masked modeling head (MMH), which leverages the high-dimensional latent embeddings extracted by the preceding modules to recover the CSI. It is worth noting that, in [?, ?], MMH is employed only during pre-training to endow LWM-1.1 with task-agnostic and generalizable feature extraction capability. In contrast, in our framework, MMH is not only used for feature learning during training, but also continuously performs CSI extrapolation for subsequent downstream tasks. Specifically, MMH employs an FFN to map the output embedding Xout,k\mathbf{X}_{\mathrm{out},k} to the extrapolated CSI. Denoting the ii-th row of Xout,k\mathbf{X}_{\mathrm{out},k} by xout,i,k\mathbf{x}_{\mathrm{out},i,k}, the corresponding output is given by

xext,i,k=FFN⁡(xout,i,k),  s.t.  [o]i=0,∀i∈M,\begin{equation} \mathbf{x}_{\mathrm{ext},i,k} = \operatorname{FFN}(\mathbf{x}_{\mathrm{out},i,k}), \; \mathrm{s.t.} \; [\mathbf{o}]_i=0, \forall i \in \mathcal{M}, \tag*{(16)} \end{equation}

and predicts the CSI associated with the ii-th candidate position, i.e., ii-th row of Hext,k\mathbf{H}_{\mathrm{ext},k}.

Connector

The connector is designed to bridge the modality gap by mapping from the embeddings of ExtLWM into the linguistic embedding space of LLM. Inspired by [?, ?], the connector employs a lightweight Q-Former architecture for efficient cross-modal alignment. With trainable parameters θCon\bm{\theta}_{\mathrm{Con}}, the connector consists of an embedding layer, a query layer, and a projection layer, which are detailed in the following.

Embedding Layer

To efficiently extract semantic information from the CSI embeddings, we employ a learnable embedding matrix to generate DqD_{\mathrm{q}} query tokens. The initialized tokens Yq∈RDq×DLWM\mathbf{Y}_{\mathrm{q}} \in\mathbb{R}^{D_{\mathrm{q}} \times D_{\mathrm{LWM}}} will serve as interface for cross-modal interaction to capture the latent manifold structure of the CSI embeddings.

Query Layer

The query layer is composed of LqL_{\mathrm{q}} transformer blocks to distill language-relevant representations from CSI features. Specifically, each block consists of the following three components:

  • Self-Attention: For the jj-th block, an HqH_{\mathrm{q}}-head self-attention module is first applied to the query embeddings Yq,k(j)\mathbf{Y}_{\mathrm{q}, k}^{(j)} to facilitate interactions among the query tokens and enhance representational diversity.

  • Cross-Attention: Subsequently, the queries Yq,k(j)\mathbf{Y}_{\mathrm{q},k}^{(j)} act as “probes” to interact with the CSI features through an HqH_{\mathrm{q}}-head cross-attention module, where the keys and values are projected from Xout,k\mathbf{X}_{\mathrm{out},k}. This enables the queries to selectively extract informative task-relevant features from the CSI embeddings.

  • FFN: Finally, the updated query tokens are passed through an FFN to produce the output for the jj-th block, denoted as Yq,k(j+1)\mathbf{Y}_{{\mathrm{q}},k}^{(j+1)}.

Projection Layer

To ensure compatibility with the downstream LLM, the output of the final query block, Yq,k(Lq+1)\mathbf{Y}_{\mathrm{q},k}^{(L_{\mathrm{q}}+1)}, is linearly projected into the GPT input embedding space, yielding Yout,k∈RDq×DLLM\mathbf{Y}_{\mathrm{out},k} \in \mathbb{R}^{D_{\mathrm{q}} \times D_{\mathrm{LLM}}}, where DLLMD_{\mathrm{LLM}} denotes the hidden dimension of GPT. The projected embeddings serve as “soft prompts”, conditioning GPT on the CSI features extracted by ExtLWM.

GPT-Based LLM

We employ a frozen LLM to unify multiple downstream wireless tasks as token-level optimization problems. In particular, for MA positioning, we exploit the pre-trained autoregressive sequential generation capability of the LLM to enable parameter-efficient adaptation. Specifically, the LLM-based cross-modal generation process can be formulated as

ti+1=LLM(Zcsi,{tj}j=1i;θLLM),\begin{equation} t_{i+1} = \mathrm{LLM} \big( \mathbf{Z}_{\mathrm{csi}}, \{t_{j}\}_{j=1}^i ; \bm{\theta}_{\mathrm{LLM}} \big), \tag*{(17)} \end{equation}

where θLLM\bm{\theta}_{\mathrm{LLM}} denotes the pre-trained LLM parameters, Zcsi∈RDx×DLLM\mathbf{Z}_{\mathrm{csi}} \in \mathbb{R}^{D_{\mathrm{x}} \times D_{\mathrm{LLM}}} denotes the aligned CSI embeddings, and {tj}j=1i\{t_{j}\}_{j=1}^i denotes the ordered sequence of ii generated token IDs, with {tj}j=10≜∅\{t_{j}\}_{j=1}^0 \triangleq \emptyset. Specifically, for LoS/NLoS identification, Zcsi=Yout,k\mathbf{Z}_{\mathrm{csi}}=\mathbf{Y}_{\mathrm{out},k} and Dx=DqD_{\mathrm{x}}=D_{\mathrm{q}}. For sum-rate maximization, Zcsi=Concat({Yout,k}k=1K)\mathbf{Z}_{\mathrm{csi}}=\mathrm{Concat}\big(\{\mathbf{Y}_{\mathrm{out},k}\}_{k=1}^{K}\big) and Dx=KDqD_{\mathrm{x}}=KD_{\mathrm{q}}. In this paper, we adopt the GPT-2 architecture [?, ?] as the LLM backbone due to its moderate parameter scale and practical implementation feasibility. Its main components are described as follows.

Embedding Layer

The embedding layer fuses the CSI embeddings with the textual embeddings. Specifically, GPT-2 employs the standard byte-pair encoding tokenizer and maps text tokens into DLLMD_{\mathrm{LLM}}-dimensional embeddings through the word-token embedding (WTE) WTE∈R50257×DLLM\mathbf{W}_{\mathrm{TE}} \in \mathbb{R}^{50257 \times D_{\mathrm{LLM}}} and word-position embedding WPE∈R1024×DLLM\mathbf{W}_{\mathrm{PE}} \in \mathbb{R}^{1024 \times D_{\mathrm{LLM}}} , which are then combined with the CSI embeddings to form the multimodal input:

ztext,iT=[WTE]ti,:+[WPE]i,:,\begin{equation} \mathbf{z}_{\mathrm{text},i}^T=[\mathbf{W}_{\mathrm{TE}}]_{t_i,:} + [\mathbf{W}_{\mathrm{PE}}]_{i,:}, \tag*{(18)} \end{equation}

where ztext,i∈RDLLM\mathbf{z}_{\mathrm{text}, i} \in \mathbb{R}^{D_{\mathrm{LLM}}} denotes the text embedding of the ii-th word token, and ti∈{0,1,...,50256}t_{i} \in \{0,1,...,50256\}. Packing all textual embedding into the matrix ZtextT=[ztext,1,...,ztext,i]\mathbf{Z}_{\mathrm{text}}^T=[\mathbf{z}_{\mathrm{text},1},...,\mathbf{z}_{\mathrm{text}, i}], the final multimodal input sequence Zemb\mathbf{Z}_{\mathrm{emb}} is constructed by prepending the CSI embeddings to the textual prefix

Zemb=[ZcsiZtext]∈R(Dx+i)×DLLM.\begin{equation} \mathbf{Z}_{\mathrm{emb}} = \left[\begin{array}{c} \mathbf{Z}_{\mathrm{csi}} \\ \mathbf{Z}_{\mathrm{text}} \end{array}\right] \in \mathbb{R}^{(D_{\mathrm{x}}+ i ) \times D_{\mathrm{LLM}}}. \\ \tag*{(19)} \end{equation}

Decoder Layer

The backbone adopts a decoder-only architecture consisting of LdecL_{\mathrm{dec}} cascaded decoder blocks with identical structures, which are employed to progressively extract hierarchical latent representations. The jj-th decoder block takes Zdec(j)\mathbf{Z}_{\mathrm{dec}}^{(j)} as the input and outputs the enhanced representation Zdec(j+1)\mathbf{Z}_{\mathrm{dec}}^{(j+1)}, where the initial input is given by Zdec(1)=Zemb\mathbf{Z}_{\mathrm{dec}}^{(1)}=\mathbf{Z}_{\mathrm{emb}}. Specifically, within each block, through normalizing and linearly projecting Zdec(j)\mathbf{Z}_{\mathrm{dec}}^{(j)}, query, key and value matrices Qdec(j)\mathbf{Q}_{\mathrm{dec}}^{(j)}, Kdec(j)\mathbf{K}_{\mathrm{dec}}^{(j)}, and Vdec(j)∈R(Dx+i)×DLLM\mathbf{V}_{\mathrm{dec}}^{(j)} \in\mathbb{R}^{(D_{\mathrm{x}}+ i)\times D_{\mathrm{LLM}}} are obtained and then split into HdecH_{\mathrm{dec}} heads to compute multi-head self-attention, with the hh-th head output given as

Zdec,h(j)=Softmax⁡(Qdec,h(j)(Kdec,h(j))TDLLMHdec+Mdec)Vdec,h(j),\begin{equation} \hspace{-0.1cm} \mathbf{Z}_{\mathrm{dec}, h}^{(j)} = \operatorname{Softmax} \Bigg(\frac{\mathbf{Q}_{\mathrm{dec}, h}^{(j)}\big( \mathbf{K}_{\mathrm{dec}, h}^{(j)} \big)^T}{\sqrt{ \frac{D_{\mathrm{LLM}}}{H_{\mathrm{dec}}}}} + \mathbf{M}_{\mathrm{dec}} \Bigg)\mathbf{V}_{\mathrm{dec}, h}^{(j)}, \tag*{(20)} \end{equation}

where Mdec∈R(Dx+i)×(Dx+i)\mathbf{M}_{\mathrm{dec}} \in\mathbb{R}^{(D_{\mathrm{x}}+ i)\times (D_{\mathrm{x}}+ i)} is the multimodal causal mask. The mask Mdec\mathbf{M}_{\mathrm{dec}} enables bidirectional attention among the DxD_{\mathrm{x}} CSI tokens while preserving causal attention for the textual tokens. The outputs Zdec,h(j)\mathbf{Z}_{\mathrm{dec}, h}^{(j)} from all attention heads are then concatenated and processed by the FFN to obtain Zdec(j+1)\mathbf{Z}_{\mathrm{dec}}^{(j+1)}.

Output Layer

The output layer maps the final decoder representation Zdec(Ldec+1)\mathbf{Z}_{\mathrm{dec}}^{(L_{\mathrm{dec}}+1)} to vocabulary logits for next-token generation. Following GPT-2, the output projection shares the token embedding matrix WTE\mathbf{W}_{\mathrm{TE}}, reducing the parameter count while maintaining consistency with the pre-trained token space. Specifically, the unnormalized logits for the (i+1)(i+1)-th token are given by:

ltext,i+1=WTE([Zdec(Ldec+1)]−1,:)T.\begin{equation} \mathbf{l}_{\mathrm{text}, i+1} = \mathbf{W}_{\mathrm{TE}} \big( [\mathbf{Z}_{\mathrm{dec}}^{(L_{\mathrm{dec}}+1)}]_{-1,:} \big)^T. \tag*{(21)} \end{equation}

The generation of the (i+1)(i+1)-th token is subsequently executed by selecting the token with the highest posterior probability:

ti+1=arg⁡max⁡t∈Tx[Softmax⁡({[ltext,i+1]t′}t′∈Tx)]t,\begin{equation} t_{i+1}=\arg \max_{t \in \mathcal{T}_{\mathrm{x}}} \bigg[ \operatorname{Softmax} \Big( \big\{ [\mathbf{l}_{\mathrm{text}, i+1}]_{t'} \big\}_{t' \in \mathcal{T}_{\mathrm{x}}} \Big) \bigg]_t, \tag*{(22)} \end{equation}

where Tx\mathcal{T}_{\mathrm{x}} denotes a task-specific restricted subset of tokens tailored for x∈{id,max}\mathrm{x} \in \{ \mathrm{id}, \mathrm{max}\}.

  • For LoS/NLoS identification, Tid={8505,3919}\mathcal{T}_{\mathrm{id}} = \{ 8505, 3919 \}, corresponding to word unit “yes\mathrm{yes}” and “no\mathrm{no}”, respectively, which enables GPT-2 to leverage its inherent linguistic priors to output classification results in a semantic manner.

  • For sum-rate maximization, Tmax\mathcal{T}_{\mathrm{max}} is defined as the set of token IDs corresponding to the position indices {“m”}m=1M\{\text{``}m\text{''}\}_{m=1}^{M}. By leveraging numerical tokens with explicit stable semantics, the discrete position optimization of NN MAs can be naturally transformed into a sequential generation problem involving NN consecutive tokens2.

Multi-Stage Training Strategy

Building upon the unified framework, this section develops a complete multi-stage training algorithm to enable parameter-efficient optimization of CSI extrapolation, LoS/NLoS identification, and sum-rate maximization under data-limited conditions. Details of training stages will be specified in the following subsections.

Stage-1: Pre-Training for ExtLWM

The pre-training stage encourages ExtLWM to learn generic structural features of MA channels from a large number of diverse and complex scenarios. Its optimization objective is to train the MMH to reconstruct the original unmasked CSI in (12) based on the output in (16). Accordingly, the self-supervised training loss is formulated as:

Lpre(θWFM)=Ek∈Kpre[∑i∈Ik,oi=0∥xext,i,k−xi,k∥2∥xi,k∥2],\begin{equation} \hspace{-0.08cm} L_{\mathrm{pre}}(\bm{\theta}_{\mathrm{WFM}})= \mathbb{E}_{k \in \mathcal{K}_{\mathrm{pre}}} \Bigg[ \sum_{i\in \mathcal{I}_k, o_{i}=0} \frac{\| \mathbf{x}_{\mathrm{ext}, i, k} - \mathbf{x}_{i,k} \|^2}{\| \mathbf{x}_{i,k} \|^2} \Bigg], \tag*{(23)} \end{equation}

where Kpre\mathcal{K}_{\mathrm{pre}} denotes the set of users used for pre-training, Ik\mathcal{I}_k denotes the set of masked patch indices (representing candidate antenna positions) for the kk-th user, with its cardinality varying across different pre-training scenarios.

To further enhance the CSI extrapolation capability of ExtLWM under highly incomplete CSI, we increase the masking ratio from about 40%40\% in LWM-1.1 to 80%80\%, exposing the model to more severe CSI incompleteness during pre-training. Meanwhile, the masking indices Ik\mathcal{I}_k are randomized in each epoch, encouraging ExtLWM to learn spatial priors that are more robust to different CSI observation patterns and thereby improving its ability to capture informative channel features from sparse CSI.

Stage-2: Fine-Tuning for CSI Extrapolation

Building on the paradigm of large-scale pre-training followed by task-specific fine-tuning, we perform full-parameter fine-tuning of θWFM\bm{\theta}_{\mathrm{WFM}} to adapt the generalized spatial representations to scenario-specific propagation characteristics. This process is particularly effective in data-scarce regimes, where ExtLWM leverages the learned channel priors to achieve high-fidelity CSI extrapolation with limited labeled samples.

Considering (?), stage-2 aims to fully exploit the CSI extrapolation capability of ExtLWM to satisfy eq_constraint_nmse, and its training objective is formulated as

min⁡θWFM Ek∈K[NMSEk].\begin{equation} \min_{\bm{\theta}_{\mathrm{WFM}}} ~ \mathbb{E}_{k \in \mathcal{K}} \big[ \mathrm{NMSE}_k \big]. \tag*{(24)} \end{equation}

Meanwhile, the training loss for stage-2 is formulated as

Lext(θWFM)=Ek∈Kext[∑i∈M,oi=0∥xext,i,k−xi,k∥2∥xi,k∥2],\begin{equation} \hspace{-0.08cm} L_{\mathrm{ext}}(\bm{\theta}_{\mathrm{WFM}})= \mathbb{E}_{k \in \mathcal{K}_{\mathrm{ext}}} \Bigg[ \sum_{i\in \mathcal{M}, o_{i}=0} \frac{\| \mathbf{x}_{\mathrm{ext}, i, k} - \mathbf{x}_{i,k} \|^2}{\| \mathbf{x}_{i,k} \|^2} \Bigg], \tag*{(25)} \end{equation}

where Kext\mathcal{K}_{\mathrm{ext}} denotes the set of users used for fine-tuning with complete CSI. Upon completion of stage-2, we will freeze θWFM\bm{\theta}_{\mathrm{WFM}} to preserve its channel feature extraction and CSI extrapolation capabilities.

Stage-3: Fine-Tuning for LoS/NLoS Identification

The fine-tuned ExtLWM can effectively extract and output scenario-specific channel feature embeddings, serving as a shared feature backbone for multiple downstream wireless tasks. Based on eq_constraint_f1, stage-3 aims to optimize the connector to enable the adaptation of LLM to LoS/NLoS identification. The training objective of stage-3 can be expressed as

max⁡θCon F1-score(K),\begin{equation} \max_{\bm{\theta}_{\mathrm{Con}}} ~ \mathrm{F1}\text{-}\mathrm{score}(\mathcal{K}), \tag*{(26)} \end{equation}

and the training loss of stage-3 can be formulated as

Lid(θCon)=Ek∈Kid[−[δklog⁡pk1+(1−δk)log⁡pk0]],\begin{equation} L_{\mathrm{id}}(\bm{\theta}_{\mathrm{Con}})= \mathbb{E}_{k \in \mathcal{K}_{\mathrm{id}}} \big[ -[\delta_k \log p_k^{1}+(1-\delta_k) \log p_k^{0}] \big], \tag*{(27)} \end{equation}

where Kid\mathcal{K}_{\mathrm{id}} denotes the set of users used for training, for which only incomplete CSI samples and the corresponding LoS/NLoS labels are required, δk∈{0,1}\delta_k \in \{0, 1\} is the ground-truth label and pki=Pr(δ^k=i)p_k^i = \text{Pr}(\hat{\delta}_k = i) is the predicted probability for class ii. Recall (22), pk1p_k^1 and pk0p_k^0 can also be interpreted as the softmax-normalized scores for the semantic tokens, i.e., “yes\mathrm{yes}” and “no\mathrm{no}”, respectively.

Stage-4: RL for Sum-Rate Maximization

In this subsection, we proceed to maximize sum-rate formulated in eq_max_rate. Distinctively, we propose a paradigm shift by analogizing the antenna positioning to token prediction in natural language processing. Just as an LLM predicts the most probable “optimal” tokens within a massive vocabulary based on linguistic context, we aim to identify the optimal antenna positions from a large number of candidates. The core philosophy is to leverage the feature extraction capability of the LLM to interpret wireless channels. By treating the previously selected antenna positions and their corresponding channel states as context, the model is empowered to predict the subsequent optimal positions.

Subsequently, we formulate MA positioning as an MDP to capture sequential dependencies and align it with LLM autoregressive generation. The key elements of the MDP are defined as follows:

  • State sn\mathsf{s}_n: sn\mathsf{s}_n encapsulates the multimodal context, comprising the CSI embeddings output by ExtLWM and the previously generated text tokens:

    sn≜{{Xout,k}k=1K,{tj}j=1n−1},∀n∈N.\begin{equation} \mathsf{s}_n \triangleq \big\{ \big\{ \mathbf{X}_{\mathrm{out}, k} \big\}_{k=1}^K, \{t_{j}\}_{j=1}^{n-1} \big\}, \forall n \in \mathcal{N}. \tag*{(28)} \end{equation}
  • Action an\mathsf{a}_n: an\mathsf{a}_n represents the prediction of nn-th token ID tnt_n. Recall that we establish a semantic mapping between the discrete space Tmax\mathcal{T}_{\mathrm{max}} and the physical coordinate pnp_n of the nn-th MA, ensuring eq_constraint_p is satisfied.

  • Action mask mn\mathsf{m}_n: We incorporate a mask vector mn∈{0,1}M\mathsf{m}_n \in \{0, 1\}^{M} to prevent LLM from generating invalid p\mathbf{p} violating the minimum antenna spacing constraint eq_constraint_pos. Specifically, mnm_n denotes the mask over Tmax\mathcal{T}_{\mathrm{max}} that identifies the valid candidate tokens for tnt_n, i.e., the feasible positions of the nn-th MA:

    mn∈{0,1}M,[mn]i={1,i∈[Mmin⁡,n,Mmax⁡,n]0,else,\begin{equation} \hspace{-0.3cm} \mathsf{m}_n \in \{0, 1\}^{M}, \big[\mathsf{m}_n \big]_i = \begin{cases} 1, i \in \big[M_{\min , n}, M_{\max , n}\big] \\ 0, \mathrm{else} \end{cases}, \tag*{(29)} \end{equation}

    where Mmin⁡,n=1M_{\min,n} = 1 for n=1n=1 and Mmin⁡,n=p n−1+dmin⁡DM_{\min,n} = p_{\,n-1} + \frac{d_{\min}}{D} for n>1n>1, while Mmax⁡,n=M−(N−n)dmin⁡DM_{\max,n} = M - (N-n)\frac{d_{\min}}{D}. After the masking process, the logits corresponding to all invalid tokens are set to −∞-\infty.

  • Policy π\pi: The policy π(⋅∣sn,mn)\pi(\cdot \mid \mathsf{s}_n, \mathsf{m}_n) depicts a probability distribution over the candidate set Tmax⁡\mathcal{T}_{\max} with the multimodal context and physical constraints given. In the training phase, the policy gradient algorithm iteratively refines π(an∣sn,mn)\pi(\mathsf{a}_n| \mathsf{s}_n, \mathsf{m}_n) , enabling the model to learn the mapping to generate high-sum-rate antenna positions.

  • Reward r\mathsf{r}: A scalar reward is computed after generating all NN actions. In data-limited scenarios, the ground-truth CSI of all users may not be available. Fortunately, the reward constructed from the high-quality extrapolated CSI provided by ExtLWM can still effectively guide policy optimization, and is defined as:

    r≜∑s=1S∑k=1Klog⁡2(1+∣h‾ext,k,sHwk,s∣2∑j≠k∣h‾ext,k,sHwj,s∣2+σc2),\begin{equation} \mathsf{r} \triangleq \sum_{s=1}^S \sum_{k=1}^{K}\log_2 \left(1 + \frac{|\mathbf{\overline{h}}_{\mathrm{ext},k,s}^H \mathbf{w}_{k, s} |^2}{\sum_{j \neq k} |\mathbf{\overline{h}}_{\mathrm{ext},k, s}^H \mathbf{w}_{j, s} |^2+\sigma_{\mathrm{c}}^2}\right), \tag*{(30)} \end{equation}

    where wk,s\mathbf{w}_{k,s} is the kk-th column of Ws\mathbf{W}_s. Various precoding schemes can be seamlessly incorporated to jointly construct the reward function, such as MMSE, weighted MMSE (WMMSE) [?, ?], and zero forcing (ZF). In this paper, we adopt MMSE precoder to achieve a favorable trade-off between sum-rate performance and computational efficiency. Specifically, assuming that the total transmit power is equally allocated across all subcarriers, the MMSE precoder can be expressed as

    Ws=PsH‾ext,s(PsH‾ext,sHH‾ext,s+σc2IK)−1∥H‾ext,s(PsH‾ext,sHH‾ext,s+σc2IK)−1∥F,\begin{equation} \mathbf{W}_s = \frac{\sqrt{P_{s}}\mathbf{\overline{H}}_{\mathrm{ext}, s} (P_{s} \mathbf{\overline{H}}_{\mathrm{ext}, s}^H \mathbf{\overline{H}}_{\mathrm{ext}, s}+\sigma_{\mathrm{c}}^2 \mathbf{I}_K)^{-1}}{\| \mathbf{\overline{H}}_{\mathrm{ext}, s} (P_{s} \mathbf{\overline{H}}_{\mathrm{ext}, s}^H \mathbf{\overline{H}}_{\mathrm{ext}, s}+\sigma_{\mathrm{c}}^2 \mathbf{I}_K)^{-1} \|_F}, \tag*{(31)} \end{equation}

    where Ps=PtxSP_{s}=\frac{P_{\mathrm{tx}}}{S} and H‾ext,s≜[h‾ext,1,s,...,h‾ext,K,s]\mathbf{\overline{H}}_{\mathrm{ext}, s} \triangleq [\mathbf{\overline{h}}_{\mathrm{ext},1, s},..., \mathbf{\overline{h}}_{\mathrm{ext},K, s}].

The constructed MDP provides a natural framework for deploying RL methods to optimize the sequential MA positioning policy. To efficiently preserve the basic semantic and LoS/NLoS classification capabilities of the LLM, we only train θCon\bm{\theta}_{\mathrm{Con}} while keeping the other modules frozen. Therefore, in stage-4, the original problem (?) can be reformulated as

max⁡θCon   Eτ∼π[r(τ)],s.t.   F1-score(K)≥ϵF1,\begin{align} \max _{\bm{\theta}_{\mathrm{Con}}} & \, \, \, \mathbb{E}_{\tau\sim\pi }\big[\mathsf{r}(\tau)\big], \\ \mathrm{ s.t. } & \,\,\, \mathrm{F1}\text{-}\mathrm{score}(\mathcal{K}) \geq \epsilon_{\mathrm{F1}}, \end{align}

where τ≜{(sn,an,mn)}n=1N\tau \triangleq \big\{(\mathsf{s}_n, \mathsf{a}_n, \mathsf{m}_n) \big\}_{n=1}^{N} denotes the roll-out trajectory. r(⋅)\mathsf{r}(\cdot) denotes the trajectory reward specified in (30). Note that the constraints in eq_constraint_pos and eq_constraint_p are guaranteed by the action space and masking mechanism, respectively, while eq_constraint_power is satisfied by the normalized precoder and eq_constraint_nmse is addressed in stage-2. Hence, only eq_constraint_f1, i.e., eq_constraint_cl, needs to be explicitly handled. For each training epoch, we denote the ii-th trajectory as τ(i)\tau^{(i)}, and define its nn-th transition tuple as bn(i)≜(sn(i),an(i),mn(i),π(an(i)∣sn(i),mn(i)),sn+1(i),r(i))\mathsf{b}_n^{(i)} \triangleq \big(\mathsf{s}_n^{(i)}, \mathsf{a}_n^{(i)}, \mathsf{m}_n^{(i)}, \pi(\mathsf{a}_n^{(i)}| \mathsf{s}_n^{(i)}, \mathsf{m}_n^{(i)}), \mathsf{s}_{n+1}^{(i)}, \mathsf{r}^{(i)} \big), which is sequentially stored into buffer B\mathcal{B}.

To fully exploit the potential of the LLM for sequential optimization, we adopt the GRPO-based RL algorithm, an LLM-specific and training-efficient method that does not require an additional critic network [?, ?]. Specifically, GRPO directly performs advantage estimation based on group-relative rewards, thereby effectively reducing the variance of advantage estimation and alleviating the credit-assignment difficulty in sequential decision making. The advantage value of the ii-th trajectory is computed based on the GG trajectories rolled out in the current training epoch:

A(i)=r(i)−mean⁡({r(j)}j=1G)std⁡({r(j)}j=1G).\begin{equation} A^{(i)}= \frac{\mathsf{r}^{(i)} - \operatorname{mean}( \{\mathsf{r}^{(j)}\}_{j=1}^G )}{\operatorname{std}( \{\mathsf{r}^{(j)}\}_{j=1}^G )}. \tag*{(32)} \end{equation}

where r(i)r^{(i)} is the terminal reward of the ii-th trajectory, which is inherently delayed and sparse since the marginal contribution of each MA cannot be explicitly measured. The GRPO loss LrlL_{\mathrm{rl}} is then formulated using the clipped surrogate objective to constrain the update magnitude:

Lrl=EBsub[min⁡(ρn(i)A(i),clip(ρn(i),1−ϵc,1+ϵc)A(i))],\begin{equation} L_{\mathrm{rl}} = \mathbb{E}_{ \mathcal{B}_{\mathrm{sub}}} \left[ \min \left( \rho_n^{(i)} A^{(i)}, \mathrm{clip}(\rho_n^{(i)}, 1-\epsilon_{\mathrm{c}}, 1+\epsilon_{\mathrm{c}}) A^{(i)} \right) \right], \tag*{(33)} \end{equation}

where Bsub\mathcal{B}_{\mathrm{sub}} denotes the mini-batch subsets randomly drawn from B\mathcal{B}, ρn(i)=πnew(an(i)∣sn(i),mn(i))π(an(i)∣sn(i),mn(i))\rho_n^{(i)}= \frac{\pi_{\mathrm{new}}(\mathsf{a}_{n}^{(i)} \mid \mathsf{s}_{n}^{(i)}, \mathsf{m}_{n}^{(i)})}{\pi (\mathsf{a}_{n}^{(i)} \mid \mathsf{s}_{n}^{(i)}, \mathsf{m}_{n}^{(i)})} represents the probability ratio between the updated and old policies.

Meanwhile, to improve the training stability of GRPO, we introduce entropy regularization to better balance exploration and exploitation. The exploration loss can be expressed as

Lexp=EBsub[H(πnew(⋅ ∣sn(i),mn(i)))],\begin{equation} \hspace{-0.05cm} L_{\mathrm{exp}} = \mathbb{E}_{\mathcal{B}_{\mathrm{sub}}} \Big[ \mathcal{H}\big( \pi_{\mathrm{new}}( \cdot ~| \mathsf{s}_{n}^{(i)}, \mathsf{m}_{n}^{(i)}) \big) \Big], \tag*{(34)} \end{equation}

where H(⋅)\mathcal{H}(\cdot) denotes the policy entropy.

Moreover, to preserve the LoS/NLoS classification performance in eq_constraint_cl during sum-rate maximization, we incorporate experience replay (ER), a widely used strategy in continual learning (CL) [?, ?], to mitigate catastrophic forgetting. Specifically, ER can be naturally integrated with the replay buffer B\mathcal{B} of the underlying RL algorithm by additionally storing a fixed set of labeled samples (δk,Hinc,k)(\delta_k,\mathbf{H}_{\mathrm{inc},k}), which provides stable supervision throughout training and regularizes policy updates. Since only a small subset Kcl⊆Kid\mathcal{K}_{\mathrm{cl}}\subseteq\mathcal{K}_{\mathrm{id}} is required to effectively preserve the classification capability, the additional memory overhead introduced by ER remains limited. Finally, within each epoch, the loss function for sum-rate maximization can be expressed as

Lmax(θCon)=−Lrl−λexpLexp+λclLcl,\begin{equation} L_{\mathrm{max}}(\bm{\theta}_{\mathrm{Con}}) = -L_{\mathrm{rl}} - \lambda_{\mathrm{exp}} L_{\mathrm{exp}} + \lambda_{\mathrm{cl}} L_{\mathrm{cl}}, \tag*{(35)} \end{equation}

where Lcl=Ek∈Kcl[−[δklog⁡pk1+(1−δk)log⁡pk0]]L_{\mathrm{cl}}= \mathbb{E}_{k \in \mathcal{K}_{\mathrm{cl}}} \big[ -[\delta_k \log p_k^{1}+(1-\delta_k) \log p_k^{0}] \big] is defined similarly to (27), λexp\lambda_{\mathrm{exp}} and λcl\lambda_{\mathrm{cl}} are hyperparameters controlling the priority of exploration and ER strategy, respectively.

Summary

We obtain ExtLWM through pre-training in stage-1. For a specific downstream MA system, stages-2-4 progressively endow the multimodal model with multi-task capabilities under data-limited conditions to solve problem (?). Notably, after stage-2, ExtLWM serves as an environment-specialized model to support a self-contained framework, providing both high-quality feature embeddings and extrapolated CSI to support the token-level optimization in stages-3/4 and the construction of the reward function (including precoder) in stage-4, respectively.

We now analyze the computational complexity of the proposed multimodal model for generating NN MA positions. Specifically, the incomplete CSI of the KK users is first processed by ExtLWM. The CSI of each user is successively passed through the embedding layer and LencL_{\mathrm{enc}} encoder blocks, resulting in a computational complexity of O(KMSDLWM+\mathcal{O}\big(KMSD_{\mathrm{LWM}}+ KLenc(M2DLWM+MDLWM2))KL_{\mathrm{enc}}(M^2D_{\mathrm{LWM}}+ MD_{\mathrm{LWM}}^2)\big). The extracted CSI embeddings are then fed into the connector, where DqD_\mathrm{q} learnable query tokens and LqL_{\mathrm{q}} query blocks are employed to project the CSI features into the embedding space of the LLM, leading to a computational complexity of O(KLq(Dq2DLWM+MDq\mathcal{O}\big(KL_{\mathrm{q}}(D_\mathrm{q}^2D_{\mathrm{LWM}}+MD_\mathrm{q} DLWM+(M+Dq)DLWM2)+KDqDLWMDLLM)D_{\mathrm{LWM}}+(M+D_\mathrm{q})D_{\mathrm{LWM}}^2)+KD_\mathrm{q}D_{\mathrm{LWM}}D_{\mathrm{LLM}}\big). Subsequently, GPT-2 autoregressively generates the NN MA position tokens based on the resulting CSI embeddings, with a computational complexity of O(Ldec(Dx2DLLM+DxDLLM2)+Ldec\mathcal{O}\big(L_{\mathrm{dec}}(D_\mathrm{x}^2D_{\mathrm{LLM}}+ D_\mathrm{x}D_{\mathrm{LLM}}^2)+L_{\mathrm{dec}} (NDLLM2+(NDx+N2)DLLM)+NMDLLM)(ND_{\mathrm{LLM}}^2+(ND_\mathrm{x}+N^2) D_{\mathrm{LLM}})+NMD_{\mathrm{LLM}}\big), where Dx=KDqD_\mathrm{x}=KD_\mathrm{q}. Therefore, by neglecting lower-order terms and constant factors, the dominant computational complexity of the proposed model can be expressed as O(KLenc(M2DLWM+\mathcal{O}\big(KL_{\mathrm{enc}}(M^2D_{\mathrm{LWM}}+ MDLWM2)+KLqMDLWM2+Ldec(NDLLM2+N2DLLM)+NMDLLM)MD_{\mathrm{LWM}}^2)+KL_{\mathrm{q}}MD_{\mathrm{LWM}}^2+L_{\mathrm{dec}}(ND_{\mathrm{LLM}}^2+N^2D_{\mathrm{LLM}})+NMD_{\mathrm{LLM}}\big).

Numerical Results

In this section, we provide comprehensive simulation results and performance analysis. We first evaluate the pre-training and CSI extrapolation performance of ExtLWM. Next, we evaluate the LoS/NLoS identification capability of the proposed network. Finally, we evaluate the LLM-based MA positioning scheme in terms of sum-rate performance. All experiments were conducted on a hardware platform equipped with two NVIDIA GeForce RTX 4090 GPUs.

Pre-Training Performance

Consistent with LWM-1.1 [?, ?], the pre-training dataset comprises 120 scenarios from DeepMIMO-V3 [?, ?] with fc=3.5f_{\mathrm{c}}=3.5 GHz and Δf=30\Delta f=30 kHz, while the ExtLWM (2.4M parameters) hyperparameters are set to αp=106\alpha_{\mathrm{p}}=10^6, [MASK]=0.1[\mathrm{MASK}]=0.1, DLWM=128D_{\mathrm{LWM}}=128, Lenc=12L_{\mathrm{enc}}=12, and Henc=8H_{\mathrm{enc}}=8. Among all users, 75%75\% of the CSI samples—corresponding to ∣Kpre∣=402409|\mathcal{K}_{\mathrm{pre}}|=402409 —are used for training. In this paper, we train three WFMs to accommodate downstream MA systems with D=λc8D=\frac{\lambda_{\mathrm{c}}}{8}, as summarized in Table 1. In terms of pre-training configuration, while retaining most settings from LWM-1.1, we set epoch Epre=800E_{\mathrm{pre}}=800 to more effectively capture the latent structure of CSI. The peak GPU memory consumption during pre-training is approximately 23.723.7GB.

Model NamePre-training candidate position ∣Ik∣| \mathcal{I}_k |SSCost
ExtLWM-sub1640 80 120 160 200 240 280 3201698h
ExtLWM-sub3230 60 90 120 1603278h
ExtLWM-sub6420 40 806476h

Table 1. Training configurations for three WFMs.

Figure 3

Figure 3. Training and validation losses for WFMs.

Figure 4

Figure 4. ExtLWM’s performance under various conditions.

Figure 3 presents the loss trajectories of the entire training phase of the three foundation models, demonstrating the effectiveness and stability of the training process. In this paper, we adopt “ExtLWM-sub16” as the foundation model for all subsequent experiments, referred to simply as ExtLWM3.

Next, we separately fine-tune ExtLWM on 14 sets of out-of-distribution CSI samples, where 50% of the samples in each unseen MA dataset are used for training. Figure 4 presents the validation NMSE under M=180M=180 and ηmsk=80%\eta_{\mathrm{msk}}=80\%. It can be observed that all curves rapidly decrease below 0.050.05 within 500 epochs, demonstrating the generalization capability of ExtLWM across diverse unseen channel distributions from multiple perspectives. Specifically, 1) We consider two DeepMIMO scenarios that are not included in the pre-training dataset, namely “O1-blockage” and “O2-dynamic”, to demonstrate the cross-scenario generalization capability of ExtLWM. 2) We evaluate ExtLWM using CSI generated by QuaDRiGa [?, ?] and Sionna [?, ?], thereby assessing its generalization capability across different channel-generation platforms beyond DeepMIMO. 3) We evaluate channel samples generated at fc=60f_{\mathrm{c}}=60 GHz and fc=140f_{\mathrm{c}}=140 GHz to investigate the cross-frequency generalization capability of ExtLWM. 4) We consider Δf=15\Delta f=15 kHz and Δf=60\Delta f=60 kHz to evaluate its generalization across different subcarrier spacings. 5) We further generate CSI using different candidate-position spacings, i.e., D=λc6D=\frac{\lambda_{\mathrm{c}}}{6} and D=λc10D=\frac{\lambda_{\mathrm{c}}}{10}, to validate the cross-array-spacing generalization capability of ExtLWM beyond the pre-training grid with λc8\frac{\lambda_{\mathrm{c}}}{8}. 6) We evaluate channel configurations with S=8S=8 and S=4S=4, thereby assessing the generalization capability of ExtLWM across different numbers of subcarriers. 7) Finally, we evaluate ExtLWM using uniform planar array (UPA) channels with M=20×9M=20\times9 and M=10×18M=10\times18, demonstrating that ExtLWM can capture spatial and propagation structures beyond a specific ULA geometry.

CSI Extrapolation Performance

In this subsection, we demonstrate the CSI extrapolation performance of fine-tuned ExtLWM in the specific downstream scenario, which is constructed on the data of 1250 users lying between row-1400 and row-1500 in “Boston5G” (M=180M=180, ηmsk=80%\eta_{\mathrm{msk}}=80\%, fc=28f_{\mathrm{c}} = 28 GHz, S=16S=16 and D=λc8D=\frac{\lambda_{\mathrm{c}}}{8}). To simulate the sample-limited environment, 50% of CSI samples, i.e., ∣Kext∣=625|\mathcal{K}_{\mathrm{ext}}|=625, are used for fine-tuning ExtLWM. The batch size, training epochs, and learning rate are set to 128, Eext=2×104E_{\mathrm{ext}}=2 \times 10^{4}, and ηext=5×10−5\eta_{\mathrm{ext}}=5 \times 10^{-5}, respectively. The peak GPU memory consumption and training time for stage-2 are approximately 8.38.3GB and 3.53.5h, respectively.

To provide a comprehensive performance comparison, we test the following benchmarks:

  • LWM (60 scenarios): The number of pre-training scenarios for ExtLWM is reduced from 120 to 60.

  • LWM (1 scenario): A dedicated extrapolation model is first trained on the “ASU Campus” scenario and then fully fine-tuned on the target scenario.

  • LWM-1.1 [?, ?]: For the case of S=16S=16, the original patch shape of 2×4×42\times4\times4 is reconfigured to 2×1×162\times1\times16, enabling LWM-1.1 to be directly applied to CSI extrapolation.

  • LWM-scratch: A transformer with the same architecture as LWM is trained from scratch.

  • Bi-LSTM: We replace the encoder blocks of LWM with a Bi-LSTM architecture having a comparable number of parameters and train the resulting model from scratch.

  • BERT [?, ?]: Following [?, ?], we replace the embedding layer and MMH of the BERT with the corresponding ExtLWM modules, and fine-tune the last four encoder blocks.

Figure 5

Figure 5. Validation loss for CSI extrapolation.

Figure 5 shows the validation loss curves of different models. It can be observed that both ExtLWM variants start from relatively low NMSE levels and then decrease rapidly during fine-tuning, indicating that MA-specific pre-training provides effective channel representations for CSI extrapolation. In comparison, LWM-1.1 remains in the region around an NMSE of 1 for a noticeably shorter period than the model trained from scratch, suggesting that its pre-training still preserves certain generic representations of geometric wireless channels. Finally, the BERT-based model consistently exhibits a higher validation loss, indicating a mismatch between its pre-trained representations and wireless channel characteristics, which cannot be sufficiently compensated for using only a limited number of downstream CSI samples.

Figure 6

Figure 6. Impact of CSI incompleteness ratio.

In Figure 6, we evaluate the CSI extrapolation performance under different CSI incompleteness ratios ηmsk\eta_{\mathrm{msk}}. The results show that ExtLWM consistently achieves superior performance. When the number of pre-training scenarios is increased from 60 to 120, ExtLWM achieves an average CSI extrapolation gain of approximately 2.152.15 dB, indicating that pre-training over more diverse scenarios facilitates the learning of more transferable spatial correlations and multipath propagation characteristics. Meanwhile, compared with the model trained from scratch, the standard ExtLWM achieves NMSE gains of more than 4.54.5 dB and 2.82.8 dB at CSI incompleteness ratios of 60%60\% and 95%95\%, respectively. Notably, even when pre-trained using only 60 scenarios, ExtLWM still outperforms the state-of-the-art LWM-1.1, further demonstrating the necessity of MA-specific WFM pre-training. Specifically, LWM-1.1 is primarily designed to learn spatial–frequency correlations from regular FPA channels and thus has limited capability in characterizing the distinctive spatial distribution patterns of MA channels. Moreover, LWM-1.1 adopts a masking ratio of only 40%40\% during pre-training, resulting in insufficient exposure to highly incomplete CSI patterns.

Figure 7

Figure 7. Impact of training data volume.

Figure 7 illustrates the impact of data volume for training, indicated by training ratio ∣Kext∣/1250|\mathcal{K}_{\mathrm{ext}}|/1250. Results show that ExtLWM exhibits increasingly pronounced performance gains over the model trained from scratch as the amount of training data decreases, achieving NMSE improvements of 8.9 dB, 9.1 dB, and 10 dB at training ratios of 20%, 10%, and 5%, respectively. Moreover, dedicated models typically capture only the propagation characteristics and spatial correlations of specific scenarios, making it difficult to effectively adapt to new channel distributions during cross-scenario transfer. When the amount of training data in the new scenario is further reduced, their performance also degrades more significantly. These results demonstrate that WFMs can achieve better CSI extrapolation performance in data-limited scenarios.

LoS/NLoS Identification Performance

In this subsection, we validate LoS/NLoS identification performance. The experiments are conducted using the data associated with “BS3” and 1250 users from row-500 in “O1-blockage” scenario from DeepMIMO-V3, with the system setting given by M=180M=180, ηmsk=90%\eta_{\mathrm{msk}}=90\%, fc=3.5f_{\mathrm{c}}=3.5 GHz, S=16S=16, D=λc8D=\frac{\lambda_{\mathrm{c}}}{8}. To simulate the sample-limited and label-scarce environment, 50% of the users with complete CSI (∣Kext∣=625|\mathcal{K}_{\mathrm{ext}}|=625) are used to fine-tune ExtLWM, while 1% of users with known labels (∣Kid∣=12|\mathcal{K}_{\mathrm{id}}| = 12) are used to train the connector. The LLM adopts “GPT-2-small” (124M parameters) from the GPT-2 family, with DLLM=768D_{\mathrm{LLM}}=768, Ldec=12L_{\mathrm{dec}}=12, and Hdec=12H_{\mathrm{dec}}=12. The connector is configured with Dq=3D_{\mathrm{q}}=3 and Lq=2L_{\mathrm{q}}=2 (0.4M parameters) [?, ?]. The training hyperparameters are set as Eid=200E_{\mathrm{id}}=200 and ηid=1×10−3\eta_{\mathrm{id}}=1\times10^{-3}. The peak GPU memory consumption and training time for stage-3 are approximately 3.53.5GB and 55min, respectively.

To compare with the proposed multi-stage training strategy, we test other training schemes as follows:

  • CLS embedding [?, ?]: The LWM architecture supports the output of a single CLS embedding that efficiently aggregates information across the entire input. We replace the CSI embedding with the CLS embedding output by the fine-tuned ExtLWM for the identification task.

  • Joint fine-tuning [?, ?]: Based on the pre-trained ExtLWM, we jointly fine-tune all parameters, i.e., θWFM\bm{\theta}_{\mathrm{WFM}} and θCon\bm{\theta}_{\mathrm{Con}}, to optimize training loss of both extrapolation in (25) and identification in (27), i.e., αextLext+αidLid\alpha_{\mathrm{ext}} L_{\mathrm{ext}} + \alpha_{\mathrm{id}} L_{\mathrm{id}}.

  • Direct fine-tuning: We directly apply the pre-trained ExtLWM to the identification task, i.e., without first going through the extrapolation task to fine-tune θWFM\bm{\theta}_{\mathrm{WFM}}.

  • Raw CSI: We bypass ExtLWM and use only the raw incomplete CSI for the identification task.

Figure 8, panel a

(a) Lab.-CSI rat.: 1%, ∣Kid∣=12|\mathcal{K}_{\mathrm{id}}|=12

Figure 8, panel b

(b) Lab.-CSI rat.: 0.5%, ∣Kid∣=6|\mathcal{K}_{\mathrm{id}}|=6

Figure 8. Performance vs. labeled CSI ratio.

In Figure 8, we plot both the F1-score of LoS/NLoS identification (on the y-axis) and the NMSE of CSI extrapolation (on the x-axis) obtained by the aforementioned training schemes using different ratios of LoS/NLoS-labeled training samples. Several key observations are elaborated in the sequel. 1) Firstly, even with very limited labeled CSI, i.e., ∣Kid∣=6|\mathcal{K}_{\mathrm{id}}| = 6, our multi-stage training delivers F1-score of 1.00 and 0.998, respectively, at high CSI incompleteness ratios of ηmsk=90%\eta_{\mathrm{msk}}=90\% and ηmsk=95%\eta_{\mathrm{msk}}=95\%, underscoring its superior robustness in extremely label-scarce regimes. 2) Secondly, as shown in Figure 8a, replacing CSI embeddings with CLS leads to F1-score degradations of 12.7% and 11%. This gap widens to 22.7% and 22.0% for fewer labels in Figure 8b. This is due to the fact that incomplete CSI renders CLS representations susceptible to capturing missing/noisy patterns over salient physical features, impairing its classification capability. 3) Finally, the Pareto frontier of joint fine-tuning is depicted in Figure 8, which is dominated by our proposed multi-stage training paradigm. This sub-optimality arises because acute label scarcity intensifies inter-task gradient conflict and exacerbates task-dominance imbalances.

Figure 9, panel a

(a) Raw CSI

Figure 9, panel b

(b) w/ CSI extrap.

Figure 9, panel c

(c) w/o CSI extrap.

Figure 9. Impact of CSI extrapolation on embedding.

Figure 9 provides tt-SNE projections of CSI embeddings comparing the post-extrapolated ExtLWM (Figure 9b) with the pre-trained counterpart (Figure 9c) to visualize the impact of extrapolation task on LoS/NLoS identification. The results reveal that integrating CSI extrapolation stimulates the model to distill latent features of MA channels more effectively and to yield more discriminative CSI representations that substantially bolster the performance of downstream tasks.

Sum-Rate Maximization Performance

In this subsection, we proceed to examine the effectiveness of RL training in resolving the sum-rate maximization task. In addition to the system settings specified in § 5.3, we set dmin=λc2d_\mathrm{min}=\frac{\lambda_{\mathrm{c}}}{2}, N=8N=8, Ptx=20P_{\mathrm{tx}}=20dBm, σc2=−70\sigma_{\mathrm{c}}^2=-70dBm, and K=6K=6. For RL algorithm, we set epoch Emax=500E_{\mathrm{max}}=500, learning rate ηmax=3×10−4\eta_{\mathrm{max}}=3\times10^{-4}, ϵc=0.2\epsilon_{\mathrm{c}}=0.2, G=20G=20, and ∣Bsub∣=GN4|\mathcal{B}_{\mathrm{sub}}|= \frac{GN}{4}, while λexp\lambda_{\mathrm{exp}} is annealed from 0.1 to 0.02. To implement ER, we set λcl=5×10−3\lambda_{\mathrm{cl}}=5\times10^{-3} and Kcl=Kid\mathcal{K}_{\mathrm{cl}}=\mathcal{K}_{\mathrm{id}}. The average execution times of the proposed framework for CSI extrapolation, LoS/NLoS identification, and sum-rate maximization are approximately 0.0160.016s, 0.0250.025s, and 0.0520.052s, respectively. The peak GPU memory consumption and training time for stage-4 are approximately 10.310.3GB and 2.32.3h, respectively.

Figure 10

Figure 10. Performance of ER method.

In Figure 10, we evaluate the efficacy of ER in maintaining LoS/NLoS identification precision using the validation set in § 5.3. Results show that ER sustains an F1-score of 1.00 during sum-rate optimization. In contrast, omitting ER leads to catastrophic forgetting at epoch 49, with model losing its previously held identification accuracy, which never recovers. Notably, even with ∣Kcl∣|\mathcal{K}_{\mathrm{cl}}| reduced from 12 to 2 (“2 samp” in legend), ER still preserves identification capabilities perfectly, whereas skipping the preliminary fine-tuning on the identification task (denoted as “JFT, 2 samp.” in the legend) results in an F1-score of only 0.824. These results demonstrate the effectiveness of the ER-based strategy and its strong training-sample efficiency. Moreover, we compare the joint fine-tuning scheme (denoted as “JFT”), which skips stage-3 and directly optimizes LoS/NLoS identification during sum-rate maximization. The results show that its sum-rate performance degrades noticeably, while the maximum F1-score reaches only 0.979. Together with the results in Fig. 8, these experiments further demonstrate the effectiveness of the proposed multi-stage training strategy. Finally, we compare ER with elastic weight consolidation (EWC) algorithm [?, ?], which mitigates forgetting by constraining parameters important to previous tasks. Experimental results show that such constraints inhibit sum-rate gains, validating the superior performance trade-off achieved by ER.

Figure 11

Figure 11. Performance of diverse LLMs.

Figure 11 validates the effectiveness and generalizability of the proposed multi-stage training paradigm. We first deploy the proposed framework on GPT-2-large (774M) and GPT-2-xl (1.5B), which adopt different configurations of DLLMD_{\mathrm{LLM}}, HdecH_{\mathrm{dec}}, and LdecL_{\mathrm{dec}}. The results demonstrate that the proposed training strategy can achieve satisfactory performance across LLMs with similar architectures but different model scales. We then further consider DialoGPT (124M) [?, ?], a dialogue-oriented variant fine-tuned from GPT-2, and show that the proposed training framework can also be directly transferred to task-specialized LLMs. Finally, we further consider Qwen3 (0.6B) [?, ?] and Gemma3 (270M) [?, ?], demonstrating that the proposed modality alignment mechanism (19), modified causal mask (20), and corresponding training strategy remain applicable to LLMs with different architectures4.

Figure 12

Figure 12. Performance of RL method.

PrecoderWMMSEMMSEZF
Execution time (s)0.4870.0520.052
Sum-rate (bps/Hz)9.6846.1801.443

Table 2. Overall comparison of different precoders.

In Figure 12, we validate the effectiveness of the proposed RL method from the following aspects: 1) The designed reward function can effectively guide the improvement of the actual sum-rate, further validating that ExtLWM can generate high-quality extrapolated CSI and feature embeddings in a self-contained manner to effectively guide MA positioning. 2) Removing the entropy loss term in (34) results in a 14.45% sum-rate degradation, demonstrating the importance of balancing exploration and exploitation during training. 3) Compared with PPO [?, ?] and the classical policy gradient (PG) method, i.e., REINFORCE [?, ?], GRPO achieves clearly better performance, mainly benefiting from its group-relative advantage estimation, which can effectively alleviate the credit-assignment difficulty. Meanwhile, replacing numerical tokens with random tokens leads to an 8.28% sum-rate degradation. These results demonstrate that the proposed design can better exploit the potential of LLMs for sequential MA positioning. 4) Replacing the frozen “GPT-2-small” with an 8.45M-parameter fully trained transformer instead results in a 6.8% sum-rate degradation. In contrast, the proposed method only trains the 0.4M parameters of the connector, demonstrating its ability to effectively exploit the trainable-parameter efficiency of LLMs. 5) Removing the pre-trained weights of ExtLWM and training it from scratch only on the downstream scenario leads to a 10.68% sum-rate degradation, a 3.84 dB degradation in CSI extrapolation performance, and a reduction of the F1-score to 96.6%. These results demonstrate that integrating the WFM with the LLM can better support data-limited MA scenarios. 6) Replacing the WTE of LLM with NN MM-class embeddings transforms MA positioning into a single-shot parallel prediction based solely on Zcsi\mathbf{Z}_{\mathrm{csi}}, resulting in an 18.03% degradation in sum-rate performance. This demonstrates that sequential generation increases the serial computational depth of the LLM, thereby enabling it to better handle problems with sequential dependencies under a fixed network depth.

In Table 2, we compare the sum-rate performance and execution time achieved under different precoding schemes. The results show that different precoders can effectively serve as part of the reward function to guide the RL agent. Specifically, WMMSE achieves an approximately 56.7% sum-rate gain over MMSE, but requires approximately 9.37 times the execution time of MMSE. This substantially higher computational overhead mainly arises from the multiple alternating iterations over auxiliary variables required by WMMSE, together with the one-dimensional search performed in each beamforming update to determine the Lagrange multiplier satisfying the transmit power constraint. In contrast, MMSE achieves an approximately 328.3% sum-rate improvement over ZF, demonstrating its effectiveness in suppressing multiuser interference under correlated channel conditions. Therefore, considering the overall trade-off between achievable sum-rate and computational complexity, we choose MMSE precoder in the sum-rate maximization task.

Next, we compare several MA positioning algorithms for sum-rate maximization, as summarized below. For fairness, all benchmark schemes adopt the MMSE precoder (31).

  • BnB [?, ?] performs a systematic tree search over the discrete antenna position space.

  • PBR [?, ?] relaxes binary variables to continuous ones and enforces discreteness through penalty terms.

  • Greedy sequentially selects the optimal MA positions based on locally maximum sum-rate.

  • FPA assumes a fixed antenna spacing of λc2\frac{\lambda_{\mathrm{c}}}{2}.

Figure 13

Figure 13. Impact of PtxP_{\mathrm{tx}} on sum-rate.

Figure 13 illustrates the sum-rate performance achieved by different methods. The results show that sequential optimization can achieve satisfactory performance for MA positioning. Specifically, at Ptx=10P_{\mathrm{tx}}=10 dBm and Ptx=26P_{\mathrm{tx}}=26 dBm, the proposed scheme achieves 90.17%90.17\% and 97.05%97.05\% of the sum rate obtained by the BnB method, respectively. This indicates that sequential optimization indeed incurs a certain degree of suboptimality compared with directly searching the original combinatorial space. On the other hand, it outperforms the PBR method, mainly because the relaxation and penalty-based approximation may introduce a performance gap from the original combinatorial optimization problem.

Figure 14

Figure 14. Impact of NN on sum-rate.

Next, we compare overall performance of different algorithms under different NN, with the results presented in Figure 14 and Table 3, respectively. The results show that, by autoregressively generating MA positions based on the learned semantic representations, the proposed method achieves approximately 96.6%96.6\% of the upper-bound sum-rate performance. In contrast, the search space of BnB grows combinatorially with NN, leading to a rapidly increasing execution time and limiting its applicability to large-scale MA configurations. Compared with PBR, the proposed method achieves an average sum-rate gain of approximately 8.0%8.0\%, while avoiding the substantial computational overhead caused by repeated alternating optimization. Although Greedy achieves the lowest execution time, its myopic position selection strategy may lead to accumulated suboptimal decisions, resulting in noticeable performance degradation as NN increases. Overall, these results demonstrate that the proposed framework can achieve a favorable trade-off between sum-rate performance and computational efficiency.

NNProposedBnBPBRGreedy
60.043s553.659s35.349s0.016s
80.052s0.443h41.553s0.017s
120.066s4.193h55.382s0.018s
160.083s58.031h71.264s0.019s
200.095s–82.720s0.021s
240.113s–93.366s0.024s
280.129s–106.527s0.025s
320.144s–111.746s0.032s

Table 3. Impact of NN on execution time.

MMExecution time (s)Sum-rate (bps/Hz)
2-7ProposedPBRGreedyProposedPBRGreedy
1000.08947.9010.0188.8858.6628.120
1400.09267.8400.0199.6318.9778.300
1800.09582.7200.02110.4519.2818.426
2200.099106.8070.02810.9249.4208.572
2600.104139.7650.03011.1389.6098.641

Table 4. Impact of MM on execution time and sum-rate.

Finally, Table 4 presents the overall performance of different algorithms under different MM, where ηmsk=80%\eta_{\text{msk}} = 80\% and N=20N = 20. The results show that, with scenario-specific fine-tuning of ExtLWM, the proposed framework can effectively learn MA channel characteristics under different candidate-position spaces. As MM increases, the growth in overall inference latency becomes noticeably milder than that observed with increasing NN, because ExtLWM, rather than GPT-2, becomes the dominant contributor to the computational overhead. In addition, the proposed multi-stage training algorithm maintains a stable average CSI extrapolation NMSE of approximately −24.6-24.6dB, while ER consistently preserves the LoS/NLoS identification performance with an F1-score of 1.01.0. These results further demonstrate the robustness and scalability of the proposed algorithm.

Conclusions

This paper proposes a unified ExtLWM-Connector-GPT architecture for multi-task processing in MA systems. Simulation results show that ExtLWM outperforms dedicated models and models trained from scratch in downstream CSI extrapolation. Meanwhile, the proposed approach effectively activates the sequential optimization capability of the LLM and achieves a favorable performance–complexity trade-off compared with conventional alternatives. In future work, we will investigate the integration of the proposed framework into multimodal semantic communication systems to extend its applicability. Moreover, we will explore partially unfreezing the LLM while preserving existing task performance to further investigate its potential performance gains.

Figure 15

Figure 15.

Figure 16

Figure 16.

Figure 17

Figure 17.

Figure 18

Figure 18.

Figure 19

Figure 19.

Figure 20

Figure 20.

References

No references were extracted for this paper.

Paper details

vlnc:W_bpxgwhjm
Version
v1 · latest
Published
Fresh — added to the Hub within the last 30 days
Added
License
CC BY 4.0
Usage

views · 5

downloads · 0

A view counts once per browser per day, a download each time the PDF is requested.

  • Engineering

Not recorded for this paper

View statement
Looking for related papers…
Contents