betaFeedback

WorksComputer Science

TSFM Meets LLM: Context as Covariates

Abstract

Textual information is ubiquitous in real-world time series datasets, appearing as news, reports, and annotations that hold key predictive signals not deducible from numerical history alone. Large Language Models (LLMs) interpret text but lack numerical precision, while Time Series Foundation Models (TSFMs) are quantitatively accurate but cannot reason about the effects of text. Moreover, the lack of large-scale paired multimodal datasets limits progress in bridging this gap. In this work, we address this challenge by providing a scalable approach for generating realistic multimodal datasets by annotating real-world time series with LLMs. Building on this, we propose Context as Covariates (CoCo), a forecasting framework in which an LLM distills its textual reasoning into numerically grounded forecast and confidence covariates that guide a TSFM backbone, thereby exploiting the strengths of both LLMs and TSFMs. We align the two models in stages, first fine-tuning the LLM via Group Relative Policy Optimization (GRPO) with a reward tied to covariate-induced improvement in forecasting performance, and then fine-tuning the TSFM on covariates generated by the aligned LLM. Our experiments show that models trained on our synthetic multimodal corpus generalize to real benchmarks spanning finance, economics, weather, traffic, and security. Notably, CoCo with Qwen3-4B-Instruct-2507 as the LLM and Chronos2 as the TSFM achieves up to 12% MAE improvement over state-of-the-art TSFMs and LLM forecasters, such as GPT-4o. Code, checkpoints, and the evaluation datasets are publicly available.

Introduction

“Forecast Apple’s stock price if revenue from iPhone sales is 15% higher than the target”

“Forecast crude oil prices given the recent tariffs on the OPEC countries”

These contextual forecasting examples across domains show that the textual context is deeply intertwined with real-world time series and can provide signals about future changes that are not captured by past numerical values alone (check Figure 1). For example, in the finance domain, analyst notes, government policies, and even social media sentiments influence market dynamics, and are commonly used to predict stock price movements [1, 2, 3, 4]. Similarly, in consumer markets, online reviews and social media discussions are used to forecast future demand [5, 6].

Figure 1. Effective use of context is essential for accurate time series forecasting. In crude oil price forecasting, textual context (e.g., recession fears on March 9) aligns with sudden price shifts. Our approach, Context as Covariates (CoCo), leverages this context to produce accurate forecasts, while unimodal models such as Chronos2 fail to capture context-induced trends.

Despite the availability of textual information, the state-of-the-art (SOTA) univariate (TimesFM [7]) and multivariate (Chronos2 [8]) time series foundation models (TSFMs) operate solely on unimodal numerical inputs. Figure 5 shows that incorporating text embeddings into SOTA multivariate forecasting models does not necessarily yield multimodal gains, even when the text clearly contains useful information. Given the substantive progress in the understanding and reasoning capabilities of large language models (LLMs) [9, 10], we address the following key question in this work: How can we leverage LLMs to process textual annotations or context and improve the performance of TSFMs? Current efforts to build such context-aware TSFMs struggle with two major barriers:

  • Insufficient Multi-Domain Multimodal Training Data: Learning complex and lagged relationships between text and future numerical values requires large volumes of aligned multimodal data from different domains. However, existing multi-domain datasets such as TimeMMD [11] and Context-Is-Key [12] contain only a few thousand aligned samples. Large-scale datasets [13] are heavily restricted to a single sub-domain, limiting cross-domain generalization.

  • LLM-Centric Multimodal Forecasting approaches neglect the predictive power of TSFMs: Numerous prior works [14, 15, 16, 17, 18, 19, 20] serialize time series as tokens and rely primarily on LLM prediction. While this enables the use of textual context, it fails to leverage the forecasting inductive biases and uncertainty modeling strengths of dedicated TSFMs.

Motivated by these gaps, we make three contributions.

  • Scalable Synthetic Multimodal Data. We introduce a pipeline that annotates real-world time series with LLM-generated text, producing large-scale, text-aligned datasets with varied lagged correlations. We empirically show that augmenting scarce and often domain-restricted real multimodal data with synthetic annotations yields strong zero-shot forecasting performance.

  • CoCo Architecture and Staged Fine-tuning. We propose CoCo (Context as Covariates), where an LLM distills text into forecast and confidence covariates that guide a Chronos2 backbone. The LLM is fine-tuned via Group Relative Policy Optimization (GRPO) using a reward tied to the forecasting performance, after which Chronos2 is fine-tuned with the LLM-generated covariates.

  • Multimodal Forecasting Gains. CoCo outperforms SOTA TSFMs and LLMs across 30 datasets and 95 settings (Figure 5) spanning finance, weather, macroeconomics, traffic, and security domains, achieving up to 70% improvement over GPT-4o in traffic forecasting (Figure 2).

Figure 2. CoCo outperforms SOTA TSFMs (Chronos2) and closed-source LLMs (GPT-4o) across datasets from diverse domains. CoCo effectively utilizes LLMs to distill textual annotations into usable forecast covariates that guide powerful TSFMs to generate accurate forecasts.

Figure 5. CoCo improves over unimodal and multimodal fine-tuning approaches, including SOTA TSFMs on real-world multimodal datasets. CoCo (ours) achieves the best average rank and mean MAE across 30 datasets (95 settings) spanning diverse domains and context lengths against 26 (TimeOmni omitted for scale) approaches. These gains demonstrate the effectiveness of GRPO-based fine-tuning along with our novel two-step fine-tuning.

Multimodal datasets for text-aided forecasting. Prior works largely study text-aided time series prediction, where auxiliary textual context improves forecasts across domains [21, 22, 16, 23, 24, 25]. However, the nature of text supervision varies significantly across domains. For example, finance offers abundant dynamic or time-varying annotations (e.g., news/social media/search trends) enabling large-scale datasets for stock price and movement predictions [13, 21, 22, 16, 23, 24, 25]. On the other hand, healthcare datasets typically provide static/global patient information in the form of medical records aligned with physiological signals and are commonly used for classification or anomaly detection [26, 27]. Other settings (e.g., weather, system monitoring, e-commerce) similarly incorporate static descriptors or dynamic textual context, but remain largely domain-specific [28, 29, 30, 31].

While TimeMMD [11] and Context-Is-Key [12] provide a collection of multimodal datasets across domains, they are primarily used for benchmarking purposes, and not large-scale training, due to their small sizes. To address these gaps, we focus on leveraging LLMs to generate synthetic multimodal datasets with realistic time-varying textual annotations aligned to real-world time series. This scales the volume of multimodal datasets, enabling large-scale training.

Multimodal forecasting approaches. Prior methods can be broadly grouped into zero-shot (training-free) and dataset-specific, training-based approaches. Zero-shot methods primarily use LLMs by encoding time series and other modalities as textual prompts, enabling LLMs to reason over temporal patterns and contextual information [32, 33, 16, 34, 15]. Such approaches offer flexibility across domains, but Park et al. [14] show that they are often sensitive to noise and typically underperform with respect to lightweight or domain-specific forecasting models [35] on standard benchmarks.

Training-based approaches learn joint representations of time series and text by fine-tuning LLMs [36, 37, 38, 18], aligning text and time series embeddings [39, 40, 41, 42], or employing fusion architectures [43, 37, 44, 45, 46]. Several works explore vision-based [47, 48, 49] or vision–language-based [50] representations by encoding time series as images or plots, or closed-loop reasoning frameworks that iteratively align text, predictions, and explanations [20]. These methods are either domain-specific or require per-dataset training.

Overall, previous works primarily (i) focus on LLMs as the key forecasters, (ii) rely on dataset-specific multimodal training, and (iii) do not exploit SOTA TSFMs [8, 7, 51]. In contrast, we develop a zero-shot, domain-agnostic setup that combines the strengths of LLMs and TSFMs to generate context-aware forecasts. We refer the readers to Appendix § 9 for a detailed discussion.

Problem Formulation

Unimodal time series forecasting models take historical numerical series XH∈XH=RlX^{H} \in\mathcal{X}^{H} = \mathbb{R}^{l}, where ll is the length of the context window, as input and generate the forecast XF∈XF=RhX^{F} \in\mathcal{X}^{F} = \mathbb{R}^{h}, where hh is the horizon or forecast length. Typically, the sample at timestamp tt is denoted as (XtH,XtF)(X^{H}_{t}, X^{F}_{t}), where XtH=[xt−l+1,xt−l+2,…,xt]∈XHX^{H}_{t} = [x_{t-l+1}, x_{t-l+2}, \ldots, x_{t}] \in\mathcal{X}^{H} and XtF=[xt+1,xt+2,…,xt+h]∈XFX^{F}_{t} = [x_{t+1}, x_{t+2}, \ldots, x_{t+h}] \in\mathcal{X}^{F}. Thus, the unimodal forecasting model parameterized by θ\theta is given by fθuni:XH→XFf_{\theta}^{\mathrm{uni}} : \mathcal{X}^{H} \to\mathcal{X}^{F}.

In the multimodal setting, the textual annotations, denoted by S∈S=Nl×Nmax⁡tokenS \in\mathcal{S} = \mathbb{N}^{l \times N_{\max}^{\mathrm{token}}}, are available for each timestamp in the context for the most general setting, with up to Nmax⁡tokenN_{\max}^{\mathrm{token}} tokens in each annotation. As with the unimodal case, the sample at timestamp tt in a dataset is denoted as (XtH,St,XtF)(X^{H}_{t}, S_{t}, X^{F}_{t}), where St=[st−l+1,st−l+2,…,st]∈SS_{t} = [s_{t-l+1}, s_{t-l+2}, \ldots, s_{t}] \in\mathcal{S}. Note that at any given timestamp tt, sts_{t} can contain a summary of the past numerical values and the predictive cues to accurately forecast the future values. To this end, the multimodal forecasting model, parameterized by θ\theta, is denoted as fθmulti:XH×S→XFf_{\theta}^{\mathrm{multi}} : \mathcal{X}^{H} \times\mathcal{S} \to\mathcal{X}^{F}.

Problem Statement: We aim to develop a multimodal forecasting model fθmultif_{\theta}^{\mathrm{multi}} such that the inclusion of textual context SS considerably improves the forecasting accuracy compared to the unimodal approach fθunif_{\theta}^{\mathrm{uni}} during zero-shot inference. More formally, given a dataset D={(XtH,St,XtF)}t=1ND = \{(X^{H}_{t}, S_{t}, X^{F}_{t})\}_{t=1}^{N}, where NN is the total number of samples, and a metric M(X^F,XF)\mathcal{M}(\hat{X}^{F}, X^{F}) that evaluates the error in the forecast X^F\hat{X}^{F} relative to the ground truth XFX^{F}, we aim to develop fθmultif_{\theta}^{\mathrm{multi}} such that M(fθmulti(XH,S),XF)≤M(fθuni(XH),XF)\mathcal{M}(f_{\theta}^{\mathrm{multi}}(X^{H}, S), X^{F}) \leq\mathcal{M}(f_{\theta}^{\mathrm{uni}}(X^{H}), X^{F}).

Synthetic Multimodal Dataset Generation

We propose a scalable approach to address the scarcity of multi-domain multimodal datasets by generating textual annotations for real-world time series using LLMs, motivated by the following key intuition: given numerical trajectories and domain descriptions, LLMs can produce semantically meaningful explanations for observed temporal trends (see Figure 11).

LLM Annotated Gasoline Price Time Series (2022)

Figure 11. Qwen3 provides realistic textual annotations. Here, we show samples of annotations generated by Qwen3-30b-A3B-Instruct-2507 [9] for gasoline prices in the United States for the year 2022. Note that the text generated is relevant to the domain queried and foreshadows the trend in the numerical data. In the plot, the green portion indicates an increasing trend and the red portion indicates a decreasing trend.

For each dataset, we randomly sample windows of fixed length and provide the numerical values, timestamps, and dataset description to an LLM (Qwen3-30B-A3B-Instruct-2507 [9], referred to as Qwen3-30B). The model is prompted to generate context-rich annotations for each timestamp tt describing plausible events or exogenous indicators explaining the observations up to tt. Additionally, the LLM produces speculative descriptions of the future trajectory without generating explicit numerical forecasts. We added key indications to the prompt to ensure sufficient diversity in the generated annotations, and refer the readers to Appendix § 17.1 for the actual prompt input to the LLM.

To assess the realism of the generated annotations, we compare the cosine similarity between the normalized BERT [52] embeddings of the true future and the predicted forecast description, for real and synthetic cases. The significant overlap in resulting distributions (Figure 8) indicates synthetic annotations capture the forecast with levels of uncertainty similar to those observed in real context.

Histogram of cosine similarity distributions for real and synthetic annotations

Figure 8. Real and synthetic annotations contain similar predictive powers to estimate the future. The significant overlap of cosine similarity distributions between the true and the predicted forecast descriptions for synthetic and real annotations validates our synthetic annotation strategy. For the real cosine similarity distribution, we compare the normalized BERT embeddings for the domain-agnostic forecast predictions from real text with the true domain-agnostic forecast description. Similarly, for the synthetic cosine similarity distribution, we compare the normalized BERT embeddings for the domain-agnostic forecast predictions from synthetic text with the true domain-agnostic forecast description. For this experiment, we consider the Media Cloud datasets for which the real annotations are available, and we generated synthetic annotations for those datasets.

To this end, we make the following observation: sim-to-real transfer, i.e., training on synthetic and testing on real-world text annotations, is feasible because both annotation types encode similar domain-agnostic descriptions of forecast trends (e.g., argmax in the second forecast timestamp) in domain-specific terms (e.g., due to tariffs, crude oil prices are expected to peak in the next two days). We validate this observation using a toy example (Figure 3). Consider the pharmaceutical sales of a company following a Seasonal Autoregressive Integrated Moving Average (SARIMA) process with shocks, explained by analyst notes. For each shock, an LLM generates a plausible domain-specific annotation. We fit SARIMA models with extracted sentiment features (scalar, domain-agnostic value ranging from -1 to 1) from both real and synthetic annotations and observe similar MAE on real annotations during the test period, demonstrating that synthetic text preserves forecast-relevant semantics useful for accurate prediction. A detailed description is provided in Appendix § 11.

SARIMA training and testing results for real and synthetic annotations

Figure 3. Training on synthetic annotations yields real-world multimodal gains. Using pharmaceutical sales data annotated with both real news and synthetic text, a SARIMA model with sentiment features achieves similar MAE when trained on synthetic versus real annotations, as both encode the same domain-agnostic forecast trends, thereby enabling sim-to-real transfer.

To formalize this observation, we introduce a domain-agnostic contextual variable Z∈SZ \in\mathcal{S} (e.g., spikes or gradual trends). Let the history, forecast, and the forecast’s domain-agnostic descriptions be sampled from a joint data distribution p(XH,Z,XF)p(X^{H}, Z, X^{F}), where the future XFX^{F} depends on history XHX^{H} and domain-agnostic context ZZ. The real annotations are naturally aligned with and describe the underlying future trends ZZ in domain-specific terms (Sr∼pr(Sr∣XH,Z))(S^{r} \sim p_{r}(S^{r} \mid X^{H}, Z)). The synthetic annotations are generated as Ss∼ps(Ss∣XH,Z,XF)S^{s} \sim p_{s}(S^{s} \mid X^{H}, Z, X^{F}), reflecting the use of future information by the LLM. To this end, for a multimodal forecaster hθmulti(XH,S)=hθmulti(XH,ϕ(S))h_{\theta}^{\mathrm{multi}}(X^{H}, S) = h_{\theta}^{\mathrm{multi}}(X^{H}, \phi(S)), we define the real and synthetic population risks as

Rmultir(θ)=Ep,pr[M(hθmulti(XH,ϕ(Sr)),XF)],Rmultis(θ)=Ep,ps[M(hθmulti(XH,ϕ(Ss)),XF)].R_{\mathrm{multi}}^{r}(\theta) = \mathbb{E}_{p,p_{r}}\left[\mathcal{M}\left(h_{\theta}^{\mathrm{multi}}(X^{H}, \phi(S^{r})), X^{F}\right)\right], \quad R_{\mathrm{multi}}^{s}(\theta) = \mathbb{E}_{p,p_{s}}\left[\mathcal{M}\left(h_{\theta}^{\mathrm{multi}}(X^{H}, \phi(S^{s})), X^{F}\right)\right].

Here, ϕ:S→Rd\phi:\mathcal{S} \to\mathbb{R}^{d} is a text encoder with embedding dimension dd, and E\mathbb{E} is the expectation operator.

Assumption 4.1 (Approximate semantic representation). There exists a semantic embedding ψ(Z)∈Rd\psi(Z) \in\mathbb{R}^{d} of the latent context and representation error bounds εr,εs≥0\varepsilon_{r}, \varepsilon_{s} \ge0 such that

∥ϕ(Sr)−ψ(Z)∥≤εr,∥ϕ(Ss)−ψ(Z)∥≤εs.(1)\lVert\phi(S^{r})-\psi(Z)\rVert\le\varepsilon_{r}, \qquad\lVert\phi(S^{s})-\psi(Z)\rVert\le\varepsilon_{s}. \tag*{(1)}

Assumption 4.2 (Lipschitz dependence on text representation). There exists a constant L>0L > 0 such that for all XHX^{H}, XFX^{F}, and hθmulti:XH×Rd→XFh_{\theta}^{\mathrm{multi}}:\mathcal{X}^{H} \times\mathbb{R}^{d} \to\mathcal{X}^{F}, and all u,v∈Rdu,v \in\mathbb{R}^{d},

∣M(hθmulti(XH,u),XF)−M(hθmulti(XH,v),XF)∣≤L∥u−v∥.(2)\left|\mathcal{M}\left(h_{\theta}^{\mathrm{multi}}(X^{H},u),X^{F}\right)-\mathcal{M}\left(h_{\theta}^{\mathrm{multi}}(X^{H},v),X^{F}\right)\right| \le L\lVert u-v\rVert. \tag*{(2)}

Here, ∣⋅∣|\cdot| represents the absolute value and ∥⋅∥\lVert\cdot\rVert represents the l2l_{2} norm functions. Assumption 4.1 states that both real (SrS^{r}) and synthetic (SsS^{s}) annotations lie close to the domain-agnostic forecast description ZZ in a latent space. Now, we provide empirical evidence to back this assumption. First, real-world text annotations (SrS^r) are typically speculations that encode the future (ZZ) and thus imply a small ϵr\epsilon_r. To justify small ϵs\epsilon_s, we refer to Figure 8. The strong overlap in cosine similarity distributions between the embeddings of forecast predictions derived from real (ϕ(Sr)\phi(S^r)) and synthetic annotations (ϕ(Ss)\phi(S^s)) with the true future descriptions (ψ(Z)\psi(Z)), i.e., ϕ(Sr)Tψ(Z)\phi(S^r)^\mathsf{T}\psi(Z) and ϕ(Ss)Tψ(Z)\phi(S^s)^\mathsf{T}\psi(Z), indicates that synthetic annotations recover the forecast nearly as well as real ones, suggesting ϵs≈ϵr\epsilon_s \approx\epsilon_r. Additionally, the generation process grounds synthetic text in numerical trajectories and domain descriptions. Here, ϕ(S)\phi(S) and ψ(Z)\psi(Z) correspond to normalized BERT embeddings of predicted and true domain-agnostic forecast descriptions, respectively.

Assumption 4.2 ensures that small perturbations in the text embedding produce limited changes in forecasting errors. Tables 5, 6, and 21 show that perturbing real-world textual annotations causes negligible variation in forecasting error when the perturbations largely preserve forecast-relevant semantics. This empirical result points towards local smoothness in the loss space with respect to small perturbations in text annotations for reasonably good forecasters.

Lemma 4.3 (Representation mismatch bound). Under Assumptions 4.1 and 4.2,

∣Rmultir(θ)−Rmultis(θ)∣≤L(ϵr+ϵs).(3)\left|R^r_{\mathrm{multi}}(\theta)-R^s_{\mathrm{multi}}(\theta)\right| \le L(\epsilon_r+\epsilon_s). \tag*{(3)}

Theorem 4.4 (Synthetic-to-real transfer). Let θ^s∈arg⁡min⁡θ∈ΘRmultis(θ)\hat{\theta}_s \in\arg\min_{\theta\in\Theta} R^s_{\mathrm{multi}}(\theta) and θr∗∈arg⁡min⁡θ∈ΘRmultir(θ)\theta_r^* \in\arg\min_{\theta\in\Theta} R^r_{\mathrm{multi}}(\theta). Under Assumptions 4.1 and 4.2,

Rmultir(θ^s)≤Rmultir(θr∗)+2L(ϵr+ϵs).(4)R^r_{\mathrm{multi}}(\hat{\theta}_s) \le R^r_{\mathrm{multi}}(\theta_r^*) + 2L(\epsilon_r+\epsilon_s). \tag*{(4)}

Lemma 4.3 states that for a forecaster, if real and synthetic embeddings are close to the same latent semantic representation (small ϵr\epsilon_r and ϵs\epsilon_s), their risks remain close. Theorem 4.4 states that a model trained with synthetic text achieves near-optimal performance under real text, with a bounded gap of 2L(ϵr+ϵs)2L(\epsilon_r+\epsilon_s) relative to the best achievable real-text risk. In practice, this gap depends on (i) how effectively the encoder ϕ\phi aligns real and synthetic text in a shared semantic space (whether ϵr≃ϵs\epsilon_r \simeq\epsilon_s), and (ii) how well synthetic annotations preserve the forecast-relevant semantics of ZZ (whether ∥ϕ(Ss)−ψ(Z)∥\|\phi(S^s)-\psi(Z)\| is small). Our results in Section 6 demonstrate that models trained on synthetic text generalize effectively to real-world multimodal datasets. Proofs of Lemma 4.3 and Theorem 4.4 and a detailed discussion on the assumptions are provided in Appendix § 10.

Our formulation is closely related to domain adaptation and invariant representations [53], where domain generalization is achieved by aligning representations or assuming a shared latent variable. Similarly, sim-to-real transfer [54] relies on learning representations robust to shifts between synthetic and real data. We adapt these concepts to multimodal forecasting and realistic synthetic annotation generation for domains where there is a lack of real textual annotations, with valid assumptions. Note that in our problem formulation and theoretical justification, the risk or the metric M(X^F,XF)\mathcal{M}(\hat{X}^{F},X^{F}) can be the mean squared error (MSE) or the mean absolute error (MAE) between the ground truth and the forecast. Without any loss of generality, the theory and algorithms developed in this work can be extended to the multivariate case.

Context as Covariates

We introduce CoCo (Context as Covariates), a multimodal forecasting framework that combines LLMs’ ability to interpret text with TSFMs’ numerical precision (Figure 4). CoCo uses the LLM to generate a text-aware forecast capturing broad contextual trends and a per-step confidence estimate, which are provided as covariates to guide a multivariate TSFM (M-TSFM), such as Chronos2, toward precise numerical forecasts. We first describe the covariate construction.

Context as Covariates (CoCo) architecture

Figure 4. Context as Covariates (CoCo) Description. The LLM θLLM\theta_{\mathrm{LLM}} takes the numerical history XtHX_t^H (blue) and textual context StS_t (violet) as input and produces a forecast covariate MtM_t (Equation (5), brown) and a rollout-based confidence covariate CtC_t (Equation (6), maroon), which are concatenated with XtHX_t^H and passed to Chronos2 for multivariate forecasting. Our key technical intuition is to utilize the confidence covariate (obtained from the per-step variance in the LLM’s predictions) to teach Chronos2 when to trust the LLM and when not to trust it (low variance →\rightarrow trust the LLM, as text induces a more decisive forecast).

LLM Forecast Covariate: Our main intuition is to model the forecast covariate as a noisy version of the next timestep. The forecast covariate, Mt∈R(l+h)M_t \in\mathbb{R}^{(l+h)}, is constructed from the numerical history XtH=[xt−l+1,…,xt]X_t^H=[x_{t-l+1},\ldots,x_t] and the text-aware LLM forecast θLLM(XtH,St)=[mt+1,…,mt+h]\theta_{\mathrm{LLM}}(X_t^H,S_t)=[m_{t+1},\ldots,m_{t+h}] as:

Mt=[xt−l+2+ϵ2, xt−l+3+ϵ3, …, xt+ϵl, mt+1, mt+2, …, mt+h, mt+h+ϵend],(5)M_t = [x_{t-l+2}+\epsilon_2,\ x_{t-l+3}+\epsilon_3,\ \ldots,\ x_t+\epsilon_l,\ m_{t+1},\ m_{t+2},\ \ldots,\ m_{t+h},\ m_{t+h}+\epsilon_{\mathrm{end}}], \tag*{(5)}

where ϵi,ϵend∼N(0,σϵ2)\epsilon_i,\epsilon_{\mathrm{end}} \sim\mathcal{N}(0,\sigma_\epsilon^2) are independent and identically distributed (i.i.d.) noise terms and we assign a very small value to σϵ2\sigma_\epsilon^2. In the history portion, the covariate at position ii is a noisy version of the next observed value xix_i. The future portion carries the LLM’s median forecast directly. This construction allows M-TSFMs to learn through in-context learning that MtM_t provides a noisy version of the true forecast, enabling text-induced trends in [mt+1,…,mt+h][m_{t+1}, \ldots, m_{t+h}] to be included in the final forecast. In other words, this setup allows M-TSFMs to interpret MtM_t as a noisy next-step predictor.

LLM Confidence Covariate: We model the confidence covariate to provide the M-TSFM with a per-step signal of how much to trust the LLM forecast. To measure confidence, we generate NrolloutsN_{\mathrm{rollouts}} independent rollouts for the same multimodal input sample (XtH,St)(X_t^H, S_t), where the ii-th rollout is denoted as [mt+1(i),…,mt+h(i)][m_{t+1}^{(i)}, \ldots, m_{t+h}^{(i)}]. The confidence at future timestep t+jt+j (1≤j≤h1 \le j \le h) is given by

ct+j=σ2({mt+j(i)}i=1Nrollouts)σ2(XtH),c_{t+j} = \frac{\sigma^2\left(\{m_{t+j}^{(i)}\}_{i=1}^{N_{\mathrm{rollouts}}}\right)}{\sigma^2(X_t^H)},

where σ\sigma denotes the standard deviation operator and dividing by σ2(XtH)\sigma^2(X_t^H) makes ct+jc_{t+j} scale invariant. To this end, we define the confidence covariate as

Ct=[ϵ2,ϵ3,…,ϵl,ct+1,ct+2,…,ct+h,ϵend].(6)C_t = [\epsilon_2, \epsilon_3, \ldots, \epsilon_l, c_{t+1}, c_{t+2}, \ldots, c_{t+h}, \epsilon_{\mathrm{end}}]. \tag*{(6)}

Small ct+jc_{t+j} indicates higher LLM confidence at step t+jt+j. The history portion of the confidence covariate is filled with ϵi∼N(0,σϵ2)\epsilon_i \sim\mathcal{N}(0, \sigma_{\epsilon}^2), where we assign a very small value to σϵ2\sigma_{\epsilon}^2 to teach the M-TSFM that the history portion of MtM_t resembles an LLM forecast with a very high confidence score.

Conditioned on MtM_t and CtC_t, an M-TSFM (θM−TSFM)(\theta_{\mathrm{M-TSFM}}) produces the forecast X^tF=θM−TSFM(XtH⊕∅∥Mt∥Ct)\hat{X}_t^F = \theta_{\mathrm{M-TSFM}}(X_t^H \oplus\varnothing\mathbin\Vert M_t \mathbin\Vert C_t). Here, XtH⊕∅X_t^H \oplus\varnothing indicates padding the history XtHX_t^H for hh forecast timestamps and ∥\mathbin\Vert denotes column-wise concatenation, such that XtH⊕∅∥Mt∥Ct∈R(l+h)×3X_t^H \oplus\varnothing\mathbin\Vert M_t \mathbin\Vert C_t \in\mathbb{R}^{(l+h)\times3} stacks the padded history and covariate channels along the feature dimension. For ease of notation, we drop ∅\varnothing and use X^tF=θM−TSFM(XtH∥Mt∥Ct)\hat{X}_t^F = \theta_{\mathrm{M-TSFM}}(X_t^H \mathbin\Vert M_t \mathbin\Vert C_t) in the rest of the manuscript.

We fine-tune both the LLM (θLLM)(\theta_{\mathrm{LLM}}) and the M-TSFM (θM−TSFM)(\theta_{\mathrm{M-TSFM}}) in sequence instead of end-to-end joint fine-tuning. The LLM is fine-tuned first against a pretrained M-TSFM, and the M-TSFM is subsequently fine-tuned using covariates pre-generated from the fine-tuned LLM.

LLM Fine-Tuning: We fine-tune the LLM with GRPO [55] to produce forecast covariates with text-induced trends by optimizing it against a pretrained M-TSFM, using forecasting improvement as the reward. Although the M-TSFM is held fixed, it can still respond to the covariate since MtM_t is constructed as a noisy next-step continuation aligned with its pretraining inductive biases (Table 25). This allows the model to partially utilize MtM_t via in-context learning, making the relative forecasting error a meaningful reward signal. For a sample (XtH,St,XtF)(X_t^H, S_t, X_t^F), we generate NrolloutsN_{\mathrm{rollouts}} forecasts {θLLM(g)(XtH,St)}g=1Nrollouts\{\theta_{\mathrm{LLM}}^{(g)}(X_t^H, S_t)\}_{g=1}^{N_{\mathrm{rollouts}}} and construct covariates Mt(g)M_t^{(g)}. The reward and GRPO advantage are:

rt(g)=11+∥θM−TSFM(XtH∥Mt(g))−XtF∥1∥θM−TSFM(XtH)−XtF∥1+δ,At(g)=rt(g)−1Nrollouts∑i=1Nrolloutsrt(i)σ({rt(i)}i=1Nrollouts)+δ.(7)r_t^{(g)} = \frac{1}{1 + \frac{\left\|\theta_{\mathrm{M-TSFM}}(X_t^H \mathbin\Vert M_t^{(g)}) - X_t^F\right\|_1}{\left\|\theta_{\mathrm{M-TSFM}}(X_t^H) - X_t^F\right\|_1 + \delta}}, \qquad A_t^{(g)} = \frac{r_t^{(g)} - \frac{1}{N_{\mathrm{rollouts}}\sum_{i=1}^{N_{\mathrm{rollouts}}} r_t^{(i)}}}{\sigma\left(\{r_t^{(i)}\}_{i=1}^{N_{\mathrm{rollouts}}}\right) + \delta}. \tag*{(7)}

where a very small δ>0\delta> 0 prevents division by zero (δ=10−8\delta= 10^{-8}). The reward lies in (0,1)(0, 1), approaching 1 when the covariate improves forecasting and 0.5 when it provides no benefit. The confidence covariate CtC_t is excluded from reward computation due to the cost of additional rollouts. We denote θM−TSFM(XtH)\theta_{\mathrm{M-TSFM}}(X_t^H), θM−TSFM(XtH∥Mt)\theta_{\mathrm{M-TSFM}}(X_t^H \Vert M_t), and θM−TSFM(XtH∥Mt∥Ct)\theta_{\mathrm{M-TSFM}}(X_t^H \Vert M_t \Vert C_t) as the unimodal, forecast-covariate, and full multimodal predictions, respectively.

M-TSFM Fine-Tuning: Following LLM fine-tuning, we pre-generate the full covariate set {Mt,Ct}t=1N\{M_t, C_t\}_{t=1}^{N} for all NN training samples using the fine-tuned LLM. The M-TSFM is then fine-tuned by minimizing the MSE between its covariate-conditioned forecast and the ground truth:

1N∑t=1N∥θM−TSFM(XtH∥Mt∥Ct)−XtF∥2\frac{1}{N}\sum_{t=1}^{N}\left\|\theta_{\mathrm{M-TSFM}}(X_t^H \Vert M_t \Vert C_t)-X_t^F\right\|^2

We note that fine-tuning is essential to teach the M-TSFM to interpret MtM_t as a soft next-step predictor and to modulate the reliance on MtM_t according to the per-step confidence signal CtC_t, producing forecasts that are context-aware and numerically precise.

This two-stage approach avoids the instability of optimizing against a moving target and keeps training tractable. The architecture choices and the hyperparameters are described in detail in Appendix E.1.

Experiments

We use Qwen3-4B-Instruct-2507 [9] (Qwen3-4B) as the LLM and Chronos2 as the M-TSFM in CoCo, with both models fine-tuned as described in Section 5.

Training Datasets and Setup: We generated synthetic annotations (Section 4) for 36,826 time series datasets (∼\sim12.4M rows) spanning retail, macroeconomics, environment, and finance, and curated ∼\sim150,000 rows of real annotations from the finance [13] domain. Details of the datasets, annotation pipeline, and training (Section 5) are provided in Appendices E.2 and E.1.

Evaluation Datasets: We evaluate CoCo on a wide range of 30 datasets (over 16,719 samples) spanning macroeconomics, finance, retail, climate, security, and traffic domains. These datasets are partly curated from existing benchmarks, such as TimeMMD [11] (8 datasets) and FNSPID [13] (11 datasets). Additionally, we curated text annotations for 11 real-world time series datasets with time-stamped news articles extracted using the Media Cloud API [56]. A description of our evaluation datasets is provided in Appendix F.1. Our experiments used 64, 128, and 256 as context lengths and 16 as the forecast length. The TimeMMD dataset is also evaluated under the original paper settings (shown as TimeMMD*).

Baselines. We compare CoCo against 27 methods, categorized by modality and adaptation: eight zero-shot unimodal, seven zero-shot multimodal, six unimodal fine-tuning, and six multimodal fine-tuning approaches. For a controlled comparison, every fine-tuned baseline uses the same synthetic text–time-series pairs as CoCo. Appendix F.2 provides the complete list and implementation details.

Metrics. We report the per-dataset and aggregated (weighted equally to avoid bias from differing sample sizes) mean MAE. To aggregate relative forecasting performance across different settings, we report two complementary metrics: Average Rank, for which the models are ranked (best model ranked 1) in each setting, and the average rank across all the settings is reported; and ELO Score

(detailed in Appendix F.3). For statistical significance, we report the Wilcoxon pp-value based on MAE computed across settings. The quantitative comparisons are summarized in Tables 1 and 2 with dataset-level metrics for each baseline category shown in Tables 13-16. For the Context-Is-Key benchmark, we report the Region of Interest Continuous Ranked Probability Score (RCRPS).

CoCo outperforms SOTA TSFMs and LLM-based forecasters consistently. We refer the readers to Tables 1 and 2, and Figure 5. The comparison against the SOTA TSFMs reveals the multimodal gains that can be achieved from textual annotations. Improvements over LLM-based forecasters showcase the ability of CoCo to effectively combine textual context and time series representations. Similarly, significant improvements over multimodal fine-tuning approaches elucidate the right approach to utilize textual annotations to improve SOTA TSFMs.

Synthetic annotations enable generalization to domains without real multimodal data. Table 24 shows that fine-tuning on real FNSPID data achieves strong in-domain performance (MAE 0.891) but degrades on out-of-distribution TimeMMD (MAE 1.528). In contrast, fine-tuning on synthetic annotations (Section 4) yields comparable FNSPID performance (MAE 0.903) and improved TimeMMD performance (MAE 1.365). Finally, fine-tuning on both real and synthetic annotations achieves the best overall results (TimeMMD MAE 1.354) without degrading FNSPID (MAE 0.894). These results show that fine-tuning on our synthetic annotations can enable generalization to domains lacking large-scale multimodal datasets, thereby showcasing sim-to-real transfer in multimodal forecasting.

Qwen3-4b Fine-tuningOverall MAE (↓\downarrow)TimeMMD MAE (↓\downarrow)FNSPID MAE (↓\downarrow)Rank (↓\downarrow)
Supervised Fine-tuning1.1871.6351.0508.289
CoCo Fine-tuning without Text1.1721.5430.9544.395
CoCo Fine-tuning with Synthetic Text1.0871.3650.9033.974
CoCo Fine-tuning with Real Text1.1401.5280.8913.605
CoCo Fine-tuning with Real and Synthetic Text1.0781.3540.8943.737

Table 24. Effect of using text and the kind of text used in fine-tuning the LLM. The results are for the Qwen3-4B-Instruct-2507 model. Real text here denotes FNSPID data and synthetic text refers to the dataset described in Appendix E.2. Though the model trained completely with FNSPID data has a marginally better rank (the wins are concentrated in the FNSPID subset, which was seen during training), we observe that it fails in the TimeMMD dataset, which is out of distribution and hence does not generalize. On the other hand, models trained on completely synthetic text or a mixture of both provide improved performance in TimeMMD as well. The MAE and average ranks reported are over the same 38 settings as Table 22.

Notably, neither supervised fine-tuning nor reinforcement learning (RL) fine-tuning without text achieves comparable gains, confirming that both the GRPO reward signal and the textual annotations are essential for fine-tuning.

CoCo’s gains are strongly dependent on how effectively the LLM utilizes the text. We use the relative MAE improvement in the LLM forecast when the textual context StS_t is provided versus withheld, as a measure of text usefulness. Positive values indicate that the text helped the LLM produce a better forecast, while negative values indicate misleading or incorrect text. Figure 7 (left) shows that text usefulness is a strong indicator of CoCo’s gains. Figure 7 (right) shows that when text is highly useful, performance peaks at high LLM confidence and degrades as confidence decreases. Conversely, when text is not useful, high LLM confidence is actively harmful, since Chronos2 is misled by confidently wrong covariates. This demonstrates that CtC_t acts as an effective gating mechanism, allowing Chronos2 to selectively incorporate MtM_t based on its per-step reliability.

Win rate and mean percentage MAE improvement of CoCo compared with Chronos2, conditioned on text usefulness and confidence

Figure 7. (Left) CoCo consistently outperforms SOTA unimodal models (Chronos2) when text is useful (Q5–Q9) and underperforms when it is not (Q1–Q3). Text usefulness is defined as the relative MAE improvement of the LLM forecast when the context StS_t is provided versus withheld. (Right) Furthermore, for the set of windows where text is useful, CoCo significantly outperforms unimodal models when the rollout variance from the LLM (Confidence CtC_t) is very low.

Fine-tuningMtM_tBoth Mt+CtM_t + C_t
Only θLLM\theta_{\mathrm{LLM}}0.211.64
Only θM−TSFM\theta_{\mathrm{M-TSFM}}2.452.55
Both θLLM\theta_{\mathrm{LLM}} and θM−TSFM\theta_{\mathrm{M-TSFM}}2.63.36

Table 3. Using both MtM_t and CtC_t consistently outperforms MtM_t alone across all fine-tuning configurations. We show the % MAE improvement over SOTA unimodal models. Sequential fine-tuning of both LLM θLLM\theta_{\mathrm{LLM}} and M-TSFM θM−TSFM\theta_{\mathrm{M-TSFM}} yields the best % MAE improvement (Table 22).

Both fine-tuning stages and both covariates are crucial for forecasting performance. Tables 3, 22 and 28 show that fine-tuning both LLM and Chronos2, and providing both the covariates, yields the largest improvement (3.36% over the Chronos2 baseline), with each component contributing incrementally. Adding the confidence covariate on top of the forecast covariate consistently improves performance across all fine-tuning configurations, except for the zero-shot setting, where neither the LLM has learned to provide good covariates nor Chronos2 has learned to interpret them.

CoCo generalizes to multiple LLM ×\times M-TSFM pairs

Tables 25 and 26, and Figure 6 demonstrate that CoCo is not specific to Qwen3-4B or Chronos2. Across five LLM configurations and three M-TSFM architectures, the forecast covariate MtM_t improves every evaluated LLM ×\times M-TSFM combination, while the confidence covariate CtC_t provides further gains with zero-shot Toto and fine-tuned Chronos2. Although Chronos2 is fine-tuned using rollouts pre-generated only by the fine-tuned Qwen3-4B, it generalizes to covariates produced by all evaluated zero-shot LLMs. The fine-tuned Qwen3-4B also outperforms the larger zero-shot LLMs as a standalone forecaster (Figure 13).

Figure 6. CoCo generalizes across model choices. MAE improvement over each numerical-only M-TSFM baseline on Media Cloud (64-16 setting).

Bar chart comparing standalone forecasting performance of LLMs

Figure 13. Comparison of standalone forecasting performance of LLMs. The plot shows the %-improvement in the MAE over the base Qwen3-4B model across the same 38 settings as Table 22. We observe that the CoCo fine-tuned Qwen3-4B outperforms larger and closed-source LLMs as a standalone forecaster.

CoCo improves text-time series alignment in pretrained LLMs

We evaluate this using the Context-Is-Key benchmark (CIK, Tables 4 and 8), where accurate forecasting requires strong cross-modal alignment between text and time series due to text-induced trends and spikes. The RCRPS metric further penalizes incorrect trends and shifted spikes. The CoCo fine-tuned LLM consistently outperforms its pretrained counterpart, showing that GRPO fine-tuning improves the LLM’s ability to combine numerical forecasting with text-based reasoning. Notably, CoCo fine-tuned Qwen3-4B matches the performance of GPT-4o. We refer readers to Appendix D for detailed results.

Limitations. First, we note that CoCo inference is slower than unimodal TSFMs due to the additional LLM inference required for extracting predictive signals from text (Table 29), a capability fundamentally absent in TSFMs. Further, CoCo assumes that textual annotations are generally predictive of future trends. When the text is misleading, the LLM can generate confidently incorrect covariates that Chronos2 cannot fully correct, thereby degrading performance (Table 21, Figure 7 (left)). Finally,

MAE (↓)
CoCo Fine-tuned Chronos2CoCo (With Misleading Text)CoCo (With Truthful Text)
2.9293.0252.802

Table 21. CoCo’s performance degrades when text is adversarial. We evaluate on a synthetically generated dataset of 1000 samples where the same numerical series is paired with two kinds of annotations: truthful text describing the correct future trend, and misleading text text deliberately describing the opposite trend. CoCo improves over the fine-tuned Chronos2 baseline with truthful text but degrades with misleading text, consistent with the gating analysis in Figure 7 (right), where confidently wrong LLM forecasts are the most harmful regime.

ModelLatency (ms/sample)
Chronos211.43 ±\pm 7.18
Toto31.15 ±\pm 39.67
TimesFM 2.537.08 ±\pm 52.43
Chronos MV75.77 ±\pm 54.26
TabPFN 2.5270.17 ±\pm 28.62
CoCo280.07 ±\pm 21.96
Toto MV312.14 ±\pm 2.47

Table 29. Inference latency (ms/sample) for CoCo and all TSFM baselines, measured on a single H100 GPU. Latency is computed by processing 128 samples (16-step forecasts with a context length of 64) in batches of 16 and averaging over batches. For CoCo, the fine-tuned Qwen3-4b-Instruct-2507 is served via a vLLM inference server on the same GPU, with k=5k = 5 rollouts generated per sample for the confidence covariate CtC_t. Despite requiring LLM inference with multiple rollouts, CoCo achieves 280 ms/sample, comparable to TabPFN 2.5. The latency overhead is a direct consequence of incorporating an LLM, which we view as necessary for extracting predictive signals from text, a capability that unimodal TSFMs fundamentally lack.

CoCo is designed for settings where numerical history and textual context carry complementary predictive signals. In settings where text is the sole driver of forecast outcomes, the TSFM backbone contributes less, and we observe that CoCo’s gains over just the LLM are limited (Appendix D).

Additional Experiments. Table 17 shows that CoCo continues to outperform zero-shot multimodal baselines on samples strictly after June 2024, suggesting robustness beyond potential pretraining overlap. To reduce sensitivity to dataset scale and outliers in aggregate MAE, Table 18 reports the geometric mean of MASE across datasets, following GIFT-Eval [57], and yields conclusions consistent with MAE. Table 19 further shows that CoCo’s relative performance remains consistent across raw and normalized evaluation metrics. Table 22 demonstrates that the strongest multimodal performance is obtained by fine-tuning both the LLM and M-TSFM and using both MtM_t and CtC_t. Finally, across 30 Media Cloud settings, CoCo outperforms GPT-5 in 70% (21/30) of cases and achieves up to an 11% MAE improvement on steel-price forecasting (Table 20).

GroupSettingDatasetNCoCoTimesFM2.5+GPT4oChronos2+GPT4oChronos2 MVToto MVGPT-4oQwen3-30BTimeOmni
Media Cloud64-16Crude Oil Price3160.8020.7520.7580.8170.7600.8770.9241.082
S&P500 Index2800.7590.7970.8050.9040.9290.8830.8590.978
Apple Stock3190.9030.9770.9771.0431.0141.0891.0591.467
Copper Futures1790.7750.7840.7690.8930.9140.8240.8241.004
Egg Price1440.7010.6510.6690.6730.6440.7140.7780.902
Electricity Price3340.8380.8870.9070.8460.8731.0290.9671.294
SPDR Gold Price3170.8830.9180.8871.0471.1050.9261.0203.396
Google Stock3190.7160.7700.7680.8670.8470.8530.8430.851
NVIDIA Stock Price3180.8480.9140.9340.9340.9371.1041.4161.001
Silver Price3320.9030.9020.9121.0891.1120.9710.9461.137
Steel Rebar Futures1620.7320.7790.7770.7460.7390.9070.9421.400
Media Cloud128-16Crude Oil Price2520.6920.6180.6410.6780.6290.7090.8500.856
S&P500 Index2160.5730.6090.6410.6320.6460.7270.6630.824
Apple Stock2550.6950.7160.7430.7360.7250.8340.8760.897
Copper Futures1150.5890.5860.6010.6040.5650.6260.6490.702
Egg Price800.5060.5060.4740.5160.5100.4840.5660.716
Electricity Price2700.5970.6680.7040.6530.6700.8100.7120.884
SPDR Gold Price2530.5120.5300.5030.6330.6390.5190.5900.775
Google Stock2550.5260.5260.5280.5590.5720.5720.6500.659
NVIDIA Stock Price2540.5480.5800.5780.5800.5940.6840.7820.649
Silver Price2680.6670.6760.6760.8050.8100.7610.7650.864
Steel Rebar Futures980.4830.4270.4420.4230.4080.4830.6311.349
Media Cloud256-16Crude Oil Price1240.4530.3410.3680.5250.4160.3960.5330.532
S&P500 Index880.2400.2650.2620.4220.4200.2620.2390.370
Apple Stock1270.4050.4400.4430.4950.5070.4780.4370.509
Electricity Price1420.4670.5010.5020.5480.5300.5560.5730.575
SPDR Gold Price1250.2860.3120.2870.3090.3350.3080.3461.085
Google Stock1270.5470.5400.5440.6760.6860.5380.5090.650
NVIDIA Stock Price1260.3520.3730.3660.4830.4410.4020.4830.459
Silver Price1400.6080.6040.5970.8090.7540.6100.6040.746
Overall Wins165403020
Mean0.6200.6320.6350.6980.6910.6980.7340.954
ELO17271667165214491475143613931200
Average Rank2.172.772.905.034.805.175.677.50
Head-to-Head Wins–20/3020/3027/3024/3027/3027/3030/30
Wilcoxon p–0.0840.0582.3e-069.9e-052.8e-063.5e-081.9e-09

Table 17. Dataset-level MAE for samples strictly after June 2024. We evaluate CoCo against multimodal baselines in a regime where the LLMs have limited or no prior exposure to the test period. The gains are consistent with those in Table 14, confirming that the improvements stem from the ability of the model to extract predictive signals from text rather than from memorized knowledge. Bold and underlined values denote the best and second-best method in each row.

Model64-16 Setting Media Cloud64-16 Setting FNSPID64-16 Setting TimeMMD128-16 Setting Media Cloud128-16 Setting FNSPID128-16 Setting TimeMMD256-16 Setting Media Cloud256-16 Setting FNSPID256-16 Setting TimeMMDNative Setting TimeMMD*Overall MASEAvg. RankELOCoCo WinsWilcoxon pp
CoCo3.0643.1361.6053.0093.0561.5332.8172.9611.5301.9152.4555.211845––
Zero-shot Unimodal
TimeX3.1363.1751.6213.0753.0641.6092.8572.9501.6621.9362.5066.62178259/959.4e-04
Chronos23.2093.3061.6253.1633.1731.6072.9573.0351.6171.9752.5569.56166879/954.9e-12
TimesFM2.53.2303.2511.5983.1843.2251.5612.9943.1311.6061.9002.5539.60167265/951.1e-08
Moirai3.2093.2041.6933.0923.2311.6312.9553.1041.6382.0022.57110.15165579/957.5e-12
Toto3.3303.3621.6613.1723.3551.6273.0503.2461.6321.9512.62513.60154283/951.3e-13
TabPFN2.53.4623.4101.7763.5593.3791.6884.1533.7241.8842.0022.85318.35138191/953.9e-16
TimeMoe3.9493.7531.9203.5483.6091.8324.6413.8941.8042.0503.01920.96128188/953.1e-15
PANDA4.3114.0162.3114.3784.1302.1623.7013.7322.0452.2603.29524.03113693/954.6e-17
Zero-shot Multimodal
TimesFM2.5+GPT4o3.1433.1501.7873.0763.1751.7392.9333.0871.6422.0702.5838.80170067/953.4e-05
Chronos2+GPT4o3.1713.2221.8083.0963.1991.7422.9513.1151.6382.1432.61110.45164276/957.2e-08
Chronos2-MV3.3653.2701.6783.2753.1961.6103.3123.1671.5941.9312.61812.22158370/957.3e-09
Toto MV3.3893.5631.6473.2483.3781.5923.2223.2781.5681.9032.63413.11155072/951.0e-09
GPT-4o3.5623.5032.2833.4573.5212.1733.3093.4541.9962.4452.98719.79133291/952.2e-16
Qwen3-30B3.6633.4882.3913.6863.6352.3823.5293.6822.3592.4853.15221.17127694/952.7e-17
TimeOmni4.9323.9873.1565.1903.7834.2524.5333.8313.3662.6163.95225.41106395/952.6e-17
Unimodal Fine-tuning
Chronos23.2813.2421.5833.1183.1951.4982.8943.1051.4812.0082.5228.96168964/952.2e-07
AutoTimes3.2703.1581.9023.1803.1581.9062.8543.1781.9721.9462.70011.55160279/951.4e-10
PatchTST3.3373.2831.9423.2523.2351.8833.0373.3331.9641.9262.72914.01152988/954.5e-16
S2IP-LLM3.4933.3421.9103.3873.2271.8053.1443.3701.9632.0112.76916.0714559/959.6e-16
iTransformer3.5093.4811.9994.0394.0562.0853.5223.9202.0551.9873.05421.45125792/954.5e-17
TEST3.9783.8052.3013.9163.8292.2213.8894.2732.5192.4283.32224.94109195/952.6e-17
Multimodal Fine-tuning
Chronos2 MV3.2243.1421.6623.0743.1361.6052.9173.1031.5801.8522.5217.47174259/952.7e-05
TaTS3.3303.2331.6293.1803.1921.5733.2613.1681.5661.9212.58210.61163568/952.5e-07
VoT3.1903.0611.8723.2503.1571.8343.5153.3451.8691.9672.69811.57160471/953.9e-09
S2IP-LLM3.3163.1471.9133.3293.2591.8723.4003.7681.9481.9452.78914.36151383/953.1e-14
AutoTimes3.4293.2712.1093.4603.3052.0963.5873.4402.0531.9792.87516.95142583/954.6e-14
TEST3.3343.3642.1293.5313.5962.1113.6933.7722.0992.0642.96819.04135191/952.9e-16

Table 18. MASE comparison across all 28 evaluated configurations and 95 settings. Each setting/group column reports the geometric mean over its constituent datasets; TimeMMD* pools the Daily, Weekly, and Monthly native-frequency benchmarks. The relative performance of CoCo under MASE is consistent with its performance under MAE in Tables 1 and 2. Bold and underlined values denote the best and second-best models.

ModelMAEMSEMAPEMAE UnscaledMASE
CoCo5.245.986.395.815.21
Zero-shot Unimodal
TiRex17.268.636.926.976.62
Chronos210.1811.169.8110.119.56
Moirai10.5711.1510.4310.4110.15
TimesFM2.510.7812.099.359.789.60
Toto13.3613.9712.8213.5413.60
TabPFN2.518.2718.7117.3917.8518.35
TimeMoE20.8219.9520.4620.3220.96
PANDA24.2123.5323.7623.9924.03
Zero-shot Multimodal
TimesFM2.5+GPT4o8.358.978.498.768.80
Chronos2+GPT4o9.6510.1610.5810.3710.45
Chronos2 MV13.4214.4511.0911.4112.22
Toto MV14.0414.9211.5812.6013.11
GPT-4o18.2317.6919.3418.9419.79
Qwen3-30B20.9621.1622.0221.4521.17
TimeOmni25.6727.0425.5525.6225.41
Unimodal Fine-tuning
Chronos29.1810.579.809.558.96
AutoTimes11.7511.1711.5611.4111.55
PatchTST13.0612.6115.1114.9614.01
S2IP-LLM14.9813.6116.9316.5716.07
iTransformer20.4918.0121.8421.5821.45
TEST24.1421.9324.9724.9824.94
Multimodal Fine-tuning
Chronos2 MV8.029.408.257.727.47
VoT11.069.6510.8510.9111.57
TaTS11.7913.089.379.9510.61
S2IP-LLM14.1812.9914.9914.5314.36
AutoTimes17.4016.5116.9716.8316.95
TEST18.9316.9319.3919.1219.04

Table 19. CoCo performs consistently across raw and normalized evaluation metrics. Average ranks are computed over the same 95 settings, similar to Table 1, Table 2, and Table 18. Bold and underlined values denote the best and second-best method for each metric.

ModelMedia CloudFNSPIDTimeMMDMedia CloudFNSPIDTimeMMDMedia CloudFNSPIDTimeMMDTimeMMD*MAE ↓Rank ↓Rating ↑Winspp
CoCo (Ours)0.8090.9070.8110.5670.6130.6720.4460.4530.6781.3290.7195.241838––
Zero-shot Unimodal
TiRex10.8350.9150.8240.5800.6160.6960.4500.4500.7381.3520.7357.26175060/955.0e-04
Chronos20.8470.9420.8320.5900.6310.7010.4610.4620.7181.3620.74510.18164381/952.0e-12
Moirai0.8430.9230.8430.5760.6470.7080.4610.4680.7261.3790.74710.57163878/959.2e-13
TimesFM2.50.8610.9410.8240.5970.6430.6990.4680.4690.7321.3420.74910.78162972/957.8e-09
Toto0.8650.9550.8360.5890.6600.7100.4730.4840.7371.3580.75813.36154880/958.1e-13
TabPFN2.50.9080.9680.8640.6470.6620.7240.6320.5380.7701.3900.80018.27138590/951.4e-16
TimeMoE1.0011.0500.9390.6470.7100.7680.6530.5740.7501.3780.83920.82129288/953.3e-15
PANDA1.0721.1331.0600.7700.7900.8560.5650.5470.8121.4990.90324.21113493/954.1e-17
Zero-shot Multimodal
TimesFM2.5+GPT-4o0.8290.8980.8570.5750.6250.7200.4560.4560.7421.3850.7428.35171260/955.0e-03
Chronos2+GPT-4o0.8310.9100.8640.5820.6290.7180.4600.4650.7341.4120.7489.65166465/956.3e-05
Chronos2 MV0.8920.9470.8700.6070.6390.7150.5160.4750.7211.3410.76313.42154274/951.8e-10
Toto MV0.9020.9620.8440.6000.6680.7060.4990.4870.7191.3480.76614.04151875/956.1e-10
GPT-4o0.9130.9661.0350.6440.6800.8530.5130.5020.8451.5860.83618.23138885/958.9e-15
Qwen3-30B0.9490.9851.0800.6910.7170.9270.5460.5390.9021.6310.87820.96129394/952.8e-17
TimeOmni1.4141.1261.5151.0990.7368.5430.7600.5641.4901.7481.75625.67105495/952.6e-17

Table 1. CoCo outperforms SOTA zero-shot unimodal and multimodal forecasting approaches on real-world multimodal datasets. CoCo (ours) achieves the best mean MAE across 30 datasets (95 settings) spanning diverse domains and context lengths. It outperforms SOTA TSFMs such as TiRex1 in over 63% (60/95) of settings. Similarly, CoCo surpasses constructed SOTA zero-shot multimodal baselines (TimesFM2.5 + GPT-4o) in 60/95 settings with a significant improvement in the aggregate MAE and the average rank metrics, highlighting effective sim-to-real transfer.

ModelMedia CloudFNSPIDTimeMMDMedia CloudFNSPIDTimeMMDMedia CloudFNSPIDTimeMMDTimeMMD*MAE ↓Rank ↓Rating ↑Winspp
CoCo (Ours)0.8090.9070.8110.5670.6130.6720.4460.4530.6781.3290.7195.241838––
Unimodal Fine-tuning
Chronos20.8510.9240.8100.5830.6370.6700.4600.4690.6751.3890.7389.18167764/951.2e-05
AutoTimes0.8510.9160.9010.5850.6330.7730.5150.4730.7841.3370.76311.75158878/953.2e-10
PatchTST0.8580.9270.9030.6030.6380.7620.4690.4920.7721.3320.76413.06155989/957.2e-15
S2IP-LLM0.8930.9390.8920.6190.6350.7360.4920.5000.7681.3630.77314.98149187/953.5e-15
iTransformer0.8940.9620.9200.6940.7560.8120.5410.5790.8081.3540.82320.49130092/956.1e-17
TEST0.9621.0401.0260.6790.7310.8600.5880.6090.8801.5900.88224.14114594/952.7e-17
Multimodal Fine-tuning
Chronos2 MV0.8450.9050.8340.5730.6250.6970.4620.4610.7021.3340.7348.02171468/951.8e-05
VoT0.8360.8880.8850.5940.6280.7530.5300.4880.7451.3390.75611.06161471/951.5e-08
TaTS0.8830.9360.8480.5940.6370.7020.5090.4730.7071.3430.75411.79159270/951.3e-08
S2IP-LLM0.8580.9100.8960.6100.6470.7580.5280.5460.7711.3380.77514.18151885/951.3e-13
AutoTimes0.8840.9490.9730.6260.6630.8350.5520.5200.7901.3500.80117.40141483/952.3e-13
TEST0.8570.9700.9770.6360.7060.8400.5560.5440.8131.3900.81518.93135991/952.7e-16

Table 2. CoCo outperforms SOTA compute-matched and fine-tuned forecasting approaches on real-world multimodal datasets. Similar to Table 1, here, we show CoCo’s improvements over fine-tuned and compute-matched baselines. Observe how, even after fine-tuning, both unimodal and multimodal baselines struggle to match CoCo’s performance.

GroupSettingDatasetNCoCoTimesFM2.5+GPT4oChronos2+GPT4oChronos2 MVToto MVGPT-4oQwen3-30BTimeOmniGPT-5
Media Cloud64-16Crude Oil Price4210.8770.8170.8310.8530.8160.9301.0271.1600.859
S&P500 Index3840.7970.8270.8430.9440.9980.8960.8550.9790.766
Apple Stock4240.8720.9480.9451.0081.0031.0430.9611.3250.854
Copper Futures1790.7750.7840.7690.8930.9140.8240.8241.0040.787
Egg Price1440.7010.6510.6690.6730.6440.7140.7780.9020.729
Electricity Price4440.7980.8540.8580.8310.8720.9480.9071.1890.829
SPDR Gold Price4220.8490.8660.8441.0101.0660.8950.9764.5650.824
Google Stock4230.7300.7570.7480.8810.8580.8220.8690.8760.757
NVIDIA Stock Price4230.8830.9390.9500.9660.9841.0841.3301.0210.948
Silver Price4400.8860.8950.9071.0031.0280.9820.9661.1310.888
Steel Rebar Futures1620.7320.7790.7770.7460.7390.9070.9421.4000.828
Media Cloud128-16Crude Oil Price3570.7140.6340.6570.7270.6580.7210.9000.9040.710
S&P500 Index3200.5620.6070.6170.6410.6560.7000.6500.8110.574
Apple Stock3600.6100.6390.6570.6570.6420.7370.7570.7680.630
Copper Futures1150.5890.5860.6010.6040.5650.6260.6490.7020.602
Egg Price800.5060.5060.4740.5160.5100.4840.5660.7160.520
Electricity Price3800.5310.5850.6090.6060.6130.6860.6310.7890.579
SPDR Gold Price3580.4860.5100.4920.6170.6100.5260.5853.8840.488
Google Stock3590.5460.5660.5750.5490.5730.6430.6780.6850.568
NVIDIA Stock Price3590.5410.5880.5900.5780.5950.6980.7640.6470.580
Silver Price3760.6650.6800.6890.7620.7710.7760.7910.8370.695
Steel Rebar Futures980.4830.4270.4420.4230.4080.4830.6311.3490.482
Media Cloud256-16Crude Oil Price2290.5090.4230.4480.5400.4630.4880.5990.5980.483
S&P500 Index1920.4220.4680.4880.4930.5190.5740.4590.5640.435
Apple Stock2320.4280.4590.4720.4800.4730.5310.5430.5350.440
Electricity Price2520.3990.4450.4560.4560.4540.5230.4670.5750.445
SPDR Gold Price2300.3050.3300.3030.3540.3530.3260.3891.9550.293
Google Stock2310.5400.5340.5340.5780.5740.5730.7680.6180.519
NVIDIA Stock Price2310.4160.4460.4350.5030.4850.4940.5610.5330.457
Silver Price2480.5470.5440.5470.7230.6700.5920.5840.7040.561
Overall Wins1632040005
Mean0.6230.6360.6410.6870.6840.7080.7471.1240.638
ELO174516771633144014871381131111731654
Average Rank2.272.973.475.675.176.437.2311.833.20
Head-to-Head Wins–22/3020/3027/3024/3028/3030/3030/3021/30
Wilcoxon p–0.0670.0283.2e-063.4e-041.0e-071.9e-091.9e-090.015

Table 20. Dataset-level MAE for 16-step forecasts across context lengths 64, 128, and 256 on the Media Cloud subset, including GPT-5 as an additional LLM baseline. Media Cloud, comprising datasets from 2024–2025, is the only subset where GPT-5 has limited prior exposure to test trends. GPT-5 performs competitively with strong TSFMAs, yet CoCo outperforms GPT-5 in 70% (21/30) of settings, demonstrating that a fine-tuned 4B LLM providing forecast-space covariates to Chronos2 surpasses a closed-source model used as a standalone forecaster. Bold and underlined values denote the best and second-best method in each row.

LLMLLM Forecast θLLM(XtH,St)\theta_{\mathrm{LLM}}(X_t^H,S_t) MAE (↓)LLM Forecast θLLM(XtH,St)\theta_{\mathrm{LLM}}(X_t^H,S_t) Rank (↓)Chronos2 Forecast θM−TSFM(XtH)\theta_{\mathrm{M-TSFM}}(X_t^H) MAE (↓)Chronos2 Forecast θM−TSFM(XtH)\theta_{\mathrm{M-TSFM}}(X_t^H) Rank (↓)CoCo θM−TSFM(XtH∥Mt)\theta_{\mathrm{M-TSFM}}(X_t^H\Vert M_t) MAE (↓)CoCo θM−TSFM(XtH∥Mt)\theta_{\mathrm{M-TSFM}}(X_t^H\Vert M_t) Rank (↓)CoCo θM−TSFM(XtH∥Mt∥Ct)\theta_{\mathrm{M-TSFM}}(X_t^H\Vert M_t\Vert C_t) MAE (↓)CoCo θM−TSFM(XtH∥Mt∥Ct)\theta_{\mathrm{M-TSFM}}(X_t^H\Vert M_t\Vert C_t) Rank (↓)
Zeroshot1.2708.8420.9800.1.0282.5791.1104.676
Qwen3-4b Fine-tuned1.0783.8160.980-0.9782.6320.9642.684
Chronos2 Fine-tuned1.2708.8420.9592.3420.9562.4590.9552.459
Both Fine-tuned1.0783.8160.9592.3420.9542.2630.9472.184

Table 22. Effect of fine-tuning and the usefulness of the covariates. Extended version of Table 3 showing MAE and average rank over the 30 datasets for the 64-16 settings and the 8 TimeMMD datasets with their native settings (38 settings in total) for different fine-tuning settings and covariate inputs. ‘-’ indicates that the rank is not valid since Chronos2 is one of the baselines.

ModelMean RCRPS ↓\downarrow
Qwen3-4B0.358
Qwen3-4B + Reasoning0.189
Qwen3-4B (CoCo)0.301
Qwen3-4B (CoCo) + Reasoning0.161
Chronos2 (Unimodal)0.265
CoCo0.256
GPT-4o (reference)0.191

Table 4. LLMs fine-tuned with CoCo have improved text-time series alignment and outperform their pretrained counterparts. Notably, Qwen3-4B fine-tuned with CoCo, with reasoning, matches the zero-shot results from GPT-4o. We refer to Table 8 for the detailed results.

ModelAverage RCRPS ↓\downarrowIntemporal InformationHistorical InformationFuture InformationCovariate InformationCausal Information
Qwen3-4b (Base)0.358 ± 0.0310.390 ± 0.0450.247 ± 0.0230.301 ± 0.0420.310 ± 0.0350.700 ± 0.149
Qwen3-4b (CoCo Fine-tuned)0.301 ± 0.0270.306 ± 0.0400.219 ± 0.0190.276 ± 0.0360.280 ± 0.0330.675 ± 0.142
Qwen3-4b (Base) + Reasoning0.189 ± 0.0360.159 ± 0.0110.134 ± 0.0110.088 ± 0.0120.204 ± 0.0470.523 ± 0.214
Qwen3-4b (CoCo Fine-tuned) + Reasoning0.161 ± 0.0130.154 ± 0.0100.130 ± 0.0120.090 ± 0.0120.171 ± 0.0160.378 ± 0.064
Chronos2 (Unimodal)0.265 ± 0.0160.254 ± 0.0210.166 ± 0.0170.293 ± 0.0310.239 ± 0.0180.438 ± 0.066
CoCo (Fine-tuned + Reasoning)0.256 ± 0.0170.242 ± 0.0230.137 ± 0.0110.281 ± 0.0290.234 ± 0.0190.438 ± 0.074
GPT-4o (reference)0.191 ± 0.0040.218 ± 0.0070.118 ± 0.0010.121 ± 0.0010.143 ± 0.0030.363 ± 0.011
Llama-3.1-405B (reference)0.143 ± 0.0060.174 ± 0.0100.146 ± 0.0010.075 ± 0.0050.144 ± 0.0080.303 ± 0.037

Table 8. RCRPS on the CiK benchmark [12]. We provide a breakdown of RCRPS by information type, and evaluate the effect of fine-tuning with CoCo on the reasoning capacity of the LLM. Bold denotes the best result among our models per column. Reference numbers for GPT-4o and Llama-3.1-405B are from the official CiK leaderboard (https://github.com/servicenow/context-is-key-forecasting) and shown in italics for context. Lower is better.

Table 12 shows CoCo’s robustness to the stochasticity induced by the LLM rollouts. Table 23 shows that increasing the number of rollouts beyond Nrollouts=5N_{\mathrm{rollouts}} = 5 provides minimal gains. We observe that generating textual annotations from historical numerical values alone produces textual annotations that do not reflect the real forecast trends. On the other hand, future-conditioned textual annotation generation reflects the true forecast trends better (Table 27).

ForecastMAE (mean ± std)
LLM Rollout (×15)0.7787 ± 0.0489
LLM Rollout Median (×3)0.7860 ± 0.0044
CoCo (×3)0.7194 ± 0.0001

Table 12. CoCo’s robustness to LLM rollout stochasticity. Overall MAE across all 95 settings, evaluated across three random seeds. Although individual LLM rollouts exhibit substantial variation, taking the median across five rollouts substantially reduces the variability, and the subsequent CoCo forecast is highly stable.

NrolloutsN_{\mathrm{rollouts}}MAE (↓\downarrow)Rank (↓\downarrow)
50.8451.833
90.8451.867
150.8461.900

Table 23. Effect of NrolloutsN_{\mathrm{rollouts}}, the number of rollouts used to compute confidence covariate on performance. The MAE and the average rank are shown over the 30 datasets for 16-step forecasts with a context length of 64.

AnnotationsDirectionMagnitude TertileArgmax Bucket
Media Cloud (Real)0.5330.3440.301
FNSPID (Real)0.5440.3400.279
Synthetic (LLM generated with ground truth future values)0.5550.3160.272
Synthetic (LLM generated without ground truth future values)0.4910.3760.346

Table 27. Synthetic textual annotations that depend on numeric history alone provide incorrect information about the true future. In this table, we show the accuracy of linear probes for predicting the domain-agnostic forecast statistics, such as the direction, magnitude tertile, and the argmax location/bucket. Note that the linear probes take an LLM’s embeddings of the context as input. In our training, we utilize synthetic annotations/context that is synthetically generated (Section 4). Here, we ablate the synthetic annotation generation strategy. We show two approaches: one where the numeric future is used to generate the annotation (row 3) and one where the numeric future is not used (row 4). We compare the accuracies of predicting domain-agnostic forecast statistics from these two variants of synthetic annotations with the accuracies of predicting the same from real-annotations. Prefix-only (without the numeric future, row 4) annotations are generated using only domain information and the observed history, without access to forecast values. Compared with real and future-conditioned annotations, they yield lower direction accuracy but substantially higher magnitude-tertile and argmax-bucket accuracies, indicating an out-of-distribution probe profile relative to the real text. This mismatch suggests that prefix-only generation may weaken sim-to-real transfer and motivates our randomized-context design (Section 4) as a middle ground between limiting future leakage and preserving real-world text statistics. However, one key limitation of this experiment is that the probes are linear, and more non-linear approaches could extract better forecast statistics from prefix-only annotations as well.

Conclusion and Future Work

In this work, we first proposed a simple yet effective recipe for generating large-scale multimodal time series datasets. Specifically, we focused on text as the alternative modality. Secondly, we proposed Context as Covariates, a novel multimodal architecture for incorporating orthogonal textual information to refine the forecasts from SOTA TSFMs. CoCo achieves up to a 12% MAE improvement over state-of-the-art TSFMs and LLM forecasters. As future work, we aim to add a text reliability assessment step to CoCo to improve the unimodal forecast only when text is useful and not misleading. This can be empirically assessed using the relationship between the text and time series in the context or history. Additionally, the current setup involves fine-tuning the LLM and M-TSFM sequentially. To this end, we aim to explore the possibility of jointly fine-tuning both models.

Acknowledgments and Disclosure of Funding

This work was supported in part by the National Science Foundation under Grant No. 2148186. Any opinions, findings, and conclusions or recommendations expressed in this material are those of the authors and do not necessarily reflect the views of the National Science Foundation.

Extended Related Works

Multimodal Datasets for Training and Evaluation

Multiple prior works have proposed multimodal time series datasets with primary focus on textual annotations for forecasting. The nature and richness of textual annotations vary widely across domains. In particular, numerous works [21, 22, 16, 23, 24, 25] focus on stock price prediction using auxiliary information, such as news articles, Twitter sentiments, and Google search trends. This is primarily due to the abundance of rich and time-varying textual annotations that describe the trends in stock price values, such as the daily opening, closing, high, and low prices. For example, FNSPID [13] contains stock prices and aligned textual annotations for over 4,000 companies from 1999 to 2023. In contrast, textual annotations in the healthcare domain, [26, 27] are typically global and static, in the form of patient medical records for electrocardiogram (ECG) signals. Such datasets have been primarily used for tasks like classification and anomaly detection. Similarly, Zheng et al. [28] have a large collection of performance metrics time series that are logged from complex systems for root-cause analysis. In the weather prediction domain, Chen et al. [29] provide static descriptions of the geography, while Xu et al. [30] focus on time-varying textual annotations that aid in forecasting temperature, rain levels, and other weather metrics. More recently, Zhou et al. [31] proposed Mo-Time, a multimodal dataset with static annotations spanning e-commerce and social media user behavior domains. From these prior works, our key takeaway is that large-scale multimodal datasets are generally domain-specific and localized (annotations are available for a short time frame and are restricted in terms of geographic locations). Additionally, based on the domains, the annotations can be static or time-varying.

TimeMMD [11] and Context-Is-Key [12] provide a collection of multimodal datasets that span healthcare, environment, finance, and retail domains. However, these efforts are restricted in the sizes of their individual datasets, and are mainly suitable for evaluation and not large-scale training. Additionally, curating these datasets is expensive owing to the significant human effort that was required.

Multimodal Forecasting Approaches

We group the existing works into zero-shot or training-free and dataset-specific training-based approaches. A common methodology for training-free approaches involves repurposing LLMs for time series forecasting, given their ability to process textual inputs efficiently.

Zero-Shot Approaches

Guo et al. [32] perform multimodal traffic flow prediction by converting information from other modalities into textual prompts for an LLM to capture complex patterns in time series and associate them with textual annotations. Kong et al. [33] fine-tuned LLMs (domain- or dataset-agnostic) with Low-Rank Adaptation (LoRA) for forecasting, imputation, and classification, by converting multimodal time series datasets and tasks into a dataset with questions and answers.

Wang et al. [16] used reasoning agents to filter relevant news articles or textual annotations along with domain- or dataset-agnostic fine-tuning and evaluation steps that update LLMs to effectively utilize time series and textual annotations, and evaluate whether all relevant annotations have been retrieved, respectively. The authors show that the proposed approach generalizes for the electricity, finance, and traffic domains.

In the finance domain, Xie et al. [34] evaluated ChatGPT’s [10] ability to forecast stock price movements. However, the results indicate that this approach does not perform better than multimodal approaches based on domain-specific training for the same task. Yu et al. [15] focused on price prediction of NASDAQ-100 stocks using historical stock price data, company metadata, and financial news, with LLMs.

Kim et al. [17] proposed Time Text Corpus, a multimodal dataset with around 10,000 timestamps from the weather and the healthcare domains. Additionally, the authors introduced the Hybrid Multi-Modal Forecaster (Hybrid-MMF) which involves a two-stage training setup to jointly predict text and time series. However, the experiments are limited in both context (history, up to 7 timestamps) and forecast lengths.

Dataset-Specific Training-Based Approaches

Li et al. [40] employed a learnable text encoder to project the time-varying textual annotations into the embedding space of learnable time series forecasting models, thereby enabling multimodal training for forecasting.

Li et al. [36] presented UrbanGPT, a hybrid approach that integrates a spatiotemporal dependency encoder with the instruction-tuning paradigm in LLMs with the objective of forecasting traffic flow patterns in New York. This provides the ability for LLMs to understand complex spatial and temporal relationships in multivariate time series datasets.

Liu et al. [37] demonstrate the ability of LLMs to perform classification and prediction for physiological and medical time series using a few-shot tuning approach.

Chattopadhyay et al. [44] introduce ContextFormer, a dataset-specific fine-tuning procedure for unimodal time series forecasting models with textual annotations provided as embedding inputs from open-source LLMs.

Jia et al. [38] fine-tuned a pretrained GPT2 [58] model with time series and textual annotations for large-scale events datasets, curated using the Global Database of Events, Language, and Tone (GDELT) tool. Jia et al. [38] generated relevant time-varying textual annotations from GDELT by filtering the required news articles using cosine similarity with respect to the natural language description of the forecasting target. The quantitative results indicate that the use of highly useful text annotations indeed helps in forecasting events more accurately than unimodal models trained on the same dataset.

Liu et al. [43] employed a cross-modal alignment setup between the time series and text embeddings, for textual annotations that describe the trend and seasonality in the time series. Here, Liu et al. [43] trained a time series embedder from scratch and utilized off-the-shelf LLMs to extract text embeddings. Though the approach outperforms other SOTA trainable approaches, the text annotations mainly provide a summary of the time series and do not provide auxiliary information. Similarly, Ekambaram et al. [59] proposed an attention-based encoder-decoder framework for image-based product sales forecasting, while also including holidays, discounts, etc.

Jiang et al. [20] proposed TimeXL, where a multimodal encoder processes both time series and textual inputs to generate initial forecasts and aided explanations. Then, a prediction LLM refines the forecasts by reasoning over or aligning the time series predictions with the explanations. Consequently, a reflection LLM compares the predicted forecast against the ground truth to identify textual inconsistencies. Overall, TimeXL proposes a closed-loop framework, where the prediction LLM aligns the text and time series predictions, and the refinement LLM iteratively enhances text quality.

TimeLLM [18] learns patch embeddings for time series to align time series representations with the token embedding space of LLMs, so that LLMs can be utilized to forecast or predict the next time series patch. Additionally, TimeLLM deals with static textual annotations, which are the descriptions of the domain and the geography corresponding to the time series dataset. However, this approach does not deal with complex time-varying annotations that provide auxiliary information to aid forecasting.

TimeCAP [45] utilizes two LLM agents, one summarizes the time series and the other uses aligned representations (learned using a multimodal encoder) of time series and its summary to forecast. Similar to other training-based approaches, TimeCAP performs dataset-specific training.

UniTime [41] and Hong et al. [42] deal with utilizing dataset descriptions (static textual annotations) to better learn time series forecasting for individual domains. Again, these annotations usually describe the dataset domain and the geographic region corresponding to where the dataset was collected, instead of time-varying auxiliary information, as in the case of finance datasets. Hong et al. [42] also generate similar synthetic annotations for time series windows. However, they do not show the forecast values as inputs and primarily summarize the historical values.

Recently, UniCA [39] proposed a unified covariate adaptation framework for TSFMs, where heterogeneous modalities such as text, categorical metadata, and numerical covariates are projected into a shared temporal representation space and fused into pretrained TSFMs using lightweight attention-based adapters. While UniCA enables covariate-aware forecasting across domains, the approach still relies on dataset-specific fine-tuning.

TimeVLM [50] uses pre-trained Vision-Language Models (VLMs), where the language encoders capture contextual information and the vision encoders capture trend and seasonality using time series features represented as images. The approach learns the parameters of the fusion layers that enrich time series representations with the image and the language embeddings for each dataset. Again, the annotations are restricted to the descriptions of the dataset domain. Daswani et al. [47] and Li et al. [48] also suggest similar empirical conclusions regarding viewing time series as images or plots. Daswani et al. [47] benchmarked the performances of the SOTA LLMs, such as GPT-4o [10], in various time series tasks, including classification, detection, and prediction. Similarly, Li et al. [48] used pre-trained vision encoders for time series classification tasks. Zhou and Yu [49] explored anomaly detection using multimodal LLMs with time series inputs in the form of images.

Sawhney et al. [46] proposed an early fusion approach that combines learnable representations for time series and text annotations to predict the stock price movements. Similarly, Chen et al. [25] employed ChatGPT [10] to predict possible stocks that might be affected for the next day, along with the direction of change (positive or negative). This is combined with learnable time series embeddings, obtained from historical time series values, to predict the movements.

Narasimhan et al. [60, 61] and Ge et al. [62] focus on developing dataset-specific models for multimodal time series generation. Here, note that the inputs provided to the model are non-causal, i.e., the inputs describe the values of all the timestamps in the time series to be generated. In contrast, to evaluate forecasting, we require annotations that are causal and contain predictions or speculations about the forecast timestamps.

Although most of the approaches mentioned above reuse SOTA LLMs for forecasting, Park et al. [14] highlighted the key issues that limit the ability of LLMs, such as sensitivity to noise. Additionally, Park et al. [14] show the inferior performance of LLMs in comparison to light-weight linear models (DLinear) on standard time series forecasting benchmarks.

From all these approaches, we find some common trends: (i) Zero-shot / few-shot approaches mainly rely on the predictive power of LLMs; (ii) Most approaches are domain-specific and fine-tune LLMs/VLMs for individual datasets or domains; (iii) Previous works did not utilize powerful TSFMs such as Chronos2 [8], TimesFM2.5 [7], TabPFN-TS [63]. We differentiate our approach from these prior works by seeking to develop a zero-shot solution that corrects/adapts unimodal forecasts from SOTA TSFMs using textual annotations. In addition, we are seeking to develop a framework that can work in multiple domains.

Proofs for Theorems in Section 4

Under our latent model, we have history, forecast description, and the forecast sampled from p(XH,Z,XF)p(X^{H}, Z, X^{F}). Real annotations are sampled as Sr∼pr(Sr∣XH,Z)S^{r} \sim p_{r}(S^{r} \mid X^{H}, Z), while synthetic annotations are sampled as Ss∼ps(Ss∣XH,Z,XF)S^{s} \sim p_{s}(S^{s} \mid X^{H}, Z, X^{F}). We define the real and synthetic population risks as

Rmultir(θ)=E(XH,Z,XF)∼p, Sr∼pr(⋅∣XH,Z)[M(hθmulti(XH,ϕ(Sr)),XF)].(8)R_{\mathrm{multi}}^{r}(\theta) = \mathbb{E}_{(X^{H},Z,X^{F})\sim p,\ S^{r}\sim p_{r}(\cdot\mid X^{H},Z)}\left[\mathcal{M}\left(h_{\theta}^{\mathrm{multi}}\left(X^{H},\phi(S^{r})\right),X^{F}\right)\right]. \tag*{(8)}
Rmultis(θ)=E(XH,Z,XF)∼p, Ss∼ps(⋅∣XH,Z,XF)[M(hθmulti(XH,ϕ(Ss)),XF)].(9)R_{\mathrm{multi}}^{s}(\theta) = \mathbb{E}_{(X^{H},Z,X^{F})\sim p,\ S^{s}\sim p_{s}(\cdot\mid X^{H},Z,X^{F})}\left[\mathcal{M}\left(h_{\theta}^{\mathrm{multi}}\left(X^{H},\phi(S^{s})\right),X^{F}\right)\right]. \tag*{(9)}

Here, note that real annotations are causal and carry information about the forecast trends in domain-specific terms. Therefore, there is no conditioning on XFX^{F} in Sr∼pr(Sr∣XH,Z)S^{r} \sim p_{r}(S^{r} \mid X^{H}, Z). However, during training, the synthetic annotations are generated using an LLM which considers the history XHX^{H} and the forecast XFX^{F} to generate the domain-specific description. Additionally, we note that ZZ can be extracted from XFX^{F} easily by computing certain domain-agnostic statistical features. Therefore, for synthetic annotations, we have Ss∼ps(Ss∣XH,Z,XF)S^{s} \sim p_{s}(S^{s} \mid X^{H}, Z, X^{F}).

For analysis, we also define the risk with the true domain-agnostic forecast description.

RmultiZ(θ)=E(XH,Z,XF)∼p[M(hθmulti(XH,ψ(Z)),XF)].(10)R_{\mathrm{multi}}^{Z}(\theta) = \mathbb{E}_{(X^{H},Z,X^{F})\sim p}\left[\mathcal{M}\left(h_{\theta}^{\mathrm{multi}}\left(X^{H},\psi(Z)\right),X^{F}\right)\right]. \tag*{(10)}

Proof of Lemma 4.3. We first introduce RmultiZ(θ)R_{\mathrm{multi}}^{Z}(\theta).

∣Rmultir(θ)−Rmultis(θ)∣=∣Rmultir(θ)−RmultiZ(θ)+RmultiZ(θ)−Rmultis(θ)∣(11)\left|R_{\mathrm{multi}}^{r}(\theta)-R_{\mathrm{multi}}^{s}(\theta)\right|=\left|R_{\mathrm{multi}}^{r}(\theta)-R_{\mathrm{multi}}^{Z}(\theta)+R_{\mathrm{multi}}^{Z}(\theta)-R_{\mathrm{multi}}^{s}(\theta)\right| \tag*{(11)}

Now, using the triangle inequality, we have

∣Rmultir(θ)−Rmultis(θ)∣≤∣Rmultir(θ)−RmultiZ(θ)∣+∣RmultiZ(θ)−Rmultis(θ)∣.(12)\left|R_{\mathrm{multi}}^{r}(\theta)-R_{\mathrm{multi}}^{s}(\theta)\right|\leq\left|R_{\mathrm{multi}}^{r}(\theta)-R_{\mathrm{multi}}^{Z}(\theta)\right|+\left|R_{\mathrm{multi}}^{Z}(\theta)-R_{\mathrm{multi}}^{s}(\theta)\right|. \tag*{(12)}

For the first term, we have

∣Rmultir(θ)−RmultiZ(θ)∣=∣Ep,pr[M(hθmulti(XH,ϕ(Sr)),XF)−M(hθmulti(XH,ψ(Z)),XF)]∣≤Ep,pr[∣M(hθmulti(XH,ϕ(Sr)),XF)−M(hθmulti(XH,ψ(Z)),XF)∣]≤LEp,pr[∥ϕ(Sr)−ψ(Z)∥](13)\begin{aligned} \left|R_{\mathrm{multi}}^{r}(\theta)-R_{\mathrm{multi}}^{Z}(\theta)\right| &=\left|\mathbb{E}_{p,p_{r}}\left[\mathcal{M}\left(h_{\theta}^{\mathrm{multi}}\left(X^{H},\phi(S^{r})\right),X^{F}\right)-\mathcal{M}\left(h_{\theta}^{\mathrm{multi}}\left(X^{H},\psi(Z)\right),X^{F}\right)\right]\right|\\ &\leq\mathbb{E}_{p,p_{r}}\left[\left|\mathcal{M}\left(h_{\theta}^{\mathrm{multi}}\left(X^{H},\phi(S^{r})\right),X^{F}\right)-\mathcal{M}\left(h_{\theta}^{\mathrm{multi}}\left(X^{H},\psi(Z)\right),X^{F}\right)\right|\right]\\ &\leq L\mathbb{E}_{p,p_{r}}\left[\left\|\phi(S^{r})-\psi(Z)\right\|\right] \tag*{(13)} \end{aligned}
≤Lϵr.(14)\leq L\epsilon_{r}. \tag*{(14)}

Here the first inequality is from Jensen’s inequality, the second inequality follows from Assumption 4.2 and the last inequality follows from Assumption 4.1.

Similarly, for the second term in Equation (12), we have

∣RmultiZ(θ)−Rmultis(θ)∣≤LEp,ps[∥ϕ(Ss)−ψ(Z)∥]≤Lϵs.(15)\left|R_{\mathrm{multi}}^{Z}(\theta)-R_{\mathrm{multi}}^{s}(\theta)\right|\leq L\mathbb{E}_{p,p_{s}}\left[\left\|\phi(S^{s})-\psi(Z)\right\|\right]\leq L\epsilon_{s}. \tag*{(15)}

Combining the two bounds gives

∣Rmultir(θ)−Rmultis(θ)∣≤L(ϵr+ϵs).(16)\left|R_{\mathrm{multi}}^{r}(\theta)-R_{\mathrm{multi}}^{s}(\theta)\right|\leq L(\epsilon_{r}+\epsilon_{s}). \tag*{(16)}

Proof of Theorem 4.4. We decompose Rmultir(θ^s)−Rmultir(θr∗)R_{\mathrm{multi}}^{r}(\hat{\theta}_{s})-R_{\mathrm{multi}}^{r}(\theta_{r}^{*}) as

Rmultir(θ^s)−Rmultir(θr∗)=Rmultir(θ^s)−Rmultis(θ^s)Rmultir(θ^s)−Rmultis(θ^s)⏟(I)(17)R_{\mathrm{multi}}^{r}(\hat{\theta}_{s})-R_{\mathrm{multi}}^{r}(\theta_{r}^{*}) =R_{\mathrm{multi}}^{r}(\hat{\theta}_{s})-R_{\mathrm{multi}}^{s}(\hat{\theta}_{s}) \underset{\mathrm{(I)}}{\underbrace{\phantom{R_{\mathrm{multi}}^{r}(\hat{\theta}_{s})-R_{\mathrm{multi}}^{s}(\hat{\theta}_{s})}}} \tag*{(17)}
+Rmultis(θ^s)−Rmultis(θr∗)Rmultis(θ^s)−Rmultis(θr∗)⏟(II)(18)+R_{\mathrm{multi}}^{s}(\hat{\theta}_{s})-R_{\mathrm{multi}}^{s}(\theta_{r}^{*}) \underset{\mathrm{(II)}}{\underbrace{\phantom{R_{\mathrm{multi}}^{s}(\hat{\theta}_{s})-R_{\mathrm{multi}}^{s}(\theta_{r}^{*})}}} \tag*{(18)}
+Rmultis(θr∗)−Rmultir(θr∗).Rmultis(θr∗)−Rmultir(θr∗)⏟(III)(19)+R_{\mathrm{multi}}^{s}(\theta_{r}^{*})-R_{\mathrm{multi}}^{r}(\theta_{r}^{*}). \underset{\mathrm{(III)}}{\underbrace{\phantom{R_{\mathrm{multi}}^{s}(\theta_{r}^{*})-R_{\mathrm{multi}}^{r}(\theta_{r}^{*})}}} \tag*{(19)}

Term (II) is non-positive since θ^s\hat{\theta}_s minimizes RmultisR^s_{\mathrm{multi}} over Θ\Theta. By Lemma 4.3, terms (I) and (III) are each bounded above by L(εr+εs)L(\varepsilon_r+\varepsilon_s). Therefore,

Rmultir(θ^s)≤Rmultir(θ⋆)+2L(εr+εs).(20)R^r_{\mathrm{multi}}(\hat{\theta}_s) \le R^r_{\mathrm{multi}}(\theta^\star) + 2L(\varepsilon_r+\varepsilon_s). \tag*{(20)}

Observe from Table 5 that changing the text only affects the MAE numbers in the third decimal for the LLM forecast as well as CoCo. To this end, we note that for reasonable forecasters, the smoothness property is generally satisfied in practice. As we explained in Section 4, the key ingredient for sim-to-real transfer is the proximity of real and synthetic annotations to the real future description in some latent space. Given the existence of foundation models that can act as powerful encoders, if the annotations contain speculations of the expected trend in the forecast, it is fair to expect small values for εs\varepsilon_s and εr\varepsilon_r, indicating good proximity in the latent space. Consequently, the bounded gap of 2L(εr+εs)2L(\varepsilon_r+\varepsilon_s) tends to zero for very small εs\varepsilon_s and εr\varepsilon_r, indicating that the model trained on synthetic annotations will perform as well as the model trained on real annotations. Additionally, our claim of real and synthetic annotations capturing similar domain-agnostic predictions in Section 4 is visually depicted in Figure 9.

t-SNE plots comparing real and synthetic annotations in domain-agnostic and domain-specific cases

Figure 9. Real and synthetic annotations contain similar domain-agnostic descriptions, but differ in semantics. Here, we show the t-SNE plots for concatenated embeddings of real and synthetic annotations. We consider the Media Cloud datasets for which the real annotations are available, and we generated synthetic annotations for those datasets. On the left, we compare the embeddings of real domain-agnostic predictions concatenated with true forecast descriptions against the synthetic counterpart. Similarly, on the right, we compare the embeddings of the real domain-specific prediction concatenated with the true forecast descriptions against the synthetic counterpart. Note that there is near-perfect overlap in the domain-agnostic case and clear separation in the domain-specific case. This conveys that for the same true forecast description (through concatenation), the domain-agnostic predictions overlap indicating that real and synthetic annotations provide similar domain-agnostic prediction. In contrast, the domain-specific predictions vary significantly, indicating that both real and synthetic domain-specific predictions have different semantic constructions.

DatasetLLM Forecast (θLLM(XtH,St)\theta_{\mathrm{LLM}}(X_t^H, S_t))CoCo (θM−TSFM(XtH∥Mt∥Ct)\theta_{\mathrm{M-TSFM}}(X_t^H \Vert M_t \Vert C_t))
Raw0.8780.858
Rephrased0.8750.859

Table 5. CoCo is less sensitive to text rephrasing. We observe that the MAE of the LLM and the MAE of CoCo change marginally with rephrasing as long as the text conveys the same information. This validates Assumption 4.2 (smoothness assumption) indicating that small perturbations in the context StS_t do not significantly affect the forecasting performance.

Assumption Validity Discussion

Assumption 4.1 focuses on the proximity of domain-agnostic forecast descriptions inferred from real and synthetic annotations to the domain-agnostic description of the true future in a common representation space. In Figure 8, we show the strong overlap in cosine similarity distributions between the embeddings of the verbal description of the forecast predictions derived from real and synthetic annotations. Note that the cosine similarity is with respect to the true future descriptions. As mentioned in Section 4, this indicates that synthetic annotations recover the forecast nearly as well as real ones, suggesting εs≈εr\varepsilon_s \approx\varepsilon_r. Additionally, the generation process grounds synthetic text in numerical trajectories and domain descriptions. In our experiments, the embeddings correspond to normalized BERT embeddings, and consequently, the cosine similarity and the Euclidean distance (used in the assumption) are equivalent measures.

To this end, we directly quantify these representation distances on our real-world evaluation datasets using the same unit-normalized BERT embeddings, on a per-domain basis, in Table 7. Across all domains, synthetic annotations exhibit nearly the same distance to the true forecast description as real annotations. Therefore, the significant overlap in the cosine similarity distribution (Figure 8) with a very high mean (∼0.93\sim0.93) and the similar representation distances between the real and synthetic annotations from the true forecast descriptions (Table 7) convey the same idea that real and synthetic annotations encode domain-agnostic information similar to the true forecast description.

DomainMean Distance Between Real and True Forecast Description RepresentationMean Distance Between Synthetic and True Forecast Description RepresentationΔ\Delta Mean
All domains0.37150.3774+0.0059
Agriculture0.37180.3735+0.0017
Energy0.37440.3745+0.0001
Macro Indicators0.37200.3805+0.0085
Retail0.37590.3744-0.0015
Finance0.37070.3775+0.0068

Table 7. Our synthetic annotations encode similar information about the true future as the real annotations. We empirically show this in Figure 8. However, as Assumption 4.1 uses the Euclidean distance measure, we also provide a per-domain breakdown of the l2l_2 norm between the real and synthetic predicted representations and the representation of the true forecast description. Here, the representation space used is the unit normalized BERT embedding space.

Additionally, we note that the Equations (13) and (15) require only the expected representation distances to be bounded rather than the per-sample bound stated in Assumption 4.1. Therefore, relaxing Assumption 4.1 to reflect the bound on the expected representation distances between the real and synthetic forecast descriptions and the true forecast descriptions would still be sufficient to prove Lemma 4.3 and Theorem 4.4.

Assumption 4.2 focuses on ensuring that small perturbations in the text embedding produce limited changes in forecasting errors. Our empirical evidence in Table 5, Table 6, and Table 21 shows that local perturbations that do not adversely affect the forecast-relevant semantics cause negligible variation in forecasting error. Conversely, for perturbations that alter the forecast relevant semantics (see Table 6 and Table 21), we observe a significant change in the forecasting performance. Consequently, we note that this empirical result points only towards local smoothness in the loss space with respect to small perturbations in text annotations for reasonably good forecasters rather than the uniform Lipschitz condition currently stated in Assumption 4.2.

Perturbation% Change in the LLM Forecast (θLLM(XtH,St)\theta_{\mathrm{LLM}}(X_t^H, S_t))MAE (↓)
Rephrased0.3
Random Word Deletion (5%)0.8
Less Verbose Annotations1.1
Sentence Reordering1.02
Using Annotations From Other Datasets5.0
Misleading Text7.9

Table 6. CoCo exhibits local smoothness with respect to text perturbations. For simple perturbations that do not affect the forecast trend indicators (first 4 rows), note that the change in the LLM forecast MAE is less than or around 1%, indicating local smoothness towards mild perturbations. However, for perturbations that change the underlying forecast (last two rows), the change is significant.

Overall, the theoretical bound on the risk in Theorem 4.4 remains conditional on these assumptions and should be interpreted as explaining when sim-to-real transfer is expected to hold, rather than guaranteeing it for arbitrary cases.

Synthetic Toy Example Description

We construct a controlled, fully synthetic toy series of length T=1100T = 1100 daily steps that combines a SARIMA-like latent process with discrete, text-driven shocks. The latent baseline ytbasey_t^{\mathrm{base}} is generated recursively from a linear trend, weekly seasonality, and an AR(1) autoregressive component centered on a fixed level:

ytbase=μ+st+ϕ(yt−1base−μ)+Asin⁡(2πtP)+εt,εt∼N(0,σ2),(21)y_t^{\mathrm{base}} = \mu+ s t + \phi\left(y_{t-1}^{\mathrm{base}} - \mu\right) + A\sin\left(\frac{2\pi t}{P}\right) + \varepsilon_t,\qquad\varepsilon_t \sim\mathcal{N}(0,\sigma^2), \tag*{(21)}

with parameters μ=50\mu= 50 (bias), s=0.008s = 0.008 (linear trend slope), ϕ=0.6\phi= 0.6 (AR coefficient), seasonal amplitude A=3.0A = 3.0 and periodicity P=7P = 7, and σ=1.2\sigma= 1.2, initialized at y0base=μy_0^{\mathrm{base}} = \mu.

The text component is added through a curated catalog of 30 discrete events, each represented by a tuple (ts,te,ℓ,label,text)(t_s, t_e, \ell, \text{label}, \text{text}) with start day tst_s, end day tet_e, signed sentiment level ℓ∈[−1,1]\ell\in[-1,1], a short tag, and a natural-language description of the event in an analyst style. The first 25 events are used for training and the last 5 are reserved as held-out test windows. From these events we construct a piecewise-constant sentiment signal ZtZ_t for length TT. Every KK timestamps before the start and after the end of the event, ZtZ_t linearly increases from 0 to ℓ\ell and linearly decreases from ℓ\ell to 0, respectively. During the event ZtZ_t stays constant at ℓ\ell. We set KK to 4. Consequently, the observed series is

Yt=ytbase+βZt,(22)Y_t = y_t^{\mathrm{base}} + \beta Z_t, \tag*{(22)}

with text coupling strength β=8.0\beta= 8.0, which makes the text-driven deviations clearly visible against the natural fluctuations of the base signal as described in Equation (21).

We first consider training on real and testing on real (TRTR). By training, we refer to regressing the parameters μ\mu, ss, ϕ\phi, AA, and β\beta. We use the least_squares function from scipy. This allows us to obtain the estimate of the underlying model used for data generation. With this model, we simulate the held-out test horizon and compute the MAE for test event timestamps. During training, we do not utilize the true sentiment levels. Instead, we use Sentence Transformer to estimate the sentiment level.

To showcase the effectiveness of training on synthetic annotations, we use LLMs to annotate the events, using the same approach described in Section 4. We extract sentiment levels from the generated synthetic annotations using the same approach as in TRTR and estimate μ\mu, ss, ϕ\phi, AA, and β\beta. Consequently, we use these estimated parameters to simulate the test events with sentiments extracted from real text and compute the MAE (TSTR).

The results in Figure 3 show that the TRTR and TSTR MAEs are very similar, thereby establishing the veracity of our sim-to-real approach for multimodal forecasting, for this specific toy scenario.

Evaluation on Context is Key (CiK)

Context is Key (CiK) [12] is a time series forecasting benchmark that pairs numerical data with diverse types of carefully crafted textual context, where crucially every task requires understanding textual context to be solved successfully. We evaluate on CiK to understand the behavior of CoCo’s components in a setting that is deliberately adversarial to unimodal numerical forecasting. We first establish where Chronos2 stands on the CiK leaderboard as a unimodal baseline. Chronos2 achieves an average RCRPS of 0.265, outperforming Chronos-Large (0.326), TimeGEN (0.353), Moirai-Large (0.520), all statistical baselines (ARIMA: 0.475, ETS: 0.530), and the majority of LLM-based methods on the leaderboard. This is notable given that CiK is explicitly designed so that numerical history alone is insufficient.

Our key takeaway from the results is that LLMs fine-tuned with CoCo consistently outperform their base, pretrained counterpart. Additionally, a Qwen3-4B model that is fine-tuned with CoCo, when prompted for reasoning, can match the performance of powerful closed-source LLMs, such as GPT-4o.

Table 8 reports the average RCRPS (Region of Interest Continuous Ranked Probability Score) on CiK for Qwen3-4b under four configurations: base model without reasoning (as evaluated on CiK), base model with a short chain-of-thought reasoning prompt, CoCo fine-tuned model without reasoning, and CoCo fine-tuned model prompted for reasoning. The reasoning prompt used is:

“Before the forecast, provide a reasoning section between <reasoning> and </rea- soning> tags.”

Adding a chain-of-thought reasoning prompt reduces RCRPS by 47% for the base model (0.358 → 0.189) and by 46% for the fine-tuned model (0.301 → 0.161). This improvement from a single prompt modification substantially exceeds the gain from fine-tuning alone in this setting. This suggests that CiK rewards explicit chain-of-thought reasoning. Fine-tuning provides complementary gains on top of reasoning. While the reasoning prompt dominates, fine-tuning still contributes meaningfully when combined with reasoning (0.189 → 0.161, a further 15% improvement). The fine-tuned model with reasoning matches GPT-4o, despite being much smaller in parameter count. This confirms that GRPO fine-tuning for covariate generation teaches transferable forecasting capabilities that generalize to a held-out benchmark the model was never trained on.

Fundamentally, CiK differs from our main evaluation benchmarks in a key respect: every CiK task is constructed so that textual context is the primary source of predictive signal, with numerical history deliberately insufficient. In this regime, the TSFM backbone adds limited value. In contrast, our evaluation datasets contain naturally occurring noisy text where numerical history remains genuinely informative, the complementary setting that CoCo is designed for, and where CoCo consistently outperforms both the LLM and the TSFM in isolation.

Training

Training Description

We fine-tune θLLM\theta_{\mathrm{LLM}} and θM−TSFM\theta_{\mathrm{M-TSFM}} in two sequential stages, as described in Section 5. Both stages use parameter-efficient LoRA adapters [64] so that the base weights remain frozen at their public checkpoints, and only the adapters are written to disk and exchanged across the training/inference processes.

Window Sampling and Normalization: Each training window is sampled by independently drawing a context length l∈{64,96,128,192,256}l \in\{64, 96, 128, 192, 256\} and a forecast horizon h∈{16,24,32,40}h \in\{16, 24, 32, 40\} uniformly at random, and then choosing a random valid start index in the source series. This exposes both stages to the full range of (l,h)(l, h) combinations rather than a single fixed pair. Sampling longer histories is deliberately included even though Chronos2’s unimodal accuracy improves rapidly with ll. It forces the LLM to remain useful in the regime where the numerical baseline is already strong, and yields a forecast covariate MtM_t whose marginal benefit (Equation (5)) does not collapse as ll grows. For every sampled window, the numerical history XtH=[xt−l+1,…,xt]X_t^H = [x_{t-l+1}, \ldots, x_t] is z-score standardized using per-window statistics μt\mu_t and σt\sigma_t (mean and standard deviation) computed over the context portion only, and the ground truth XtF=[xt+1,…,xt+h]X_t^F = [x_{t+1}, \ldots, x_{t+h}] is normalized with the same (μt,σt)(\mu_t, \sigma_t). The LLM is conditioned on raw (unscaled) values in the prompt but is supervised in the normalized space (i.e., the reward MAE in Section 5 is computed after applying (μt,σt)(\mu_t, \sigma_t) to the parsed forecast), while Chronos2 operates entirely in the per-window normalized space. The evaluation MAEs reported are in the normalized space, so that they are comparable across datasets. We observe that the relative performance of models does not change significantly for metrics computed in the de-normalized space as well (Table 19).

LLM Fine-Tuning: The base LLM θLLM\theta_{\mathrm{LLM}} is initialized from Qwen3-4B-Instruct-2507 and adapted with rank-32 LoRA modules (α=64\alpha= 64, dropout 0, no bias) attached to all attention and MLP projection matrices (qq, kk, vv, oo, gate, up, down) across the 36 transformer blocks, yielding ∼61.6\sim61.6M trainable parameters (∼1.5%\sim1.5\% of the ∼4\sim4B base). We optimize the GRPO objective of Section 5 with Nrollouts=8N_{\mathrm{rollouts}} = 8 rollouts per prompt sampled at temperature 1.0, a prompt batch size of 256, and AdamW at a learning rate of 10−610^{-6}, for up to 500 optimizer steps. The reward rt(g)r_t^{(g)} is computed against the frozen pretrained Chronos2 (amazon/chronos-2) and the resulting GRPO advantage At(g)A_t^{(g)} updates only the LoRA adapters of θLLM\theta_{\mathrm{LLM}}. Training prompts are drawn at a 0.5/0.5 ratio from our synthetic corpus and FNSPID real-world annotations with a periodicity stratifier that balances periodic and aperiodic series. We run the training loop with PrimeRL [65], splitting GPUs into 2 trainers and 2 vLLM [66] inference workers (4 H100s).

Textual Context: Although the LLM prompt always includes the full numerical history XtX_t of length ll, the textual annotation StS_t is restricted to only the most recent Ktrain=10K_{\mathrm{train}} = 10 timesteps during fine-tuning. Concretely, only the text fields associated with {xt−Ktrain+1,…,xt}\{x_{t-K_{\mathrm{train}}+1}, \ldots, x_t\} are inserted into the prompt’s annotation block. Older annotations, if present, are discarded. We deliberately use a smaller KtrainK_{\mathrm{train}} at training time than the Keval=16K_{\mathrm{eval}} = 16 used at evaluation. Textual annotations are dense and verbose, and including all 16 steps in every prompt, in combination with the longest contexts l=256l = 256, the Nrollouts=8N_{\mathrm{rollouts} } = 8 on-policy completions per GRPO update, and LoRA activations stored across transformer blocks, pushed the per-rollout sequence length above the 8,192-token budget that fits within the LoRA fine-tuning memory on our 2-trainer setup. Truncating to Ktrain=10K_{\mathrm{train}} = 10 keeps the per-rollout sequence length comfortably under that budget while preserving the most causally relevant annotations. At inference time, this constraint disappears and we expand the textual window back to Keval=16K_{\mathrm{eval}} = 16, matching the value used by all closed-source LLM baselines for a fair comparison.

Chronos2 Fine-Tuning: Once θLLM\theta_{\mathrm{LLM}} has converged, we pre-generate the covariate set {Mt,Ct}t=1N\{M_t, C_t\}_{t=1}^{N} from Equations (5) and (6) using Nrollouts=5N_{\mathrm{rollouts}} = 5 rollouts per sample for the confidence channel. The forecaster θM−TSFM\theta_{\mathrm{M-TSFM}} is initialized from amazon/chronos-2 (∼119.5\sim119.5M parameters, dmodel=768d_{\mathrm{model}} = 768, dff=3072d_{\mathrm{ff}} = 3072, 12 encoder blocks) and adapted with rank-8 LoRA modules (α=16\alpha= 16, dropout 0, no bias) attached to the self-attention qq, kk, vv, oo projections in every encoder block and to the output projection of the patch-embedding head, totaling ∼1.21\sim1.21M trainable parameters (∼1.0%\sim1.0\% of the base). The base weights are kept frozen and only the LoRA adapters are trained by minimizing 1N∑t=1N∥θM−TSFM(XtH∥Mt∥Ct)−XtF∥2\frac{1}{N}\sum_{t=1}^{N}\left\|\theta_{\mathrm{M-TSFM}}(X_t^H \mathbin{\|} M_t \mathbin{\|} C_t) - X_t^F\right\|^2 with AdamW for 10,000 steps, with model selection by validation MAE on a held-out 5% split. The fine-tuned adapters are merged with the base amazon/chronos-2 weights only at inference time.

Noise Level for Covariate Construction. The forecast covariate MtM_t (Eq. (5)) and confidence covariate CtC_t (Eq. (6)) include i.i.d. noise ϵi∼N(0,σϵ2)\epsilon_i \sim\mathcal{N}(0,\sigma_{\epsilon}^{2}) in the history portion to allow Chronos2 to interpret it as a noisy next-step predictor. Table 9 shows the effect of σϵ\sigma_{\epsilon} on forecasting performance. Neither removing noise entirely (σϵ=0.00\sigma_{\epsilon}=0.00) nor using high noise levels (σϵ=1.00\sigma_{\epsilon}=1.00) yields the best performance. The former provides no signal diversity for Chronos2 to calibrate against, while the latter obscures the next-step information in the history portion. A small noise level of σϵ=0.05\sigma_{\epsilon}=0.05 achieves the best MAE and is used throughout all experiments for both fine-tuning and inference.

σϵ\sigma_{\epsilon}CoCo MAE (↓\downarrow)
0.000.810
0.050.801
0.250.802
0.500.805
1.000.814

Table 9. Effect of noise standard deviation σϵ\sigma_{\epsilon} on CoCo’s forecasting performance. σϵ\sigma_{\epsilon} controls the noise added to the history portion of the forecast covariate MtM_t (Eq. (5)) and the confidence covariate CtC_t (Eq. (6)). Results are for 16-step forecasts with context length 64 on the Media Cloud dataset, using the base pretrained Chronos2 as the TSFM backbone. Bold denotes the best result. σϵ=0.05\sigma_{\epsilon}=0.05 is selected and used in all experiments.

GRPO Training Curve

Figure 10. Average GRPO training reward rtr_t (Eq. (7)) over training steps, computed as the mean over NrolloutsN_{\mathrm{rollouts}} rollouts per prompt in a batch. The reward increases steadily reflecting that the LLM progressively learns to generate forecast covariates MtM_t that provide better forecasts (θM−TSFM(XtH∥Mt))(\theta_{\mathrm{M-TSFM}}(X_t^{H}\mathbin{\|}M_t)) relative to the unimodal baseline (θM−TSFM(XtH))(\theta_{\mathrm{M-TSFM}}(X_t^{H})).

Training Datasets

For training, as described in Section 4, we annotate real-world time series datasets using a large language model (Qwen3-30B-A3B-Instruct-2507) with the prompt detailed in § 17.1. Specifically, we provide windows of 30 timestamps to the LLM for annotation in each batch for a dataset. Since we evaluate forecast horizons of up to 16 steps, this procedure yields annotations with varying degrees of predictive information. In particular, the first annotation within a window has access to the full future horizon and may therefore contain stronger predictive signals, whereas annotations closer to the end of the window are increasingly speculative and may carry little to no predictive power.

We focus on the macroeconomic domain, as the relevant datasets are publicly available, abundant, and well-studied. Importantly, macroeconomic time series are known to be influenced by textual information from news and reports, making them a natural testbed for evaluating multimodal models that incorporate textual context. The primary sources of real-world time series used in our study are:

  • FRED: A public-domain repository of domestic and international economic indicators maintained by the Federal Reserve Bank of St. Louis. We select around 1200 datasets from the domains included in the FRED-MD collection, which is curated for empirical analysis in economics and finance. These datasets span multiple frequencies and economic domains.

  • Other Sources: We collect time series from two major European statistical agencies: Eurostat, the statistical office of the European Union; and Statistics Denmark (StatBank), Denmark’s national statistical institute. From these sources, we collect time series datasets corresponding to various economic indicators, such as energy utility, unemployment rates, etc. These sources offer datasets at multiple frequencies (daily to monthly).

A summary of the number of datasets generated from each source, with the frequency and the number of samples annotated, is provided in Table 10. Leveraging the synthetic annotation process, we were able to create a large-scale multimodal training dataset with more than 12 million samples, spanning over 36,968 target variables.

SourceFrequencyDatasetsSamples
FREDDaily95107,993
Weekly9197,010
Monthly807433,750
Quarterly9318,891
Annual523,176
EIA DataDaily118252,581
Monthly1,596428,449
DK DataMonthly26,7665,314,577
EU DataMonthly7,0011,471,900
FavoritaDaily87148,469
FNSPID (Real)Daily142140,977
Air QualityHourly1204,135,865
Total36,96812,553,638

Table 10. Summary of the training datasets used. Datasets from FNSPID contain real annotations

To mitigate memorization and unintended exposure to test sets, we enforce a strict date cutoff of December 2022 when creating the training dataset. While some existing multimodal datasets we use for evaluation contain samples from before this cutoff, the datasets used (summarized in Table 11) also consist of time series from diverse domains, which are substantially out-of-distribution relative to the datasets used during training. To further enforce a strict separation between training and evaluation and to explicitly address the knowledge cutoff issue of LLMs, we additionally curate a new multimodal evaluation suite consisting exclusively of data from 2024–2025. This dataset enables evaluation under both temporal shift and domain mismatch, providing a test of generalization, and is described in detail in Section F.1.

Text sourceDatasetTarget sourceFrequencyDate rangeTotal≥2024-06
Media CloudApple Stock PriceGoogle FinanceDaily2024-01 – 2025-12503398
Copper Futures PriceGoogle FinanceDaily2025-01 – 2026-01258258
Crude Oil PriceFREDDaily2024-01 – 2026-01500395
Egg PriceTrading EconomicsDaily2025-02 – 2025-12223223
Google Stock PriceGoogle FinanceDaily2024-01 – 2025-12502398
NASDAQ ElectricityFREDDaily2024-01 – 2025-12523413
NVIDIA Stock PriceGoogle FinanceDaily2024-01 – 2025-12502397
S&P500 IndexFREDDaily2024-01 – 2025-10463359
SPDR Gold Shares PriceGoogle FinanceDaily2024-01 – 2025-12501396
Silver PriceTrading EconomicsDaily2024-01 – 2026-01519411
Steel Rebar Futures PriceTrading EconomicsDaily2025-01 – 2026-01241241
TimeMMDBroiler Price (Agriculture)TimeMMDMonthly1980-01 – 2024-045320
Drought Level (Climate)TimeMMDWeekly2000-01 – 2024-051,2720
US Trade Balance (Economy)TimeMMDMonthly1987-01 – 2024-034470
Gasoline Price (Energy)TimeMMDWeekly1993-04 – 2024-041,6220
Air Quality (Environment)TimeMMDDaily2019-09 – 2023-091,4800
Disaster Grant (Security)TimeMMDMonthly1998-09 – 2024-053090
Unemployment Rate (Social Good)TimeMMDMonthly1980-01 – 2024-045320
Traffic Volume (Traffic)TimeMMDMonthly1979-12 – 2024-035320
FNSPIDBank of America Stock PriceFNSPIDDaily2018-06 – 2020-064990
Boeing Stock PriceFNSPIDDaily2021-12 – 2023-125000
Costco Stock PriceFNSPIDDaily2021-12 – 2023-125000
Exxon Mobil Stock PriceFNSPIDDaily2021-12 – 2023-125000
Goldman Sachs Stock PriceFNSPIDDaily2021-12 – 2023-125000
JPMC Stock PriceFNSPIDDaily2018-09 – 2020-064370
McD Stock PriceFNSPIDDaily2018-06 – 2020-065000
Microsoft Stock PriceFNSPIDDaily2022-04 – 2023-124120
Oracle Stock PriceFNSPIDDaily2021-12 – 2023-125000
Tesla Stock PriceFNSPIDDaily2022-05 – 2023-124100
Walmart Stock PriceFNSPIDDaily2021-12 – 2023-125000
Total16,7193,889

Table 11. Summary of the multimodal evaluation datasets grouped by source.

Apart from the synthetic dataset generated, we also include a subset of the FNSPID dataset during training to evaluate the usefulness of synthetic annotations when compared to real-world text for training (results in Table 24). The top 150 stocks where the text availability is highest are chosen for training. Since FNSPID is also part of the evaluation set, we exclude the stocks in the evaluation set from the training set to prevent leakage during training.

Additionally, we use the Beijing Air Quality dataset [67] as part of real-world multimodal datasets used along with our synthetically annotated multimodal datasets. The dataset contains air quality metrics and environmental variables recorded at 12 different stations. For each metric/variable, we construct textual annotations using the remaining features. The dataset has time series samples with hourly frequency.

Evaluation

Evaluation Datasets

A comprehensive summary of the datasets used for evaluation is provided in Table 11. Our study utilizes existing multimodal benchmarks, specifically the TimeMMD benchmark and stock prices from FNSPID. However, these datasets lack samples beyond 2023. Given that our methodology leverages the recent LLMs for forecasting, evaluating solely on legacy data introduces a risk of temporal data leakage, given that they have knowledge cutoffs of up to late 2024. To ensure a rigorous and “fair” evaluation in which the LLM has no prior exposure to test trends, we curated a new multimodal evaluation suite comprising datasets with samples following the 2024 cutoff. We show that CoCo’s performance does not degrade when evaluated separately in samples after the cutoff date (Table 17).

TimeMMD We evaluate on eight TimeMMD datasets (spanning Agriculture, Climate, Economy, Energy, Security, Social Good and Traffic). While TimeMMD provides heterogeneous text sources (reports and search queries), we aggregate and summarize all available information for each timestamp to provide a unified context.

FNSPID We include stock price data and timestamped NASDAQ news for a subset of the S&P 500’s top 25 companies from different sectors. To handle the high volume of daily articles, we summarize multiple news entries into a single, daily textual summary.

Media Cloud For the curation of the text for the new evaluation datasets, we leveraged the Media Cloud API4 for news articles with timestamps and used the pipeline shown in Figure 12 to obtain text annotations on a daily basis. The LLM prompts used in the pipeline are provided in I.3. We also provide the Media Cloud queries used to fetch news articles from the API for each dataset.

Workflow for generating textual annotations from real-world time series datasets

Figure 12. Our approach to generate textual annotations, for real-world time series datasets, extracted from the news articles that are scraped from the internet. Let us consider the task of generating textual annotations for the NVIDIA stock price dataset. First, we use the Media Cloud API to extract the relevant news articles between two successive timestamps t−1t-1 and tt. These news articles are summarized into Key Events and Facts and Predictive Information by an LLM. This summary is treated as the text annotation for the timestamp tt.

Target variables were aggregated from three primary sources:

  • Google Finance5: Daily market pricing and equity data were retrieved using Google Finance, via the Google Sheets integration. This source provides broad accessibility to global exchange data. Specific tickers are provided to ensure exact dataset reconstruction.

  • FRED: Macroeconomic indices and benchmarks were sourced from the FRED database, a public-domain repository for domestic and international economic indicators maintained by the Federal Reserve Bank of St. Louis.

  • Trading Economics6: Commodity prices were obtained from the Trading Economics platform. This source provides curated global economic data. Researchers seeking to replicate this specific subset may require a platform subscription.

Below are the specific tickers and Media Cloud queries used for the curation:

  • S&P500 Index

  • Source : FRED - SP5007

  • Media Cloud Query : “s&p 500” OR s&p500

  • Crude Oil Price

    • Source : FRED - DCOILWTICO8

    • Media Cloud Query : “crude oil” OR wti OR brent

  • NASDAQ Electricity Benchmark

    • Source : FRED - NASDAQNQUSB6510109

    • Media Cloud Query : nasdaq AND (electricity OR energy) AND sector

  • NVIDIA Stock Price

    • Source : Google Finance, Ticker: NASDAQ:NVDA

    • Media Cloud Query : nvidia AND analyst

  • SPDR Gold Shares Price

    • Source : Google Finance, Ticker: NYSEARCA:GLD

    • Media Cloud Query : gold AND analyst

  • Egg Prices

    • Source : Trading Economics - Eggs US10

    • Media Cloud Query : egg AND (sales OR price)

  • Apple Stock Price

    • Source : Google Finance, Ticker: NASDAQ:AAPL

    • Media Cloud Query : apple AND tech AND analyst

  • Silver Price

    • Source : Trading Economics - Silver11

    • Media Cloud Query : silver AND (sales OR price)

  • Steel Rebar Futures Price

    • Source : Trading Economics - Steel Rebar Futures12

    • Media Cloud Query : steel AND (NOT amazon) AND (price OR sale OR export OR import)

  • Google Stock Price

  • Source : Google Finance, Ticker: NASDAQ:GOOG

  • Media Cloud Query : (google OR alphabet) AND (analyst)

  • Copper Futures Price

    • Source : Google Finance, Ticker: COMEX:HGW00

    • Media Cloud Query : copper AND (NOT amazon) AND (price OR sale OR export OR import)

Evaluation Baselines

We organize the 27 baseline configurations according to whether they incorporate textual context and whether their parameters are fine-tuned. This categorization is also used in Figure 5.

  1. Zero-shot unimodal (8). Chronos2 [8], TimesFM2.5 [7], TiRex [68], Moirai [69], Toto [70], TabPFN2.5 [51], TimeMoE [71], and Panda [72]. These methods forecast directly from the numerical history without textual context or task-specific fine-tuning.

  1. Zero-shot multimodal (7). GPT-4o, Qwen3-30B, and TimeOmni directly forecast from the numerical history and paired text. Chronos2 MV and Toto MV receive Sentence Transformer embeddings of the text as additional covariates. We also include Chronos2+GPT-4o and TimesFM2.5+GPT-4o mean ensembles [73], which average the corresponding TSFM and LLM forecasts.

  1. Unimodal fine-tuning (6). Chronos2, PatchTST [74], iTransformer [75], AutoTimes [76], S2IP-LLM [77], and TEST [78] are fine-tuned using only the numerical histories from the synthetic training data.

  1. Multimodal fine-tuning (6). Chronos2 MV, AutoTimes, VoT [79], TaTS [40], S2IP-LLM, and TEST incorporate both numerical history and textual context during fine-tuning and inference. Where a method was originally unimodal, we adapt its input representation to incorporate text.

Many fine-tuning baselines were originally designed for dataset-specific training. To ensure a controlled comparison, we instead train every fine-tuned configuration using the same synthetic text–time-series pairs used to train CoCo and evaluate it on the same held-out real-world datasets.

Evaluation Metrics

Normalization: As described in Appendix § 13.1, we apply z-score normalization during inference, and all reported MAE values are computed in the normalized space.

MASE: Along with the MAE, we also report Mean Absolute Scaled Error (MASE) (Table 18) as a scale-free accuracy metric. For a forecast [x^t+1,x^t+2,…,x^t+h][\hat{x}_{t+1}, \hat{x}_{t+2}, \ldots, \hat{x}_{t+h}] and ground-truth [xt+1,xt+2,…,xt+h][x_{t+1}, x_{t+2}, \ldots, x_{t+h}] over horizon length hh, we compute

MASE=1n∑t=1n∣xt−x^t∣1n−1∑t=2n∣xt−xt−1∣\mathrm{MASE} = \frac{\frac{1}{n}\sum_{t=1}^{n} |x_t-\hat{x}_t|}{\frac{1}{n-1}\sum_{t=2}^{n} |x_t-x_{t-1}|}

The numerator is the model MAE over the forecast window, while the denominator is a naive lag-1 scaling term computed on the same window. Lower is better; values <1< 1 indicate performance better than the naive scale baseline. This follows the MASE usage in the GIFT-Eval benchmark for comparable, scale-normalized time-series evaluation across heterogeneous datasets. The aggregate MASE over datasets is computed by taking the geometric mean across datasets as per convention.

Wilcoxon Signed-Rank p-value: For each model pair, we form paired per-setting errors and run the two-sided Wilcoxon signed-rank test. The test ranks absolute paired differences and uses their signs, so it reflects both effect size and direction consistency. The null hypothesis is that the median paired difference is zero (no systematic difference). Small p-values indicate stronger evidence of a systematic difference while mixed positive and negative differences typically yield larger p-values.

ELO Ratings: We rank forecasting models across datasets, context lengths, and forecast horizons using an ELO rating system [80], which estimates latent skill levels from relative outcomes. Each evaluation setting induces a multiplayer matchup among all models, converted into relative outcomes based on the MAE (lower values indicate better performance). Model ratings are updated after each matchup using the standard ELO update rule:

Ri′=Ri+K⋅(Si−Ei),(23)R'_i = R_i + K \cdot(S_i - E_i), \tag*{(23)}

where RiR_i and Ri′R'_i denote the pre- and post-match ratings of model ii, SiS_i is the observed score derived from its rank in the matchup, and EiE_i is the expected score computed from all participating models’ current ratings using a logistic model with deviation parameter D=400D = 400. All models are initialized with a base rating of 1500, and K=32K = 32 is the update factor controlling sensitivity.

The conventional ELO system is sensitive to the order of matchups and can become unstable as the number of models and evaluation settings increases [81]. To mitigate this, we repeat the rating procedure across 32 random orderings of the matchups and report the mean rating for each model. This approach yields stable ratings at our scale of models and settings. This evaluation protocol is in the same spirit as rating-based leaderboards used for LLMs [82], which employ ELO-style updates to aggregate performance comparisons across many evaluation settings rather than relying on simple averages of ranks or win counts.

CoCo’s Robustness to Stochasticity

We evaluate the stability of CoCo across all 95 dataset–setting combinations under three sources of stochasticity: (1) LLM rollout sampling, (2) noise injection into the historical covariates, and (3) stochasticity from LLM fine-tuning checkpoints.

LLM rollout stochasticity: We generate LLM forecasts using three different random seeds, with five rollouts per seed at temperature 1.0, resulting in 15 rollouts per sample. While individual rollout forecasts exhibit substantial variation, aggregating the rollouts using their median substantially reduces this variability. Across the three runs, the LLM rollout median has a coefficient of variation (CV) of only 0.56%, with the CV remaining below 2% for most dataset–setting combinations. More importantly, this residual variation is effectively absorbed by the TSFM: CoCo’s final forecast has a CV of only 0.01%. The corresponding aggregate MAE statistics are shown in Table 12.

Robustness to covariate noise: CoCo additionally injects Gaussian noise with σ=0.05\sigma= 0.05 into the historical covariates during Chronos2 inference as a regularization mechanism. To isolate the effect of this stochasticity, we evaluate CoCo using five random seeds while keeping the LLM rollouts fixed. CoCo achieves an overall MAE of 0.7194 ± 0.0005, corresponding to a CV of only 0.06%. Thus, the small amount of noise introduced during inference does not materially affect the final forecast.

Fine-tuning stochasticity: Finally, we evaluate the sensitivity of CoCo to stochasticity during LLM fine-tuning. We train the LoRA adapter for Qwen3-4B using three different training seeds, producing three independent checkpoints. At temperature 0, the resulting LLM forecasts exhibit noticeable variation, with an MAE of 0.818 ± 0.009 (CV = 1.05%). In contrast, the corresponding CoCo forecasts remain highly consistent, achieving an MAE of 0.7194 ± 0.0001 (CV = 0.01%). This demonstrates that CoCo is not sensitive to the particular LLM fine-tuning checkpoint used.

Overall, CoCo remains highly robust across all three sources of stochasticity. We primarily attribute this stability to the use of the median across multiple LLM rollouts, which substantially suppresses variability before the resulting covariate is provided to the TSFM. The TSFM further absorbs the small amount of residual variation, resulting in a highly stable final forecast.

Extended Results

GroupDatasetCoCoTiRex1Chronos2MoiraiTimesFM2.5TotoTabPFN2.5TimeMoEPANDA
Media CloudCrude Oil Price0.7000.6820.6900.6870.6630.7030.7770.7700.774
S&P500 Index0.5940.6010.6410.6250.6520.6670.7780.8350.816
Apple Stock0.6370.6660.6660.6510.6760.6830.7600.7970.846
Copper Futures0.6820.7350.7270.6790.7510.7010.7810.7750.941
Egg Price0.6040.5870.5830.6070.5870.5830.6550.5980.865
Electricity Price0.5760.6130.6050.6080.5920.6250.7100.7070.751
SPDR Gold Price0.5470.5760.5910.5820.6420.6260.6820.8210.832
Google Stock0.6060.6000.6240.6270.6360.6400.6590.8090.790
NVIDIA Stock Price0.6130.6260.6340.6330.6550.6590.7080.7620.785
Silver Price0.6990.7250.7530.7370.7580.7380.8990.9260.964
Steel Rebar Futures0.6070.6260.6330.6400.6420.5920.6850.6670.748
FNSPIDBoeing Stock0.5850.5870.5980.6090.6110.6170.6220.6890.728
Bank of America Stock0.8200.8260.8500.8430.8310.8710.8900.9411.005
Costco Stock0.5670.5690.5830.6040.5840.6020.6720.7000.728
Goldman Sachs Stock0.7430.7410.7430.7670.7250.7670.7310.7190.854
JPMC Stock0.7990.8040.8370.8160.8490.8610.8761.0401.008
McDonald’s Stock0.6840.6840.7120.6810.7360.7120.7420.8060.857
Microsoft Stock0.5960.6100.6460.6250.6450.6540.7120.8010.830
Oracle Stock0.5940.5820.6000.6240.6260.6480.6750.7810.807
Tesla Stock0.5660.5770.5930.5620.5590.6100.5870.6340.709
Walmart Stock0.5910.6160.6150.6340.6330.6410.6680.7160.765
Exxon Mobil Stock0.6900.6640.6840.7060.7300.7120.7720.7310.766
TimeMMDBroiler Price0.6530.6740.7050.6820.7350.7120.8090.8530.934
Drought Level0.6890.6970.7120.7180.7150.7300.7400.7590.821
US Trade Balance0.6500.7050.6750.6670.6550.7160.7270.8130.844
Gasoline Price0.6710.6800.6970.7070.6960.7230.7750.7660.855
Air Quality Index0.8370.8220.8300.8490.8260.8340.8490.8560.897
Disaster Grants1.4501.5771.5471.5391.5801.5591.4651.5221.588
Unemployment Rate0.5500.5310.5650.5960.5550.5440.6340.6690.733
Travel Volume0.2640.3320.2700.3160.2520.2710.2920.3150.602
TimeMMD DailyAir Quality Index0.9740.9280.9450.9570.9330.9480.9631.0700.969
TimeMMD WeeklyDrought Level1.0401.0631.1021.0621.0791.0811.1111.1861.496
Gasoline Price1.0411.0381.0991.0911.0931.0961.0981.1891.648
TimeMMD MonthlyBroiler Price1.6521.6821.6811.7121.7071.7741.7151.7521.671
US Trade Balance1.3561.4361.3901.5221.3711.3431.4431.3351.450
Disaster Grants1.6101.6041.6511.6621.6271.6341.6311.6421.758
Unemployment Rate1.5391.5881.5391.6351.6011.6171.5651.5991.574
Travel Volume1.4191.4761.4851.3871.3251.3721.5931.2541.425
Overall Wins382506132461
Mean0.7190.7350.7450.7470.7490.7580.8000.8390.903
ELO182017151602158215931487134412661090
Average Rank2.242.984.074.314.175.226.527.198.31
Head-to-Head Wins—60/9581/9578/9572/9580/9590/9588/9593/95
Wilcoxon pp—5.0e-042.0e-129.2e-137.8e-098.1e-131.4e-163.3e-154.1e-17

Table 13. Dataset-level MAE for zero-shot unimodal methods. Each dataset reports the mean over the 64-16, 128-16, and 256-16 settings. TimeMMD Daily, Weekly, and Monthly retain their native settings as separate blocks. Summary statistics are computed across all 95 settings. CoCo outperforms multiple state-of-the-art TSFMs. Bold and underlined values denote the best and second-best method in each row.

GroupDatasetCoCoTimesFM2.5+GPT4oChronos2+GPT4oChronos2Toto MVMVGPT-4oQwen3-30BTimeOmni
Media CloudCrude Oil Price0.7000.625_0.645_0.7070.6460.7130.8420.887
S&P500 Index0.594_0.634_0.6490.6930.7240.7230.6550.785
Apple Stock0.637_0.682_0.6920.7150.7060.7700.7540.876
Copper Futures0.682_0.685_0.6850.7480.7400.7250.7360.853
Egg Price0.6040.5790.5720.594_0.577_0.5990.6720.809
Electricity Price0.5760.6280.6410.631_0.646_0.7190.6680.851
SPDR Gold Price_0.547_0.5680.5460.6600.6760.5820.6503.468
Google Stock0.6060.619_0.619_0.6690.6680.6800.7720.726
NVIDIA Stock Price0.613_0.658_0.6580.6830.6880.7580.8850.734
Silver Price0.699_0.706_0.7140.8300.8230.7840.7810.891
Steel Rebar Futures0.6070.6030.610_0.584_0.5740.6950.7871.375
FNSPIDBoeing Stock0.5850.5880.5910.5970.6200.6200.7370.764
Bank of America Stock0.8200.802_0.820_0.8600.8700.8620.8891.006
Costco Stock0.5670.5670.5770.6210.6290.6270.6320.713
Goldman Sachs Stock0.7430.7560.7740.689_0.733_0.8660.8340.879
JPMC Stock0.7990.763_0.768_0.8650.8620.7800.8401.077
McDonald’s Stock0.6840.645_0.670_0.6910.7450.6950.8780.875
Microsoft Stock0.596_0.607_0.6150.6780.6940.6320.6260.662
Oracle Stock0.594_0.635_0.6340.6510.6580.7140.6890.744
Tesla Stock0.566_0.565_0.5830.5520.5850.6250.6350.628
Walmart Stock0.5910.6420.6380.6340.6390.7170.6630.722
Exxon Mobil Stock0.690_0.687_0.6790.7180.7250.7410.7950.826
TimeMMDBroiler Price0.6530.6690.663_0.786_0.7530.7250.7791.394
Drought Level0.689_0.699_0.7050.7210.7160.7700.7700.841
US Trade Balance0.6500.6970.6960.703_0.662_0.9001.1531.436
Gasoline Price0.6710.667_0.671_0.7140.7010.7100.6830.828
Air Quality Index0.8370.9170.9210.833_0.833_1.1261.1581.146
Disaster Grants1.4501.6401.629_1.552_1.5671.8471.7642.908
Unemployment Rate0.5500.461_0.463_0.5710.5460.5220.6370.786
Travel Volume0.2640.4340.430_0.271_0.2710.6900.81221.457
TimeMMD DailyAir Quality Index_0.974_1.5161.5350.943_0.947_2.5061.9101.275
TimeMMD WeeklyDrought Level_1.040_1.0351.0551.1121.0811.1011.1001.225
Gasoline Price1.0411.0091.0211.0941.1331.0421.0321.237
TimeMMD MonthlyBroiler Price1.6521.6701.697_1.657_1.7721.7972.2142.170
US Trade Balance1.3561.3991.423_1.347_1.3161.5151.7002.373
Disaster Grants1.6101.6631.6741.639_1.611_1.7671.7731.981
Unemployment Rate1.5391.400_1.423_1.5361.5711.4221.6101.691
Travel Volume1.419_1.386_1.4661.4031.3441.5381.7122.031
Overall Wins41257139000
Mean0.7190.7420.7480.7630.7660.8360.8781.756
ELO17951744167615491528136512891053
Average Rank2.232.643.144.174.345.656.257.58
Head-to-Head Wins–60/9565/9574/9575/9585/9594/9595/95
Wilcoxon pp–0.0056.3e-051.8e-106.1e-108.9e-152.8e-172.6e-17

Table 14. Dataset-level MAE for zero-shot multimodal methods. Each dataset reports the mean over the 64-16, 128-16, and 256-16 settings. TimeMMD Daily, Weekly, and Monthly retain their native settings as separate blocks. Summary statistics are computed across all 95 settings. Mean ensembles are included because they combine zero-shot forecasts. CoCo outperforms state-of-the-art LLM forecasters as well as multivariate TSFMs that directly incorporate Sentence Transformer text embeddings as covariates. Bold and underlined values denote the best and second-best method in each row.

GroupDatasetCoCoChronos2 (FT)AutoTimesPatchTSTS2IP-LLMiTransformerTEST
Media CloudCrude Oil Price0.7000.6810.6910.7070.7230.7630.769
S&P500 Index0.5940.6600.6520.6260.6610.7400.764
Apple Stock0.6370.6830.6690.6830.6980.7720.827
Copper Futures0.6820.7290.7310.7170.7760.7790.767
Egg Price0.6040.6230.7050.6980.7740.7650.805
Electricity Price0.5760.5600.6030.6030.6140.6270.667
SPDR Gold Price0.5470.5770.6060.5870.6000.6330.680
Google Stock0.6060.6430.6630.6510.6850.7430.789
NVIDIA Stock Price0.6130.6590.6670.6510.6830.7380.759
Silver Price0.6990.7220.7620.7190.7230.7640.809
Steel Rebar Futures0.6070.6020.5500.6580.6520.6790.707
FNSPIDBoeing Stock0.5850.6050.6130.6180.6180.7580.716
Bank of America Stock0.8200.8480.8550.8620.8710.9380.998
Costco Stock0.5670.5930.5950.5760.5970.6510.629
Goldman Sachs Stock0.7430.7370.7130.7630.7640.8190.824
JPMC Stock0.7990.8120.8270.8150.8490.8911.000
McDonald’s Stock0.6840.7090.7000.7390.7180.8520.809
Microsoft Stock0.5960.6130.6380.6160.6200.6850.761
Oracle Stock0.5940.5890.6190.6220.6040.6970.690
Tesla Stock0.5660.5720.5710.5980.6080.6810.722
Walmart Stock0.5910.6310.6150.6340.6530.6990.762
Exxon Mobil Stock0.6900.7360.6700.7010.7060.7520.815
TimeMMDBroiler Price0.6530.6680.6850.6920.6910.7290.800
Drought Level0.6890.7250.7090.7430.7750.8180.868
US Trade Balance0.6500.6200.8130.7090.6930.6780.796
Gasoline Price0.6710.6960.7340.7320.7570.8070.840
Air Quality Index0.8370.8310.8690.9220.8930.9300.985
Disaster Grants1.4501.4621.5251.5381.4881.5371.546
Unemployment Rate0.5500.5010.7400.6890.6470.8100.794
Travel Volume0.2640.2440.4780.4740.4440.4660.747
TimeMMD DailyAir Quality Index0.9740.9350.9711.0240.9901.0581.124
TimeMMD WeeklyDrought Level1.0401.0721.0351.0611.0911.1131.239
Gasoline Price1.0411.0651.1111.0891.1731.1201.315
TimeMMD MonthlyBroiler Price1.6521.7251.7991.6851.7671.6842.144
US Trade Balance1.3561.4781.2941.3001.3431.3161.589
Disaster Grants1.6101.6211.6361.6161.6301.6131.980
Unemployment Rate1.5391.5391.6191.5511.6221.5421.663
Travel Volume1.4191.6801.2301.3291.2861.3871.665
Overall Wins5223161300
Mean0.7190.7380.7630.7640.7730.8230.882
ELO1860169715951549147212491078
Average Rank1.692.633.403.744.255.716.58
Head-to-Head Wins–64/9578/9589/9587/9592/9594/95
Wilcoxon pp–1.2e-053.2e-107.2e-153.5e-156.1e-172.7e-17

Table 15. Dataset-level MAE for unimodal fine-tuning methods. Each dataset reports the mean over the 64-16, 128-16, and 256-16 settings. TimeMMD Daily, Weekly, and Monthly retain their native settings as separate blocks. Summary statistics are computed across all 95 settings. CoCo outperforms unimodal fine-tuning approaches, indicating that exogenous textual context is essential for improved forecasting and is difficult to recover from numerical history alone. Bold and underlined values denote the best and second-best method in each row.

GroupDatasetCoCoChronos2MV (FT)VoTTaTSS2IP-LLMAutoTimesTEST
Media CloudCrude Oil Price0.700<u>0.680</u>0.6670.6980.7150.6990.704
S&P500 Index0.594<u>0.641</u>0.6790.6940.6660.6820.727
Apple Stock0.6370.6570.6610.7020.6980.7300.699
Copper Futures<u>0.682</u>0.6790.7070.7300.7550.7550.732
Egg Price<u>0.604</u>0.6670.7090.5920.7370.7810.749
Electricity Price0.576<u>0.587</u>0.5960.6230.6010.6170.622
SPDR Gold Price0.547<u>0.561</u>0.6420.6460.6030.6610.660
Google Stock0.606<u>0.622</u>0.6410.6680.7110.7070.671
NVIDIA Stock Price0.613<u>0.656</u>0.6620.6720.6700.6790.682
Silver Price0.699<u>0.726</u>0.7750.8100.7460.8280.798
Steel Rebar Futures0.6070.6080.5850.5670.583<u>0.567</u>0.608
FNSPIDBoeing Stock0.5850.617<u>0.613</u>0.6310.6650.6650.704
Bank of America Stock<u>0.820</u>0.8060.8560.8430.8640.9150.928
Costco Stock<u>0.567</u>0.5660.5880.6130.6140.6200.670
Goldman Sachs Stock0.7430.741<u>0.725</u>0.6840.7650.7270.793
JPMC Stock<u>0.799</u>0.7960.8130.8410.8200.8920.895
McDonald’s Stock<u>0.684</u>0.6630.6850.6950.7030.7280.737
Microsoft Stock0.596<u>0.608</u>0.6230.6690.6380.6790.683
Oracle Stock0.5940.5920.6120.6340.6440.6710.669
Tesla Stock0.5660.586<u>0.565</u>0.5520.6640.5950.676
Walmart Stock0.5910.617<u>0.606</u>0.6270.6340.6370.670
Exxon Mobil Stock0.6900.7080.6620.7110.704<u>0.686</u>0.713
TimeMMDBroiler Price0.653<u>0.670</u>0.6760.7770.6720.7090.696
Drought Level0.6890.723<u>0.710</u>0.7220.7400.7730.745
US Trade Balance0.650<u>0.637</u>0.7280.6300.7320.9180.860
Gasoline Price0.6710.7110.732<u>0.708</u>0.7390.7540.763
Air Quality Index0.837<u>0.842</u>0.9090.8590.9010.8670.879
Disaster Grants1.4501.4821.4801.511<u>1.467</u>1.4501.501
Unemployment Rate0.5500.5860.715<u>0.552</u>0.7270.7940.765
Travel Volume<u>0.264</u>0.3030.4060.2620.4890.6630.802
TimeMMD DailyAir Quality Index0.9740.9411.0130.9610.999<u>0.960</u>1.010
TimeMMD WeeklyDrought Level<u>1.040</u>1.0271.0651.0711.0671.1111.104
Gasoline Price1.0411.0861.102<u>1.066</u>1.1491.1931.177
TimeMMD MonthlyBroiler Price1.6521.751<u>1.694</u>1.7031.7341.7801.710
US Trade Balance1.3561.2931.3461.3501.307<u>1.294</u>1.436
Disaster Grants1.6101.6141.6291.6411.623<u>1.633</u>1.707
Unemployment Rate1.5391.5141.586<u>1.520</u>1.5951.6211.650
Travel Volume1.4191.4461.2751.433<u>1.229</u>1.2051.325
Overall Wins47191113140
Mean0.7190.7340.7560.7540.7750.8010.815
ELO1761164215721499143813251263
Average Rank2.072.803.453.914.555.385.84
Head-to-Head Wins–68/9571/9570/9585/9583/9591/95
Wilcoxon pp–1.8e-051.5e-081.3e-081.3e-132.3e-132.7e-16

Table 16. Dataset-level MAE for multimodal fine-tuning methods. Each dataset reports the mean over the 64-16, 128-16, and 256-16 settings. TimeMMD Daily, Weekly, and Monthly retain their native settings as separate blocks. Summary statistics are computed across all 95 settings. The CoCo fine-tuning approach for incorporating text into forecasting outperforms other multimodal fine-tuning methods that jointly incorporate numerical history and text. Bold and underlined values denote the best and second-best method in each row.

M-TSFMLLMθLLM(XtH,St)\theta_{\mathrm{LLM}}(X_t^H, S_t)θM−TSFM(XtH)\theta_{\mathrm{M-TSFM}}(X_t^H)θM−TSFM(XtH∣Mt)\theta_{\mathrm{M-TSFM}}(X_t^H \mathbin{|} M_t)θM−TSFM(XtH∣Mt∣Ct)\theta_{\mathrm{M-TSFM}}(X_t^H \mathbin{|} M_t \mathbin{|} C_t)
Chronos2GPT-OSS-120B0.8850.8470.8280.842
Qwen3-30B0.9490.8470.8160.851
Gemma3-27B0.8710.8470.8210.827
Qwen3-4B1.0190.8470.8240.895
Qwen3-4B (FT)0.8620.8470.8180.801
TotoGPT-OSS-120B0.8850.8650.8640.854
Qwen3-30B0.9490.8650.8590.851
Gemma3-27B0.8710.8650.8590.855
Qwen3-4B1.0190.8650.8630.858
Qwen3-4B (FT)0.8620.8650.8620.857
MoiraiGPT-OSS-120B0.8850.8430.8380.844
Qwen3-30B0.9490.8430.8410.839
Gemma3-27B0.8710.8430.8390.843
Qwen3-4B1.0190.8430.8390.841
Qwen3-4B (FT)0.8620.8430.8410.845
Chronos2 (LoRA FT)GPT-OSS-120B0.8850.8240.8160.813
Qwen3-30B0.9490.8240.8140.811
Gemma3-27B0.8710.8240.8140.810
Qwen3-4B1.0190.8240.8150.815
Qwen3-4B (FT)0.8620.8240.8150.809

Table 25. Table 25: CoCo generalizes across LLM and M-TSFM choices. Mean MAE is reported on the Media Cloud datasets for the 64–16 context–horizon setting. Across five LLM configurations and three M-TSFM backbones, the forecast covariate MtM_t improves over the corresponding M-TSFM baseline in every evaluated combination. The confidence covariate CtC_t provides further gains with the zero-shot Toto backbone and after fine-tuning Chronos2. Importantly, Chronos2 is fine-tuned using rollouts pre-generated only by the fine-tuned Qwen3-4B, yet it generalizes to covariates from every evaluated zero-shot LLM. These results demonstrate that CoCo is not reliant on properties specific to Qwen3-4B or Chronos2. Bold denotes the best result in each row.

GroupSettingDatasetNTSFM TotoTSFM MV TotoCoCo (With Toto) Only Forecast CovariateCoCo (With Toto) Both Covariates
Suite64-16Crude Oil Price4210.9000.8160.899__0.899__
S&P500 Index3840.9260.9980.959__0.948__
Apple Stock4240.9591.003__0.951__0.941
Copper Futures1790.8270.9140.814__0.815__
Egg Price144__0.666__0.6440.7380.694
Electricity Price4440.8470.872__0.840__0.838
SPDR Gold Price4220.9871.066__0.968__0.953
Google Stock4230.7940.858__0.780__0.775
NVIDIA Stock Price4230.9350.984__0.926__0.914
Silver Price4400.9241.0280.925__0.925__
Steel Rebar Futures1620.747__0.739__0.7400.739
FNSPID64-16Boeing Stock4210.834__0.822__0.8240.813
Bank of America Stock4201.1031.0841.111__1.101__
Costco Stock421__0.842__0.8750.8430.829
Goldman Sachs Stock4210.9680.9350.959__0.951__
JPMC Stock358__1.131__1.1171.1341.132
McDonald’s Stock4210.9571.005__0.946__0.938
Microsoft Stock3330.9470.987__0.943__0.938
Oracle Stock4210.9130.926__0.906__0.894
Tesla Stock3310.848__0.829__0.8300.817
Walmart Stock4210.9870.991__0.970__0.958
Exxon Mobil Stock4210.9721.008__0.956__0.952
TimeMMD64-16Broiler Price4530.9841.063__0.978__0.974
Drought Level11930.9090.920__0.907__0.904
US Trade Balance3680.8820.8370.9010.899
Gasoline Price15430.9260.9370.907__0.912__
Air Quality Index14010.8860.8820.884__0.884__
Disaster Grants230__1.139__1.1441.1361.140
Unemployment Rate4530.659__0.660__0.6700.680
Travel Volume4530.3040.3080.3190.316
Overall Wins47316
Mean0.8900.9080.8890.882
ELO1458141514861640
Average Rank2.7672.9672.5671.700
Head-to-Head Wins22/3021/3026/30-
Wilcoxon p1.28e-029.3e-034.97e-05-

Table 26. Results with Toto as the TSFM backbone, using covariates generated by the fine-tuned Qwen3-4b LLM without any Toto-specific fine-tuning. CoCo (with Toto) outperforms Toto in 73% (22/30) of datasets and outperforms sentence transformer augmented Toto (TSFM MV) in 70% (21/30) of the datasets, demonstrating that the covariate design transfers across TSFM backbones and is not specific to Chronos2. Bold and __underlined__ indicate the best and second best per dataset respectively.

Fine-tuned ComponentsMean MAE (↓\downarrow)Avg. Rank among Baselines (↓\downarrow)ELO among Baselines (↑\uparrow)Avg. Rank among Variants (↓\downarrow)ELO among Variants (↑\uparrow)
M-TSFM & LLM0.7195.2418291.871535
LLM0.7326.4417851.931508
M-TSFM0.7247.1617582.191457

Table 28. Ablation of fine-tuned components in CoCo across all 95 settings. We compare variants with only the LLM fine-tuned, only the M-TSFM fine-tuned, and both components fine-tuned, evaluating each against all the 27 methods as well as against the other CoCo variants. Fine-tuning both components jointly (in a sequential manner as described in Section 5) achieves the best mean MAE, average rank, and ELO in both comparisons. Fine-tuning either component individually also outperforms the zero-shot baselines, demonstrating that both stages contribute independently. Notably, the advantage of jointly fine-tuning both components is more pronounced when evaluated against the full baseline pool than in the direct three-way comparison.

LLM Prompts

Synthetic Annotation Prompt

The following prompt is used to generate the large-scale training dataset in Section E.2.

You are given a time series for the economic indicator: { Series Title
/ Information }. The data consists of dates (t) and corresponding
values in appropriate units. Your task is to annotate each data point
with a text column that serves as an analytical briefing note. For
each data point (t, y_t), follow this structured thinking process to
generate the text:

1.  Observation: State the current value and its immediate change. Is
it an increase or decrease? Is the pace of change accelerating,
decelerating, or stable compared to the prior few periods?

2.  Contextualization: Explain this movement by citing 2-3 plausible,
interconnected macroeconomic drivers. Make sure to use factors that
align and are relevant to the time period / date of the data point.

3.  Outlook: Provide a speculative, forward-looking statement. This
outlook must NOT reveal future data. Instead, frame it in terms of
risks, potential, and key indicators to monitor for confirmation. Use
cautious but insightful language.

You don't need to have explicit sections in your response. Just the
text.

Key Directives:
-   Report-Like Tone: The text must be formal, objective, and
analytical.
-   Predictive Value: The combination of context and outlook should
provide a rich narrative basis for forecasting without leaking future
values.
-   Dynamic Narrative: Do not use repetitive sentence structures or
explanations. Each annotation should feel like a unique daily or
monthly report.
-   Conciseness: Keep each annotation under 150 words.
-   Output Format: Your entire output must be in CSV format with three
columns:`t`, `y_t`, `text`. Do not add headers, markdown, or any text
outside of the CSV rows.

Begin generating the CSV output now.

Data:
{Data as rows containing timestamp and value.}

LLM Forecast Prompt

The following prompt is used to obtain forecasts from LLMs, given the numerical history and textual annotations. It is used to both evaluate LLMs for forecasting performance and to obtain the covariates required for CoCo.

You are a time series forecasting expert. Analyze the historical data
with text annotations to predict future values.

FULL TIME SERIES ({l: context length} timesteps):
{Data as rows containing timestamps and values}
DETAILED CONTEXT (most recent 16 timesteps with text annotations):
[
{Textual context as rows containing timestamps and annotations}
]

TASK:
Analyze the full time series pattern and the text annotations for:
1.  FACTUAL PATTERNS: What trends, events, or behaviors are described
in the recent history?
2.  PREDICTIVE SIGNALS: Are there any forward-looking statements,
expectations, or indicators of future change?

Based on this analysis, provide a forecast for the next {h: prediction
horizon} steps. FORECAST TARGET TIMESTAMPS (next steps, in order):
[
{Timestamps to forecast}
]

IMPORTANT: Your forecast numbers MUST correspond to these timestamps
in the same order. Output ONLY {h: prediction horizon} comma-separated
numbers representing your forecast. Do not include any explanations or
other text.

Forecast:

Media Cloud Dataset Collection

Prompt used to filter irrelevant articles and summarize relevant articles

First, determine if the following article is related to {topic} in any
way.

If the article is NOT related to {topic}, respond with exactly: NA

If the article IS related to {topic}, provide a concise summary (3–4
sentences) focusing on:
1. Key facts and events
2. Forward-looking statements or predictions
Only use the provided article to create the summary.

Title: {article title}

Content:
{article content}

Response:

Prompt used to generate summaries from collection of articles

You are analyzing news articles from {date} to provide a comprehensive
daily summary for financial forecasting.

Below are summaries of {N} articles from this date:

{Article 1 Summary: ....}
{Article 2 Summary: ....}
...
{Article N Summary: ....}
Based on these article summaries, provide a comprehensive daily
summary with TWO sections:

SECTION 1 - KEY EVENTS AND FACTS:
Synthesize the main events, developments, and factual information from
the day. Focus on market movements, economic data, corporate
announcements, and significant news. (3-5 sentences)

SECTION 2 - FORWARD-LOOKING SIGNALS:
Identify and synthesize any forward-looking information, predictions,
market expectations, or signals that could indicate future trends.
Include analyst forecasts, anticipated policy changes, and emerging
patterns. (3-5 sentences)

Please provide your response in exactly this format:

KEY EVENTS AND FACTS:
[Your synthesis here]

FORWARD-LOOKING SIGNALS:
[Your synthesis here]

Broader Societal Impacts

Our main contribution in this work is Context as Covariates (CoCo), which is a novel forecasting pipeline that combines the strengths of LLMs and TSFMs to perform accurate multimodal time series forecasting. We also propose a novel approach to generate synthetic multimodal annotations, thereby enabling sim-to-real transfer in domains that lack multimodal time series datasets. We note that improved forecasting using exogenous text has a lot of positive impacts in the time series domain, as it can enable accurate decision making based on precise forecasts. Additionally, CoCo unlocks the ability to perform accurate scenario simulation by providing speculative text annotations for the future. As such, the societal consequences of our work are limited to those arising from improved quality of generated synthetic data. Apart from that, one of the limitations (Section 6) of CoCo is that it is sensitive to incorrect or misleading text. In other words, if the text is deliberately misleading or false, CoCo can generate erroneous forecasts.

NeurIPS Paper Checklist

Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper’s contributions and scope?

Answer: [Yes]

Justification: We prove the effectiveness of CoCo to interpret text accurately and improve multimodal forecasting in Section 6. Additionally, we also showcase the effectiveness of training on synthetic multimodal data that is generated using the approach outlined in Section 4 (see Table 24).

Guidelines:

  • The answer [N/A] means that the abstract and introduction do not include the claims made in the paper.

  • The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A [No] or [N/A] answer to this question will not be perceived well by the reviewers.

  • The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

  • It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: In Section 6, we list three key limitations of our approach pertaining to inference latency, lack of evaluating the reliability of text, and limitations in benchmarks where numerical history has the least significance for forecasting.

Guidelines:

  • The answer [N/A] means that the paper has no limitation while the answer [No] means that the paper has limitations, but those are not discussed in the paper.

  • The authors are encouraged to create a separate “Limitations” section in their paper.

  • The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

  • The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

  • The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

  • The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

  • If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

  • While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren’t acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

Theory assumptions and proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [Yes]

Justification: We validate Assumptions 4.1 and 4.2 through real-world experiments. Lemma 4.3 and Theorem 4.4 are proved in Appendix B.

Guidelines:

  • The answer [N/A] means that the paper does not include theoretical results.

  • All the theorems, formulas, and proofs in the paper should be numbered and cross-referenced.

  • All assumptions should be clearly stated or referenced in the statement of any theorems.

  • The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

  • Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

  • Theorems and Lemmas that the proof relies upon should be properly referenced.

Experimental result reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: All training and inference hyperparameters are clearly discussed in Appendix E.1.

Guidelines:

  • The answer [N/A] means that the paper does not include experiments.

  • If the paper includes experiments, a [No] answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

  • If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

  • Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general, releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

  • While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example (a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm. (b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully. (c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset). (d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [Yes]

Justification: We have provided the code, data, and instructions to reproduce the results in Section 6, as part of our supplementary material. Additionally, we have publicly released the code and data, with the links provided in the manuscript.

Guidelines:

  • The answer [N/A] means that paper does not include experiments requiring code.

  • Please see the NeurIPS code and data submission guidelines (https://neurips.cc/public/guides/CodeSubmissionPolicy) for more details.

  • While we encourage the release of code and data, we understand that this might not be possible, so [No] is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

  • The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https://neurips.cc/public/guides/CodeSubmissionPolicy) for more details.

  • The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

  • The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

  • At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

  • Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

Experimental setting/details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer) necessary to understand the results?

Answer: [Yes]

Justification: Appendices E.1 and F.1 contain the exact training and testing related hyperparameters.

Guidelines:

  • The answer [N/A] means that the paper does not include experiments.

  • The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

  • The full details can be provided either with the code, in appendix, or as supplemental material.

Experiment statistical significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [Yes]

Justification: In all the tables, we report the Wilcoxon p-value for statistical significance.

Guidelines:

  • The answer [N/A] means that the paper does not include experiments.

  • The authors should answer [Yes] if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

  • The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

  • The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

  • The assumptions made should be given (e.g., Normally distributed errors).

  • It should be clear whether the error bar is the standard deviation or the standard error of the mean.

  • It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

  • For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g., negative error rates).

  • If error bars are reported in tables or plots, the authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

Experiments compute resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Justification: Appendix E.1 describes the required compute resources for the LLM finetuning experiments.

Guidelines:

  • The answer [N/A] means that the paper does not include experiments.

  • The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

  • The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

  • The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn’t make it into the paper).

Code of ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines?

Answer: [Yes]

Justification: Yes, our research conforms with the NeurIPS Code of Ethics in every respect.

Guidelines:

  • The answer [N/A] means that the authors have not reviewed the NeurIPS Code of Ethics.

  • If the authors answer [No], they should explain the special circumstances that require a deviation from the Code of Ethics.

  • The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

Broader impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [Yes]

Justification: Our work mainly focuses on improving time series forecasting with multimodal context. As explained in the limitations (see Section 6), we note that our approach utilizes the textual context without guardrails on how much the text should be trusted. We provided a detailed discussion regarding the same in Appendix J. Guidelines:

  • The answer [N/A] means that there is no societal impact of the work performed.

  • If the authors answer [N/A] or [No], they should explain why their work has no societal impact or why the paper does not address societal impact.

  • Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

  • The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate Deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

  • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

  • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pre-trained language models, image generators, or scraped datasets)?

Answer: [N/A]

Justification: We believe our work does not suffer from high-risk misuse.

Guidelines:

  • The answer [N/A] means that the paper poses no such risks.

  • Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

  • Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

  • We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [Yes]

Justification: All datasets and models used in this work are properly credited via citations in the main paper. Specifically: (1) Qwen3-4b [Qwen/Qwen3-4B-Instruct-2507] is released under Apache 2.0; (2) Chronos2 [8] (amazon/chronos-2, version 2.1.0) is released under Apache 2.0; (3) TimeMMD [11] is released under CC BY 4.0; (4) FN-SPID [13] is released under CC BY-NC 4.0, and is used exclusively for research purposes consistent with its non-commercial license. Media Cloud data is accessed via the public Media Cloud API [56] in accordance with its terms of service. All model checkpoints are accessed from their official HuggingFace repositories.

Guidelines:

  • The answer [N/A] means that the paper does not use existing assets.

  • The authors should cite the original paper that produced the code package or dataset.

  • The authors should state which version of the asset is used and, if possible, include a URL.

  • The name of the license (e.g., CC-BY 4.0) should be included for each asset.

  • For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

  • If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

  • For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

  • If this information is not available online, the authors are encouraged to reach out to the asset’s creators.

13. New assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [Yes]

Justification: Our paper introduces three new assets: (1) a synthetic multimodal training dataset, (2) a curated real-world evaluation dataset, and (3) CoCo fine-tuned TSFM and LLM checkpoints. We provide detailed documentation for all assets to ensure reproducibility. Appendix E.2 and Appendix I describe the synthetic data generation pipeline, along with the prompts used. Appendix F.1 provides details on the numerical time series and textual annotations used to curate the real-world evaluation dataset. Appendix E.1 documents the fine-tuning procedures, hyperparameters, and training configurations for both the TSFM and LLM components. We additionally provide the fine-tuned checkpoints and inference pipeline as supplementary material, and have open-sourced the curated datasets (check the main manuscript for the links).

Guidelines:

  • The answer [N/A] means that the paper does not release new assets.

  • Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

  • The paper should discuss whether and how consent was obtained from people whose asset is used.

  • At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and research with human subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [N/A]

Justification: Our paper does not involve crowdsourcing nor research with human subjects.

Guidelines:

  • The answer [N/A] means that the paper does not involve crowdsourcing nor research with human subjects.

  • Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

  • According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional review board (IRB) approvals or equivalent for research with human subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [N/A]

Justification: Our paper does not involve crowdsourcing nor research with human subjects.

Guidelines:

  • The answer [N/A] means that the paper does not involve crowdsourcing nor research with human subjects.

  • Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

  • We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

  • For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

16. Declaration of LLM usage

Question: Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigor, or originality of the research, declaration is not required.

Answer: [Yes]

Justification: Our work utilizes LLMs to understand and interpret the effect of text on time series forecasting. Our approach involves fine-tuning LLMs for multimodal forecasting. We have described the fine-tuning procedure in Section 5 and described all the prompts that we used in the Appendix.

Guidelines:

  • The answer [N/A] means that the core method development in this research does not involve LLMs as any important, original, or non-standard components.

  • Please refer to our LLM policy in the NeurIPS handbook for what should or should not be described.

References

  1. [1]Rui Liu, Jiayou Liang, Haolong Chen, and Yujia Hu. Analyst reports and stock performance: Evidence from the chinese market, 2025. URL https://arxiv.org/abs/2411.08726.
  2. [2]Kevin Taylor and Jerry Ng. Natural language processing and multimodal stock price prediction, 2024. URL https://arxiv.org/abs/2401.01487.
  3. [3]Kamaladdin Fataliyev, Aneesh Chivukula, Mukesh Prasad, and Wei Liu. Stock market analysis with text data: A review, 2021. URL https://arxiv.org/abs/2106.12985.
  4. [4]Kemal Kirtac and Guido Germano. Sentiment trading with large language models. Finance Research Letters, 62:105227, April 2024. ISSN 1544-6123. doi: 10.1016/j.frl.2024.105227. URL http://dx.doi.org/10.1016/j.frl.2024.105227.
  5. [5]Ruize Gao, Feng Xiao, Jinpu Li, and Shaoze Cui. Textual semantics and machine learning methods for data product pricing, 2025. URL https://arxiv.org/abs/2511.22185.
  6. [6]Nway Nway Aung, Yuejingxian Mao, Chandra Suwandi Wijaya, Aryel Beck, Miura Koji, and Tajika Yosuke. Demand forecasting: Integrating textual and market data to navigate irregular market disruption. ADMIT ’24, page 270–280, New York, NY, USA, 2025. Association for Computing Machinery. ISBN 9798400718120. doi: 10.1145/3701100.3701158. URL https://doi.org/10.1145/3701100.3701158.
  7. [7]Abhimanyu Das, Weihao Kong, Rajat Sen, and Yichen Zhou. A decoder-only foundation model for time-series forecasting, 2024. URL https://arxiv.org/abs/2310.10688.
  8. [8]Abdul Fatir Ansari, Oleksandr Schur, Jaris Küken, Andreas Auer, Boran Han, Pedro Mercado, Syama Sundar Rangapuram, Huibin Shen, Lorenzo Stella, Xiyuan Zhang, Mononito Goswami, Shubham Kapoor, Danielle C. Maddix, Pablo Guerron, Tony Hu, Junming Yin, Nick Erickson, Prateek Mutalik Desai, Hao Wang, Huzefa Rangwala, George Karypis, Yuyang Wang, and Michael Bohlke-Schneider. Chronos-2: From univariate to universal forecasting, 2025. URL https://arxiv.org/abs/2510.15821.
  9. [9]An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, Chujie Zheng, Dayiheng Liu, Fan Zhou, Fei Huang, Feng Hu, Hao Ge, Haoran Wei, Huan Lin, Jialong Tang, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Yang, Jiaxi Yang, Jing Zhou, Jingren Zhou, Junyang Lin, Kai Dai, Keging Bao, Kexin Yang, Ke Yang, Lianqiao Deng, Mei Li, Mingfeng Xue, Mingze Li, Pei Zhang, Peng Wang, Qin Zhu, Rui Men, Ruize Gao, Shixuan Liu, Shuang Luo, Tianhao Li, Tianyi Tang, Wenbiao Yin, Xingzhang Ren, Xinyu Wang, Xinyu Zhang, Xuancheng Ren, Yang Fan, Yang Su, Yichang Zhang, Yinger Zhang, Yu Wan, Yuqiang Liu, Zekun Wang, Zeyu Cui, Zhenru Zhang, Zhipeng Zhou, and Zihan Qiu. Qwen3 technical report, 2025. URL https://arxiv.org/abs/2505.09388.
  10. [10]OpenAI. Gpt-4 technical report, 2024. URL https://arxiv.org/abs/2303.08774.
  11. [11]Haoxin Liu, Shangqing Xu, Zhiyuan Zhao, Lingkai Kong, Harshavardhan Kamarthi, Aditya B. Sasanur, Megha Sharma, Jiaming Cui, Qingsong Wen, Chao Zhang, and B. Aditya Prakash. Time-MMD: Multi-domain multimodal dataset for time series analysis, 2025. URL https://arxiv.org/abs/2406.08627.
  12. [12]Andrew Robert Williams, Arjun Ashok, Étienne Marcotte, Valentina Zantedeschi, Jithendaraa Subramanian, Roland Riachi, James Requima, Alexandre Lacoste, Irina Rish, Nicolas Chapados, and Alexandre Drouin. Context is key: A benchmark for forecasting with essential textual information. In Forty-second International Conference on Machine Learning, 2025. URL https://openreview.net/forum?id=jh2WuBT1Fn.
  13. [13]Zihan Dong, Xinyu Fan, and Zhiyuan Peng. Fnspid: A comprehensive financial news dataset in time series, 2024. URL https://arxiv.org/abs/2402.06698.
  14. [14]Junwoo Park, Hyuck Lee, Dohyun Lee, Daehoon Gwak, and Jaegul Choo. Revisiting llms as zero-shot time-series forecasters: Small noise can break large models, 2025. URL https://arxiv.org/abs/2506.00457.
  15. [15]Xinli Yu, Zheng Chen, Yuan Ling, Shujing Dong, Zongyi Liu, and Yanbin Lu. Temporal data meets llm–explainable financial time series forecasting. arXiv preprint arXiv:2306.11025, 2023.arxiv.org/abs/2306.11025
  16. [16]Xinlei Wang, Maike Feng, Jing Qiu, Jinjing Gu, and Junhua Zhao. From news to forecast: Integrating event analysis in llm-based time series forecasting with reflection. Advances in Neural Information Processing Systems, 37:58118–58153, 2024.arxiv.org/abs/2409.17515
  17. [17]Kai Kim, Howard Tsai, Rajat Sen, Abhimanyu Das, Zihao Zhou, Abhishek Tanpure, Mathew Luo, and Rose Yu. Multi-modal forecaster: Jointly predicting time series and textual data, 2024. URL https://arxiv.org/abs/2411.06735.
  18. [18]Ming Jin, Shiyu Wang, Lintao Ma, Zhixuan Chu, James Y. Zhang, Xiaoming Shi, Pin-Yu Chen, Yuxuan Liang, Yuan-Fang Li, Shirui Pan, and Qingsong Wen. Time-llm: Time series forecasting by reprogramming large language models, 2024. URL https://arxiv.org/abs/2310.01728.
  19. [19]Yueyang Yao, Jiajun Li, Xingyuan Dai, MengMeng Zhang, Xiaoyan Gong, Fei-Yue Wang, and Yisheng Lv. Context-aware probabilistic modeling with llm for multimodal time series forecasting, 2025. URL https://arxiv.org/abs/2505.10774.
  20. [20]Yushan Jiang, Wenchao Yu, Geon Lee, Dongjin Song, Kijung Shin, Wei Cheng, Yanchi Liu, and Haifeng Chen. Explainable multi-modal time series prediction with llm-in-the-loop. arXiv preprint arXiv:2503.01013, 2025.arxiv.org/abs/2503.01013
  21. [21]Mengpu Liu, Mengying Zhu, Xiuyuan Wang, Guofang Ma, Jianwei Yin, and Xiaolin Zheng. Echo-gl: Earnings calls-driven heterogeneous graph learning for stock movement prediction. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pages 13972–13980, 2024.
  22. [22]Ramit Sawhney, Arnay Wadhwa, Shiyam Agarwal, and Rajiv Shah. Fast: Financial news and tweet based time aware network for stock trading. In Proceedings of the 16th conference of the european chapter of the association for computational linguistics: main volume, pages 2164–2175, 2021.DOI
  23. [23]Yumo Xu and Shay B Cohen. Stock movement prediction from tweets and historical prices. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1970–1979, 2018.DOI
  24. [24]Huizhe Wu, Wei Zhang, Weiwei Shen, and Jun Wang. Hybrid deep sequential modeling for social text-driven stock prediction. In Proceedings of the 27th ACM international conference on information and knowledge management, pages 1627–1630, 2018.DOI
  25. [25]Zihan Chen, Lei Nico Zheng, Cheng Lu, Jialu Yuan, and Di Zhu. Chatgpt informed graph neural network for stock movement prediction. arXiv preprint arXiv:2306.03763, 2023.DOI
  26. [26]Shirly Wang, Matthew BA McDermott, Geeticka Chauhan, Marzyeh Ghassemi, Michael C Hughes, and Tristan Naumann. Mimic-extract: A data extraction, preprocessing, and representation pipeline for mimic-iii. In Proceedings of the ACM conference on health, inference, and learning, pages 222–235, 2020.DOI
  27. [27]Sandra Śmigiel, Krzysztof Pałczyński, and Damian Ledziński. Ecg signal classification using deep learning techniques based on the ptb-xl dataset. Entropy, 23(9):1121, 2021.DOI
  28. [28]Lecheng Zheng, Zhengzhang Chen, Dongjie Wang, Chengyuan Deng, Reon Matsuoka, and Haifeng Chen. Lemma-rca: A large multi-modal multi-domain dataset for root cause analysis, 2025. URL https://arxiv.org/abs/2406.05375.
  29. [29]Wei Chen, Xixuan Hao, Yuankai Wu, and Yuxuan Liang. Terra: A multimodal spatio-temporal dataset spanning the earth. Advances in Neural Information Processing Systems, 37:66329–66356, 2024.DOI
  30. [30]Zhijian Xu, Yuxuan Bian, Jianyuan Zhong, Xiangyu Wen, and Qiang Xu. Beyond trend and periodicity: Guide time series forecasting with textual cues.
  31. [31]Xin Zhou, Weiqing Wang, Francisco J. Baldán, Wray Buntine, and Christoph Bergmeir. Motime: A dataset suite for multimodal time series forecasting, 2025. URL https://arxiv.org/abs/2505.15072.
  32. [32]Xusen Guo, Qiming Zhang, Junyue Jiang, Mingxing Peng, Meixin Zhu, and Hao Frank Yang. Towards explainable traffic flow prediction with large language models. Communications in Transportation Research, 4:100150, 2024.DOI
  33. [33]Yaxuan Kong, Yiyuan Yang, Yoon tae Hwang, Wenjie Du, Stefan Zohren, Zhangyang Wang, Ming Jin, and Qingsong Wen. Time-mqa: Time series multi-task question answering with context enhancement. arXiv preprint arXiv:2503.01875, 2025.arxiv.org/abs/2503.01875
  34. [34]Qianqian Xie, Weiguang Han, Yanzhao Lai, Min Peng, and Jimin Huang. The wall street neophyte: A zero-shot analysis of chatgpt over multimodal stock movement prediction challenges. arXiv preprint arXiv:2304.05351, 2023.arxiv.org/abs/2304.05351
  35. [35]Ailing Zeng, Muxi Chen, Lei Zhang, and Qiang Xu. Are transformers effective for time series forecasting?, 2022. URL https://arxiv.org/abs/2205.13504.
  36. [36]Zhonghang Li, Lianghao Xia, Jiabin Tang, Yong Xu, Lei Shi, Long Xia, Dawei Yin, and Chao Huang. Urbangpt: Spatio-temporal large language models. In Proceedings of the 30th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, pages 5351–5362, 2024.arxiv.org/abs/2403.00813
  37. [37]Xin Liu, Daniel McDuff, Geza Kovacs, Isaac Galatzer-Levy, Jacob Sunshine, Jiening Zhan, Ming-Zher Poh, Shun Liao, Paolo Di Achille, and Shwetak Patel. Large language models are few-shot health learners. arXiv preprint arXiv:2305.15525, 2023.arxiv.org/abs/2305.15525
  38. [38]Furong Jia, Kevin Wang, Yixiang Zheng, Defu Cao, and Yan Liu. Gpt4mts: Prompt-based large language model for multimodal time-series forecasting. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pages 23343–23351, 2024.
  39. [39]Lu Han, Yu Liu, Lan Li, Qiwen Deng, Jian Jiang, Yinbo sun, Zhe Yu, Binfeng Wang, Xingyu Lu, Lintao Ma, Han-Jia Ye, and De-Chuan Zhan. UniCa: Unified covariate adaptation for time series foundation model. In The Fourteenth International Conference on Learning Representations, 2026. URL https://openreview.net/forum?id=I8q4MZb4OP.
  40. [40]Zihao Li, Xiao Lin, Zhining Liu, Jiaru Zou, Ziwei Wu, Lecheng Zheng, Dongqi Fu, Yada Zhu, Hendrik Hamann, Hanghang Tong, et al. Language in the flow of time: Time-series-paired texts weaved into a unified temporal narrative. In International Conference on Learning Representations, volume 2026, pages 24437–24484, 2026.
  41. [41]Xu Liu, Junfeng Hu, Yuan Li, Shizhe Diao, Yuxuan Liang, Bryan Hooi, and Roger Zimmermann. Unitime: A language-empowered unified model for cross-domain time series forecasting. In Proceedings of the ACM Web Conference 2024, pages 4095–4106, 2024.
  42. [42]Xiaobin Hong, Jiawen Zhang, Wenzhong Li, Sanglu Lu, and Jia Li. Unify and anchor: A context-aware transformer for cross-domain time series forecasting, 2025. URL https://arxiv.org/abs/2503.01157.
  43. [43]Chenxi Liu, Qianxiong Xu, Hao Miao, Sun Yang, Lingzheng Zhang, Cheng Long, Ziyue Li, and Rui Zhao. Timecma: Towards llm-empowered multivariate time series forecasting via cross-modality alignment, 2025. URL https://arxiv.org/abs/2406.01638.
  44. [44]Sameep Chattopadhyay, Pulkit Paliwal, Sai Shankar Narasimhan, Shubhankar Agarwal, and Sandeep P Chinchali. Context matters: Leveraging contextual features for time series forecasting. arXiv preprint arXiv:2410.12672, 2024.
  45. [45]Geon Lee, Wenchao Yu, Kijung Shin, Wei Cheng, and Haifeng Chen. Timecap: Learning to contextualize, augment, and predict time series events with large language model agents. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 39, pages 18082–18090, 2025.
  46. [46]Ramit Sawhney, Shivam Agarwal, Arnav Wadhwa, and Rajiv Shah. Deep attentive learning for stock movement prediction from social media text and company correlations. In Proceedings of the 2020 conference on empirical methods in natural language processing (EMNLP), pages 8415–8426, 2020.
  47. [47]Mayank Daswani, Mathias M. J. Bellaiche, Marc Wilson, Desislav Ivanov, Mikhail Papkov, Eva Schnider, Jing Tang, Kay Lamerigts, Gabriela Botea, Michael A. Sanchez, Yojan Patel, Shruthi Prabhakara, Shravya Shetty, and Umesh Telang. Plots unlock time-series understanding in multimodal models, 2024. URL https://arxiv.org/abs/2410.02637.
  48. [48]Zekun Li, Shiyang Li, and Xifeng Yan. Time series as images: Vision transformer for irregularly sampled time series, 2023. URL https://arxiv.org/abs/2303.12799.
  49. [49]Zihao Zhou and Rose Yu. Can llms understand time series anomalies?, 2025. URL https://arxiv.org/abs/2410.05440.
  50. [50]Siru Zhong, Weilin Ruan, Ming Jin, Huan Li, Qingsong Wen, and Yuxuan Liang. Time-vlm: Exploring multimodal vision-language models for augmented time series forecasting. arXiv preprint arXiv:2502.04395, 2025.
  51. [51]Léo Grinsztajn, Klemens Flöge, Oscar Key, Felix Birkel, Philipp Jund, Brendan Roof, Benjamin Jäger, Dominik Safaric, Simone Alessi, Adrian Hayler, Mihir Manium, Rosen Yu, Felix Jablonski, Shi Bin Hoo, Anurag Garg, Jake Robertson, Magnus Bühler, Vladyslav Moroshan, Lennart Purucker, Clara Cornu, Lilly Charlotte Wehrhahn, Alessandro Bonetto, Bernhard Schölkopf, Sauraj Gambhir, Noah Hollmann, and Frank Hutter. Tabpfn-2.5: Advancing the state of the art in tabular foundation models, 2026. URL https://arxiv.org/abs/2511.08667.
  52. [52]Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: pre-training of deep bidirectional transformers for language understanding. CoRR, abs/1810.04805, 2018. URL http://arxiv.org/abs/1810.04805.
  53. [53]Martin Arjovsky, Léon Bottou, Ishaan Gulrajani, and David Lopez-Paz. Invariant risk minimization. arXiv preprint arXiv:1907.02893, 2019.arxiv.org/abs/1907.02893
  54. [54]Josh Tobin, Rachel Fong, Alex Ray, Jonas Schneider, Wojciech Zaremba, and Pieter Abbeel. Domain randomization for transferring deep neural networks from simulation to the real world. In 2017 IEEE/RSJ international conference on intelligent robots and systems (IROS), pages 23–30. IEEE, 2017.
  55. [55]Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Minchuan Zhang, Y. K. Li, Y. Wu, and Daya Guo. Deepseekmath: Pushing the limits of mathematical reasoning in open language models, 2024. URL https://arxiv.org/abs/2402.03300.
  56. [56]Hal Roberts, Rahul Bhargava, Linas Valı̄ukas, Dennis Jen, Momin M. Malik, Cindy Sherman Bishop, Emily B. Nudule, Aashka Dave, Justin Clark, Bruce Etling, Robert Faris, Anushka Shah, Jasmin Rubinovitz, Alexis Hope, Catherine D’Ignazio, Fernando Bermejo, Yochai Benkler, and Ethan Zuckerman. Media cloud: Massive open source collection of global news on the open web. Proceedings of the International AAAI Conference on Web and Social Media, 15(1):1034–1045, May 2021. doi: 10.1609/icwsm.v15i1.18127. URL https://ojs.aaai.org/index.php/ICWSM/article/view/18127.
  57. [57]Taha Aksu, Gerald Woo, Juncheng Liu, Xu Liu, Chenghao Liu, Silvio Savarese, Caiming Xiong, and Doyen Sahoo. Gift-eval: A benchmark for general time series forecasting model evaluation. arXiv preprint arXiv:2410.10393, 2024.arxiv.org/abs/2410.10393
  58. [58]Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. Language models are unsupervised multitask learners. OpenAI blog, 1(8):9, 2019.
  59. [59]Vijay Ekambaram, Kushagra Manglik, Sumanta Mukherjee, Surya Shravan Kumar Sajja, Satyam Dwivedi, and Vikas Raykar. Attention based multi-modal new product sales time-series forecasting. In Proceedings of the 26th ACM SIGKDD international conference on knowledge discovery & data mining, pages 3110–3118, 2020.
  60. [60]Sai Shankar Narasimhan, Shubhankar Agarwal, Oguzhan Akcin, sujay sanghavi, and Sandeep P. Chinchali. Time weaver: A conditional time series generation model. In Forty-first International Conference on Machine Learning, 2024. URL https://openreview.net/forum?id=WpKDeixmFr.
  61. [61]Sai Shankar Narasimhan, Shubhankar Agarwal, Litu Rout, Sanjay Shakkottai, and Sandeep P. Chinchali. Constrained posterior sampling: Time series generation with hard constraints. In The Thirty-ninth Annual Conference on Neural Information Processing Systems, 2025. URL https://openreview.net/forum?id=iBLAYaEKdL.
  62. [62]Yunfeng Ge, Jiawei Li, Yiji Zhao, Haomin Wen, Zhao Li, Meikang Qiu, Hongyan Li, Ming Jin, and Shirui Pan. T2s: High-resolution time series generation with text-to-series diffusion models, 2025. URL https://arxiv.org/abs/2505.02417.
  63. [63]Noah Hollmann, Samuel Müller, Katharina Eggensperger, and Frank Hutter. Tabpfn: A transformer that solves small tabular classification problems in a second, 2023. URL https://arxiv.org/abs/2207.01848.
  64. [64]Edward J Hu, yelong shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. LoRA: Low-rank adaptation of large language models. In International Conference on Learning Representations, 2022. URL https://openreview.net/forum?id=nZeVKeeFYf9.arxiv.org/abs/2106.09685
  65. [65]Prime Intellect. Prime-rl: Agentic rl training at scale, 2025. URL https://github.com/PrimeIntellect-ai/prime-rl.
  66. [66]Woosuk Kwon, Z uohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph E. Gonzalez, Hao Zhang, and Ion Stoica. Efficient memory management for large language model serving with pagedattention, 2023. URL https://arxiv.org/abs/2309.06180.
  67. [67]Song Chen. Beijing Multi-Site Air Quality. UCI Machine Learning Repository, 2017. DOI: https://doi.org/10.24432/C5RK5G.
  68. [68]Andreas Auer, Patrick Podest, Daniel Klotz, Sebastian Böck, Günter Klambauer, and Sepp Hochreiter. Tixer: Zero-shot forecasting across long and short horizons with enhanced in-context learning, 2025. URL https://arxiv.org/abs/2505.23719.
  69. [69]Gerald Woo, Chenghao Liu, Akshat Kumar, Caiming Xiong, Silvio Savarese, and Doyen Sahoo. Unified training of universal time series forecasting transformers, 2024. URL https://arxiv.org/abs/2402.02592.
  70. [70]Ben Cohen, Emad Khwaja, Kan Wang, Charles Masson, Elise Ramé, Youssef Doubli, and Othmane Abou-Amal. Toto: Time series optimized transformer for observability, 2024. URL https://arxiv.org/abs/2407.07874.
  71. [71]Xiaoming Shi, Shiyu Wang, Yuqi Nie, Dianqi Li, Zhou Ye, Qingsong Wen, and Ming Jin. Time-moe: Billion-scale time series foundation models with mixture of experts, 2025. URL https://arxiv.org/abs/2409.16040.
  72. [72]Jeffrey Lai, Anthony Bao, and William Gilpin. Panda: A pretrained forecast model for chaotic dynamics, 2026. URL https://arxiv.org/abs/2505.13755.
  73. [73]Haoxin Liu, Yichen Zhou, Rajat Sen, B. Aditya Prakash, and Abhimanyu Das. Rethinking multimodal time-series forecasting evaluation, 2026. URL https://arxiv.org/abs/2607.06973.
  74. [74]Yuqi Nie, Nam H. Nguyen, Phanwidee Sinthong, and Jayant Kalagnanam. A time series is worth 64 words: Long-term forecasting with transformers, 2023. URL https://arxiv.org/abs/2211.14730.
  75. [75]Yong Liu, Tengge Hu, Haoran Zhang, Haixu Wu, Shiyu Wang, Lintao Ma, and Mingsheng Long. itransformer: Inverted transformers are effective for time series forecasting, 2024. URL https://arxiv.org/abs/2310.06625.
  76. [76]Yong Liu, Guo Qin, Xiangdong Huang, Jianmin Wang, and Mingsheng Long. Automates: Autoregressive time series forecasters via large language models, 2024. URL https://arxiv.org/abs/2402.02370.
  77. [77]Zijie Pan, Yushan Jiang, Sahil Garg, Anderson Schneider, Yuriy Nevmyvaka, and Dongjin Song. S²ip-llm: Semantic space informed prompt learning with llm for time series forecasting, 2024. URL https://arxiv.org/abs/2403.05798.
  78. [78]Chenxi Sun, Hongyan Li, Yaliang Li, and Shenda Hong. Test: Text prototype aligned embedding to activate llm’s ability for time series, 2024. URL https://arxiv.org/abs/2308.08241.
  79. [79]Siyuan Wang, Peng Chen, Yihang Wang, Wanhui Qiu, Chenjuan Guo, Bin Yang, and Yang Shu. Unlocking the value of text: Event-driven reasoning and multi-level alignment for time series forecasting, 2026. URL https://arxiv.org/abs/2603.15452.
  80. [80]Arpad E Elo. The proposed uscf rating system, its development, theory, and applications. Chess life, 22(8): 242–247, 1967.
  81. [81]Mer iem Boubdir, Edward Kim, Beyza Ermis, Sara Hooker, and Marzieh Fadaee. Elo uncovered: Robustness and best practices in language model evaluation, 2023. URL https://arxiv.org/abs/2311.17295.
  82. [82]Wei-Lin Chiang, Lianmin Zheng, Ying Sheng, Anastasios Nikolaos Angelopoulos, Tianle Li, Dacheng Li, Hao Zhang, Banghua Zhu, Michael Jordan, Joseph E. Gonzalez, and Ion Stoica. Chatbot arena: An open platform for evaluating llms by human preference, 2024. URL https://arxiv.org/abs/2403.04132.

Paper details

vlnc:W_7zv84amd
Version
v1 · latest
Published
Fresh — added to the Hub within the last 30 days
Added
License
CC BY 4.0
Usage

views · 2

downloads · 0

A view counts once per browser per day, a download each time the PDF is requested.

  • Computer Science

Not recorded for this paper

View statement
Looking for related papers…
Contents