betaFeedback

WorksComputer Science

Out of Support, Not Out of Reach: In-Context Learning with Unsupported Instructions and Composed Tasks

Abstract

In-Context Learning (ICL) enables pretrained LLMs to adapt to downstream tasks from a small set of input-output demonstrations, without parameter updates. Yet existing theory primarily explains ICL as identifying a latent task already represented in pretraining, leaving unclear whether ICL can succeed when either the instruction or the target task falls outside the pretraining support. We develop a theoretical framework under mild assumptions that separates these two forms of distribution shift. For target tasks represented in pretraining, we show that demonstrations can identify the task and reduce sensitivity to pretrain-supported prompt templates as their number increases. For an unsupported instruction, we show that under a fixed model-induced reference extension, demonstration evidence can dominate an NN-independent instruction shift while preserving the exponential identification rate. We then use the same ICL framework as an interpretive lens for long-horizon target tasks that are not represented in pretraining but admit a plan that decomposes them into pretraining-supported subtasks. Applying the supported-task analysis step by step yields a standard accumulated-error bound and clarifies how invoking familiar subtasks along an intermediate-state trajectory can avoid directly selecting the unsupported complete task. Together, our results characterize when demonstrations, instructions, and task decomposition can support ICL beyond the corresponding pretraining supports.

Introduction

In-Context Learning (ICL) ([5]) is widely used to adapt Large Language Models (LLMs) through an instruction and a small set of input–output demonstrations, without updating model parameters. A prominent line of theory explains ICL as implicit Bayesian inference over a latent task ([30, 26, 22]): the demonstrations provide evidence about which task generated the prompt, and prediction improves as the posterior concentrates on that task. The standard account relies on two support assumptions. First, the posterior ranges over latent tasks or concepts already represented by the pretraining mixture, so the target has positive prior probability under that mixture ([30, 29, 26]). Second, the instruction is typically omitted or absorbed into a fixed canonical prompt format, rather than modeled as a task-conditional variable that may differ across demonstrations ([3, 25, 2]). In deployment, however, both assumptions can fail.

Consider first the instruction. Real prompts can depart from the pretraining distribution in systematic ways. A user of Indian English may write please revert with the numbers, using revert to mean “reply,” while the same word is associated predominantly with “undo” in the model’s coding data. The term bias may denote a regression intercept to one user and a fairness property to another. A company’s internal service may retain the name of an open-source project even after its behavior has diverged. Other deviations are inconsistent rather than repeated — typos that differ from slot to slot, or demonstrations assembled by retrieval or by multiple annotators, each carrying its own rendering. What matters is the interaction of two axes: whether the instruction was seen paired with the intended task during pretraining, and whether one instruction governs the prompt or each demonstration carries its own. Table 1 formalizes these seven structures. They lead to our first research question: *When can additional demonstrations overcome an instruction outside the task-conditional pretraining support, and when do additional instructions introduce enough new shift to offset the evidence supplied by those demonstrations?

#SupportAppearancePrompt formatExample
1SupportedSinglez=(i,(xn,yn)n=1N,xquery)z = (i, (x_n, y_n)_{n=1}^{N}, x_{\mathrm{query}})A standard instruction stated once: “Classify ticket priority.”
2UnsupportedSinglez=(i~,(xn,yn)n=1N,xquery)z = (\tilde{i}, (x_n, y_n)_{n=1}^{N}, x_{\mathrm{query}})Indian-English revert used once to mean “reply.”
3SupportedRepeatedz=((i,xn,yn)n=1N,(i,xquery))z = ((i, x_n, y_n)_{n=1}^{N}, (i, x_{\mathrm{query}}))A standard schema key, e.g. task: sentiment, repeated in every record.
4UnsupportedRepeatedz=((i~,xn,yn)n=1N,(i~,xquery))z = ((\tilde{i}, x_n, y_n)_{n=1}^{N}, (\tilde{i}, x_{\mathrm{query}}))One private API label, e.g. operation: Atlas, reused in every record.
5SupportedVaryingz=((in,xn,yn)n=1N,(iquery,xquery))z = ((i_n, x_n, y_n)_{n=1}^{N}, (i_{\mathrm{query}}, x_{\mathrm{query}}))Retrieved examples using familiar paraphrases such as “sentiment” and “tone.”
6UnsupportedVaryingz=((i~n,xn,yn)n=1N,(i~query,xquery))z = ((\tilde{i}_n, x_n, y_n)_{n=1}^{N}, (\tilde{i}_{\mathrm{query}}, x_{\mathrm{query}}))Multi-tenant examples carrying distinct private labels such as triage_acme and route_7G.
7MixedCorruptedz=((in,xn,yn)n=1N,(iquery,xquery)), in∈{i,i~n}z = ((i_n, x_n, y_n)_{n=1}^{N}, (i_{\mathrm{query}}, x_{\mathrm{query}})),\ i_n \in\{i,\tilde{i}_n\}A standard template with sparse typos such as sentimnet and sentimemt.

Table 1. Seven prompt formats classified by support and instruction placement. Supported means that the instruction was paired with the intended task with positive probability during pretraining; unsupported means that this probability is zero. Appendix D.2 gives complete examples.

The other support assumption fails at the level of the target task itself. Language-model agents are increasingly asked to execute long-horizon objectives that may never have appeared as complete tasks during pretraining. At the same time, many such objectives admit plans composed of pretraining-supported skills: extracting a field, transforming a record, invoking a known operation, checking an intermediate state, and aggregating the results. A latent-task posterior cannot directly select a target task with zero pretraining probability, but an agent need not execute that task as one indivisible mapping. As an application of the ICL perspective, we ask: When can an out-of-support target task be realized by a plan that decomposes it into pretraining-supported subtasks, and how do step-wise errors accumulate over a long execution horizon? We study execution under a given task decomposition, rather than the separate problem of how an agent discovers that decomposition.

We address both questions in a common latent-task framework. For unsupported instructions, task evidence from demonstrations competes with the shift induced by the instruction structure. For long-horizon tasks, the same perspective is applied along an intermediate-state trajectory of supported subtasks. A pathwise Lipschitz analysis quantifies sensitivity along both types of paths.

Our Contributions. We distinguish instruction-level support from task-level support and characterize how generalization at these two levels is shaped by the number and provenance of demonstrations, instruction placement, and task decomposition. Our main contributions are summarized as follows.

  • Characterizing ICL with Supported and Unsupported Instructions (§ 3). Holding the target task within the pretraining task support, we distinguish instructions that lie inside or outside the corresponding task-conditional instruction support and analyze the seven prompt structures in Table 1. We show that informative demonstrations identify the target task and, under a fixed model-induced reference extension, preserve their exponential rate despite a fixed unsupported instruction whose continuation cost does not grow with the demonstration count. This is a conditional statement about the specified extension, rather than an identification of its off-support values from the pretraining distribution. For mixed or varying instructions, the instruction-shift cost can instead accumulate; convergence depends on whether task evidence grows faster than that cost.

  • Applying the ICL View to Long-Horizon Tasks through Supported Subtasks (§ 4). We use the preceding ICL framework to interpret a target task with zero pretraining probability. Although such a task cannot be selected directly by the pretrained latent-task posterior, it may be realized by a plan that composes supported subtasks along an intermediate-state trajectory. We bound the final error by the accumulated errors of the execution steps, revealing why a single poorly supported or unstable subtask can become the bottleneck.

  • Validating the Theoretical Statements through Experiments (§ 5). We test the predicted interaction between instruction structure and demonstration count, the pathwise characterization of prompt sensitivity, and the effect of decomposition quality on multi-step task execution.

Taken together, our instruction results offer a unified view of ICL beyond pretraining support. For a specified off-support reference extension, demonstrations can supply evidence for the intended latent task that dominates a fixed instruction-shift cost; realizing this behavior in the actual model further requires controlled model–reference error. Task decomposition, meanwhile, brings a novel long-horizon objective within the model’s repertoire of pretrained skills. In both cases, success depends on whether the useful evidence dominates the sensitivity accumulated along the path from the test prompt to what the model has already learned.

Problem Setup

This section formalizes the latent-task view of ICL and defines the seven prompt formats used throughout our analysis by distinguishing task-conditional instruction support and instruction placement.

Latent Task Model. Let V\mathcal{V} be a finite vocabulary and let S\mathcal{S} be a finite space of individual inputs, outputs, and intermediate states. Following Xie et al. [30], we assume that each sample (x,y)∈S×S(x,y) \in\mathcal{S} \times\mathcal{S} is generated from a latent task tt in a finite task set T\mathcal{T}. Conditioned on tt, X∼PX∣tX \sim P_{X\mid t}, Y=ft(X)Y = f_t(X), where ft:S→Sf_t : \mathcal{S} \to\mathcal{S} is the task-specific mapping.

We write P(⋅)P(\cdot) for distributions and p(⋅)p(\cdot) for their mass or density functions, and assume π(t)=P(T=t)>0\pi(t)=\mathbb{P}(T=t)>0 for every t∈Tt\in\mathcal{T}. § 4 will additionally consider downstream tasks t†∉Tt^\dagger\notin\mathcal{T}, for which we extend the pretraining prior by π(t†)=0\pi(t^\dagger)=0. An ICL prompt with NN demonstrations is written as zN=(i,(x1,y1),…,(xN,yN),xquery)z_N=(i,(x_1,y_1),\ldots,(x_N,y_N),x_{\mathrm{query}}); we omit the subscript and write zz when NN is clear from context. Here ii collects the instruction variables under the chosen prompt format and xqueryx_{\mathrm{query}} is the query input. Let D0D_0 denote the support of the pretraining prompt distribution across all values of NN. All almost-sure statements as N→∞N\to\infty refer to the length-NN prefixes of one infinite sequence of demonstrations generated by the stated mechanism.

Instructions. A single task may admit multiple instructions and prompt formats. For a pretraining-supported task t∈Tt\in\mathcal{T}, we model its instructions as draws from the task-conditional distribution Pi∣tP_{i\mid t}. Support is therefore task-conditional: for a target task t∗∈Tt^*\in\mathcal{T}, an instruction ii is supported if p(i∣t∗)>0p(i\mid t^*)>0 and unsupported if p(i∣t∗)=0p(i\mid t^*)=0. This definition asks whether pretraining ever paired the instruction with the intended task, not whether the words appeared in pretraining or whether a user regards them as semantically correct. For example, revert may be common in coding data but unsupported for the task of drafting a reply. The distinction matters because p(i∣t∗)p(i\mid t^*) enters the latent-task posterior directly: ordinary Bayesian conditioning applies to supported instructions, whereas an unsupported instruction requires the off-support analysis developed below.

Fix a maximum instruction length LiL_i, let e:V∪{⟨pad⟩}→Rde:\mathcal{V}\cup\{\langle\mathrm{pad}\rangle\}\to\mathbb{R}^d be the token embedding map with e(⟨pad⟩)=0e(\langle\mathrm{pad}\rangle)=0, and right-pad every instruction to length LiL_i. Write V≤Li:=⋃m=0LiVm\mathcal{V}^{\leq L_i}:=\bigcup_{m=0}^{L_i}\mathcal{V}^m. For i=(v1,…,vm)∈V≤Lii=(v_1,\ldots,v_m)\in\mathcal{V}^{\leq L_i}, define its padded embedding matrix as E(i):=[e(v1),…,e(vm),0,…,0]⊤∈RLi×dE(i):=[e(v_1),\ldots,e(v_m),0,\ldots,0]^\top\in\mathbb{R}^{L_i\times d}. Thus instructions with different token lengths lie in one common ambient space; Appendix § 14 provides the corresponding padding-stability justification. For each competing task t≠t∗t\ne t^* and supported instruction ii, define the task-relative instruction score rt(i):=log⁡(p(i∣t)/p(i∣t∗))r_t(i):=\log\left(p(i\mid t)/p(i\mid t^*)\right). The score compares how likely the instruction is under the competing task tt and the target task t∗t^*. For the continuous-path analysis, we use a fixed model-induced extension of this score along the instruction-embedding path defined below. We do not require the off-support extension to arise from separate normalized densities on the continuous instruction-embedding space. Its values away from the symbolic embeddings are additional structure made explicit in the continuation condition below, rather than something identified by the finite symbolic support. Appendix § 11 shows that an analytic Transformer scoring map supplies one concrete such extension and implies the required finite continuation chain.

Prompt Formats. The formats below model common ways in which prompts are assembled. A user or system message supplies one global instruction; a structured dataset may copy one field or operator into every record; and retrieval or multiple annotators may attach a different rendering to each demonstration. These placement patterns matter because a shared instruction contributes one piece of task evidence, whereas independently varying instructions can contribute a new likelihood factor with every demonstration. Crossing the three placement patterns—single, repeated, and varying—with supported versus unsupported instructions gives formats 1–6. Format 7 captures partial corruption, where a supported template remains in most slots but typos or schema errors affect a subset (Table 1). This distinction follows from the assumed generative factorization rather than from the Transformer architecture alone. Appendix § 15.§ 4 gives a controlled experiment showing that a Transformer can learn the corresponding effective instruction sample sizes from cross-position correlation structure.

The support labels in the table follow directly from the sampling mechanism. In formats 1, 3, and 5, every instruction is drawn from Pi∣t∗P_{i\mid t^*} and is therefore supported. Formats 2, 4, and 6 instead select instructions outside its support, so p(i∣t∗)=0p(i\mid t^*)=0. Format 7 contains both the shared supported template and distinct unsupported corruptions. For notational convenience, we use ii to denote the collection of all instructions appearing in a prompt, regardless of the specific prompt format. Unless otherwise specified, input-output pairs (X,Y)(X,Y) are drawn i.i.d. conditional on the latent task.

ICL with Supported and Unsupported Instructions

Throughout this section, the target task is pretraining-supported: t∗∈Tt^*\in\mathcal{T} and π(t∗)>0\pi(t^*)>0. The distinction studied below concerns only whether the instruction is supported conditional on that task.

Let M(y∣z)M(y\mid z) denote the output probability of the actual pretrained model. The Bayesian predictor p(y∣z)p(y\mid z) on supported prompts, and its fixed continuation pˉ(y∣z)\bar{p}(y\mid z) on unsupported prompts, serve as theoretical reference predictors, which need not coincide with MM exactly (see Appendix D.§ 1).

ICL with Pretraining-Supported Instructions

We first consider prompt formats whose instructions lie in the support of the pretraining distribution.

Assumption 3.1 (Identifiability of task evidence). For every pair of distinct tasks t1,t2∈Tt_1, t_2 \in\mathcal{T}, the demonstration distributions satisfy KL⁡(PX,Y∣t1∥PX,Y∣t2)>0\operatorname{KL}(P_{X,Y\mid t_1} \mathbin{\|} P_{X,Y\mid t_2}) > 0, and the corresponding demonstration log-likelihood ratio is integrable under PX,Y∣t1P_{X,Y\mid t_1}. For format 5, the log-likelihood ratio of the joint instruction–demonstration distribution is also integrable. Under the format-specific generation mechanism described in § 2,

KL⁡(Pi∣t1⊗PX,Y∣t1∥Pi∣t2⊗PX,Y∣t2)=KL⁡(Pi∣t1∥Pi∣t2)+KL⁡(PX,Y∣t1∥PX,Y∣t2)>0.(1)\operatorname{KL}(P_{i\mid t_1} \otimes P_{X,Y\mid t_1} \mathbin{\|} P_{i\mid t_2} \otimes P_{X,Y\mid t_2}) = \operatorname{KL}(P_{i\mid t_1} \mathbin{\|} P_{i\mid t_2}) + \operatorname{KL}(P_{X,Y\mid t_1} \mathbin{\|} P_{X,Y\mid t_2}) > 0. \tag*{(1)}

Thus the task evidence is identifiable in formats 1, 3, and 5, while the demonstration KL condition above also supplies the gap used in the unsupported-instruction results below.

Under this assumption, the demonstrations provide exponentially increasing evidence for the target task. In format 5, the independently sampled paraphrases form part of this task evidence rather than a single shared instruction factor.

Theorem 3.2 (Informal: model bound with supported instructions). Consider formats 1, 3, and 5 under Assumption 3.1. Let t∗t^* be the target task. Then there exist constants α,β>0\alpha, \beta> 0 and a finite model-approximation error εM≥0\varepsilon_M \ge0 such that, almost surely, for all sufficiently large NN,

∣M(yquery∣z)−p(yquery∣xquery,t∗)∣≤βe−αN+εM.(2)\left|M(y_{\mathrm{query}}\mid z) - p(y_{\mathrm{query}}\mid x_{\mathrm{query}}, t^*)\right| \le\beta e^{-\alpha N} + \varepsilon_M. \tag*{(2)}

The bound shows several key insights. The exponent α\alpha is controlled by the least distinguishable competing task: when every demonstration provides positive evidence for t∗t^*, posterior mass on the competitors decays exponentially with NN. The prefactor β\beta collects effects that do not grow with the demonstration count, including the task prior, the query distribution, and a shared instruction factor. Thus, in formats 1 and 3, copying the same supported instruction into more positions does not create additional independent evidence; the improvement with NN comes from the new input–output pairs. In format 5, by contrast, each supported paraphrase is part of an instruction–demonstration triple, so it can contribute additional evidence for identifying the task. See Appendix D.§ 4 for complete proof.

ICL with Pretraining-Unsupported Instructions

Single or repeated unsupported instructions.

We first consider formats 2 and 4 when the same unsupported instruction is used throughout the prompt. Format 2 contains one displayed occurrence; format 4 contains N+1N + 1 copies of one shared realization. The repeated copies in format 4 are not treated as N+1N + 1 independent task observations: the instruction template is sampled once and copied deterministically into all displayed slots. Thus, for every task tt,

p(z∣t)=p(i∣t)p(xquery∣t)∏n=1Np(xn,yn∣t).(3)p(z \mid t) = p(i \mid t)p(x_{\mathrm{query}} \mid t)\prod_{n=1}^{N}p(x_n, y_n \mid t). \tag*{(3)}

Format 2 is the corresponding single-occurrence special case.

Let i0i_0 be a supported anchor and i~\tilde{i} the unsupported endpoint. In the padded instruction-embedding space of § 2, define

Es:=(1−s)E(i0)+sE(i~),s∈[0,1].(4)E_s := (1-s)E(i_0) + sE(\tilde{i}), \qquad s \in[0,1]. \tag*{(4)}

For each competing task, we analytically continue its supported instruction log-likelihood ratio along this path. We denote the corresponding continued predictive probability by pˉ(yquery∣z)\bar{p}(y_{\mathrm{query}} \mid z). This continuation is a model-induced reference extension; because the endpoint has zero probability, its value is not identified by the pretraining distribution alone.

Under this condition, we can give an exponential bound on the continued reference predictor and a corresponding bound for the actual model.

References

  1. [1]Rishabh Agarwal, Avi Singh, Lei Zhang, Bernd Bohnet, Luis Rosias, Stephanie Chan, Biao Zhang, Ankesh Anand, Zaheer Abbas, Azade Nova, John D. Co-Reyes, Eric Chu, Feryal Behbahani, Aleksandra Faust, and Hugo Larochelle. Many-shot in-context learning. Advances in Neural Information Processing Systems, 37:76930–76966, 2024.arxiv.org/abs/2404.11018
  2. [2]Kwangjun Ahn, Xiang Cheng, Hadi Daneshmand, and Suvrit Sra. Transformers learn to implement preconditioned gradient descent for in-context learning. Advances in Neural Information Processing Systems, 36, 2023.arxiv.org/abs/2306.00297
  3. [3]Ekin Akyürek, Dale Schuurmans, Jacob Andreas, Tengyu Ma, and Denny Zhou. What learning algorithm is in-context learning? investigations with linear models. International Conference on Learning Representations, 2023.arxiv.org/abs/2211.15661
  4. [4]Yu Bai, Fan Chen, Huan Wang, Caiming Xiong, and Song Mei. Transformers as statisticians: Provable in-context learning with in-context algorithm selection. In Thirty-seventh Conference on Neural Information Processing Systems, 2023.arxiv.org/abs/2306.04637
  5. [5]Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 33:1877–1901, 2020.arxiv.org/abs/2005.14165
  6. [6]Nelson Elhage, Neel Nanda, Catherine Olsson, Tom Henighan, Nicholas Joseph, Ben Mann, Amanda Askell, Yuntao Bai, Anna Chen, Tom Conerly, Nova DasSarma, Dawn Drain, Deep Ganguli, Zac Hatfield-Dodds, Danny Hernandez, Andy Jones, Jackson Kernion, Liane Lovitt, Kamal Ndousse, Dario Amodei, Tom Brown, Jack Clark, Jared Kaplan, Sam McCandlish, and Chris Olah. A mathematical framework for transformer circuits. Transformer Circuits Thread, 2021.
  7. [7]Guhao Feng, Bohang Zhang, Yuntian Gu, Haotian Ye, Di He, and Liwei Wang. Towards revealing the mystery behind chain of thought: A theoretical perspective. Advances in Neural Information Processing Systems, 2023.arxiv.org/abs/2305.15408
  8. [8]Zixuan Gong, Xiaolin Hu, Huayi Tang, and Yong Liu. Towards auto-regressive next-token prediction: In-context learning emerges from generalization. In The Thirteenth International Conference on Learning Representations, 2025. URL https://openreview.net/forum?id=gK1rI98VRp.arxiv.org/abs/2502.17024
  9. [9]Jianliang He, Xintian Pan, Siyu Chen, and Zhuoran Yang. In-context linear regression demystified: Training dynamics and mechanistic interpretability of multi-head softmax attention. In Forty-second International Conference on Machine Learning, 2025. URL https://openreview.net/forum?id=3TM3fxwTps.arxiv.org/abs/2503.12734
  10. [12]Takeshi Kojima, Shixiang Shane Gu, Machel Reid, Yutaka Matsuo, and Yusuke Iwasawa. Large language models are zero-shot reasoners, 2023. URL https://arxiv.org/abs/2205.11916.
  11. [13]Hongbo Li, Lingjie Duan, and Yingbin Liang. Provable in-context learning of nonlinear regression with transformers, 2025. URL https://arxiv.org/abs/2507.20443.
  12. [14]Zhiyuan Li, Hong Liu, Denny Zhou, and Tengyu Ma. Chain of thought empowers transformers to solve inherently serial problems. International Conference on Learning Representations, 2024.arxiv.org/abs/2402.12875
  13. [15]Jiachang Liu, Dinghan Shen, Yizhe Zhang, William B. Dolan, Lawrence Carin, and Weizhu Chen. What makes good in-context examples for gpt-3? Proceedings of Deep Learning Inside Out (DeeLIO 2022): The 3rd Workshop on Knowledge Extraction and Integration for Deep Learning Architectures, pages 100–114, 2022.
  14. [16]Fei Lu and Yue Yu. Transformer learns the cross-task prior and regularization for in-context learning, 2025. URL https://arxiv.org/abs/2505.12138.
  15. [17]Arvind V. Mahankali, Tatsunori Hashimoto, and Tengyu Ma. One step of gradient descent is provably the optimal in-context learner with one layer of linear self-attention. In The Twelfth International Conference on Learning Representations, 2024.
  16. [18]William Merrill and Ashish Sabharwal. The expressive power of transformers with chain of thought. International Conference on Learning Representations, 2024.arxiv.org/abs/2310.07923
  17. [19]Sewon Min, Xinxi Lyu, Ari Holtzman, Mikel Artetxe, Mike Lewis, Hannaneh Hajishirzi, and Luke Zettlemoyer. Rethinking the role of demonstrations: What makes in-context learning work? Empirical Methods in Natural Language Processing, 2022.
  18. [20]Maxwell Nye, Anders Johan Andreassen, Guy Gur-Ari, Henryk Michałewski, Jacob Austin, David Bieber, David Dohan, Aitor Lewkowycz, Maarten Bosma, David Luan, Charles Sutton, and Augustus Odena. Show your work: Scratchpads for intermediate computation with language models, 2021. URL https://arxiv.org/abs/2112.00114.
  19. [21]Catherine Olsson, Nelson Elhage, Neel Nanda, Nicholas Joseph, Nova DasSarma, Tom Henighan, Ben Mann, Amanda Askell, Yuntao Bai, Anna Chen, Tom Conerly, Dawn Drain, Deep Ganguli, Zac Hatfield-Dodds, Danny Hernandez, Scott Johnston, Andy Jones, Jackson Kernion, Liane Lovitt, Kamal N dous se, Dario Amodei, Tom Brown, Jack Clark, Jared Kaplan, Sam McCandlish, and Chris Olah. In-context learning and induction heads, 2022. URL https://arxiv.org/abs/2209.11895.
  20. [22]Madhur Panwar, Kabir Ahuja, and Navin Goyal. In-context learning through the bayesian prism. In The Twelfth International Conference on Learning Representations, 2024. URL https://openreview.net/forum?id=HX5ujdsSon.arxiv.org/abs/2306.04891
  21. [23]Hongjin Su, Jungo Kasai, Chen Henry Wu, Weijia Shi, Tianlu Wang, Jiayi Xin, Rui Zhang, Mari Ostendorf, Luke Zettlemoyer, Noah A. Smith, and Tao Yu. Selective annotation makes language models better few-shot learners. In The Eleventh International Conference on Learning Representations, 2023.arxiv.org/abs/2209.01975
  22. [24]Eric Todd, Millicent L. Li, Arnab Sen Sharma, Aaron Mueller, Byron C. Wallace, and David Bau. Function vectors in large language models, 2024. URL https://arxiv.org/abs/2310.15213.
  23. [25]Johannes von Oswald, Eyvind Niklasson, Ettore Randazzo, João Sacramento, Alexander Mordvintsev, Andrey Zhmoginov, and Max Vladymyrov. Transformers learn in-context by gradient descent. International Conference on Machine Learning, 2023.arxiv.org/abs/2212.07677
  24. [26]Xinyi Wang, Wanrong Zhu, and William Yang Wang. Large language models are implicitly topic models: Explaining and finding good demonstrations for in-context learning. arXiv preprint arXiv:2301.11916, 2023.
  25. [27]Albert Webson and Ellie Pavlick. Do prompt-based models really understand the meaning of their prompts? In Marine Carpuat, Marie-Catherine de Marneffe, and Ivan Vladimir Meza Ruiz, editors, Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 2300–2344, Seattle, United States, July 2022. Association for Computational Linguistics. doi: 10.18653/v1/2022.naacl-main.167. URL https://aclanthology.org/2022.naacl-main.167/.DOI
  26. [28]Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia, Ed H. Chi, Quoc V. Le, and Denny Zhou. Chain-of-thought prompting elicits reasoning in large language models. Advances in neural information processing systems, 35:24824–24837, 2022.arxiv.org/abs/2201.11903
  27. [29]Noam Wies, Yoav Levine, and Amnon Shashua. The learnability of in-context learning. Advances in Neural Information Processing Systems, 2023.arxiv.org/abs/2303.07895
  28. [30]Sang Michael Xie, Aditi Raghunathan, Percy Liang, and Tengyu Ma. An explanation of in-context learning as implicit bayesian inference. International Conference on Learning Representations, 2022.arxiv.org/abs/2111.02080
  29. [31]Seonghyeon Ye, Hyeonbin Hwang, Sohee Yang, Hyeongu Yun, Yireun Kim, and Minjoon Seo. Investigating the effectiveness of task-agnostic prefix prompt for instruction following. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pages 19386–19394, 2024.arxiv.org/abs/2302.14691
  30. [32]Ruigi Zhang, Spencer Frei, and Peter L. Bartlett. Trained transformers learn linear models in-context. Journal of Machine Learning Research, 25(49):1–55, 2024.arxiv.org/abs/2306.09927
  31. [33]Zihao Zhao, Eric Wallace, Shi Feng, Dan Klein, and Sameer Singh. Calibrate before use: Improving few-shot performance of language models. International Conference on Machine Learning, 139:12697–12706, 2021.arxiv.org/abs/2102.09690

Paper details

vlnc:W_gxndujyd
Version
v1 · latest
Published
Fresh — added to the Hub within the last 30 days
Added
License
CC BY 4.0
Usage

views · 0

downloads · 0

A view counts once per browser per day, a download each time the PDF is requested.

  • Computer Science

Generative AI tools

View statement
Looking for related papers…
Contents