WorksComputer Science
Sampling Is Not Curiosity: Why LLM Agents Should Investigate
Abstract
LLM agents are increasingly used for open-ended tasks, very common in research scenarios where they are required to explore, investigate, or act curiously. At inference time, an LLM is a fixed map from context to next-token distribution, the pretrained weights determine the function, and only its conditioning changes as context accumulates. Mechanisms like temperature, best-of-N, tree search, or RAG vary how the distribution is sampled or what it is conditioned on, but do not update the agent's model of the environment. We argue that current LLM agents do not explore curiously; they explore stochastically. From how curiosity is understood in the RL literature, it requires a predictive structure over how the environment responds to the agent's actions, which improves online from current interaction, drives action selection through its predicted improvement, and persists beyond the context window. Additionally, in its open-ended form, the agent should intrinsically originate what to investigate. Thus, we propose a framework defining curiosity in LLM agents, reserving the term for systems that satisfy it. In doing so, we identify the research directions its implementation opens up.
Figure 1. Design of conceptual components for a curious agent system . The predictive component targets environment responses [C1] and updates from observations [C2]; its predicted improvement supplies the intrinsic reward shaping action selection through [C3]; persistent state retains improvements beyond the context window [C4]. In open-ended settings, it can also design its own prediction targets and environments [C5].
40th Conference on Neural Information Processing Systems (NeurIPS 2026). Position Paper Track.
Introduction
As LLM agents are deployed in more scenarios, they are increasingly requested to have agency and autonomy. However, the agents are not designed with that scenario in mind. In particular, as they go into more open-ended domains, such as research, we expect AI agents to propose and validate hypotheses, conduct experiments, and analyze results for further inquiry. Recent systems such as The AI Scientist [22], CORAL [26], or AlphaEvolve [23] frame their goal as open-ended exploration and report results that suggest the field is moving beyond task completion as the primary capability of interest. We identify a clear trajectory going forward, moving from requesting agents to complete tasks to open-ended tasks where the agent has to figure out how the successful task is defined. This shift in capability is not about better task-following, but instead the agent needs to decide what to do.
With this evolution of capabilities, the field has responded by adopting a vocabulary of curiosity, exploration, novelty, and intrinsic motivation. Search-style methods like Tree-of-Thoughts [45] or Graph-of-Thoughts [4] are framed as exploration of reasoning paths; Retrieval-Augmented (RAG) memory is offered as a foundation for learning from experience [21]; intrinsic-reward post-training methods are explicitly labeled curiosity-driven [6]. These terms are inspired by and borrowed from the reinforcement-learning tradition and the previous works on artificial curiosity [35], but the architectural commitments that gave them their meaning have not been carried over.
Traditionally, scientific research can be divided into two distinct tasks. On one side, (I) finding answers to given questions, and on the other side (II) coming up with good questions. Current LLM agent development is focused on improving their performance on (I), but it is not so focused on fulfilling (II). At inference time, an LLM samples from a fixed distribution determined by its pre-trained weights and conditioned on the given context. In agentic settings, the agent interacts with its environment; it takes actions, receives observations and conditions subsequent generations on what it has observed. Exploration in this sense happens during interaction. But it happens at the level of which inputs enter the predictive computation, not at the level of the computation itself. The function from context to next-token distribution remains unchanged. We call this distinction interaction-level exploration versus model-level adaptation. This distinction reflects an architectural fact about LLMs, that they discover useful information in an environment but observe without learning [9]. What varies in these mechanisms is how the model is sampled, not what the model encodes about the environment.
In this paper, we argue that current LLM agents do not explore curiously, they explore stochastically. Curiosity, in the sense inherited from the RL tradition, requires that the agent maintain a model of the domain it investigates, improve it from online interaction, drive action selection through its predicted improvement, and retain the knowledge in some way across the episode. In its open-ended form, it requires a further commitment that the agent originate what to investigate without being externally specified. The field should reserve the term curiosity for systems that satisfy these commitments to some degree, and recognize each of these commitments as an open problem for the future development of LLM agents.
Paper Position
Current LLM agents do not explore curiously. They explore stochastically, changing how they sample from a fixed distribution instead of updating a predictive model based on interaction. The field’s curiosity vocabulary should be reserved for systems that keep a model of the domain under investigation. These systems improve it from online interactions, use it to drive action selection, and retain it throughout the episode. In its open-ended form, curiosity further requires that the agent generate the prediction targets, instead of receiving them externally.
In this paper we do not introduce a new mechanism, as each ingredient of the framework exists in the curiosity tradition. Instead, we adapt it to LLM agents, whose conceptual environments, open-ended action spaces, rich priors, and undetermined prediction targets differ from the settings they were developed for. From these differences, we derive commitments that make the claims of curiosity explicit. The remainder of this paper discusses the requirements of curiosity from the RL tradition and its adaption to an LLM-agent setting (§ 2), deriving five commitments for curious agents (§ 3). We then use these commitments to diagnose the current practice (§ 4), and discuss alternative views (§ 5). We close with the open problems and future research directions (§ 6).
What Curiosity requires
The vocabulary of curiosity and exploration comes from decades of research in reinforcement learning, intrinsic motivation, and robotics that originated these terms. This section identifies the traditional requirements of curiosity and adapts them for LLM-based agents.
Curiosity at the System Level
An LLM at inference time is a function from context to next-token distribution , where are the pretrained weights, is the current context, and is the predictive distribution over the next token. Mechanisms often described as performing exploration in the reasoning space like temperature sampling, tree search [45], or retrieval augmentation [21], change how is sampled or what it is conditioned on, producing action distributions of the form , where is a sampling configuration and is a transformation (sampling rule, search strategy, retrieval mechanism) that produces an action distribution from ’s output. In multi-step scenarios, the context grows across the trajectory as actions are taken and come back from the environment, incrementally conditioned on the accumulating history. This is genuine interaction with an environment, in the traditional sense of environment exploration. However, in all cases, is invariant, and the function remains the same function before and after the agent observes any response from the environment. What varies in these cases is and the context on which is conditioned.
We refer to this distinction as interaction-level exploration, which contrasts with the model-level adaptation. Interaction-level exploration is the variation in actions, contexts, sampling configuration, or trajectories during the agent’s behavior, while maintaining the predictive function fixed. Model-level adaptation is the update of the predictive structure within the system such that the predictions about the environment improve over the episode. Throughout this work, we develop on top of model-level adaptation, under the premise that interaction-level exploration is necessary but not sufficient for curiosity given the traditional understanding of it. Whether some in-context computation could nonetheless implement some aspects of model-level adaptation is an open question that we comment on in Section 5.
Interaction-level Exploration vs. Model-level Adaptation. Interaction-level exploration varies actions and contexts while the predictive function remains fixed. Model-level adaptation updates the predictive structure itself from environment interaction.
Thus, when building LLM agents, we treat the LLM as a component of a larger system that can be represented as:
where the LLM, , forms part of the system, alongside a predictive structure over the environment responses, persistent state , and an action-selection policy . We treat the system , irrespective of the methods used to implement each one of the components, which may or may not contain different calls to the LLM. For instance, the predictive structure can live in many places, like in natural-language hypotheses updated by Bayesian inference [20], in distilled-memory representations, structured data, latent variables, or in different updates of .
The Curiosity Tradition
Curiosity can be defined as the intrinsic drive to explore, seek understanding, and ask questions in order to learn and make sense of the unknown. In psychology, Berlyne [3] described curiosity as a motivational drive triggered by collative variables such as novelty, surprise, complexity, and incongruity that create a state of epistemic conflict or uncertainty, which increases the level of arousal. People search for an optimal arousal level such that when it is too high, they seek information to reduce uncertainty (curiosity), or when it is too low, they search for how to increase it (exploration).
For decades, researchers have been exploring how to transform curiosity into a computational feature [38]. This translation is not trivial, but there is a notable convergence on what curiosity as an intrinsic reward requires. In early formulations, the agent is rewarded for encountering surprising observations [33], grounding the intrinsic motivation in the prediction error of a learnable world model. This formulation faces a well-known difficulty in stochastic environments, known as the noisy TV problem [35, 5]. The reward can cause the agent to focus on irreducible noise because its prediction error remains high. This was later refined by introducing the learning progress which does not reward the error itself, but its rate reduction [32]. This shift relocated curiosity from “what surprises the agent” to “what the agent is coming to understand”. Subsequent works have formalized this in various ways, such as expected information gain about the environmental dynamics [40, 14, 41] or developmental skill acquisition [25]. But the core structural claim is about getting better at predicting the environment from interacting with it.
Schmidhuber Schmidhuber [35] provides the formalization of the framework, unifying the threads of compression progress where the agent maintains a predictive component of the environment, and intrinsic reward proportional to the rate at which the agent’s history shortens as new observations arrive. Additionally, PowerPlay [36] extends the framework further to the case when the agent must originate its own environments, generating its own questions in pursuit of compression progress. We see a similar behavior in [12] proposing an “artificial scientist” that does not only answer given questions, but continually invents its own questions driven by intrinsic rewards for surprising outcomes.
In these formulations from the traditional understanding of computational curiosity, we identify four recurring ingredients that are defining features: (i) a learnable predictive component over the environment, (ii) online update of that component from current interaction, (iii) intrinsic reward derived from the system improvement, (iv) action selection in relation to the reward. An extra fifth element (v), the generation of prediction targets, is required when the environment does not provide them. These are the core elements identified from the curiosity tradition. Proposals for learned world models share the same predictive core, using predictions of the environment to plan [11], but our focus is on what is required for that prediction to drive exploration in LLM agents.
Adapting the Framework to LLM Agents
The artificial curiosity tradition was developed for problems of embodied agents that interact in environments with regular state-transition dynamics. However, the community is currently applying these ideas to LLMs, and we need make the differences explicit to inform our adaptation:
State-space environments. The original predictive components were designed for pixel transitions, sensor readings, or game states. LLM agents operate in more conceptual environments such as codebases, literature, web ecosystems, or scientific domains. These environments have some notion of dynamics, but they are mainly conceptual and rarely admit tidy state-space representations.
Given action spaces. Embodied agents have a discrete set of actions, and the agent selects from a well-defined set. By contrast, LLM agents act through natural language, tool calls, and queries. This expands the action space, and its structure is something the agent must reason about.
No priors. Embodied-agent formulations typically assume that the agent starts ignorant and learns from scratch. LLMs, however, have a vast amount of general-knowledge data pre-loaded in their weights, which is already doing real work before any episode begins.
Externally given prediction targets. In embodied environments, the prediction target is provided by the environment itself. State transitions are what the agents are meant to learn and model. In an open-ended LLM-agent setting, the prediction target is itself undetermined; the agent must decide what subgoals, hypotheses, or mechanisms are worth understanding.
These differences motivate three adaptations of the framework, each grounding one of the commitments derived below:
(A) Prediction target. The prediction does not need to target specific state transitions or operate in a fixed action space. Now, the prediction target can be a model of how a codebase responds to commands, a hypothesis about a physical system, or any other structure whose accuracy improves as the agent gathers evidence. What survives the generalization is that the agent must predict how the environment responds to the agent’s actions, not properties of the agent’s own outputs, which rules out signals defined over the LLM’s own generation. (B) Priors. Pretraining gives the agent a head start, but this cannot be confused with episode-time learning. The agent now has to distinguish what the agent already knows, from what the agent learned from interacting with the environment. Without this distinction, improvement due to priors could be mistaken for learning.
(C) Open-endedness. In embodied environments, the environment determines what is worth investigating. In open-ended settings, nothing plays this role, so the agent must choose its own lines of inquiry, and that choice becomes part of curiosity and not something external to it.
The Curiosity Framework: Commitments for Curious LLM Agents
From the curiosity tradition and our adaptation, we derive five architectural commitments for curiosity in LLM agents. The first four are derived from the ideas on prediction improvement driving actions. The fifth one derives from the open-endedness adaptation and applies specifically to settings where the prediction targets are not externally supplied.
[C1] Predict Environment. The agent maintains a predictive component over how the environment responds to its actions, with the prediction target being environmental dynamics rather than properties of the agent’s own generation.
The agent maintains predictions about how the environment responds to its actions. The predictive structure can live anywhere in the system, not necessarily as part of the LLM. It can live in parametric updates, hypothesis state, a memory, or structured data. But the target of the prediction is the environmental dynamics, not the agent’s own outputs. A system can be uncertain about its own outputs in ways unrelated to what it knows about the world it is acting in.
[C2] Update from Interaction. The prediction improves within the episode, and the improvement is caused by current observations.
Predictions improve during the episode, grounded in the current interaction with this environment and not retrieved from priors. By online, we mean within-episode, and grounded means caused by the observations of the environment. This rules out improvements that derive from pretraining alone, as the LLM already encodes a substantial amount of world knowledge in its weights. It also rules out improvements that come from retrieval over a fixed external corpus. Both can produce better-conditioned predictions without any update driven by what the agent has actually observed. The agent has to learn something from the current interaction.
[C3] Act to Learn. Action selection is shaped by where the predictive component is expected to improve, so the agent acts to learn rather than learning while acting.
The first two commitments give the agent a predictive component and a mechanism to update it. This commitment closes the loop, stating that actions are chosen because their outcomes are expected to be informative for the predictive component, and the resulting observations feed back into the update. Without this, updates may occur, but they do not shape what the agent does next. This separates the framework from systems where action selection is driven entirely by task reward. Curiosity requires that action selection respond to expected predictive improvement, whether measured as prediction error, information gain, or learning progress.
[C4] Retain the Improvement. Improvements to the predictive component persist in some form beyond the context window in which they were formed.
The persistence requirement is about availability for future action selection, not about storage. Improvements derived from interaction remain accessible to shape what the agent does later, and this access cannot rely on the observations remaining in context. The commitment is method-agnostic, but it rules out systems in which the only source of improvement is the immediate context. The commitment requires the system to carry the improvement beyond the boundary of the context.
[C5] Choose What to Investigate (Open-Endedness). The agent originates the prediction target itself, instead of receiving it from the environment or from a human specification of the problem.
The first four commitments together define curiosity in its task-driven form, where an agent investigates a given domain, improving its predictions about that domain, and acting to drive that improvement. In the division of the two important scientific tasks, this would correspond to (I) finding answers, which is externally specified. The fifth commitment addresses the second task, (II) coming up with good questions. It applies specifically to the open-ended setting where prediction targets are not externally supplied. The agents originates what to model and what counts as worth understanding. It can propose its own fitness function, benchmark, or problem domain. This commitment does not raise the bar for curiosity in general, an agent that designs experiments and hypotheses within a human-provided domain is a curious agent in the task-driven form under C1–C4. C5 distinguishes an additional frontier of open-ended curiosity, where the prediction target itself is not given.
♦ Curious Agents We call systems satisfying C1–C4 curious agents in the task-driven form, and systems satisfying C1–C5 open-ended curious agents, with C5 extending curiosity to settings where the prediction target is not supplied. The category is computational, not behavioral; a system is a curious agent if it demonstrably satisfies the commitments, and the rest of the paper takes the commitments as our criterion for the term.
Example To make this concrete, consider an agent investigating a new material:
Stochastic Exploration. The agent generates diverse candidate hypotheses (e.g., via temperature or best-of-N), tests them in arbitrary order, and appends failed experiments to its context. It has no prediction of which experiment would most reduce its uncertainty.
Curious Exploration. The agent maintains competing hypotheses about the material [C1], predicts the outcome of candidate experiments under each and runs the most informative one [C3], updates its hypotheses on the result [C2], and carries the update forward to shape the next experiment [C4].
Both agents produce multiple actions, interact with the environment, and may use memory, but only the second closes the loop from prediction to action update, selecting actions that maximize learning.
Diagnosing Current Practice
In the previous sections, we went over what is required for curiosity and the translation of the framework to LLM agents. We now turn that framework on the field to see how current works relate to our commitments. Applying the commitments requires distinguishing two kinds of systems:
Explicit systems. The components of the system, , the predictive component , update rule, and the policy are externally inspectable. Whether the commitments are met can be analyzed from the system’s design.
Implicit systems. The relevant computation may happen inside the architecture, and causal identification is substantially harder. In these cases, the conclusion is that curiosity has not been demonstrated, not that the system is not curious.
The systems we discuss here are explicit, and our verdicts follow from the designs stated in their own papers. We close with examples that show how to achieve the architectural targets.
The wrong prediction target A common pattern is the implementation of methods using curiosity or exploration signals that are computed over the model’s own output rather than over the environmental responses from the actions. Curiosity-Driven Exploration (CDE) [6] uses per-response perplexity as a curiosity bonus, computing novelty over generated trajectories; CoVo [47] combines a consistency-of-trajectory intrinsic reward with a perplexity-based curiosity bonus, both computed over the model’s own reasoning paths; and a study of ideation diversity measures the variation of model proposals [2]. In these cases, the signal driving exploration is internal to the language model and mainly tracks whether the model produces varied output, and not whether the agent is uncovering new knowledge in the environment.
This differs from our framework, where commitment [C1] concerns a prediction over environmental responses, distinct from the language model’s generation distribution, improving through interaction. A coverage objective over output space gives the agent no insight into how the world responds, because the question is never asked. In these cases, the systems are doing real work, like improving robustness, escaping local minima, or broadening hypothesis search, but the work is happening at a different level than curiosity understood in the traditional sense.
One could argue that diverse generation is doing the productive work by broadening the search space. Audran-Reiss et al. [2] documents this by mentioning that ideation diversity provides robustness through redundancy. This is indeed useful. But we would say this differs from what is known as computational curiosity. This idea of robustness through redundancy works by sampling more broadly from a fixed distribution, which can hardly change the agent’s model of the world.
Memory mistaken for adaptation Another pattern identified is the use of persistent storage as the basis of online learning. A popular standard is to maintain a memory of reflections on prior failures [39]. Similar works like Voyager [42] accumulate a growing library of skills of executable code, retrieved by task-embedding similarity. These systems retain the structure of the episode from multiple attempts, and their stored content updates during interaction.
The architectural question is what gets updated. The framework describes a predictive component that improves from the interaction with the environment, with the improvement shaping the next actions. In this line of work, we see that the retrieved content can change what the policy does, but the loop from prediction to action, which involves selecting actions because their outcomes are expected to reduce predictive error, is not present. Neither Reflexion nor Voyager contains a component that scores predictions about environmental responses, so there is no signal to which action selection could respond. Action selection is driven by task reward and the augmented context, not by where predictions are expected to improve.
Adjusting which stored traces to retrieve is a meaningful operation, but it is a different operation from selecting actions whose outcomes the agent expects to be informative. The first changes how the agent acts on what it already believes, the second changes what the agent investigates. The limitation is therefore not memory itself, but what is stored and how it is used. A memory-based system could satisfy the commitments, but it is eventually limited because memory acts only through the bounded context, so at scale, weight updates may be needed for learning to last. The relation between memory and action-driven prediction remains an open architectural question.
Externally-supplied inquiry direction A third direction is the range of systems that orchestrate exploration over a problem space defined by humans, which have made a significant impact and outstanding progress. The AI Scientist [22], its v2 successor [44], AlphaEvolve [23], FunSearch [29], or Coral [26], to name some popular ones, all execute impressive long-horizon work. They generate research hypotheses, evolve programs toward score-maximizing objectives, and discover mathematical algorithms inside human-defined search spaces.
Several of these systems explicitly position themselves as open-ended discovery, targeting open scientific problems. This sense of open-endedness, where there is a large search space, unbounded progress against an unknown optimum point, is real, and these systems make great progress. They can generate their own subgoals within a task, which candidate to inspect or what is still to distill. This is a genuine inquiry direction at the sub-goal level. Under our open-ended commitment [C5], the agents should eventually be able to formulate the problems themselves, for instance by proposing their own fitness function, benchmark, or problem domain, and refine them based on their own predictive structure. In current systems, the frame is still supplied by humans and remains fixed; the agent’s autonomy operates inside it. The framework’s open-ended form asks for agents that generate inquiry direction, achieving the scientific frontier of generating their own unsolved questions.
What architectural engagement looks like The framework’s commitments describe an achievable target, not an aspirational one. WorldLLM [20] shows an example of the four task-driven commitments, without modifying the weights. The system maintains a set of natural-language hypotheses over environmental responses, which are updated during interaction through an LLM inference loop, conditioned on observed outcomes. With no extrinsic reward, action selection is driven entirely by intrinsic signals from these hypotheses, where prediction error performing the best, similar to Schmidhuber [33], while a learning-progress variant underperformed for estimation reasons. SENSEI [30] shows the potential with parameter updates, training a world model learned from interaction, with an exploration policy that optimizes high epistemic uncertainty, so improvements persist in the weights by construction. AutoDiscovery [1] applies the loop to data-driven discovery. The LLM’s belief in a hypothesis is generated before and after an experiment runs on real data, and the resulting Bayesian surprise rewards a tree search over hypotheses, with results persisting in the tree and not in an updated model. In all three, what the agent does next depends on what it expects to learn.
None of these fully engages the open-ended commitment [C5]. WorldLLM’s hypothesis space is defined by human specification of the environment, and SENSEI’s notion of interestingness is fixed before exploration begins. AutoDiscovery comes closest, choosing which questions to pursue without a specified research goal, but within human-supplied datasets. Work on autotelic agents [10, 46] explores parts of these problems. Nevertheless, engaging all five commitments simultaneously remains an unsolved task, but the architectural components are available in different forms. Stating the commitments jointly is what makes this gap visible.
Alternative Views
In-Context Learning (ICL) is Compression Progress As an LLM agent accumulates context from observations, prior actions, and retrieved memory, its conditional distribution differs from the one induced by the task input alone, . The amount of context agents can maintain has grown alongside progress in LLMs, although sometimes at the cost of quality. The conditional becomes more accurate over the course of an episode as more relevant context accumulates. This is, in a precise sense, predictive improvement driven by interaction. It has been shown that in-context learning can be seen as a form of implicit Bayesian inference over latent concepts in the pretraining distribution [43], where the conditional distribution updates as context accumulates and concentrates on the concept consistent with the prompt examples. Under this view, pretrained knowledge can supply the predictive component [C1], and the in-context update is caused by current observations rather than by priors or retrieval, so it can satisfy [C2] as stated, but with some limitations.
In-context Learning can be seen as an explicit boundary case of the framework, instead of a counterexample. What changes during ICL is the conditioning of the model but not the model itself, the map from context to next-token distribution is fixed and the input varies. This is where we can claim that current agents sample from a fixed distribution. Granting [C1] and [C2], the commitments that separate stochastic from curious exploration are [C3] and [C4]. By itself, ICL does not fulfill [C3], the conditional distribution can update as context accumulates, but action selection is not driven by where the predictive distribution is expected to improve. The agent’s actions are driven by task reward, by retrieved exemplars, and by the policy’s response to the augmented context. ICL provides the predictive update, but not the action-selection criterion that makes the update operative as curiosity. Second it does not satisfy [C4], the model context limit puts a hard constraint on the persistence of the updates, so improvements that live only in the conditioning are evicted with the observations that produced them. Neither failure can be closed by prompting or sampling differently. The framework does not rule out in-context learning as a direction towards curious agents; on the contrary. A system built around in-context learning as a direction towards curious agents; on the contrary. A system built around in-context predictive improvement can lead to curiosity when designed around predictive improvement, with action selection shaped by where the improvement is expected and the persistent state. [8] shows an example toward [C3], where they train a policy to maximize learning progress of a frozen in-context learner, but show this works only when actions do not change the environment, also leaving [C4] unaddressed.
Tool-using agents are interacting with the environment Modern deployed agents, such as coding agents, web agents, or scientific agents, have access to tools and interact with an environment. They can launch commands, observe the result, and review their behavior. One could argue that environmental interaction is a basis of curiosity, and these systems already have it.
Interaction, however, is necessary, but not sufficient. We argue that interaction-level exploration, where an agent performs actions and observes the results, is different from model-level adaptation, where the agent has a predictive component that improves from those observations. Tool usage by itself satisfies the condition of interaction. But a current tool-using LLM agent does not measurably improve a prediction from its observations. This has been reported by Engländer et al. [9], highlighting that agents discover information in their environment, but fail to act on it.
RL-finetuning encodes in weights At inference time, the map is fixed and only its conditioning varies, but RL fine-tuning does update the model weights. Across training, the model’s distribution shifts to better match the environment. This can partially match the requirement of prediction improvement from our commitment [C2]. However, there is a mismatch in timescale, as current RL finetuning method still still separates training from deployment. Weight updates from standard RL fine-tuning happen across training episodes, not within an episode in response to current observations. Curiosity is operative in the moment of action selection; the predictive improvement has to be available to shape what the agent does next. Weight updates from yesterday’s training run produce a better-calibrated policy, but not a system that revises its predictions in response to what it just observed and acts on the revision.
This drives the argument to approaches closer to Meta-RL and Continual Learning, which complicates this picture in a different way. Meta-RL [34, 37, 17, 19] learn to implement an RL algorithm at inference time, allowing easy deployment to new tasks. Continual learning [28, 27, 18, 15] targets a sequential task stream and seeks weight updates that persist throughout different tasks without catastrophic forgetting. Streaming online RL pushes weight updates to episode time directly. Where standard RL fine-tuning places the predictive update across training episodes, these approaches push the update closer to the episode time.
Research Directions
Throughout this paper, we have discussed what curiosity in LLM-based agents requires. We have kept the formulation technology-agnostic and used it to diagnose where current practice falls short. The framework names four commitments and one extension. We identify some areas of research derived from these commitments.
Open Problems
Recognizing predictive components Would we recognize a curious system if we saw one? We have established a framework of curiosity for LLMs that is agnostic to where the predictive component lives, whether it is in parametric updates, hypothesis state, or structured memory. As a result, how we recognize it depends on the system. For explicit systems, the commitments can be checked against the design, as we do in § 4. For implicit systems, where the computation may happen inside the architecture, causal identification is substantially harder. In these cases, we suggest behavioral tests, which apply without inspecting internals. They are a starting point, where passing them shows that a system behaves as the commitments require, but not whether or how its predictive component is implemented. Until an implicit system is tested, the correct conclusion is that curiosity has not been demonstrated, not that the system is not curious. In particular, if a transformer-based system contains a predictive component, how could we evaluate it? Mechanistic interpretability [24] can provide tools to better evaluate LLMs in this context, but developing such operational notion remains an open question.
| Behavioral tests for the task-driven commitments | |
| [C1] Predict Environment | [C2] Update from Interaction |
| Before acting, the agent predicts the outcome of action in a state ; the prediction is scored against the observed response. | Prediction error on held-out responses drops after a controlled set of observations, more than for a control agent that does not receive them. |
| [C3] Act to Learn | [C4] Retain the Improvement |
| Among actions with similar task value, the agent prefers the more informative one, and the preference disappears when the intrinsic signal is ablated. | Prediction accuracy survives a context reset that keeps the persistent state , but not one that also wipes . |
Table 1. Behavioral tests for the task-driven commitments
Within-episode measurement Does the agent’s prediction actually improve as it acts? In large part, the commitments presented in the framework assume that the predictions improve from current observations [C2] and that action selection responds to where the improvement is expected [C3]. Current agent benchmarks for agentic discovery and exploration, like DiscoveryWorld [16] or MLAgentBench [13], do not expose these dynamics. They are mainly conceived to score their outcomes, not predictive trajectories. Even ambitious open-ended works like the AI Scientist [22] are scored on the artifacts produced, not what the agent discovered. The key difficulty is attribution, distinguishing improvement caused by the current observations from improvement attributable to priors, retrieval, or in-context patterns.
Combining with task reward How does the curiosity signal coexist with the agent’s actual job? Real agents have task rewards, user requests, and score-maximization objectives (extrinsic signal). The intrinsic signal has to deal with the extrinsic rewards without overwhelming or being overwhelmed by them. This is a version of the exploration vs. exploitation trade-off. The RL literature offers partial answers. Some forms of adversarial curiosity, in which two modules bet against each other on the outcomes of computational experiments, were even shown to accelerate the intake of external reward [31]. Meta-RL and continual-learning approaches bring updates closer to episode time. However, a key problem when applying intrinsic rewards to LLMs is how an intrinsic-improvement signal is weighted against an extrinsic objective in agents whose action space is open natural language and tool calls. This area requires further ideas for deployment.
Choosing what to predict What should the agent be trying to predict in the first place? A key aspect of curiosity is that it requires improvement over an environment’s predictions [C1], but the target of those predictions remains unspecified; natural-language hypotheses, latent variables, or a distilled memory can qualify, but we are not limited by them. For instance, WorldLLM [20] gives the agent structured natural-language hypotheses to refine, but the hypothesis space comes from a human specification of the environment. It remains an open problem to select what the agent should predict, and even more in the open-ended scenario [C5]. Beyond single agents, curious agents could also learn together, sharing what they learn about the environment so that each one’s progress informs what the others investigate, as explored for cooperative bandits with heterogeneous agents [7].
The path of the field forward
Few LLM agents today can satisfy these commitments in a meaningful sense (§ 6.1). But agents are deployed in increasingly open-ended tasks, where the difficulty is no longer to complete a specific task, but to figure out the job and what is worth knowing about it. The gap between sampling and curiosity becomes harder to avoid. However, we do not know which architecture closes that gap. Sampling does not produce what curiosity vocabulary is meant to describe. Reserving curiosity costs little, since a claim need only state which commitments it meets, while using the term loosely makes the open problem impossible to state. Building toward systems that satisfy the commitments is worth the work, and worth doing under a name that brings it together.
Conclusion
Our work argues that current LLM agents do not explore curiously, they explore stochastically. The map from context to output distribution is fixed, while what changes is how it is sampled and what it is conditioned on. From the curiosity tradition, we identify that curiosity requires a prediction over environment responses, updated online from current interaction, driving action selection through the improvement it is expected to yield, and persisting beyond the context that produced it. In open-ended settings, it further requires the agent to produce its own prediction targets. From this, we derived commitments that translate this concept to LLM agents, and used them to diagnose current developments in the field, showing where they fall short or where they approach this concept. This analysis identifies the components of a curious agent, and with them the problems to solve to build more curious agents.
As agents increase their capabilities and are deployed in more scenarios, the challenge shifts from completing specified tasks to deciding what is worth investigating, and from there to closing the gap between task completion and curiosity while keeping that curiosity aligned with human interests.
References
- [1]Dhruv Agarwal, Bodhisattwa Prasad Majumder, Reece Adamson, Megha Chakravorty, Satvika Reddy Gavireddy, Aditya Parashar, Harshit Surana, Bhavana Dalvi Mishra, Andrew McCallum, Ashish Sabharwal, et al. Autodiscovery: Open-ended scientific discovery via bayesian surprise. Advances in Neural Information Processing Systems, 38:25181–25219, 2026.arxiv.org/abs/2507.00310
- [2]Alexis Audran-Reiss, Jordi Armengol-EstapÊ, Karen Hambardzumyan, Amar Budhiraja, Martin Josifoski, Edan Toledo, Rishi Hazra, Despoina Magka, Michael Shvartsman, Parth Pathak, et al. What does it take to be a good ai research agent? studying the role of ideation diversity. arXiv preprint arXiv:2511.15593, 2025.
- [3]Daniel E Berlyne. Conflict, arousal, and curiosity. 1960.DOI
- [4]Maciej Besta, Nils Blach, Ales Kubicek, Robert Gerstenberger, Michal Podstawski, Lukas Gianinazzi, Joanna Gajda, Tomasz Lehmann, Hubert Niewiadomski, Piotr Nyczyk, et al. Graph of thoughts: Solving elaborate problems with large language models. In Proceedings of the AAAI conference on artificial intelligence, volume 38, pages 17682–17690, 2024.DOI
- [5]Yuri Burda, Harri Edwards, Deepak Pathak, Amos Storkey, Trevor Darrell, and Alexei A Efros. Large-scale study of curiosity-driven learning. arXiv preprint arXiv:1808.04355, 2018.arxiv.org/abs/1808.04355
- [6]Runpeng Dai, Linfeng Song, Haolin Liu, Zhenwen Liang, Dian Yu, Haitao Mi, Zhaopeng Tu, Rui Liu, Tong Zheng, Hongtu Zhu, et al. Cde: Curiosity-driven exploration for efficient reinforcement learning in large language models. arXiv preprint arXiv:2509.09675, 2025.arxiv.org/abs/2509.09675
- [7]Abhimanyu Dubey and Alex `Sandy’ Pentland. Kernel methods for cooperative multi-agent contextual bandits. In Hal Daumé III and Aarti Singh, editors, Proceedings of the 37th International Conference on Machine Learning, volume 119 of Proceedings of Machine Learning Research, pages 2740–2750. PMLR, 13–18 Jul 2020. URL https://proceedings.mlr.press/v119/dubey20b.html.arxiv.org/abs/2008.06220
- [8]Eric Elmoznino, Sangnie Bhardwaj, Johannes von Oswald, Rajai Nasser, João Sacramento, Rif A Saurous, Guillaume Lajoie, et al. Can in-context learning support intrinsic curiosity? arXiv preprint arXiv:2606.19476, 2026.arxiv.org/abs/2606.19476
- [9]Leon Engländer, Sophia Althammer, Ahmet Üstün, Matthias Gallé, and Tom Sherborne. Agents explore but agents ignore: Llms lack environmental curiosity. arXiv preprint arXiv:2604.17609, 2026.
- [10]Sébastien Forestier, Rémy Portelas, Yoan Mollard, and Pierre-Yves Oudeyer. Intrinsically motivated goal exploration processes with automatic curriculum learning. Journal of Machine Learning Research, 23(152):1–41, 2022.
- [11]David Ha and Jürgen Schmidhuber. World models. arXiv preprint arXiv:1803.10122, 2(3):440, 2018.DOI
- [12]Vincent Herrmann, Louis Kirsch, and Jürgen Schmidhuber. Learning one abstract bit at a time through self-invented experiments encoded as neural networks. In International Workshop on Active Inference, pages 254–274. Springer, 2023.arxiv.org/abs/2212.14374
- [13]Qian Huang, Jian Vora, Percy Liang, and Jure Leskovec. Mlagentbench: Evaluating language agents on machine learning experimentation. arXiv preprint arXiv:2310.03302, 2023.arxiv.org/abs/2310.03302
- [14]Laurent Itti and Pierre Baldi. Bayesian surprise attracts human attention. Advances in neural information processing systems, 18, 2005.
- [15]Joel Jang, Seonghyeon Ye, Sohee Yang, Joongbo Shin, Janghoon Han, Gyeonghun Kim, Stanley Jungkyu Choi, and Minjoon Seo. Towards continual knowledge learning of language models. arXiv preprint arXiv:2110.03215, 2021.arxiv.org/abs/2110.03215
- [16]Peter Jansen, Marc-Alexandre Côté, Tushar Khot, Erin Bransom, Bhavana Dalvi Mishra, Bodhisattwa Prasad Majumder, Oyvind Tafjord, and Peter Clark. Discoveryworld: A virtual environment for developing and evaluating automated scientific discovery agents. Advances in Neural Information Processing Systems, 37:10088–10116, 2024.arxiv.org/abs/2406.06769
- [17]Yulun Jiang, Liangze Jiang, Damien Teney, Michael Moor, and Maria Brbic. Meta-rl induces exploration in language agents. arXiv preprint arXiv:2512.16848, 2025.arxiv.org/abs/2512.16848
- [18]James Kirkpatrick, Razvan Pascanu, Neil Rabinowitz, Joel Veness, Guillaume Desjardins, Andrei A Rusul, Kieran Milan, John Quan, Tiago Ramalho, Agnieszka Grabska-Barwinska, et al. Overcoming catastrophic forgetting in neural networks. Proceedings of the national academy of sciences, 114(13):3521–3526, 2017.DOI
- [19]Michael Laskin, Luyu Wang, Junhyuk Oh, Emilio Parisotto, Stephen Spencer, Richie Steigerwald, DJ Strouse, Steven Hansen, Angelos Filos, Ethan Brooks, et al. In-context reinforcement learning with algorithm distillation. arXiv preprint arXiv:2210.14215, 2022.arxiv.org/abs/2210.14215
- [20]Guillaume Levy, Cédric Colas, Pierre-Yves Oudeyer, Thomas Carta, and Clément Romac. Worldllm: Improving llms’ world modeling using curiosity-driven theory-making. arXiv preprint arXiv:2506.06725, 2025.arxiv.org/abs/2506.06725
- [21]Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler, Mike Lewis, Wen-tau Yih, Tim Rocktäschel, et al. Retrieval-augmented generation for knowledge-intensive nlp tasks. Advances in neural information processing systems, 33:9459–9474, 2020.arxiv.org/abs/2005.11401
- [22]Chris Lu, Cong Lu, Robert Tjarko Lange, Jakob Foerster, Jeff Clune, and David Ha. The ai scientist: Towards fully automated open-ended scientific discovery. arXiv preprint arXiv:2408.06292, 2024.arxiv.org/abs/2408.06292
- [23]Alexander Novikov, Ngân Vũ, Marvin Eisenberger, Emilien Dupont, Po-Sen Huang, Adam Zsolt Wagner, Sergey Shirobokov, Borislav Kozlovskii, Francisco JR Ruiz, Abbas Mehrabian, et al. Alphaevolve: A coding agent for scientific and algorithmic discovery. arXiv preprint arXiv:2506.13131, 2025.arxiv.org/abs/2506.13131
- [24]Catherine Olsson, Nelson Elhage, Neel Nanda, Nicholas Joseph, Nova DasSarma, Tom Henighan, Ben Mann, Amanda Askell, Yuntao Bai, Anna Chen, et al. In-context learning and induction heads. arXiv preprint arXiv:2209.11895, 2022.arxiv.org/abs/2209.11895
- [25]Pierre-Yves Oudeyer, Frédric Kaplan, and Verena V Hafner. Intrinsic motivation systems for autonomous mental development. IEEE transactions on evolutionary computation, 11(2):265–286, 2007.DOI
- [26]Ao Ou, Han Zheng, Zijian Zhou, Yihao Yan, Yihong Tang, Shao Yong Ong, Fenglu Hong, Kaichen Zhou, Chonghe Jiang, Minwei Kong, et al. Coral: Towards autonomous multi-agent evolution for open-ended discovery. arXiv preprint arXiv:2604.01658, 2026.
- [27]Mark B Ring. Child: A first step towards continual learning. Machine Learning, 28(1):77–104, 1997.
- [28]Mark Bishop Ring. Continual learning in reinforcement environments. The University of Texas at Austin, 1994.
- [29]Bernardino Romera-Paredes, Mohammadamin Barekatain, Alexander Novikov, Matej Balog, M Pawan Kumar, Emilien Dupont, Francisco JR Ruiz, Jordan S Ellenberg, Pengming Wang, Omar Fawzi, et al. Mathematical discoveries from program search with large language models. Nature, 625(7995):468–475, 2024.
- [30]Cansu Sancaktar, Christian Gumbsch, Andrii Zadaianchuk, Pavel Kolev, and Georg Martius. SENSEI: Semantic exploration guided by foundation models to learn versatile world models. In Aarti Singh, Maryam Fazel, Daniel Hsu, Simon Lacoste-Julien, Felix Berkenkamp, Tegan Maharaj, Kiri Wagstaff, and Jerry Zhu, editors, Proceedings of the 42nd International Conference on Machine Learning, volume 267 of Proceedings of Machine Learning Research, pages 52745–52777. PMLR, 13–19 Jul 2025. URL https://proceedings.mlr.press/v267/sancaktar25a.html.arxiv.org/abs/2503.01584
- [31]Juergen Schmidhuber. Exploring the predictable. In Advances in evolutionary computing: theory and applications, pages 579–612. Springer, 2003.DOI
- [32]Jürgen Schmidhuber. Curious model-building control systems. In Proc. international joint conference on neural networks, pages 1458–1463, 1991.DOI
- [33]Jürgen Schmidhuber. A possibility for implementing curiosity and boredom in model-building neural controllers. In Proc. of the international conference on simulation of adaptive behavior: From animals to animats, pages 222–227, 1991.DOI
- [34]Jürgen Schmidhuber. On learning how to learn learning strategies. Technical Report FKI-198-94, Fakultät für Informatik, Technische Universität München, November 1994. URL https://people.idsia.ch/~juergen/FKI-198-94ocr.pdf.
- [35]Jürgen Schmidhuber. Formal theory of creativity, fun, and intrinsic motivation (1990–2010). IEEE transactions on autonomous mental development, 2(3):230–247, 2010.DOI
- [36]Jürgen Schmidhuber. Powerplay: Training an increasingly general problem solver by continually searching for the simplest still unsolvable problem. Frontiers in psychology, 4:313, 2013.DOI
- [37]Jürgen Schmidhuber, Jieyu Zhao, and Marco Wiering. Shifting inductive bias with success-story algorithm, adaptive levin search, and incremental self-improvement. Machine Learning, 28(1): 105–130, 1997.DOI
- [38]Jürgen Schmidhuber. Artificial curiosity & creativity since 1990-91. https://people.idsia.ch/~juergen/artificial-curiosity-since-1990.html, 2021, 2022, 2023, 2025. Accessed: 2026-02-17.
- [39]Noah Shinn, Federico Cassano, Ashwin Gopinath, Karthik Narasimhan, and Shunyu Yao. Reflexion: Language agents with verbal reinforcement learning. Advances in neural information processing systems, 36:8634–8652, 2023.arxiv.org/abs/2303.11366
- [40]Jan Storck, Sepp Hochreiter, Jürgen Schmidhuber, et al. Reinforcement driven information acquisition in non-deterministic environments. In Proceedings of the international conference on artificial neural networks, Paris, volume 2, pages 159–164, 1995.
- [41]Yi Sun, Faustino Gomez, and Jürgen Schmidhuber. Planning to be surprised: Optimal bayesian exploration in dynamic environments. In International conference on artificial general intelligence, pages 41–51. Springer, 2011.arxiv.org/abs/1103.5708
- [42]Guanzhi Wang, Yuqi Xie, Yunfan Jiang, Ajay Mandlekar, Chaowei Xiao, Yuke Zhu, Linxi Fan, and Anima Anandkumar. Voyager: An open-ended embodied agent with large language models. arXiv preprint arXiv:2305.16291, 2023.arxiv.org/abs/2305.16291
- [43]Sang Michael Xie, Aditi Raghunathan, Percy Liang, and Tengyu Ma. An explanation of in-context learning as implicit bayesian inference. arXiv preprint arXiv:2111.02080, 2021.arxiv.org/abs/2111.02080
- [44]Yutaro Yamada, Robert Tjarko Lange, Cong Lu, Shengran Hu, Chris Lu, Jakob Foerster, Jeff Clune, and David Ha. The ai scientist-v2: Workshop-level automated scientific discovery via agentic tree search. arXiv preprint arXiv:2504.08066, 2025.arxiv.org/abs/2504.08066
- [45]Shunyu Yao, Dian Yu, Jeffrey Zhao, Izhak Shafran, Tom Griffiths, Yuan Cao, and Karthik Narasimhan. Tree of thoughts: Deliberate problem solving with large language models. Advances in neural information processing systems, 36:11809–11822, 2023.arxiv.org/abs/2305.10601
- [46]Jenny Zhang, Joel Lehman, Kenneth Stanley, and Jeff Clune. Omni: Open-endedness via models of human notions of interestingness. arXiv preprint arXiv:2306.01711, 2023.arxiv.org/abs/2306.01711
- [47]Kongcheng Zhang, Qi Yao, Shunyu Liu, Yingjie Wang, Baisheng Lai, Jieping Ye, Mingli Song, and Dacheng Tao. Consistent paths lead to truth: Self-rewarding reinforcement learning for llm reasoning. arXiv preprint arXiv:2506.08745, 2025.arxiv.org/abs/2506.08745
