AI Cosmos: An Agentic Infra for Situated Learning

“For education becomes transformative when knowledge is not merely expressed, but enacted, shared, and turned into reusable intellectual assets.” – Manifesto of AI Cosmos

Introduction

Introduction

In 1929, philosopher Alfred North Whitehead described the problem of inert knowledge in his seminal collection “The Aims of Education”: “...the problem of keeping knowledge alive, of preventing it from becoming inert, which is the central problem of all education” [1]. Nearly a century later, Whitehead’s diagnosis remains relevant, despite successive advances in educational technologies. Today, generative AI appears to offer a promising remedy to this enduring problem by providing intelligence that is continuously accessible, highly interactive, and readily adaptable into explanations and personalized tutorials [2, 3, 4, 5].

Yet, this remedy may not close the gap between knowing and doing but instead widen it in a new form. Across educational settings, learners become more fluent in reproducing AI-provided concepts, explanations, and solutions [6] without commensurate gains in their ability to apply that knowledge independently [7, 8], a discrepancy we refer to as performance–competence decoupling [9]. For instance, an undergraduate student may be able to explain the divide-and-conquer tactic of quicksort yet struggle to implement it robustly [10]; similarly, a bar-exam candidate can fluently state a legal doctrine yet fails to apply it to an unfamiliar case [11]. These examples reflect a structural asymmetry: generative AI has substantially reduced the cost of accessing and reproducing declarative knowledge (“knowing what”) [12], without producing an equivalent improvement in procedural knowledge (“knowing how”) [13].

We should note that this asymmetry no longer stems from LLM capability, but from the interaction scheme in which these models are used [14, 15]. Despite the diversity of AI-assisted educational applications, from content creation to lesson planning [16, 17, 18], a common interaction pattern is the request–artifact exchange, where the learner submits a request, the system produces an explanation or artifact, and the interaction often ends once that artifact is delivered [19]. Question answering is such an epistemic outsourcing example, more accurate and pedagogically appropriate responses may improve immediate task performance without requiring learners to reason through, test, or revise solutions themselves. The limitation, therefore, is not simply the quality of generated outputs, but whether the interaction supports the activities through which competence develops. Learning research emphasizes active participation, feedback, and iterative revision within meaningful, scaffolded activities [20, 21, 22, 23]. When assistance is organized primarily around output delivery, learners’ actions, feedback, and revisions may remain outside the interaction, limiting opportunities to support continued practice and use its outcomes to inform subsequent learning [24, 25].

Recent LLM-based tutoring systems structure pedagogical dialogue and feedback to support learners’ reasoning [27] (Chowdhury et al. 2024), while general-purpose multi-agent frameworks coordinate role-specialized agents to complete complex tasks [28, 29, 30]. Yet the effectiveness of these systems in supporting learners’ sustained competence development cannot be inferred from the quality of instructional guidance or the success of agent task execution alone. One consideration is agency-preserving participation, which emphasizes learners’ active engagement in explaining, testing, and revising their own work within authentic or scaffolded activities, rather than merely receiving generated solutions [21, 20]. Tool-using agents are often designed for autonomous task execution [31, 32]; when applied to educational settings, such designs may position learners primarily as requesters or evaluators of completed work. A second consideration is maintaining cumulative, inspectable record of practice. Constructionism emphasizes the educational value of artifacts that learners can create, examine, and improve over time [33]. Agentic systems also accumulate and refine reusable skills from past experience [34], although their reported gains primarily concern subsequent agent task performance rather than learners’ use of prior practice.

Taken together, these considerations motivate an approach to AI-assisted learning that centers learners’ own reasoning and action within learning activities, while enabling subsequent activities to build on prior practice.

We translate these considerations into two complementary architectural shifts. The first shift moves from request-artifact exchange to scenario-based participation, which organizes learning around learners’ active involvement in socially mediated, tool-supported activities (Lave and Wenger, [21]; Kaptelinin and Nardi, [35]). Each scenario integrates three configurable dimensions of practice: social (learners and role-specialized agents as co-participants), epistemic (task-relevant knowledge resources), and operational (stateful, executable workbenches, i.g., a coding IDE„ when needed). Learners and agents coordinate through dialogue and, when applicable, perform task-relevant operations on shared workbench states. The resulting observations inform subsequent dialogue and actions, supporting learners in applying knowledge, evaluating outcomes, and adapting their approaches through practice. The second shift moves from one-off artifacts to cumulative assets of practice. Rather than treating final outputs as the endpoint of learning activities, this shift makes the processes and outcomes of practice available for reuse. Subsequent scenarios can thus build on accumulated experience, supporting continuity across learning activities. Together, these shifts place learners’ active participation and the accumulation of practical experience at the center of AI-assisted learning, shifting the emphasis from task completion to the development of learners’ competence (Kolb, [36]; Schön, [37]).

We instantiate these shifts in AI Cosmos (https://aicosmos.ai/), an agentic infrastructure for situated learning that organizes learner–agent participation in structured scenarios and preserves records of practice for reuse. A scenario control plane configures each learning activity from a learner’s request by specifying its objective, participant roles, relevant knowledge resources, and executable workbenches when needed. A task execution plane coordinates dialogue and workbench operations over shared task states, dynamically scheduling learners and agents to participate in iterative problem solving, execution, and revision. To connect successive scenarios, an asset–memory loop extracts and validates reusable assets from scenario configurations and records of practice. By retaining provenance and dependencies, these assets make prior outcomes and the associated reasoning, feedback, and revisions available for inspection and adaptation in subsequent activities. We evaluate scenario-based interaction through two preregistered randomized controlled studies examining the acquisition of legal reasoning and programming competence, finding greater learning gains than a conventional LLM in the former and stronger programming mastery than an LLM-as-Teacher baseline in the latter.

The Scenario–Asset Framework

The Scenario

The epistemic and operational components may be empty, where $\mathcal{W}_S = \varnothing$, participation is entirely discursive, and $\mathcal{K}_S = \varnothing$, no external knowledge resource is explicitly bound to the activity. A scenario proceeds through three stages: initialization configures its components, execution advances the scenario through successive state transitions directed toward $o_S$, and completion packages its residue and outputs for asset extraction.

The Asset

By preserving content together with its provenance and dependencies, assets can be inspected, retrieved, adapted, and extended across scenarios. In particular, scenario specifications, process records, and experience traces retain the conditions and actions surrounding an outcome, making records of prior practice available to future participants rather than preserving only the final product.

The Asset–Memory Feedback Loop

With these three operations, later scenarios can inspect, compose, and extend validated contributions from prior activities, enabling assets and records of practice to accumulate across interactions.

System Architecture of AI Cosmos

AI Cosmos implements the Scenario–Asset Framework via three interconnected components (Figure 2). The scenario control plane configures learning objectives, participant roles, knowledge resources, and optional workbenches from learner requests and relevant prior assets. The task execution plane coordinates learner–agent interactions and workbench operations via history-dependent scheduling. Learners and agents operate on shared workbench states, with resulting observations informing subsequent interactions. The asset–memory substrate derives reusable assets from scenario configurations and records of practice, manages their validation and admission, and preserves provenance and dependencies for reuse in subsequent scenarios.

The Scenario Control Plane

The Scenario Control Plane

This plane instantiates the scenario-configuration operator $\mathcal{F}_{\mathrm{config}}$, mapping an initial learning request to an executable scenario for the task execution plane. Rather than relying on a fixed agent team and tool set, it configures the three dimensions according to the learning objective. Let $u$ be a user input, comprising a request and a (possibly empty) set $\mathcal{U}$ of uploaded resources. The operator is defined as:

It first grounds the request into the objective $o_S$ through intent parsing. This objective then conditions the scenario configuration that may reuse previously admitted assets through relevance-gated retrieval: $\mathcal{A}^{\mathrm{in}}_S = R(o_S,\mathcal{M})$, which returns only assets whose relevance to $o_S$ exceeds a predefined threshold, otherwise $\mathcal{A}^{\mathrm{in}}_S = \varnothing$. We write $\mathcal{A}^{\mathrm{in},\mathcal{P}}_S$, $\mathcal{A}^{\mathrm{in},\mathcal{K}}_S$, and $\mathcal{A}^{\mathrm{in},\mathcal{W}}_S$ for the asset subsets used in three dimensions.

Social configuration. This dimension is realized as a shared, multiparty interaction space comprising both agents and human participants: $\mathcal{P}_S = \mathcal{P}^{\mathrm{agent}}_S \cup\mathcal{P}^{\mathrm{human}}$. Given the objective $o_S$ and any retrieved role specifications, it determines the roles required by the scenario and instantiates one agent per role, with both the number of roles $n$ and each profile determined adaptively:

Each role profile $ $r_i$ specifies the participant’s task responsibility, expertise, interaction policy, and access to knowledge resources and workbench operations. When $\mathcal{A}^{\mathrm{in},\mathcal{P}}_S \ne\varnothing$, $\operatorname{Roles}(\cdot)$ reuses or adapts roles from previously admitted configurations; otherwise, it synthesizes new role profiles from scratch.

Epistemic configuration. This dimension is realized as a knowledge base that remains queryable throughout scenario execution. AI Cosmos holds a library $\Sigma$ of modular skills that can be loaded on demand ((Anthropic, 2025)). For each uploaded resource $\rho\in\mathcal{U}$, the control plane selects a compatible skill $\sigma_\rho\in\Sigma$ and applies it to produce a set of indexed, retrievable units $\sigma_{\rho}(\rho)$. Let $\mathcal{A}_{S}^{\mathrm{in},\mathcal{K}} \subseteq\mathcal{A}_{S}^{\mathrm{in}}$ denote the retrieved knowledge-bearing assets, and $\operatorname{Index}(\mathcal{A}_{S}^{\mathrm{in},\mathcal{K}})$ denote the indexed units derived from the retrieved assets. The knowledge base is

During execution, authorized participants access $\mathcal{K}_{S}$ through a retrieval service. When neither uploaded resources nor previously accumulated knowledge is required, $\mathcal{K}_{S} = \varnothing$.

Operational configuration. This dimension is realized by selecting and binding workbenches through which participants execute task-relevant objects. Let $\mathcal{W}$ denote the registry of available workbenches (e.g., a coding IDE or domain-specific instrument), the plane selects the subset required by the objective:

For a purely discursive objective, $\mathcal{W}_{S} = \varnothing$. Each workbench $w \in\mathcal{W}_{S}$ maintains a shared, mutable state $x_{w}$ and exposes a typed action interface $\mathcal{I}_{w}$. The interface consists of named operations for inspecting or transforming that state, such as creating, modifying, executing, or querying a task-relevant object. Binding a workbench connects its action interface to the scenario’s shared interaction stream. An authorized agent invokes an operation in $\mathcal{I}_{w}$ through a structured tool call, whereas a human participant acts through a graphical interface whose interactions are mapped to the same underlying operations. An operation updates $x_{w}$ and emits an observation that is recorded in the shared interaction history. Thus, conversation and operation remain synchronized: dialogue can initiate changes to the workbench, and workbench observations can immediately inform subsequent dialogue and action. The access scope encoded in each role profile $r_{i}$ determines which workbenches in $\mathcal{W}_{S}$, and which operations in their $\mathcal{I}_{w}$, the corresponding agent is authorized to invoke.

The Task Execution Plane

The task execution plane runs a configured scenario $S$ toward its objective $o_{S}$ by coupling deliberation in the shared interaction space with action in the bound workbenches.

Outer loop: multi-party scheduling. At each outer step, the execution plane selects the next system-mediated participant according to the current interaction history $H_{t}$ and objective:

The selected participant contributes to the shared interaction or invokes a bound workbench. Since scheduling is conditioned on the evolving history rather than a fixed turn order, the scenario can move dynamically among explanation, questioning, critique, tool use, and human handoff. All agent actions remain subject to the role and access policies established by the scenario control plane. Each scheduled action and its resulting observation are recorded in the history and inform subsequent scheduling.

Grounding agent turns with hybrid retrieval. When a knowledge base $\mathcal{K}_{S}$ is available, the task execution plane derives a query $q_{t}^{K}$ from the interaction history $H_{t}$ and scores each entry $k$ by combining normalized BM25 ([39]) and embedding-based cosine similarity:

where $\lambda\in[0,1]$ balances lexical and semantic relevance, and both similarity scores are normalized to a common range. Entries with $\operatorname{score}(q_{t}^{K}, k) \geq\tau$ are provided as grounding context for subsequent agent actions.

The workbench: sandboxed state and intent-level operations. Each workbench $w \in\mathcal{W}_{S}$ comprises an isolated execution environment, which maintains its persistent state $x_{w}$ and a typed action interface $\mathcal{I}_{w}$. The sandbox preserves task state across operations and confines untrusted model-generated execution from the host environment and other scenarios. The interface exposes a small vocabulary of operations:

Here, $\mathrm{CREATE}$ creates or initializes a task-relevant object, $\mathrm{MODIFY}$ changes an existing object, $\mathrm{EXECUTE}$ runs a computation or executable object, and $\mathrm{QUERY}$ inspects the current state. Each workbench implements only the operations supported by its capabilities. For agent invocations, operations are specified at the level of intent: the agent provides an operation and a natural-language sub-goal, which a workbench-specific adapter translates into platform primitives [40]. Thus, heterogeneous environments can share the same higher-level scheduling and execution logic while differing only in their runtime and adapter.

Inner loop: executing a workbench operation. When a scheduled agent action requires workbench $w$, the agent invokes an operation $\iota_{w} \in\mathcal{I}_{w}$, authorized by its role profile $r_{i}$, with a sub-goal $o_{g}$ through a structured tool call [31]. The operation executes within the sandbox through iterative generation and execution of platform-specific primitives, using diagnostic feedback to guide retries. Let $j$ index execution attempts, with $x_{w}^{(0)} = x_{w}$ denoting the initial workbench state. At each attempt,

where $\operatorname{Gen}_{w}$ is an LLM-based generator of workbench-specific primitives $c_{w}^{(j)}$ (e.g., Python cells in Jupyter), while $\operatorname{Exec}_{w}$ executes these primitives in an isolated sandbox, yielding the updated state $x_{w}^{(j+1)}$ and observation $y_{w}^{(j)}$. The diagnostic input $\epsilon^{(j)}$ captures feedback from the preceding failed attempt, with $\epsilon^{(0)} = \varnothing$. After each attempt, the workbench evaluates the result against the sub-goal:

If verification fails and the retry budget permits, diagnostic feedback guides the next generation attempt. The loop terminates upon successful verification or budget exhaustion. The complete invocation trace, including intermediate failures and repairs, is retained for later inspection and asset extraction, while the terminal observation and execution status are incorporated into the shared interaction history $H_{t+1}$ without exposing every intermediate attempt.

Interruptible human participation. Human participants may intervene during scenario execution without waiting for scheduler-selected turns. Subject to scenario permissions, they can inspect or modify bound workbenches through a graphical interface, operating on the same persistent state and underlying operations as authorized agents. These interventions are recorded in the shared interaction history, allowing subsequent scheduling and agent actions to account for the updated workbench state.

The Asset–Memory Substrate

The two planes above orchestrate an individual scenario, while the asset–memory substrate connects successive scenarios through the feedback loop in § 3.3, which manages four stages of the asset lifecycle.

Production: deriving assets from practice. At configuration, each instantiated role profile $r_{i}$ yields a role specification, and the configured tuple $\langle O_{S}, P_{S}, K_{S}, W_{S}\rangle$ yields a scenario specification. During execution, durable task-relevant objects created or materially revised in a workbench become candidate knowledge artifacts, while dialogue, retrieval events, workbench operations, observations, failures and repairs, and human interventions accumulate in the execution history. At completion, the resulting history $H_S$ is persisted as a process record. An offline pass further distills reusable experience traces:

Here, $\operatorname{Seg}(\cdot)$ partitions the execution record at dialogue-turn and workbench-attempt boundaries. For each episode $\eta$, $\kappa_\eta$ summarizes the triggering context, $\beta_\eta$ abstracts the action taken, and $\ell_\eta\in\{+,-\}$ denotes its task-level outcome:

where $\operatorname{Prog}(\cdot)$ is the scheduler’s progress assessment, $\operatorname{Verify}(\cdot)$ the inner-loop self-check, and $\mu^{\pm}[\Delta]$ returns + if $\Delta$ holds and − otherwise. A failed attempt often followed by a successful repair therefore yields distinct negative and positive traces. These labels capture task-level utility rather than pedagogical effectiveness.

Admission: type-specific validation. Process records are admitted without quality filtering to preserve execution histories. For other candidate assets, an LLM-based evaluator applies a type-specific acceptance rubric $\Phi_{\tau_a}(o_S)$, with access to asset content, provenance, and scenario context:

where $\tau_a$ denotes the asset type. Knowledge artifacts additionally require human approval. The final admission criterion is

Only assets satisfying $\mathcal{V}(a)=1$ are admitted to reusable memory.

Persistence: memory and provenance. Each admitted asset is stored as $a=\langle\tau_a,c_a,s_a,\mathcal{A}_a^{\mathrm{prior}},\operatorname{st}_a\rangle$, where $\operatorname{st}_a\in\{\mathrm{active},\mathrm{deprecated}\}$ denotes its lifecycle status. Assets form a directed provenance graph $\mathcal{G}$, with an edge $a'\to a$ for each dependency $a'\in\mathcal{A}_a^{\mathrm{prior}}$. Asset content and dependencies remain immutable: when a new asset $a''$ supersedes $a$, it references $a$ as a dependency, and $\operatorname{st}_a$ is set to deprecated. By default, retrieval is restricted to active assets:

Deprecated assets remain available for provenance inspection and rollback. Since dependencies point from earlier to later assets, $\mathcal{G}$ remains acyclic.

Reuse: asset-specific integration. During scenario configuration, retrieved scenario specifications can initialize new configurations, while role specifications can be reused or adapted. Validated knowledge artifacts are indexed into $\mathcal{K}_S$ for retrieval, whereas process records remain available for provenance and inspection without being treated as trusted knowledge by default. During execution, experience traces are retrieved using a query $q_t^{\mathrm{exp}}$ derived from the current interaction history:

Positive traces identify actions that advanced similar task states, while negative traces denote actions associated with failure. These traces provide auxiliary context for $\operatorname{Schedule}(\cdot)$ without determining its decisions.

Compounding via reuse. The substrate closes the feedback loop by allowing validated assets and execution experience from one scenario to inform subsequent scenarios. When relevant prior assets exist, reuse can reduce repeated configuration and reconstruction while preserving the provenance of the reused material.

Experiments

Human Study: Situated Learning for Bar-Examination Preparation

Legal problem solving provides a demanding test of application-oriented learning: success requires not only recalling legal doctrine but also identifying relevant rules and applying them to novel fact patterns. We therefore use bar-examination preparation to test whether restructuring LLM interaction from conventional chat into a structured scenario improves learning of applied legal reasoning.

Study Design and Measures

Design and participants. We conducted a pre-registered, three-arm, parallel-group randomized controlled trial. A total of 131 students were assigned in a 1:1:1 ratio to one of three learning conditions: (i) vanilla LLM, a conventional request–response assistant, (ii) LLM-as-Teacher in which the same LLM was prompted to adopt an expert-tutor, or (iii) AI Cosmos. All conditions used the same underlying LLM, holding model capability constant while varying the interaction architecture. Following our preregistered data-integrity criteria, we excluded participants whose learning logs failed to meet minimum-engagement requirements (e.g., insufficient interaction or exceptionally short inputs), leaving an analyzed sample of $N = 111$ (37 per arm). We observed no statistically significant between-group differences in the measured pre-intervention characteristics, including sex, prior legal training, age, AI literacy, self-efficacy, and pre-test performance (all $p \ge0.27$; Table 1 in Appendix § 10.§ 2).

	Vanilla LLM	LLM-as-Teacher	AI Cosmos	Test ($p$)
Female (%)	73.0	81.1	86.5	$\chi^2$, 0.34
Prior legal training (%)	24.3	27.0	21.6	$\chi^2$, 0.86
Age (years)	$23.2 \pm2.1$	$23.8 \pm3.0$	$23.0 \pm2.0$	$F$, 0.41
AI literacy (/170)	$119.7 \pm15.0$	$119.8 \pm13.8$	$116.1 \pm14.3$	$F$, 0.44
Self-efficacy, GSES	$15.1 \pm4.0$	$15.1 \pm4.3$	$13.6 \pm3.7$	$F$, 0.27
Pre-test, total (%)	$50.3 \pm11.8$	$50.4 \pm13.9$	$47.0 \pm11.0$	$F$, 0.42
- objective items (%)	$48.8 \pm12.9$	$48.6 \pm16.0$	$44.3 \pm12.5$	$F$, 0.29
- open-response items (%)	$56.2 \pm18.4$	$57.4 \pm13.8$	$58.1 \pm15.2$	$F$, 0.88

Procedure and measures. Participants completed four phases: (1) Pre-test: An AI-literacy inventory, the General Self-Efficacy Scale (GSES, Schwarzer and Jerusalem [41]), and a legal-knowledge assessment covering civil and criminal law. This assessment combined objective single- and multiple-choice items with open-response questions. (2) Learning: Participants studied matched civil- and criminal-law problems using only their assigned system. (3) Immediate post-test: A new, difficulty-matched legal assessment with the same composition, followed by the GSES and the NASA-TLX workload index (Hart and Staveland [42]). (4) Delayed post-test ($\approx4$ weeks later): A randomly selected subsample ($n = 18$ per condition) completed a new assessment of the same composition to probe retention. All legal assessments were scored on a 0–100 scale, with objective items weighted at 80% and open-response items at 20%. The open responses were scored by three law PhDs whose ratings showed high agreement (Cronbach’s $\alpha= 0.95$ pre-test, 0.96 post-test) and were averaged. The outcome was the learning gain, defined as the post-test minus pre-test change in total score.

Statistical analysis. We report group means and standard deviations for all primary outcomes. Unadjusted differences in learning gain were examined using one-way ANOVA¹, with the Kruskal–Wallis test as a nonparametric robustness check. Shapiro–Wilk tests did not reject normality of the gain distributions (all $p > 0.05$), and Levene’s test indicated no evidence of unequal variances ($p = 0.95$). The primary between-group inference used ANCOVA on immediate post-test scores, with pre-test score entered as a covariate. This approach adjusts for chance baseline variation and generally provides greater precision than analysis of raw change scores. The homogeneity-of-regression-slopes assumption was supported by the condition-by-pre-test interaction test ($p = 0.77$). We report the omnibus ANCOVA effect together with $\eta^2$ and model-adjusted contrasts between AI Cosmos and the control conditions. For complementary unadjusted comparisons of raw learning gains, we report Welch’s $t$ tests and standardized effect sizes (Cohen’s $d$ or bias-corrected Hedges’ $g$) with bootstrap 95% confidence intervals [43]. Subgroup and assessment-component analyses were exploratory and were not adjusted for multiplicity. All hypothesis tests were two-sided with $\alpha= 0.05$.

Results

Learning efficacy. All three conditions showed large pre–post gains (paired $t$, all $p < 10^{-8}$; $d_z = 1.22$, 1.45, and 1.88 for the vanilla, LLM-as-teacher, and AI Cosmos). AI Cosmos yielded the largest raw gain (22.8 ± 12.1 pp), followed by LLM-as-teacher (18.5 ± 12.7) and vanilla LLM (15.7 ± 12.9; Figure 3b). The omnibus ANOVA on raw gains approached, but did not reach, statistical significance ($F(2, 108) = 2.94$, $p = 0.057$, $\eta^2 = 0.052$). However, the primary planned contrast was clear: AI Cosmos outperformed the vanilla LLM by 7.0 pp (Welch $t(71.8) = 2.42$, $p = 0.018$), a medium effect size ($d = 0.56$, bootstrap 95% CI [0.10, 1.05]). An ANCOVA adjusting for the AI Cosmos group’s slightly lower baseline reproduced this advantage on adjusted post-test means (i.e., post-test scores predicted at the common pre-test mean, i.e., adjusted means 70.2 vs. 65.8; group effect $F(2, 107) = 3.00$, $p = 0.054$, partial $\eta^2 = 0.053$; AI Cosmos vs. vanilla $b = 4.47$, $p = 0.018$), with the pre-test serving as a significant covariate ($b = 0.20$, $p = 0.001$). The advantage of AI Cosmos over the stronger LLM-as-teacher was positive but not statistically significant ($\Delta= 4.2$ pp, $p = 0.146$, $d = 0.34$), while LLM-as-teacher itself did not significantly outperform vanilla LLM ($\Delta= 2.8$ pp, $p = 0.121$ adjusted). With the underlying model held constant, these results indicate a reliable advantage of AI Cosmos over conventional LLM interaction, whereas prompting the same model with a teacher persona alone did not yield a statistically detectable benefit.

Interaction structure. The clearest behavioral difference emerged in the structure of interaction (Figure 3c). Total student input did not differ across conditions (median approximately 7,100 Chinese characters; $F(2, 108) = 0.13$, $p = 0.88$), nor did session counts ($p = 0.71$). However, AI Cosmos elicited approximately 1.7× as many dialogue turns as either control (51.0 ± 21.1 vs. 29.6 ± 13.2 and 29.2 ± 13.2; $F(2, 108) = 21.8$, $p = 1.1 \times10^{-8}$, $\eta^2 = 0.288$; both pairwise $p < 10^{-5}$; $g = 1.23$ [0.79, 1.76]). Correspondingly, students produced approximately 40% fewer characters per turn (162 vs. 259/276; $F(2, 108) = 7.66$, $p < 0.001$; AI Cosmos vs. Vanilla LLM $p \approx0.001$, $g = -0.80$). Thus, despite similar total input volume, AI Cosmos produced a substantially more iterative, turn-rich interaction pattern. Raw interaction volume itself was only weakly associated with learning gain (rounds: $r = 0.16$, $p = 0.09$; characters: $r = 0.07$, n.s.), and the association between dialogue turns and gain was further attenuated after controlling for condition ($r = 0.06$, n.s.). These results suggest that the observed advantage cannot be explained simply by greater interaction volume and is instead consistent with differences in how participation was structured.

Durability. At the four-week follow-up, the descriptive ordering observed immediately after learning was preserved. Relative to pre-test, the AI Cosmos subsample showed the largest retained gain (+18.9 pp), followed by LLM-as-Teacher (+16.8 pp) and Vanilla LLM (+14.7 pp; Figure 3d). The AI Cosmos–Vanilla LLM difference remained positive ($\Delta= 4.2$ pp, $d = 0.34$) but was smaller than the immediate effect. With only 18 participants per condition, however, the follow-up analysis was underpowered and yielded no significant between-group difference ($F(2, 51) = 0.48$, $p = 0.62$). Forgetting from the immediate to delayed post-test also did not differ significantly across conditions ($p = 0.43$). The follow-up therefore provides directionally consistent but inconclusive evidence regarding retention, and a larger delayed-test sample is needed to determine whether the immediate advantage persists over time.

Structured application versus open argumentation. Decomposing the learning gains by item type suggested that the overall advantage was concentrated in the objective, scenario-based items (Figure 4). Gains on these items were 18.2, 21.2, and 26.5 pp for Vanilla LLM, LLM-as-Teacher, and AI Cosmos ($F(2, 108) = 2.89$, $p = 0.060$, $\eta^2 = 0.051$). AI Cosmos exceeded Vanilla LLM by 8.3 pp (95% CI [1.8, 14.9], $d = 0.56$, $p = 0.018; ANCOVA-adjusted $p = 0.038$). By contrast, gains on open-response items were smaller and did not differ across conditions (5.6–7.8 pp; $F(2, 108) = 0.23$, $p = 0.80$; AI Cosmos vs. Vanilla LLM $d = 0.13$, n.s.). A mixed-design ANOVA showed a strong main effect of item type ($F(1, 108) = 55.6$, $p < 0.001$,

Human Study: An Instrumented Workbench for Skill Acquisition in Competitive Programming

$\eta_p^2 = 0.34$ but no condition-by-item-type interaction ($F(2, 108) = 0.95$, $p = 0.39$). Accordingly, the apparent concentration of the AI Cosmos advantage in structured application items should be interpreted descriptively rather than as evidence of moderation by item type. The present study thus provides stronger evidence for gains in applying legal doctrine to structured fact patterns than for improvements in extended written argumentation.

Robustness across learners. Exploratory subgroup analyses examined whether the AI Cosmos advantage was restricted to particular learner profiles. Across median splits on prior knowledge and AI literacy, the AI Cosmos–Vanilla LLM effect remained directionally positive, although the small subgroup sizes (15–22 participants per condition) produced wide confidence intervals and no multiplicity correction was applied (Figure 5). These analyses are therefore treated as consistency checks rather than tests of moderation. A complementary split at the median pre-test score (46.7) showed the same descriptive ordering in both prior-knowledge strata. Among lower-scoring learners, gains were +28.4, +26.0, and +21.8 pp across the three conditions, whereas among higher-scoring learners they were +14.5, +9.6, and +10.6 pp (Figure 6). Lower-pre-test learners gained more overall ($F(1, 105) = 46.8$, $p < 10^{-4}$), but the condition-by-prior-knowledge interaction was not significant ($F(2, 105) = 0.55$, $p = 0.58$). Together with the pre-test-adjusted ANCOVA, these results argue against the overall AI Cosmos effect being attributable solely to its slightly lower baseline performance.

Self-efficacy and cognitive load. Self-efficacy gains were descriptively largest for AI Cosmos (+5.2 vs. +4.8 and +3.9 GSES points; $d = 0.35$ vs. Vanilla LLM), but the between-group difference was not significant ($F(2, 80) = 1.02$, $p = 0.37$). Perceived workload also did not differ significantly across conditions (NASA-TLX: 19.4, 18.2, 17.5; $F(2, 78) = 2.31$, $p = 0.11$). Thus, we found no evidence that the larger learning gains and more iterative interaction observed with AI Cosmos were accompanied by greater perceived workload.

Human Study: An Instrumented Workbench for Skill Acquisition in Competitive Programming

Whereas the bar-examination study (§ 5.1) primarily examined scenario-based social interaction, this study focuses on the operational dimension: the executable workbench. Competitive programming provides a demanding test of procedural competence, because success requires learners not only to understand an algorithm but also to implement, execute, and debug it as a correct and efficient program. We thus examine whether integrating an executable coding workbench into the learning interaction improves the translation of conceptual understanding into implementation skill, and whether the effect varies with prior aptitude.

Study Design and Measures

Design and participants. We conducted a preregistered, three-arm randomized controlled study stratified by two aptitude-defined curriculum tracks. From the 140 students recruited, 100 completed the diagnostic pre-test, and 54 completed the full multi-session protocol and constituted the analysis sample (93% male; mean age 18.9 years; 89% first-year undergraduates). Participants were assigned to one of three learning conditions: (i) Vanilla LLM, a unconstrained conventional request–response assistant; (ii) LLM-as-Teacher baseline; and (iii) AI Cosmos, which used the same underlying model within a configured scenario equipped with an executable coding workbench for implementation, execution, and iterative debugging. Both LLM-as-Teacher and AI Cosmos conditions followed the same restriction against supplying complete solutions, providing a focused comparison of the additional operational affordances available in AI Cosmos. The Vanilla LLM condition did not impose this restriction. All three conditions used the same base model, which was independently verified to be capable of solving the experimental tasks. The diagnostic pre-test was designed to align training content with prior mastery. Participants who had not yet mastered search and dynamic programming entered the basic-algorithm track, whereas those who had mastered these topics but not the advanced material entered the data-structure track, which covered segment trees and leftist heaps. Participants already proficient in both sets of material did not proceed to the training phase. Within each track, participants were randomized across the three conditions ($n$ = 19 AI Cosmos, 18 LLM-as-Teacher, and 17 Vanilla LLM overall; cell sizes 7–11). Thus, the tracks determined what participants studied, whereas the experimental conditions determined how they studied it. We observed no statistically significant between-condition differences in measured baseline characteristics, including pre-test performance within and across tracks and track composition (all $p \ge0.20$; Table 2 in Appendix § 10.§ 3; Figure 7a).

	Vanilla LLM	LLM-as-Teacher	AI Cosmos	Test ($p$)
Female (%)	11.8	11.1	0.0	$\chi^2$, 0.31
Age (years)	$18.5 \pm0.6$	$18.7 \pm0.7$	$19.3 \pm1.9$	$F$, 0.20
First-year (%)	94.1	88.9	84.2	$\chi^2$, 0.64
Basic-algorithm track (%)	47.1	38.9	47.4	$\chi^2$, 0.85
Data-structure track (%)	52.9	61.1	52.6	$\chi^2$, 0.85
Pre-test (overall score)	$33.0 \pm24.0$	$31.8 \pm26.5$	$36.0 \pm29.0$	$F$, 0.89
- basic-algorithm track (score)	45.0	56.2	52.1	$F$, 0.72
- data-structure track (score)	22.4	16.4	21.5	$F$, 0.67
Pre-test problems below criterion (%)	91.2	86.1	86.8	$\chi^2$, 0.75

Procedure and measures. Each participant completed two training sessions organized around a fixed three-stage practice cycle: (1) Learn: participants studied one problem interactively with their assigned system and submitted a solution to the online judge (OJ). (2) Self-check: they attempted a second problem targeting the same algorithmic concept, with assistance available only when needed. (3) Quiz: they independently solved a third problem that tested the same concept under a different problem framing. Advancement required a full-score OJ submission on all three problems. One week later, participants completed a contest-style post-test containing two previously unseen problems that required variants or combinations of the trained algorithms. The post-test thus assessed generalization of the trained implementation skills to novel problems. To minimize benchmark contamination and memorization, all problems used in the study were newly authored by two competitive-programming experts with NOI/ACM gold-medal credentials and had no online counterparts. The base model used in all three conditions was separately verified to be capable of solving each task, ensuring that task difficulty did not exceed the model’s underlying problem-solving capability. Because conventional OJ scores provide limited diagnostic information about partially correct implementations, the two problem authors also rescored all submitted source code while blind to experimental condition. They first established a common error codebook covering compilation failures, incomplete algorithmic structures, invariant-maintenance errors, boundary and memory faults, complexity or termination failures, and degeneration to brute-force solutions (Appendix § 10.§ 4). These error categories were mapped to standardized deductions on a five-level mastery scale (G1–G5), anchored in the completeness and correctness of the target algorithm’s implementation. The experts independently scored each submission, and their scores were then averaged. Expert scores were strongly correlated with OJ scores (Pearson $r = 0.92$–$0.96$ across assessment phases) but captured partial algorithmic competence that binary test-case outcomes could miss. At post-test, expert rescoring increased the scores of 33 submissions, decreased none, and produced a mean adjustment of +10.8 points. We report three complementary post-test outcomes: (i) continuous expert-rescored performance (0–100); (ii) criterion mastery, defined as grade G4 or higher, corresponding to a substantially complete and correct implementation of the target algorithm; and (iii) score–mastery dissociation, capturing cases in which a submission earns partial credit without demonstrating criterion-level implementation of the target algorithm.

Statistical analysis. We report condition means and standard deviations and examine unadjusted between-condition differences using one-way ANOVA, with the Kruskal–Wallis test as a nonparametric robustness check. The primary between-group inference uses ANCOVA on post-test score, adjusting for pre-test performance and curriculum track. For complementary unadjusted pairwise comparisons, we report Welch’s $t$ tests and bias-corrected Hedges’ $g$ with bootstrap $95\%$ confidence intervals. Given the modest per-condition sample sizes ($n = 17$–$19$) and the bounded or zero-inflated distributions of several outcomes, we additionally report permutation-based $p$-values and Mann–Whitney $U$ tests where appropriate. Criterion-mastery proportions are compared using Fisher’s exact test, whereas participant-level mastery counts are compared using the Mann–Whitney $U$ test. Score-based results are interpreted jointly with criterion mastery, which captures algorithm-specific implementation competence not fully reflected by OJ performance alone. Track-stratified and submission-level analyses are exploratory and are not adjusted for multiplicity. All statistical tests are two-sided with $\alpha= 0.05$.

Results

Skill acquisition. Post-test scores showed a descriptive advantage for AI Cosmos (68.0 $\pm$ 32.3), followed by the vanilla LLM (61.5 $\pm$ 28.1) and LLM-as-Teacher (49.8 $\pm$ 31.7; Figure 7b). The omnibus difference was not statistically significant at this sample size ($F(2, 51) = 1.65$, $p = 0.20$, $\eta^2 = 0.061$; Kruskal–Wallis $p = 0.22$), and the ANCOVA group effect after adjusting for pre-test score and training track showed a similar pattern ($F(2, 49) = 1.89$, $p = 0.16$, partial $\eta^2 = 0.072$). Despite the limited statistical power, the pairwise estimates reveal a consistent architectural advantage. AI Cosmos outperformed the LLM-as-Teacher baseline by 18.2 points, yielding a medium effect size (Welch $t(35.0) = 1.73$, $p = 0.092$; permutation $p = 0.091$; Mann–Whitney $p = 0.096$; Hedges’ $g = 0.56$, 95% CI [−0.06, 1.30]). In contrast, the LLM-as-Teacher condition produced the lowest numerical performance, scoring 11.7 points below the vanilla assistant ($g = -0.38$). Within-condition analyses further show that relative to their respective pre-tests, participants using AI Cosmos and vanilla LLM both improved significantly on trained material (paired $t$-tests: $p = 0.0004$ and $p = 0.0008$, respectively; $d_z = 1.00$), whereas the LLM-as-Teacher condition showed only a non-significant increase ($p = 0.12$, $d_z = 0.39$). Together, these results suggest that integrating guidance with an executable environment provides a stronger pathway from conceptual explanation to procedural programming ability than dialogue-based guidance alone.

Equal scores, unequal mastery. The similar aggregate scores of AI Cosmos and vanilla LLM conceal a more fundamental difference in what learners actually achieved (Figure 8). Both conditions produced successful online-judge submissions at comparable rates (82% vs. 85%), explaining their similar raw scores. Yet, automated judging primarily measures whether a program passes test cases, not whether the learner has acquired the intended algorithmic skill. Among submissions receiving positive OJ credit, AI Cosmos achieved criterion mastery (grade $\ge G4$) in 68% of cases, compared with 45% for vanilla LLM and 32% for LLM-as-Teacher (AI Cosmos vs. LLM-as-Teacher, Fisher’s exact $p = 0.009$; vs. vanilla, $p = 0.12$). Conversely, non-mastery passing solutions, i.e., submissions that obtained test credit without substantially implementing the target algorithm, were less frequent under AI Cosmos (26%) than under vanilla LLM (47%) or LLM-as-Teacher (53%; AI Cosmos vs. LLM-as-Teacher, $p = 0.031$). The results reveal a central limitation of score-based evaluation in AI-assisted programming: passing tests and acquiring transferable procedural knowledge are not identical outcomes. While a conventional assistant may help learners produce partially successful solutions, AI Cosmos’s executable workbench encourages learners to construct, test, and repair their own implementations, resulting in a higher proportion of solutions that reflect genuine algorithmic understanding. Thus, the primary advantage of AI Cosmos is not merely higher task performance, but a closer alignment between measured performance and underlying competence.

Criterion mastery. Because automated scores can obscure differences in solution quality, we next examined criterion mastery: whether learners produced substantially complete and correct implementations of the target algorithms (grade $\ge G4$). This measure captures a stricter form of procedural competence than passing test cases alone (Figure 9a). AI Cosmos learners achieved criterion mastery on an average of 1.11 out of the two post-test problems, compared with 0.76 for vanilla LLM and 0.50 for LLM-as-Teacher. The difference between AI Cosmos and LLM-as-Teacher was significant (Mann–Whitney $p = 0.024$). Moreover, 36.8% of AI Cosmos learners achieved mastery on both problems, compared with 17.6% for vanilla LLM and 11.1% for LLM-as-Teacher. The expert-derived error taxonomy further reveals how this advantage manifested in learners’ code (Figure 9b). AI Cosmos produced the highest proportion of flawless implementations (E0K: 52.6% vs. 38.2% for vanilla LLM and 25.0% for LLM-as-Teacher; Fisher’s exact test, AI Cosmos vs. LLM-as-Teacher, $p = 0.019$), and the lowest proportion of incomplete algorithmic structures (E02: 5.3% vs. 8.8–11.1% in the control conditions). The ordinal mastery analysis showed a similar pattern: post-test submissions were concentrated at the two extremes of the scale, corresponding to either incomplete attempts or fully correct implementations. A proportional-odds model further indicated that AI Cosmos learners had higher odds of achieving a stronger mastery level than LLM-as-Teacher learners (odds ratio = 2.59, 95% CI [1.10, 6.07], $p = 0.029$). Together, these results suggest that AI Cosmos primarily improves the quality of acquired programming skill rather than merely increasing the likelihood of obtaining a passing submission.

Why the workbench matters: executable practice beyond guidance. The preceding analyses suggest that the advantage of AI Cosmos does not arise from providing more instructional content, but from changing how learners interact with that content via an executable environment (Figure 10). First, the benefit of the workbench was largest among learners who required greater external scaffolding. Decomposing the AI Cosmos advantage over LLM-as-Teacher by curriculum track revealed a substantial difference in the basic-algorithm track: AI Cosmos improved post-test scores by 31.4 points, corresponding to Hedges’ $g = 0.94$ for scores and $g = 1.16$ for criterion mastery (the latter reaching significance, Mann–Whitney $p = 0.039$). The advantage was smaller in the more advanced data-structure track (+10.5 points; $g = 0.32$ for scores and $g = 0.58$ for mastery). Although the condition $\times$ track interaction was not significant ($p = 0.34$), this pattern is consistent with the interpretation that executable environments provide greater support when learners have not yet developed stable internal models of algorithm execution. For novice programmers, the ability to construct, execute, and debug within the same environment provides a concrete feedback loop that abstract explanations alone cannot provide. Second, the programming study complements the pattern observed in the bar-examination study. In the legal domain, where competence is expressed primarily through argumentation, additional dialogue structure produced progressively larger benefits. Yet, in programming, competence must ultimately be expressed through executable artifacts. Accordingly, instructional guidance without an integrated environment did not produce the same benefit as a workbench-linked scenario. Together, the two studies suggest that effective AI learning systems must align interaction structure with the form of the target competence. Structured dialogue supports domains where reasoning itself is the object of practice, whereas executable workbenches support domains where competence emerges through construction, testing, and revision. AI Cosmos operationalizes this alignment by coupling guidance with the artifacts through which learners practice.

Validity: whether performance reflects learner capability. Finally, we conducted an exploratory construct-validity analysis examining whether observed programming performance remained associated with learners’ prior ability (Figure 11). Within the AI Cosmos condition, pre-test standing was positively correlated with subsequent criterion mastery (Pearson $r = 0.59$, $p = 0.008$; Spearman $\rho= 0.57$), and this relationship remained stable under leave-one-out resampling ($r \in[0.53, 0.65]$). In contrast, the corresponding correlations were near zero in both control conditions (vanilla LLM: $r = 0.08$, $p = 0.76$; LLM-as-Teacher: $r = -0.06$, $p = 0.80$). The difference between AI Cosmos and LLM-as-Teacher was significant (Fisher’s $z$, $p = 0.040$), with a similar but non-significant trend against vanilla LLM ($p = 0.11$). This pattern suggests that different interaction architectures may produce different relationships between measured performance and learner capability. When learners can rely heavily on generated code, final submissions may partially reflect system assistance rather than the learner’s own implementation ability. By requiring learners to participate in construction, execution, and debugging, AI Cosmos makes observed performance more closely coupled with the learner’s demonstrated competence. Thus, the findings highlight an important consideration for AI-assisted learning evaluation: improving performance is insufficient if the resulting measure no longer reflects what learners can independently do.

Related Work

AI-Assisted Learning and Pedagogically Structured Interaction Evidence from mathematics and programming education shows that improved performance with LLM assistance does not necessarily result in better performance on subsequent unassisted assessments [44, 45]. Research on LLM tutoring has investigated ways to guide students during problem solving rather than directly supply answers. Earlier work uses annotated tutoring dialogues [46] and explicit pedagogical policies to study and organize multi-turn guidance [48] (Pal Chowdhury et al. 2024). More recent methods train pedagogical policies [49], adapt feedback using learner-state estimates and task verification [50], and maintain instructional plans across interactions [51, 52]. Tutoring benchmarks further assess instructional behavior beyond answer correctness [53, 54]. These studies demonstrate progress in adaptive instructional support. However, evaluations of tutoring quality or success on guided tasks do not by themselves establish what learners can do independently. An important question is how students’ own actions during assisted problem solving affect their later performance on unfamiliar tasks without assistance.

Human–Agent Participation in Interactive Task Environments Agent coordination can support human participation rather than only automate task completion. Co-STORM allows users to guide discussions among agents and reports higher-quality exploratory outputs than its comparison systems [55]. In a course-based multi-agent learning environment, students exhibited distinct patterns of knowledge co-construction and co-regulation, associated with prior knowledge and learning gains [56]. Participation in executable tasks also requires actions that change task state and produce feedback. CodeAct and SWE-agent improve agent performance through execution feedback and task-specific interfaces [57, 58], while TutorGym evaluates tutoring agents on intermediate actions in interactive learning systems and identifies substantial errors in their judgments of incorrect steps [59]. Magentic-UI and Collaborative Gym further allow users to participate during agent execution [60]; the latter reports better collaborative task outcomes than autonomous execution [61]. These results establish capabilities for conversational participation and shared task execution. For learning applications, however, task success alone does not establish which reasoning and operations learners performed themselves.

Knowledge Building and Reusable Practice Records Learning activities can produce resources that remain useful after a task is completed. In knowledge-building environments, students improve shared ideas through successive contributions, while Learning by Design incorporates earlier design attempts into later inquiry [62, 63]. More recent tools make learners’ activity records accessible: Watanabe et al. [64] combines programming and browsing histories and identifies distinct problem-solving patterns that are not evident from final submissions alone. Related research on LLM agents shows that experience can also be reused for subsequent execution. Agent Workflow Memory extracts reusable routines and improves web-task success [65], G-Memory retrieves collaboration histories together with generalized insights [66], and Skill-Pro validates procedural skills for reuse across tasks and agents [67]. These results establish two uses of prior activity: improving an agent’s later actions and helping people examine and revise earlier work [34, 68]. The distinction matters for educational systems, where a reusable solution does not necessarily preserve the decisions, feedback, and revisions through which learners developed it.

Conclusions

In this paper, we introduce AI Cosmos, an agentic infrastructure for situated learning that organizes AI-assisted learning around two complementary principles: scenario-based participation and cumulative assets of practice. Through the Scenario–Asset Framework, AI Cosmos enables learners and role-specialized agents to participate in structured learning activities, while preserving the processes and outcomes of practice as reusable assets across scenarios. Two preregistered controlled studies in learning legal reasoning and competitive programming demonstrate the potential of structured participation to improve learners’ independent competence.

Contribution

Contributors’ names are listed in alphabetical order by first name.

Core Contributors

Fangming Chang, Mingyu Su, Yifan Zhu, Yuan Wang

Contributors

Bo Zhang, Bosi Wen, Chenhao Yuan, Chenwen Liao, Dazhi Jiang, Deyao Hong, Erle Zhu, Genrui Zhang, Guchan Li, Hongwen Gan, Huichao Bi, Jiahao Yan, Jincheng Fan, Kehan Zheng, Qinren Luo, Qiran Liu, Ruoqi Cao, Shizhi Zhang, Sihan Chen, Siyan Deng, Xuehan Zhou, Xujun Li, Yang Dang, Yanzhuo Zhu, Yu Zhang, Zheyu Chen

Tech Leads

Jinfeng Zhou, Yuxuan Chen

Advisors

Hongning Wang, Manli Li

Experiments

Human Study: Situated Learning for Bar-Examination Preparation

Participant Statistics

Detailed distribution statistics of the participants are presented in Table 1.

Human Study: An Instrumented Workbench for Skill Acquisition in Competitive Programming

Participant Statistics

Detailed distribution statistics of the participants are presented in Table 2.

Expert Re-grading Rubric: Mastery Scale and Error Codebook

To convert the coarse, test-case-based online-judge (OJ) score into a measure of genuine algorithmic competence (§ 6), two NOI/ACM gold-medalist experts re-graded every submission, blind to arm, using the two-part rubric reproduced here: a five-level mastery scale (G1–G5; Table 3) that summarizes how completely and correctly a submission realizes the problem’s target algorithm, and an error codebook (Table 4) that records the specific failure mode underlying that grade. Each submission is tagged in the form [code:name] together with a grade. Throughout, the “target algorithm” is the intended solution for the problem at hand (e.g., flood-fill or dynamic programming on the basic-algorithm track; a segment tree or leftist heap on the data-structure track). Two grading conventions are central and explain why the expert grade can diverge from the raw OJ score. First, a submission that merely passes test cases by means of a brute-force or otherwise non-target method is not credited as mastering the target algorithm: it is normally graded G1, regardless of the points it earns. Second, every submission that “looks like the target algorithm but falls short of full marks” was re-run locally and assigned a specific failure code before being graded, so that the grade reflects a diagnosed failure mode rather than the OJ verdict alone. The criterion-mastery threshold used in the main text is grade $\ge$ G4 (i.e., G4 or G5); the “flawless-implementation” rate (EOK) reported there is exactly the G5 rate.

Grade	Anchor	Description
G1	Target algorithm not formed	Implementation direction is inconsistent with the target algorithm, or no valid algorithmic intent can be identified. A brute-force or non-target solution is graded here even when it earns partial credit.
G2	Intent present, key errors	The target algorithm is attempted, but with defects in complexity, boundary handling, memory safety, or a critical invariant.
G3	Main flow present, details unstable	The target algorithm’s main framework exists but still contains key implementation errors.
G4	Framework essentially complete	Only minor, non-fundamental defects remain. (Criterion-mastery threshold.)
G5	Correct and complete	The target algorithm is implemented correctly and passes; corresponds to error code EOK.

Code	Name	Definition	Severity
E00	Not submitted / incomplete	No valid submission, or no assessable result was produced.	medium
E01	Compilation / syntax error	Code does not compile, or the submitted file is not executable source.	high
E02	Incomplete skeleton	The basic structure of the target algorithm was never formed.	high
E03	Critical maintenance error	A core invariant or key maintenance step is wrong, yielding incorrect results.	high
E04	State / boundary error	The main flow is present, but a conditional branch or boundary case is mishandled.	medium
E05	Complexity / termination error	Complexity is inadequate or the termination strategy is unstable, failing some test cases.	high
E06	Brute-force degeneration	A non-target, brute-force implementation degrades the complexity (time-limit exceeded).	high
E07	Boundary / memory-safety error	Out-of-bounds or illegal access, stack-overflow risk, or another runtime fault.	high
E99-S0	Source missing	No analyzable source code is available for the problem.	medium
E99-S1	Intent unclear	A file exists, but no valid algorithmic intent can be determined.	medium
EOK	No error	The target algorithm is implemented correctly and passes (grade G5).	none
N01	Non-standard but passing	A non-standard yet theoretically valid solution that passes all local tests (flagged separately).	note
N02	Non-target implementation	The approach is inconsistent with the target algorithm; not counted as target-algorithm mastery.	note

The two parts are coupled: the error code localizes what went wrong, and the mastery grade places that failure on a single ordinal scale of target-algorithm competence. Thus an E06 (brute-force) or N02 (non-target) submission is graded G1 however many test cases it passes, an E02 (incomplete skeleton) typically maps to G2–G3, a submission with only minor defects maps to G4, and an EOK submission is G5. The main-text failure-mode analysis (Figure ??b) reports the rates of the three most diagnostic outcomes on the post-test problems—EOK (correct), E06 (brute-force degeneration), and E02 (incomplete skeleton)—computed over all submissions in each arm.