betaFeedback

WorksComputer Science

RECAST: REconstruction of Chain-of-Thought via Auditable Structured Traces

Abstract

Large closed-source reasoning models expose only their final answers. The intermediate reasoning that produces those answers is hidden behind API surfaces, tool-call policies, and safety filters. Existing chain-of-thought (CoT) extraction methods assume a side channel: a visible reasoning field, a scratchpad carried through tool calls, or a signed/encrypted reasoning block. We reproduce three representative state-of-the-art extraction methods and find that, on Claude-Sonnet-5.5 accessed through the standard chat-completions interface, all three return zero CoT tokens. The failure is structural rather than a bug: the reasoning field is absent, named tool selection is rejected, free-text scratchpad arguments are filtered, and no replayable reasoning block is exposed. We therefore take a different route. We present RECAST, a reconstruction-based framework that conditions a decoder on the question and the target model's final answer, reconstructs a first-person structured rationale, cleans it, and builds SFT-ready data. To the best of our knowledge, RECAST is the first system to reconstruct CoT from frontier closed models such as Claude-Sonnet-5.5, GPT-6.1, and Gemini 3 without using hidden reasoning channels. We instantiate RECAST on Claude-Sonnet-5.5 and additionally obtain successful qualitative reconstructions on GPT-6.1 and Gemini 3 with the same answer-only protocol. The system is open-source and packaged as an agent skill: one prompt deploys it and runs the complete pipeline to produce SFT-ready CoT data. This preprint describes the system, its threat model, and a controlled artifact validation; it does not claim large-scale benchmark improvements.

Introduction

Frontier closed-source reasoning models are increasingly deployed as black boxes: a user supplies a question and receives a final answer, while the intermediate reasoning that produced the answer remains hidden. This is a problem for reasoning distillation. If we only observe answers, a student model can imitate what the teacher answered but receives little supervision on how to reason. If we can obtain an auditable rationale for each answer, we can instead train the student on the reasoning process itself.

A growing line of work attempts to recover such rationales from closed models. The representative strategies are (i) forcing the model to place its reasoning into a custom tool argument [9], (ii) replaying a scratchpad state through multi-step tool injections [14], and (iii) replaying a signed or encrypted reasoning block with a weaker decoder [13]. These methods are extraction methods: each assumes a side channel through which hidden reasoning becomes observable.

We reproduced all three methods and evaluated them on Claude-Sonnet-5.5 [1] through a standard OpenAI-compatible chat-completions interface. None of them produced a single CoT token. The failures were not implementation errors. The interface rejected named tool selection, filtered free-text tool arguments, omitted reasoning fields, and exposed no signed or encrypted reasoning block. In one probe, the target model consumed 9,949 completion tokens but returned only a final answer, confirming that substantial internal computation took place while the reasoning layer remained invisible.

The failure is structural. The three methods fail for different surface reasons, but the underlying pattern is the same: they need a channel that a modern frontier deployment does not provide. We identify four constraints that jointly close the extraction surface:

  1. No reasoning field. Responses contain final content and usage, but not reasoning_content, thinking, signature, or encrypted_content.

  1. No named tool forcing. The endpoint accepts only automatic tool choice; requests for a specific function or “any” function are rejected.

  1. Filtered free-text tool arguments. Tools with a free-text scratchpad argument are blocked by the safety layer, including when the tool is given a neutral name.

  1. No replayable block. There is no provider-issued reasoning artifact that a weaker decoder could replay.

These constraints are properties of the deployment, not of our scripts: ordinary tools with short structured arguments still work, and a standalone decoder call succeeds. We therefore do not try to bypass them.

From extraction to reconstruction. RECAST takes the only reliably observable signal—the final answer—and reconstructs a rationale from it. A decoder model receives the question and the target model’s complete answer, writes a first-person, stage-structured solution trace, and is then cleaned by a second model to remove dialogue, meta-commentary, and answer statements. A final-answer stage attaches a canonical answer, and a dataset builder produces chat-formatted SFT data with assistant-only supervision. Because the method only needs the final answer, it does not depend on hidden reasoning fields, tool-call state, or provider-issued encrypted blocks. The same protocol applies to any model whose answers are observable through a chat interface.

Auditability. We use “auditable” in a procedural sense: the reconstructed trace is organized into stages, is self-contained, does not mention dialogue/meta artifacts, and does not reveal the final answer before the derivation. The decoder is answer-conditioned, while the student sees only the question at inference. This is the property the title names; it is a condition for usable supervision, not a claim that the trace is the target’s true hidden computation.

Contributions.

  • First reconstruction-based system for frontier closed models. To the best of our knowledge, RECAST is the first system to reconstruct CoT from frontier closed models such as Claude-Sonnet-5.5, GPT-6.1, and Gemini 3 without access to hidden reasoning fields, provider-specific tool side channels, or signed/encrypted reasoning blocks.1

  • An end-to-end reconstruction pipeline. We implement collection, reconstruction, cleaning, answer extraction, SFT dataset construction, and validation as a reproducible job-based pipeline.

  • Dataset-aware elicitation. We design six teacher–student patterns and a router that selects a pattern from the dataset profile, with secondary-pattern and short/medium prompt fallbacks for content filters.

  • Defensive cleaning and validation. We introduce a JSON-contract cleaner, leak and length guards, answer canonicalization, and schema/duplicate/leakage checks for SFT data.

  • One-prompt automation. We package the entire project as an agent skill: a single prompt deploys it and runs the complete pipeline to produce SFT-ready CoT data.

Scope and artifact. This is a system report, not a large-scale empirical study. Section 3 formalizes the extraction surface and explains why the three extraction methods fail. Section 4 presents the reconstruction pipeline. Section 4.8 reports a controlled artifact validation on a single MATH-500 item; we deliberately do not report a benchmark comparison in this preprint. The complete system, prompts, patterns, and reproduction commands are available at github.com/C1ove-ai/RECAST ([16]).

Related Work

Chain-of-thought and rationale distillation. Chain-of-thought prompting asks a model to produce intermediate reasoning before its final answer ([19]). This idea underlies a large family of distillation methods that train a smaller student on teacher-generated rationales. Distilling Step-by-Step trains a student on both rationales and answers ([6]). STaR bootstraps rationales that lead to correct answers and iteratively fine-tunes on them ([21]). Orca studies explanation-style distillation from stronger teachers ([11]). LIMO and s1 show that carefully curated reasoning traces and test-time scaling can produce strong small reasoning models ([20, 10]). DeepSeek-R1 demonstrates large-scale reinforcement learning and distillation of reasoning traces ([3]). These methods assume access to rationales, either from an open model, a visible thinking channel, or human annotation. RECAST targets the setting in which none of these is available and only the final answer is observable.

Extraction of hidden reasoning. A recent line of work attempts to make hidden reasoning observable through the API surface. Luo et al. [9] use a custom tool with a free-text argument and a forced tool choice to induce a “forced reasoning” trace. Qu et al. [14] replay a scratchpad state through a sequence of tool injections and search over injection trajectories. Panfilov et al. [13] replay signed or encrypted reasoning blocks with a weaker decoder model. These methods are important because they show that reasoning can sometimes be extracted without weight access. They are, however, extraction methods: each depends on a side channel that the deployment must expose. Our reproduction shows that on Claude-Sonnet-5.5 these channels are closed (§ 3). RECAST is complementary rather than adversarial: it does not attempt to open the channel, but reconstructs a rationale from the answer that the channel does expose.

Faithfulness of rationales. A rationale that looks plausible need not faithfully reflect the computation that produced the answer. Lanham et al. [7] introduce interventions that test whether a model’s stated reasoning actually causally affects its predictions. Turpin et al. [15] show that chain-of-thought explanations can be biased by post-hoc rationalization and can omit influential features. Process supervision and verifier-based methods provide complementary signals for reasoning quality ([8, 17]). We treat the reconstructed trace as a proxy, not as the model’s true hidden CoT, and we include answer consistency, leakage, length, and downstream utility as part of the evaluation protocol rather than assuming faithfulness from surface fluency.

Data synthesis and agent tooling. Synthetic data pipelines have become a practical way to obtain supervision at scale ([18]). Our work is also a systems contribution: a job-based pipeline, a prompt router, and an agent skill that packages the workflow for one-prompt deployment. The agent skill format used here follows the standard SKILL.md convention supported by Codex, Claude Code, and DeepSeek Harness; the implementation is model-agnostic and configures the target, decoder, and cleaner independently.

Positioning. RECAST differs from prior rationale distillation in two ways. First, it does not require a teacher rationale; it reconstructs one from the answer. Second, it does not require a hidden reasoning channel; it is designed for the answer-only interface that frontier deployments actually expose. The price is that the reconstructed trace is a proxy whose faithfulness must be evaluated, which we make explicit throughout.

Threat Model

We formalize the reconstruction setting as an extraction threat model. The model is the target; the reconstructor is an external agent with only black-box query access. The goal is to obtain an auditable rationale that is useful for training a student, not to recover the target’s true hidden computation.

Target model and observable surface. Let TT be a closed-source reasoning model and let qq be a question. We write zz for the model’s hidden chain-of-thought and y=T(q)y=T(q) for the final answer that the API returns. The API exposes yy, token usage, and a finish reason. It does not expose weights, logits, hidden states, or zz. We write O(T)\mathcal{O}(T) for this observable surface.

Reconstructor capabilities. A reconstructor RR can (i) query TT with chosen questions and prompts, (ii) use an external decoder model DD, (iii) choose among a set of elicitation patterns P\mathcal{P}, and (iv) post-process answers and traces locally. It cannot read the target’s weights, hidden states, reasoning fields, or provider-internal logs.

Goal. Given a dataset {(qi,yi)}i=1n\{(q_i,y_i)\}_{i=1}^{n}, RR produces a trace ci=D(qi,yi,pi)c_i=D(q_i,y_i,p_i) for a pattern pi∈Pp_i \in\mathcal{P}. The trace is used to train a student SS on (qi,ci)(q_i,c_i). Success is functional: SS should solve held-out questions, and cic_i should pass auditability checks (answer consistency, leakage, length, structure). Exact recovery of ziz_i is neither required nor claimed.

Auditability condition. We use auditable to describe a trace that is structured, self-contained, free of dialogue/meta artifacts, and does not reveal the final answer before the derivation. The student receives only the question at inference; the decoder sees yy while constructing cc. Auditability is a property of the reconstruction procedure, not a claim that the trace is the target’s true hidden computation.

Constraints. The reconstructor operates under realistic deployment constraints: rate limits and query budgets; provider safety filters; terms of service; and no access to privileged interfaces. We do not attempt to bypass content filters, exploit tool implementations, or use unauthorized endpoints. The threat model is therefore an interface-limited one: the question is what can be reconstructed from the surface a legitimate user is allowed to observe.

Why the three extraction methods fail. Table 1 summarizes our controlled reproduction. The three methods require different side channels, and on Claude-Sonnet-5.5 through standard chat-completions all three channels are unavailable. The result is zero CoT tokens in all cases. The source model in the P2 probe consumed 9,949 completion tokens but returned only a final answer, which is consistent with substantial hidden reasoning that never reaches the client.

Table 1: Controlled reproduction of three representative CoT extraction methods on Claude-Sonnet-5.5 through a standard chat-completions interface. All three obtain zero CoT tokens. The table reports the observed interface behavior, not a benchmark score; runs use one or two questions and are intended as mechanism checks.

MethodRequired side channelObserved on Claude-Sonnet-5.5CoT tokens
P3 Forced-Reasoning ([9])Named/any tool_choice plus a free-text tool argumentNamed tool choice rejected with HTTP 400; free-text scratchpad arguments blocked with content_filter0
P1 EchoCoT ([14])Scratchpad tool state across multiple turnsFirst scratchpad call blocked with content_filter; tool invocation rate 0%0
P2 Stealing Reasoning Traces ([13])Signed/encrypted reasoning block to replayNo signature, encrypted_content, or reasoning_text; source billed 9,949 completion tokens; extraction error 1.00
RECAST (ours)Final answer yy plus an external decoder DDOnly the answer is required; no hidden channelstructured trace

Table 1.

The four closing constraints. The observed failures are not independent accidents. They reflect four interface properties:

  1. Reasoning fields are absent. The response contains final content and usage, but not reasoning_content, thinking, signature, or encrypted_content. P3’s native baseline and P2’s source stage both require such a field.

  1. Named tool choice is rejected. The endpoint accepts only automatic tool choice. P3’s first protocol step—forcing a specific function—cannot be executed.

  1. Free-text tool arguments are filtered. A tool with a free-text scratchpad argument is blocked even under a neutral name. P1’s scratchpad and P3’s soft-force fallback both depend on this channel.

  1. No replayable block is exposed. P2 needs an opaque provider-issued block. Chat-completions does not produce one, so there is nothing to replay.

These constraints are deployment-level. Ordinary tools with short structured arguments still work, and a standalone decoder call succeeds; the failure is specific to the channels that extraction methods require.

Why reconstruction succeeds. RECAST operates entirely inside the observable surface. It consumes qq and yy, both of which are available under the threat model, and uses a decoder to synthesize a structured rationale. It does not require a reasoning field, a tool call, a named tool choice, or an encrypted block. This shifts the problem from opening a closed channel to reconstructing a plausible and auditable trace from an open one. The shift is what makes the method model-agnostic: any model whose final answer is observable can be processed by the same protocol.

Success criteria. We evaluate the reconstructed trace along four axes: (i) task utility, whether a student trained on the trace improves on held-out questions; (ii) answer consistency, whether the trace supports the reference answer; (iii) auditability, whether the trace is structured, self-contained, and free of dialogue/meta artifacts; and (iv) cost, the number of target, decoder, and cleaner calls per accepted sample. This preprint reports the system and a controlled artifact validation; the large-scale version of these axes is the subject of a follow-up study.

Methodology

Overview and Design Principles

RECAST turns a dataset of questions into SFT-ready reasoning data through five stages: (1) collect the target model’s answers, (2) reconstruct a structured rationale with a decoder, (3) clean the rationale, (4) extract the final answer, and (5) build and validate the SFT dataset. Figure 1 illustrates the flow. The four model roles are deliberately separated: a target model produces answers, a decoder reconstructs traces, a cleaner removes artifacts, and a student is trained on the result. Each role can use a different OpenAI-compatible endpoint; the default configuration uses a relay-served target model and DeepSeek for decoding and cleaning, but the code has no vendor-specific dependency.

The RECAST pipeline

Figure 1. The RECAST pipeline. The target model is treated as a black box; only its final answer enters the reconstruction stage. The decoder produces a structured first-person trace, the cleaner removes dialogue/meta artifacts, and the dataset builder attaches a canonical answer and validates the result.

We follow six design principles. Answer-only: the reconstruction stage never requires a hidden reasoning field. Role separation: target, decoder, and cleaner are independent, so a failure in one role does not invalidate the others. Dataset-aware elicitation: the prompt used to collect the answer is selected from the dataset profile, because some patterns elicit more explicit reasoning than others. Defensive cleaning: every generated trace passes through leak, length, and answer-consistency checks. Reproducibility: every run creates a timestamped job with logs, configuration, and a manifest. Automation: the same pipeline is exposed as a command-line tool and as an agent skill.

Target Answer Collection and Pattern Routing

The target model is queried with a dataset-specific elicitation pattern. A pattern is a short teacher–student script that encourages the target to expose intermediate steps without explicitly asking for a hidden CoT. Table 2 lists the six patterns used by the router.

PatternProfileStructure and purpose
math_solvermathTeacher reads the problem; student derives and verifies. For proof problems, the verifier attacks the argument for gaps.
socraticlogic, planningRepeated questions on goal, constraints, assumptions, and counterexamples before solving.
expert_panelscienceMultiple expert roles decompose the problem, check assumptions, and converge on a solution.
pair_programmingcodeDriver–navigator pair writes code, tests, and counterexamples.
red_bluesecondaryBlue team proposes a solution; red team attacks edge cases and counterexamples; a judge converges.
retrospectiveopen-endedSolve first, then use a retrospective/PDCA cycle to surface requirements and extensions.

Table 2. Elicitation patterns and their intended profiles. The pattern controls how the target answer is elicited; it does not control the decoder or the cleaner.

Routing. The router first checks a dataset-name map (e.g., MATH-500 and GSM8K map to math; HumanEval and MBPP map to code). For an unknown dataset, it scores the first rows with profile keywords and falls back to general. The selected profile determines a primary pattern and a secondary pattern (Table 2 shows the primary mapping; the registry contains the full mapping). The router is exposed as a standalone CLI so that users can inspect the decision before running the pipeline.

Filter-resilient collection. Relay deployments may reject prompts that ask for structured reasoning. The collector therefore maintains a prompt chain: primary pattern, secondary pattern, medium prompt, and short prompt. Each prompt is tried first with the original question and then with a LaTeX-light variant. The attempt metadata (pattern, prompt variant, question variant) is stored with every record, so a job can be audited after the fact. A failed question does not abort the batch; it is recorded as an error and the pipeline continues.

Rationale Reconstruction

The decoder receives the question and the target model’s complete answer. It is instructed to write in the first person, in the order in which the solution unfolds, and to cover five stages: task identification, requirement/diagnostic analysis, strategy selection, key derivation, and verification/closing. The prompt forbids dialogue and meta-commentary (“teacher”, “student”, “template”, “decode”, etc.), forbids restating the target answer verbatim, and forbids announcing the final answer in the middle of the trace. A compact version of the prompt is shown in Appendix § 8.

Length floor. Short reconstructions tend to collapse into generic summaries. We therefore enforce a minimum output length

Lmin⁡=max⁡(0, α⋅∣y∣),L_{\min} = \max(0,\ \alpha\cdot|y|),

where ∣y∣|y| is the approximate token length of the target answer and α\alpha is a configurable floor (default 1.0). If the decoder returns a trace shorter than Lmin⁡L_{\min}, it is re-prompted with an expansion instruction that asks for omitted derivations, alternatives, and verification steps. The floor is a coverage heuristic, not a quality guarantee; § 6 discusses the risk of padding.

Template variants. The pipeline ships two decoder templates. The solution template is the default: it asks for a natural first-person solution trace with the five stages above. The audit template is a legacy seven-stage version that adds explicit output organization and self-check sections. The solution template produces shorter, more natural traces and is used for the artifact validation in § 4.8; the audit template is retained for ablation and for datasets that need a more explicit structure.

Cleaning and Guardrails

The decoder can leak artifacts from the target answer: dialogue framing, classroom metaphors, statements about the task or the reconstruction process, and explicit “final answer” lines. The cleaner rewrites the trace into a self-contained first-person rationale. It returns a JSON object with three fields:

{cleaned_cot, removed, notes}.\{\texttt{cleaned\_cot},\ \texttt{removed},\ \texttt{notes}\}.

The cleaner is not allowed to add new mathematical content; it may remove or rewrite noise, and it must preserve the stage headings and their order.

The removed categories are recorded for auditing (dialogue, shell, task meta, answer line).

The cleaner runs at temperature zero with a JSON response mode. It retries on parse errors and on leak/length violations, and it receives the specific violation as feedback. A rule-based safety net runs after the model: it strips residual meta sentences, removes explicit answer declarations, normalizes whitespace, and checks that the trace is not empty. The cleaned trace is stored as an overlay keyed by record; the raw decoded trace is never overwritten.

Answer Extraction and Canonicalization

Each training sample needs a final answer. When the dataset provides a reference answer, the pipeline uses it directly (for MATH-500 and GSM8K, the test-set answer is used). When no reference is available, a hybrid extractor first applies pattern-based parsing and then asks a model to extract the final answer from the trace. Answers are normalized (whitespace, \boxed{}, units, and common equivalent forms) before being appended to the supervision target. The extraction method is recorded per record (testset_reference, llm_extraction, or regex), so a dataset can be filtered by answer provenance.

SFT Data Pipeline

The dataset builder converts each accepted trace into a chat sample. The system prompt frames the student as a reasoning assistant; the user message is the question; the assistant message contains the cleaned trace followed by a canonical final-answer line. Loss is applied only to the assistant tokens. We support four target variants (Table 4) for ablations and for different training objectives.

TargetAssistant content
cot_answerCleaned CoT followed by the final answer (default).
cot_onlyCleaned CoT only; the answer is kept in metadata and not trained on.
source_onlyThe target model’s original answer; used as an answer-only baseline.
cot_plus_sourceCleaned CoT plus the original answer; used to test whether answer style adds signal beyond the reconstructed rationale.

Table 4. Supervision target variants produced by the dataset builder.

StatusMeaning and downstream handling
okCleaned trace passed JSON parsing, leak checks, and length checks; used as the supervision target.
still_leakyDialogue or meta terms remain after retries; raw trace is used and the leak is recorded.
rejected_shortCleaned trace is shorter than the minimum ratio; raw trace is used and the record is flagged.
rejected_longCleaned trace exceeds the maximum ratio (possible duplication); raw trace is used and the record is flagged.
api_errorCleaner call failed; raw trace is used and the error is recorded.

Table 3. Cleaner outcomes and how the dataset builder handles them. A non-ok outcome never silently replaces the raw trace with a shorter or cleaner-looking one.

Before writing the splits, the builder removes duplicate questions, enforces minimum and maximum lengths, filters traces whose final answer disagrees with the reference, and checks for benchmark leakage. The output contains sft_train.jsonl, sft_val.jsonl, sft_test.jsonl, per-variant evaluation files, and dataset_stats.json with counts, dropped reasons, answer methods, and CoT sources. The builtin benchmark loaders use MATH-500 (Hendrycks et al., [5]) and GSM8K (Cobbe et al., [2]). A validator checks the schema, duplicate rates, and leakage terms; a failed validation stops the job without deleting the raw artifacts.

Agent Skill and One-Prompt Automation

The complete workflow is packaged as an agent skill. The skill directory contains a SKILL.md entry point, a pipeline reference, and a small wrapper that locates the repository from any working directory. Installing the skill copies it into the host’s skill directory (project-level .dsh/skills or .claude/skills, or user-level ~/.codex/skills); a new session then exposes the skill to the agent. The intended interaction is a single prompt such as “run CoT reconstruction on MATH-500 with model X”. The agent loads the skill, resolves the repository, chooses the dataset, target model, and pattern, runs the pipeline, and reports the job directory and summary.

The skill does not embed credentials. It reads the three configuration files (config.relay.json, api.txt, api_clean.txt) and supports swapping the target, decoder, and cleaner independently through command-line flags (--target-*, --decoder-*, --cleaner-*). This is important for reproducibility: a different lab can reuse the skill with entirely different model providers.

Implementation and Artifact Validation

The repository contains the pipeline entry point, the collector, the decoder runner, the router, the cleaning/answer/dataset/validation scripts, the prompt templates, the patterns, the evaluation harness, and the skill. A run creates a job directory jobs/ with raw/, decoded/, cleaned/, answers/, sft/processed/, logs/, config/, manifest.json, and summary.md. The manifest records every stage’s status, elapsed time, and output path; the summary contains counts and a ready-to-run training command.

To validate the artifact rather than the method’s accuracy, we ran the complete pipeline on a single MATH-500 item with Claude-Sonnet-5.5 as the target and the default DeepSeek roles. All six stages completed (Table 5). The collector produced one successful answer; the decoder produced a 1,297-token trace against a 938-token floor; the cleaner returned ok with no leak terms; the answer stage used the MATH-500 reference; the builder produced one cot_answer training sample; and the validator passed. We use this run only to verify that the stages compose and that the artifacts are self-consistent.

StageTime (s)
Collect target answer12.8
Decode reconstructed CoT9.9
Clean trace8.4
Extract/canonicalize answer2.2
Build + validate SFT data0.2

Table 5. Single-item artifact validation. Times are wall-clock seconds for one question; the run is a pipeline smoke test, not a benchmark.

Cross-model qualitative validation. Beyond Claude-Sonnet-5.5, we applied the same answer-only protocol to GPT-6.1 ([12]) and Gemini 3 ([4]) and obtained successful qualitative reconstructions: each run produced a structured, first-person trace that passed the cleaner and the dataset validator. We do not report quantitative comparisons for these models in this preprint; they serve as existence demonstrations that the protocol is not tied to a particular vendor. The system accepts any OpenAI-compatible chat endpoint, so the same commands apply when the target, decoder, or cleaner is changed.

Discussion

Extraction versus reconstruction. The three reproduced methods are elegant when their side channel exists. They fail on Claude-Sonnet-5.5 not because the implementations are wrong, but because the deployment closes the channel: reasoning fields are omitted, named tool choice is rejected, free-text tool arguments are filtered, and encrypted blocks are absent. A system that depends on any one of these assumptions inherits a single point of failure. RECAST removes the dependency altogether. It treats the final answer as the observable signal and reconstructs a rationale that is consistent with it. The trade-off is explicit: the reconstructed trace is a proxy, and its faithfulness must be tested rather than assumed.

Why the answer is a useful anchor. The final answer constrains the space of plausible rationales: it fixes the destination and rules out derivations that end elsewhere. This is weaker than observing the true trajectory, but it is stronger than answer-only training, which provides no intermediate supervision. The decoder uses the answer as an anchor, not as a script: the prompt asks for a first-person derivation and forbids restating the answer as the explanation. The supporting pattern is also important. A pattern that elicits a detailed answer gives the decoder more material to reconstruct from; a terse answer leaves more of the rationale to the decoder’s prior.

Is the trace auditable? We use “auditable” in a procedural sense: the trace is structured into stages, is self-contained, does not mention teachers/students/templates/decoding, and does not announce the final answer in the middle of the derivation. Cleaning enforces these properties, and the validator checks length, answer consistency, and leakage. This is not a guarantee of faithfulness. A trace can be well structured and still be a post-hoc rationalization. We therefore treat auditability as a necessary condition for usable supervision, not as evidence that the trace equals the target’s hidden computation.

Auditable traces. The title uses “auditable” to emphasize that the trace is organized, self-contained, and checkable. Auditability is weaker than faithfulness: a well-structured trace can still be a post-hoc rationalization. The decoder is answer-conditioned; the student sees only the question at inference. A training-time answer-masking variant is a natural extension, but it is outside the scope of this preprint, and we do not conflate the deployment condition with a stronger experimental claim.

Cross-model behavior. The failure analysis is specific to Claude-Sonnet-5.5 through a standard chat-completions interface; it is not a claim that the three extraction methods fail everywhere. The positive result is more general: because RECAST only needs the final answer, the same protocol applies to GPT-6.1, Gemini 3, and any other model with an observable answer. We have obtained qualitative successes on all three model families, but we do not report a controlled cross-model comparison in this preprint.

Cost and automation. Each accepted sample uses one target call, one decoder call, one cleaner call, and (optionally) one answer-extraction call. This is more expensive than answer-only distillation but much cheaper than human rationale annotation. The skill reduces the operational cost further: one prompt deploys the pipeline and produces a self-contained job, so a user who is not a Python developer can still obtain SFT-ready data. The skill inherits the pipeline’s model-agnostic configuration, so the same prompt works with a different target, decoder, or cleaner.

Limitations and Responsible Use

No large-scale efficacy claim. This preprint does not report a benchmark comparison. We do not claim that a student trained on reconstructed traces outperforms a base model, an answer-only baseline, or a teacher. The artifact validation in § 4.8 is a pipeline smoke test, not evidence of accuracy. A full evaluation requires multiple seeds, multiple datasets, and a controlled comparison of target variants.

Endpoint-specific failure analysis. The failure of the three extraction methods is observed on Claude-Sonnet-5.5 through one chat-completions interface. A different endpoint, a native SDK, or a different model version may expose a channel that this deployment closes. Our claim is about the interface we studied, not about the impossibility of extraction in general.

Reconstructed traces are proxies. The decoder produces a plausible rationale, not the target’s hidden computation. The trace may contain post-hoc rationalization, may omit influential features, or may be overconfident. Cleaning improves presentation but does not make the trace faithful. Downstream evaluation and faithfulness probes are necessary; surface fluency is not enough.

Answer leakage and padding. The decoder sees the answer and may leak it into the rationale; the cleaner removes explicit answer statements, but subtle leakage can remain. The length floor reduces short, generic traces but can also encourage padding. The dataset builder records the trace source and length, and users should inspect a sample before large-scale training.

Model and provider drift. Relay endpoints, model versions, and safety policies change. A prompt that succeeds today may be filtered tomorrow, and a decoder update may change the style of the reconstructed trace. The job manifest records the configuration and timestamp, but it cannot guarantee that a later run is identical. Reproduction should record model identifiers and endpoint classes.

Responsible use. RECAST is a research tool for reconstructing auditable rationales from answers. It is not intended to bypass safety controls, extract proprietary reasoning at scale, or circumvent provider terms. The threat model in § 3 assumes authorized access to an API and forbids filter evasion. Users are responsible for complying with their model provider’s terms and for not redistributing model outputs without permission. The benchmark data used by the pipeline follows its upstream license (MATH and GSM8K are MIT-licensed in their original distributions), and redistributed derivatives should preserve the corresponding notices.

Training scope. The optional training scripts target LoRA/QLoRA SFT on a single GPU and were not executed at scale for this preprint. Reported GPU results should be treated as environment dependent. The paper’s claims are about the reconstruction pipeline and its artifact, not about a particular training recipe.

Conclusion

We studied the practical limits of extracting chain-of-thought from frontier closed models. Three representative state-of-the-art extraction methods—forced reasoning, scratchpad state replay, and encrypted-block replay—all returned zero CoT tokens on Claude-Sonnet-5.5 through a standard chat-completions interface. The failures follow from four interface constraints, not from implementation errors. We therefore proposed RECAST, which reframes the problem from extracting a hidden rationale to reconstructing an auditable rationale from the final answer. RECAST combines dataset-aware answer elicitation, first-person reconstruction, defensive cleaning, answer canonicalization, and a validated SFT data pipeline. To the best of our knowledge, it is the first system to reconstruct CoT from frontier closed models such as Claude-Sonnet-5.5, GPT-6.1, and Gemini 3 without using hidden reasoning channels. The system is open-source and packaged as an agent skill, so a single prompt can deploy it and run the complete pipeline to produce SFT-ready data. The next step is a controlled large-scale evaluation of student accuracy, faithfulness, and cost; the present preprint deliberately reports the system and its artifact validation rather than an efficacy claim.

Prompt Templates

The repository contains the exact prompts used in the runs. For readability, this appendix gives English translations of the two central templates; the original templates are available under templates/ and eval/data/ in the artifact.

Decoder prompt. The decoder receives the question and the full target answer and is asked for a first-person solution trace:

[Question]
{question}

[Complete answer from the closed model]
{answer}

Write the reasoning process that I went through
    when solving this problem, in the first-
person and in the order in which the solution
    unfolds:

1. how I understood the problem, identified the
    constraints, and noticed likely traps;
2. which approaches I considered, why I chose
    one, and why I abandoned another;
3. how the key derivation proceeds, including
    the necessary mathematical expressions
    and the evidence for each step;
4. how I verified the key steps (substitution,
    simplification, boundary checks, or
    counterexamples);
  1. how I closed the solution.

Requirements:

  • give concrete reasoning, not filler such as “I analyzed it” or “I checked it”;

  • do not restate sentences from the complete answer;

  • do not mention teacher/student/classroom/dialogue/role/template/reconstruction/decoding, and do not describe how the answer is being organized;

  • do not reveal the final answer before the end of the reasoning;

  • output only the reasoning process, with no opening or closing remarks.

Cleaner prompt. The cleaner is asked for a strict JSON object and is told that it may remove or rewrite noise but must not invent mathematical content:

You are a training-data cleaner. The input is a CoT reconstructed from a closed model's complete answer; it may contain (1) dialogue/classroom artifacts, (2) meta-commentary about reconstruction, templates, or the task, and (3) explicit answer declarations.

Rewrite it as a self-contained first-person rationale. Remove all dialogue/classroom traces and all sentences about how the answer was produced, reconstructed, or formatted. Keep the stage headings and their order. Do not add new mathematical content. Remove explicit "final answer:" lines; the answer is appended later.

Return exactly one JSON object:
{"cleaned_cot": "...", "removed": ["dialogue", "shell", "task_meta", "answer_line"], "notes": "one sentence, may be empty"}

Question:
{question}

Trace to clean:
{cot}

Pattern Registry and Routing Examples

The router combines a dataset-name map with keyword scoring over the first rows. The registry is data, not code: adding a profile or a pattern does not require changing the pipeline. Table 6 gives representative decisions.

DatasetProfilePrimarySecondary
MATH-500, GSM8K, AIMEmathmath_solverred_blue
HumanEval, MBPP, LiveCodeBenchcodepair_programmingred_blue
GPQA, ScienceQAscienceexpert_panelred_blue
BBH, LogiQAlogicsocraticexpert_panel
PlanBench, agent tasksplanningsocraticretrospective
Open-ended writingopen_endedretrospectivesocratic
Unknowngeneralsocraticmath_solver

Table 6. Representative routing decisions. Unknown datasets fall back to keyword scoring and then to general.

Each collection record stores the selected pattern, the prompt variant, the question variant, and the attempt number. This makes a failed content_filter attempt traceable without rerunning the batch.

Job Schema

A run creates a self-contained job directory:

jobs/<stamp>/
  raw/        collected.jsonl
  decoded/    decoded_<stamp>.json/.md
  cleaned/    cleaned_cots.json
  answers/    answers.json
  sft/        processed/sft_*.jsonl
  logs/       per-stage logs
  config/     prompts, run config
  manifest.json
  summary.md

The manifest is the machine-readable index. It records the dataset and target configuration, the selected pattern and routing reason, each stage’s status, the number of collected/decoded/cleaned records, the number of kept SFT samples, the dropped-sample reasons, and the paths to every artifact. The summary is generated from the manifest and contains the same counts plus a ready-to-run training command.

Reproduction Commands

Setup.

git clone \
  https://github.com/C1ove-ai/RECAST-Reconstruction-of-Chain-of-Thought-via-Answer-Masked-Structured-Traces.git \
  recast-cot-reverse
cd recast-cot-reverse
pip install -r requirements.txt
cp config.relay.example.json config.relay.json
cp api.example.txt api.txt
cp api_clean.example.txt api_clean.txt
# edit the three files; never commit them

Smoke run.

python recast_pipeline.py \
  --dataset math500 \
  --target-model <model-id> --limit 2

The command prints the job path and writes summary.md and manifest.json. --dry-run resolves the dataset, target, pattern, and stage commands without calling an API.

Model roles. The target, decoder, and cleaner are independent OpenAI-compatible roles:

python recast_pipeline.py --dataset math500 \
  --target-config my_teacher_relay.json \
  --target-model teacher-x \
  --decoder-model decoder-y \
  --decoder-url https://decoder.example/v1 \
  --decoder-api my_decoder_key.txt \
  --cleaner-model cleaner-z \
  --cleaner-url https://cleaner.example/v1 \
  --cleaner-api my_cleaner_key.txt

Agent skill.

powershell -NoProfile -ExecutionPolicy Bypass `
  -File skills\install_skill.ps1

After installation, a new agent session can run the pipeline from a single prompt. The skill only orchestrates the local repository; it does not embed keys or model outputs.

References

  1. [1]Anthropic. 2026. Claude sonnet 5.5. https://www.anthropic.com.
  2. [2]Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman. 2021. Training verifiers to solve math word problems. arXiv preprint.arxiv.org/abs/2110.14168
  3. [3]DeepSeek-AI. 2025. DeepSeek-R1: Incentivizing reasoning capability in LLMs via reinforcement learning. arXiv preprint.DOI
  4. [4]Google DeepMind. 2026. Gemini 3. https://deepmind.google.
  5. [5]Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. 2021. Measuring mathematical problem solving with the MATH dataset. arXiv preprint.
  6. [6]Cheng-Yu Hsieh, Chun-Liang Li, Chih-Kuan Yeh, Hootan Nakhost, Yasuhisa Fujii, Alexander Ratner, Ranjay Krishna, Danqi Chen, and Tomas Pfister. 2023. Distilling step-by-step: outperforming larger language models with less training data and smaller model sizes. Findings of ACL.arxiv.org/abs/2305.02301
  7. [7]Tamera Lanham, Anna Chen, Ansh Radhakrishnan, Benoit Steiner, Carson Denison, Danny Hernandez, Dustin Li, Esin Durmus, Evan Hubinger, Jackson Kruthoff, et al. 2022. Measuring faithfulness in chain-of-thought reasoning. arXiv preprint.arxiv.org/abs/2307.13702
  8. [8]Hunter Lightman, Vineet Kosaraju, Yura Burda, Harrison Edwards, Bowen Baker, Teddy Lee, Jan Leike, John Schulman, Ilya Sutskever, and Karl Cobbe. 2023. Let’s verify step by step. arXiv preprint.arxiv.org/abs/2305.20050
  9. [9]Xiaoyu Luo, Tao Ren, Wenrui Yu, Xiao Li, Qiongxiu Li, and Johannes Bjerva. 2026. Capable yet parsimonious: Extracting and characterizing hidden chain-of-thought in frontier models. arXiv preprint.arxiv.org/abs/2609.26637
  10. [10]Niklas Muennighoff, Zitong Yang, Weijia Shi, Xiang Lisa Li, Li Fei-Fei, Hannah Hajishirzi, Luke Zettlemoyer, Percy Liang, Emmanuel Candès, and Tatsunori Hashimoto. 2025. s1: Simple test-time scaling. arXiv preprint.arxiv.org/abs/2501.19393
  11. [11]Subhabrata Mukherjee, Arindam Mitra, Ganesh Jawahar, Saket Agarwal, Harsh Pal, Yash Biswas, et al. 2023. Orca: Progressive learning from complex explanation traces of GPT-4. arXiv preprint.arxiv.org/abs/2306.02707
  12. [12]OpenAI. 2026. GPT-6.1. https://openai.com.
  13. [13]Alexander Panfilov, David Schmotz, Ilia Shumailov, Luca Beurer-Kellner, Joachim Schlaeffer, Ameya Prabhu, Jonas Geiping, and Maksym Andriushchenko. 2026. Stealing reasoning traces from proprietary LLM APIs. arXiv preprint.arxiv.org/abs/2608.09867
  14. [14]Yiting Qu, Ziqing Yang, Chi Cui, Ye Leng, Junjie Chu, and Yang Zhang. 2026. EchoCoT: Extracting hidden chain-of-thought from black-box large reasoning models. arXiv preprint.DOI
  15. [15]Miles Turpin, Julian Michael, Ethan Perez, and Samuel R. Bowman. 2023. Language models don’t always say what they think: Unfaithful explanations in chain-of-thought prompting. Advances in Neural Information Processing Systems.arxiv.org/abs/2305.04388
  16. [16]Qi Wang, Kang Xu, Chengcheng Wan, and Jiangtao Wang. 2026. RECAST: REConstruction of Chain-of-Thought via Auditable Structured Traces. github.com/C1ove-ai/RECAST. Open-source artifact.arxiv.org/abs/2610.41813
  17. [17]Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc Le, Ed Chi, Sharan Narang, Aakanksha Chowdhery, and Denny Zhou. 2023a. Self-consistency improves chain of thought reasoning in language models. International Conference on Learning Representations.arxiv.org/abs/2203.11171
  18. [18]Yizhong Wang, Yeganeh Kordi, Swaroop Mishra, Alisa Liu, Noah A. Smith, Daniel Khashabi, and Hannaneh Hajishirzi. 2023b. Self-instruct: Aligning language models with self-generated instructions. Proceedings of ACL.arxiv.org/abs/2212.10560
  19. [19]Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia, Ed Chi, Quoc Le, and Denny Zhou. 2022. Chain-of-thought prompting elicits reasoning in large language models. Advances in Neural Information Processing Systems.arxiv.org/abs/2201.11903
  20. [20]Yixin Ye, Binyuan Hui, Mian Yang, Binhua Li, Fei Huang, et al. 2025. LIMO: Less is more for reasoning. arXiv preprint.arxiv.org/abs/2502.03387
  21. [21]Eric Zelikman, Yuhuai Wu, Jesse Mu, and Noah D. Goodman. 2022. STaR: Bootstrapping reasoning with reasoning. Advances in Neural Information Processing Systems.arxiv.org/abs/2203.14465

Paper details

vlnc:W_dwu3xyw9
Version
v1 · latest
Published
Fresh — added to the Hub within the last 30 days
Added
License
CC BY 4.0
Usage

views · 0

downloads · 0

A view counts once per browser per day, a download each time the PDF is requested.

  • Computer Science

Not recorded for this paper

View statement
Looking for related papers…
Contents