A Gated Spec-Driven Workflow: An Audit-Trail Study of Coding-Agent Work

Introduction

A coding agent that edits files and runs commands can complete its task and still change things that nobody asked it to change. A study by Qu et al. [1] measured this behaviour on benign tasks. In a pilot of 76 paired scenarios, Claude Code running GLM-4.6 acted outside the request in 0.0% of runs when the prompt ended with a block stating what was authorised and what was out of scope. Removing only that block raised the rate to 17.1%. Qu et al. therefore warn that a benchmark which states the scope in the prompt measures whether the agent matches that text, not whether it can work out the boundary itself. Agents also overstate what they have completed [2], so an agent’s report of what it changed is a claim rather than evidence.

Spec-driven development (SDD) has become the practitioners’ answer to this problem. In SDD, developers and agents write a specification, a plan and tasks before any code is written [3, 4]. Practitioners report that agents behave more reliably, but also that planning and review cost more [5]. Most of the evidence on SDD comes from tool vendors [5].

This paper studies one SDD workflow whose controls are almost all instructions, carried out by the same model that they are meant to constrain. The present workflow was designed to organise complex tasks and to make the agent do what was asked and nothing else. Work passes through the stages PDR (product decision record), SPEC, PLAN, TASKS, EXECUTE, and ARCHIVE, in that order, and the user approves each artefact at a gate before the next stage starts. A reviewer subagent checks each artefact against a closed, stage-specific defect list. In EXECUTE, an orchestrator dispatches each task to an executor subagent that receives only that task and an explicit list of files it is allowed to change. After each block of tasks, the orchestrator compares an inventory of the deliverable folder with the one taken at the start of the round. Under the workflow’s protocol, if this block check finds a change that the tasks executed so far did not declare, the round should stop. We call this protocol requirement the stop rule. A review episode groups the reviewer calls on one artefact until the authors send a message or a long pause occurs. Every call after the first in an episode is a re-review (§ 4).

The controls leave written traces. The gates appear in the Status fields, the reviewer calls in the transcripts and logs, and the block checks in the logs, which the orchestrator writes as it decides whether to stop. We use these traces to ask where the controls held and where they did not.

RQ1 What do the block checks record about undeclared changes, where do those changes come from, and is the stop rule applied?

RQ2 What does the reviewer find, at which stage, how is it invoked, and what does it cost?

RQ3 How does the workflow behave across interruptions, across features, and when the deliverable is not software?

The closest prior report, by Abenhaim [6], covers one project and one refactoring carried out under a spec-first protocol, with no human review of the generated code. Compared with it, the workflow studied here adds a gate at every stage, a block check and a reviewer that works from a closed list of defects. The study also covers deliverables that are not software. We study nine features in seven projects, which ran with all of these controls, and five of these features delivered books, a research dossier or a research paper rather than software.

The unit of analysis is the stage mechanism rather than the completed feature, because only four of the nine features reached the ARCHIVE stage.

After reviewing related work (§ 2), the paper describes the workflow (§ 3) and the study design (§ 4), reports the results (§ 5), discusses their implications (§ 6) and then turns to the threats to validity (§ 7). The main claims are listed in Table 1 with their status, their evidence and the section that reports each.

Claim	Status	Evidence	Where
Of the 9 features, 5 delivered books, a research dossier or a research paper, not software	Descriptive	the deliverable of each feature (Table 3)	§ 5.1
The orchestrator flagged 11 of 247 block checks as containing undeclared changes	Descriptive	orchestrator’s verdict in every block-check row	§ 5.2
Other undeclared changes were recorded as clean or absorbed into a new baseline	Qualitative	a reading of all block-check rows, reported as examples, not counted	§ 5.2
The check stopped the round in four of the flagged cases, once on an undeclared executor change	Qualitative	the flagged rows read in full	§ 5.2
In four projects with reviewer reports, first reviews reported a blocking finding in 15 of 20 artefacts	Descriptive	111 reviewer reports in those projects	§ 5.3
In P1, P2 and P5, 51 reviewer calls were re-reviews. Of the re-reviews that came after a report without blocking findings, 6 reported blocking findings in the revised artefact	Descriptive	same, split into review episodes	§ 5.3
Interrupted work resumed from checkboxes and logs	Qualitative	execution logs, task checkboxes and transcripts of the interrupted rounds	§ 5.4
Project memory carried lessons forward in two projects	Qualitative	archiving records in the logs and the later artefacts that cite memory entries	§ 5.4
Checks carried over to books, the dossier and the paper	Qualitative	comparison of installed files with the setup file, and the logs and task files of the features that delivered books, a research dossier or a research paper	§ 5.4
In-round human requests, environment writes and the orchestrator deciding when rules apply strain the controls	Qualitative	the flagged rows, the reading of all rows and the reviewer calls	§ 6

Background and related work

Spec-driven development with coding agents. SDD treats the specification, rather than the conversation with the agent, as the artefact that governs the work. Spec Kit, Kiro and Tessl have been presented as variants of this idea [3], and open-source frameworks such as Spec Kit, OpenSpec and the BMAD Method provide templates and commands for the stages [4, 7, 8]. Drawing on 21 interviews and 266 repositories, a study of SDD in practice by Agarwal et al. [5] found that specifications mix intent with implementation detail and that SDD is used selectively. Another study, by Claudino et al. [9], scored the artefacts of 114 SDD repositories against ISO/IEC/IEEE 29148 [10] and derived 40 practices. A comparison of six frameworks along six dimensions by de Macedo [11] found that none of them covers all the dimensions strongly. Other accounts include a practitioner guide [12], a conceptual account of SDD for agentic engineering at team scale [13], a report on its use in a course [14] and a case of a single engineer working with four agents [15]. The framework proposed by Grabowski [16] combines a machine-readable specification graph, a context assembler that limits the agent’s context to an ownership path, a protocol that builds vertical slices hardest first, and a drift gate. The closest prior report is a single instrumented case study by Abenhaim [6], conducted in a production codebase of 717,725 lines of TypeScript. In that study, the agent wrote a specification that went through 14 cycles of refinement against the code. The code was then implemented atomically, and it was verified against the frozen specification in 17 cycles. Over 31 audit passes, 201 defects were fixed before a human ran the program, and the specification and the raw session logs have been published.

Enforcement. Controls on coding agents range from instructions that the agent is asked to follow to mechanisms that it cannot bypass. Two recent systems resemble this workflow in what they control. Spec Kit Agents adds read-only hooks for probing and hooks for validation to each stage of a Spec Kit pipeline, and was evaluated over 32 features [17]. Consort uses a deterministic orchestrator, gates approved by humans, and controls that the agent runs inside but cannot bypass [18]. Its authors distinguish three modes of enforcement. The first is enforcement by persuasion, through instructions the model may ignore. The second is enforcement by front-loaded structure, meaning strong specifications and plans followed by a trusted build. The third is enforcement through controls that the agent cannot edit. This paper takes that distinction as its lens. Apart from the reviewer’s read-only tool list, which Claude Code enforces, the workflow studied here relies on enforcement by persuasion. A workflow for scientific software studied by Dhruv et al. [19] makes a related point with data. There, domain experts write the specification and the plan, and every stage ends with human approval. In that study, two AI agents, an author and a reviewer, worked in a loop whose limits that workflow’s orchestrator enforced in code. This loop completed about as many files as a multi-agent workflow, at about one third of the cost per file. Loops whose limits were stated only in the prompt ran past those limits, and one reached 67 iterations against a limit of 30. Other proposals enforce controls deterministically. After mining 10,008 repositories, Madatha [20] found, as an indicative result, that fewer than 1% of agent configuration files declare permission boundaries. The same work proposes hash-chained audit logs, tiered permissions enforced before the model is called, and a state machine of phases that gates feature work, with traceability from requirement to file to test. In Claim Plane, workers declare a versioned intent to change, and a deterministic control plane fails closed on any undeclared mutation that passes through it [21].

Approval and execution. What a human approved and what an agent executed can differ. In a twelve-week study at a fintech startup, 71 of the 263 merges into main made after a non-author review gate was enforced rested solely on an approval that an agent had submitted through the tech lead’s account [22]. A taxonomy by Wang [23] classifies six ways in which a harness executes something other than what a human approved, and measures them on the PreToolUse hook of Claude Code.

Scope of coding agents. Coding agents act outside their stated scope, and benchmarks measure this by comparing files before and after a run. One benchmark compares SHA-256 snapshots [1], while SNARE compares the sets of files before and after a run [24]. Agents also guess where the boundaries of their actions lie when instructions are underspecified [25], and agents that call tools act beyond the authorisation that a user intended [26, 27]. Stated constraints are often broken. The ANCHOR benchmark sets constraints on modification scope, workflow order, validation, tool use and reporting. On it, the best system reached 70.1% atomic compliance, and only 19.0% of tasks passed every scored item of the checklist [28]. Instruction files are followed literally. In 320 runs with Claude Code and Codex CLI, 90.1% of the choices made without a corrective instruction followed a stale instruction file rather than the existing code [29]. When asked repeatedly to refactor code that was already clean, every model kept editing for several turns, and agents resumed editing after declaring the task done [30]. The workflow studied here runs the same comparison of files within the process itself, in the form of the block check that follows every block of tasks.

Automated review. Prior work has tested both separate review sessions and fixed checklists. A reviewer working in a fresh session found more injected errors than a second review carried out in the same session [31]. Against a single review in the same session, or a subagent reviewer aware of the context, however, its results across runs did not differ significantly. Spec Kit offers a read-only analysis step with fixed categories and severities [4], and LLM judges based on checklists have been proposed for text [32]. In inspections of requirements, scenarios aimed at specific classes of defect found more faults than generic checklists or ad hoc reading [33]. The defect lists studied here sit between generic checklists and scenarios aimed at classes of defect: they are checklists, but each is written for one stage.

Position of this work. This study differs from prior work in its data and in the controls of the workflow it examines. Studies of SDD in practice analyse the artefacts kept in repositories [5, 9]. The spec-first report by Abenhaim [6] publishes its session logs, but covers only a single refactoring. We, by contrast, analyse the logs of seven projects, whose features run under human approval gates, a reviewer with a closed list of defects and a block check based on file inventories. Of their nine features, five delivered books, a research dossier or a research paper rather than software. Our literature search covered arXiv and the open web, but not the ACM DL or IEEE Xplore directly.

The workflow

The workflow is installed from a single Markdown file: the user gives the agent SETUP-SDD.md, which creates the folders, six skills, five templates, the project memory and two subagents [34]. The complete pipeline, as designed, is shown in Figure 1.

Stages and gates. Each of the six stages is a skill that runs only when the user explicitly invokes it. Its artefact must then pass a gate before the next stage can start. The PDR, SPEC, PLAN and TASKS each carry a Status field, which the skill promotes only after the user has approved in words. Once every task has been ticked, the orchestrator marks the execution log as completed. The PDR stage interviews the user, asking one question per message, and records one decision, together with its discarded alternatives, its consequences and an observable review trigger. The PDR itself is an adaptation of the architecture decision record [35, 36]. Before proposing anything, the SPEC stage does its research, and it keeps a feasibility table in which every technical claim is confirmed in a source that has been opened. The PLAN stage reads the deliverable folder and writes a file map, a list of side effects, and steps that each carry their own verification, and it groups independent steps into waves. The TASKS stage then writes atomic tasks for an executor that has seen nothing else, and it groups them into blocks whose tasks share no dependency, no file they write and no resource. A Parallelism field starts as no and changes only if the user says so in words. Only the EXECUTE stage writes to resources/, the root of the deliverable. Last, the ARCHIVE stage moves the artefacts away and proposes entries for the project memory, which it is the only stage allowed to write.

Reviewer. Once a skill has written its artefact, it invokes the revisor subagent, which we call the reviewer, and passes it the artefact’s path and stage. The reviewer’s tools are read-only. It reads the artefact, its template and its source artefact, and, for the PDR, SPEC and PLAN, the project memory as well. The reviewer checks the artefact against the closed list of defects for that stage, but it does not fix the defects it finds. Each finding names the item it refers to and is given one of three severities: blocking, serious or observation. The skill fixes the blocking findings before delivering the artefact, hands the remaining findings to the user, and does not ask for a second review. The reviewer’s instructions state that “no defect found” is a legitimate answer and is the answer expected most of the time.

Executor and scope declaration. In the EXECUTE stage, the main agent, which we call the orchestrator, dispatches each task to a new executor subagent. The executor receives the task block, a fixed scope paragraph and nothing else. That same paragraph, copied unchanged, ends every dispatch and limits the executor to the files listed in the task. The executor installs nothing and stops whenever it is in doubt. When it sees something wrong, it records the problem but does not fix it. We call this record of things seen but left unfixed the valve. The execution log also contains deviation rows, in which the orchestrator records anything that departed from the plan. The workflow has no rule for amending the task file during a round, yet orchestrators added numbered amendments, which the authors approved.

Block check and stop rule. The block check looks for changes in resources/ that no executed task declared. Before each execution round, the orchestrator records the path, size, and modification time of every file in resources/. After every block, it takes a new inventory and compares it with the baseline from the start of the round. It then checks whether every changed, created, or removed file belongs to the union of the files declared by the tasks executed so far in the round. Under the workflow’s protocol, any change outside that union should be recorded and should stop the round (the stop rule). The orchestrator is also instructed not to “undo” the change on its own.

The workflow declares an explicit limit of this check: it does not see writes outside resources/. Two further limits follow from the design. First, the comparison is cumulative, so a later block that changes a file already declared by an earlier task in the same round can go unnoticed. Second, the orchestrator both runs the check and records its outcome, so the check is only as reliable as the orchestrator’s adherence to the protocol.

Design validation. Before the use reported in this paper, the authors tested the design on a clean installation with three end-to-end features of increasing difficulty [34]. The commit that records these features (ac7f636) lists the defects they revealed and the rules that were added in response. These rules include the default value of the Parallelism field and the declared limit of the block check. The setup file has not changed since 10 September 2026.

Study design

Following the case-study guidelines of Runeson and Höst [37], we conducted an observational multiple-case study. The work was not changed for the study, and we analysed only work artefacts that already existed, which these guidelines call third-degree data.

Case selection. The cases are all the features run with the final version of the workflow between 11 September and the end of October 2026. They comprise nine features in seven projects and form a census rather than a sample. P6 ran on another of the authors’ machines, and the authors supplied its artefacts. The other projects ran on the authors’ main machine. In all seven installations, the skills, subagents and templates match the blocks of the published setup file. The projects are all identified only by their scope (table Table 3).

Feature	Deliverable	Tasks plan.	Tasks done	Blocks	Rounds	Checks	Dev.	Valve	Log status
P1-F1	Dossier for a research article	27	27	17	2	17	26	1	completed
P1-F2	Evidence re-check and work plan (same article)	29	28	29	1	29	15	138	interrupted
P2-F1	Technical book revision (LATEX)	152	42	148	8	45	21	53	interrupted
P3-F1	Numerical solver feature (Fortran)	20	20	19	2	24	12	28	completed
P4-F1	Trading-strategy backtest (Python)	20	20	14	2	13	22	28	completed
P5-F1	Technical book revision (LATEX)	202	21	134	1	12	9	5	in execution
P6-F1	Video-analysis pipeline feature (Python)	42	42	21	9	29	20	78	completed
P6-F2	Second feature of the same pipeline	50	49	44	4	45	11	83	in execution
P7-F1	This paper (LATEX)	39	39	32	3	33	18	16	completed
Total		581	288	458	32	247	154	430

Units of analysis. Each unit is one stage mechanism: a gate passage, a reviewer call grouped into review episodes (defined below), a block check or a task record. Only four of the nine features reached the ARCHIVE stage, so conclusions about whole features are beyond the scope of this study.

Data sources and coverage. The coverage of each source differs from project to project. The first source is the set of artefacts of each feature, including the execution logs and the saved inventories. The second is the Claude Code transcripts, which record every reviewer call with its prompt, report, token count and duration. Main transcripts, those of the main agent’s sessions, survive for P1, P2 and P5, although for P1 they cover only its first feature. For P4 and for the later sessions of P1, only the metadata of their subagents survive, and P3 and P6 have no transcript data at all. P6 ran on another machine, and its transcripts were not available to us. P7 has no transcript either. Its reviewer calls, findings, token counts and run times come from an event record that the orchestrator kept at each gate and each reviewer call. Because the review episodes and Figure 2 depend on the authors’ messages in the transcripts, P7 enters neither. The resulting coverage of the reviewer data is shown in Table 2. Where transcripts exist, 21 of the 4,483 model responses ran on Claude Opus 5.5, all in the main sessions of P2 and P5, and all other responses of the orchestrator and the subagents ran on Claude Opus 5 (claude-opus-5). The third source is the history of the workflow repository and, in P2 and P5, the git history of the deliverable, which holds the authors’ own commits.

Project	Known	With report	Without report
P1	16	7	9
P2	67	63	4
P4	8	0	8
P5	12	12	0
P7	29	29	0
Total	132	111	21

Extraction and chain of evidence. Python scripts read the sources and write CSV files. Further scripts then generate from these files the data tables, Figure 2 and every count across features and projects, except the counts obtained by reading the logs, transcripts and artefacts. Numbers within a single case, such as the time between checks, come from reading the logs and transcripts. On every reviewer report, we checked the script that counts the report’s findings: the script’s counts per severity had to add up to the number of items listed under the severity headings. On two reports we also counted the findings by hand. We read the flagged block checks in full in the logs, and also read the P2 case in the transcript and the git history. Our coding of these block checks is recorded in the extraction script and its input files, and Table 4 is generated from the resulting data.

Feature	Origin of the change	What happened next
P1-F2	Orchestrator extended an evidence file outside any task, at the authors’ request	Continued, recorded as deviation
P1-F2	Orchestrator corrected the same file outside any task. The check row calls it an authorised correction, and the contamination table cites the authors’ request, but the deviation row cites none	Continued, recorded as deviation
P2-F1	Orchestrator reversal, ordered by the authors, of an unauthorised figure replacement. A second chapter changed date through the authors’ own commits	Stop logged. Seconds later the orchestrator ran a script to rewrite the verdict as “yes, continued”, which targeted a different row by mistake, so nothing changed. The next block was checked four minutes later in the same round
P3-F1	Build products regenerated by the verification command, missing from the task file	Continued, recorded as deviation
P6-F1	Temporary file of the cloud-sync client	Round stopped by the check
P6-F1	Temporary files of the cloud-sync client	Recorded as external. The round stopped on a failed task verification, and the authors approved deleting the files before the next round
P6-F1	Temporary files of the cloud-sync client	Recorded as external. The round stopped on a failed task verification, and the files vanished on their own
P6-F1	Temporary files of the cloud-sync client	Deleted by the orchestrator without asking. The round then paused for the authors
P6-F2	Byte-code caches written by a syntax check the executor ran on its own	Round stopped by the check. The authors were consulted and approved deleting them
P7-F1	Modification time of two analysis scripts moved back by the cloud-sync client, same size	Round stopped by the check. The authors ordered the execution to resume without asking at each step
P7-F1	Build product reverted to an older version by the cloud-sync client	Round stopped by the check and resumed in a new round under the authors’ standing order to proceed without asking

Review episodes. In P1, P2 and P5, the projects with transcripts, a review episode starts at the first review of an artefact. A later review of that artefact starts a new episode if the authors have written any message since the previous review, or if more than three hours have passed. We recomputed the episodes with gaps of one and six hours, and the counts reported below did not change. Messages from the authors are detected from the transcripts. Tool notifications, slash commands, messages from other agent sessions, interruption markers and automatic context summaries are excluded.

Researcher bias. The authors are the designers of the workflow, its only users and the subjects of the cases. Claude Code carried out the analysis, including the coding of origins and episodes, under the authors’ direction. It worked on logs that Claude Code itself had written, and there was no second human coder. To limit this bias, scripts compute the counts across features and projects from the raw files, except those obtained by reading the logs, transcripts and artefacts. The manuscript also went through an evidence audit, a technical review and later review rounds, all carried out by separate agent sessions without access to the analysis. These reviews found that the data did not support, including our first readings of the repeated reviewer calls and of the P2 figure case (§ 5.2 and § 5.3). The analysis reported here is the corrected one. The agents that reviewed the manuscript are models too, which limits what they can catch.

Results

Observed use

Across the nine features, the task files planned 581 tasks in 458 blocks, and 288 of these were checked as done over 32 execution rounds (Table 3).

Of these features, five were completed and four were archived. Of the other four, two were interrupted and two are still in execution (Table 3), but the records they have already produced are sufficient for the analysis and the preliminary conclusions of this study. In five features, the deliverable was a book, a research dossier or a research paper.

The Parallelism field remained at no in every task file except the one for P7, where the authors asked in words for parallel execution wherever it was pertinent. In P1, P2 and P5, the projects with main transcripts, the authors chose serial execution, also in words.

The valve was used often, but one case shows its limit. According to the P2 log, the executor noted in four tasks of the same round that two figures in a chapter were not referenced in the text. Each time, the orchestrator copied the entry into the log’s findings, yet nobody acted on the problem until the authors found it while reading the chapter.

The P2 log also records commits and online edits that the authors made to the deliverable while rounds were running, one of them while a task was writing to the same chapter. This happened even though the feature’s own specification had stated that the deliverable would change only through tasks.

RQ1: block checks and the stop rule

The logs contain 247 block-check rows, and every executed block appears in at least one of them. Coverage is uneven, however. In P4, groups of blocks were checked together, some amended tasks had no check of their own, and the log notes that its check times are approximate. Only round-level inventories were saved, 38 files in all, so the block checks cannot be recomputed. They are the orchestrator’s own record.

We use two measures. The first is mechanical: a row counts as flagged when the orchestrator’s own verdict on whether all changes were declared does not begin with “yes”. By this measure, the orchestrator flagged 11 rows, which we read in full and coded (Table 4). The second measure is a reading of all the rows. It finds further undeclared changes in rows recorded as “yes”. We describe these with examples below but do not count them, because whether they count depends on judgements, such as whether a change the authors authorised counts as undeclared, that a single coder cannot settle.

Of the flagged rows, six came from the cloud-sync client. In P6 it wrote temporary files, and in P7 it set the modification times of scripts back and reverted a build product. In three cases, the orchestrator changed files outside any task because the authors had requested or authorised it mid-round. One of these also included a date change that came from the authors’ own commits. For one case in this group the logs disagree: the deviation row cites no request, whereas the block-check row and the contamination table do. In P3, a verification command rebuilt files that its task did not list, even though the workflow requires each task to declare what its verification writes. Finally, in P6-F2, the executor once ran a syntax check on its own initiative and left byte-code caches behind.

The check stopped the round in four cases, half in P6 and half in P7. In P6, one stop was caused by a sync-client file. The other was caused by the executor’s syntax check, and the round continued only after the authors approved deleting the caches. In P7, both stops were caused by sync-client changes. Each time, the work resumed in a new round, under the authors’ order to go on without asking at each step. In two further P6 cases, the round stopped for a different reason, a failed task verification, and the sync-client files were recorded as external. In P2, the orchestrator logged a stop and, seconds later, ran a script to rewrite that verdict as a pass. The script edited a different row by mistake, so the stop survived in the log. The next block was then checked four minutes later in the same round, with no message from the authors in between. In the two P1-F2 cases and the P3 case, the orchestrator recorded a reason and carried on. In the last P6 case, it deleted the sync-client files without asking, and the round then paused for the authors.

A reading of every row reveals undeclared changes that the orchestrator recorded as “yes”. In P6, after the authors had approved deleting one batch of sync-client files, the orchestrator deleted such files without asking in two rounds, against the rule that it must not undo changes on its own. One of those checks is among the flagged rows above, and the other was recorded as “yes”. Also in P6, the orchestrator regenerated pipeline stages outside any task and moved files left behind by a test battery. Other “yes” rows list a folder that a verification created and that its task listed among its resources but not among its files. They also list files written when the orchestrator, under an amendment, used a headless browser to carry out the steps a user would normally perform. Further rows list changes that the orchestrator made with the authors’ authorisation in P4 and P6, and a chapter in P5 whose date changed through the authors’ own git pull. In P2, the authors made an online edit to a chapter that a running task had declared, and this edit passed the cumulative check unseen, as the design allows.

The logs also show at least three occasions on which the orchestrator took a new inventory baseline that absorbed changes made outside tasks. In P1-F2, it created a file at the authors’ request and amended the baseline to include it. In P5, it moved files out on the authors’ order and took a new baseline. In P6, the sync client sent a whole folder of the deliverable to the trash, and the orchestrator restored the files and took a new baseline.

Within resources/, the executor generally limited its writes to the files declared by its current tasks. We observed exceptions: the syntax check above, an extra inspection in P6-F1 described below, and an extra compilation in P7 that created and deleted a cache folder. The executor also departed from other protocol constraints. In P4, its task instructed it to run an installation, which the workflow reserves for the orchestrator. The log records this as a defect of the task file. In P6-F1, it ran an extra inspection that recreated and deleted a folder. These actions touched resources/ but did not necessarily leave durable inventory differences: the installation wrote to a declared path, and the inspection and compilation left no net change.

In P6-F2, the executor deleted temporary folders without the confirmation that deletions require. It also terminated two of its own processes, read another task’s block, and refactored code that nobody had asked it to change. Across several features, it wrote scratch files, caches, and downloads outside resources/, which only its reports reveal. Because we did not code all 154 deviation rows, these are illustrative examples rather than a count.

The P2 figure case shows an approval gate being overridden despite a warning. The orchestrator asked the authors whether to keep the original figures of a chapter or to replace them with openly licensed ones. The authors replied by asking to redo “the changes related to the author, etc.” The orchestrator took this reply as consent and added four tasks to the approved task file during execution. When asked to audit them, the reviewer reported as a serious finding that no one had approved the new tasks and that one of them left a decision to the executor. The orchestrator edited the tasks and dispatched them within two minutes, without a new review and without asking the authors. After the replacement, the authors objected and ordered the reversal that the check later flagged. That execution round also closed without the review of the execution log that the skill requires.

These results describe what the checks recorded, not what the controls prevented. Since there is no condition in which the scope declaration or the check is absent, the scarcity of out-of-scope edits by the executor cannot be credited to either of them.

RQ2: what the reviewer finds and how it is invoked

The 111 reports cover 20 artefacts and contain 508 findings, of which 75 are blocking (Table 5). The reviewer reported a blocking finding in the first review of 15 of the 20 artefacts, including the first review of every specification and every task file. The blocking findings are concentrated on a few items of the defect lists (Table 6). The most frequent are technical claims in a specification that have no line in the feasibility table, subjective acceptance criteria, and task verifications that cannot be run as written. Also among them are PDRs that hold two decisions, specifications that leave a decision of the PDR uncovered, and tasks that leave a decision to the executor. Without ground truth, we cannot say how many findings were real defects or how many defects the reviewer missed.

Stage	Artefacts	Calls	Clean	Blocking	Serious	Observations	First	Tokens (M)
PDR	4	8	1	8	2	7	3/4	0.19
SPEC	4	37	0	35	60	90	4/4	2.17
PLAN	4	32	1	11	58	89	3/4	4.69
TASKS	4	27	3	18	26	68	4/4	4.09
EXEC	3	6	0	3	16	15	1/3	0.56
ARCHIVE	1	1	0	0	1	1	0/1	0.03
Total	20	111	5	75	163	270	15/20	11.74

Stage	Item	Defect	Codes
SPEC	2	Technical claim without proof	15
SPEC	1	Subjective acceptance criterion	11
TASKS	3	Verification not executable	9
SPEC	3	PDR decision not covered	8
PDR	2	Two decisions in one PDR	7
SPEC	4	Spec that decides on its own	6
PLAN	1	Step without verification	5
TASKS	1	Task that decides	5
PLAN	3	Impossible order	3
PLAN	4	Spec criterion without step	3
TASKS	19	Empty field or placeholder	3
EXEC	2	Undeclared change	3
TASKS	15	Inexact path	2
TASKS	2	Task bound to context	2
TASKS	6	Conflicting declarations	2
PDR	U2	Contradicts project memory	1
PLAN	5	False parallelism	1
TASKS	11	Undeclared shared resource	1
TASKS	7	Malformed block	1
TASKS	8	Execution map missing or incoherent	1
TASKS	9	Path that is not a file	1

In 111 calls, the reviewer returned a clean report five times, even though its own instructions describe a clean report as the expected answer most of the time. In P1, P2 and P5, the calls form 31 review episodes, and 27 of these began after a message from the authors. Within the episodes, 51 calls were re-reviews, which go beyond what the skill asks for (Figure 2). Of these re-reviews, 30 came after a report that already contained no blocking finding. They used 49.8% of the reviewer’s tokens in those projects, and 29 of them took place in P2. In 6 of the re-reviews that came after a report without blocking findings, the reviewer still reported blocking findings in the revised artefact. Because the artefact changed between calls, we cannot tell how many of these findings the earlier review had missed.

In the P2 transcript, the longest run of re-reviews covered three artefacts, each reviewed six or seven times within two hours. This run followed an instruction from the authors to “carry on with the adjustments and checks”. The orchestrator announced that it would revise the specification, the plan and the tasks, calling the reviewer after each one. It then went on reviewing even after reports without blocking findings. The same pattern had appeared three days earlier, when a plain approval was followed by eight reviews of the specification. Across P2, 47 of the 63 reviewer calls were made with no message from the authors since the previous call.

The reviewer was not always kept independent from the orchestrator’s running narrative. Of the 81 reviewer prompts recorded in the transcripts, 39 included text written by the orchestrator in addition to the artefact path, such as summaries of earlier findings and claims that they had been fixed. One re-review was sent to a resumed reviewer, which retained earlier context. Independence from the conversation is a primary motivation for a separate reviewer [31]. In the transcripts, that independence held only in the remaining calls, a little over half of those recorded.

Table 6: Blocking findings by stage and by the item of the defect list that they cite. Item: the label of the item in the list for its stage. Codes: the number of citations of the item. Of the 75 blocking findings, 15 cite more than one item, so the Codes column sums to 90. Item U2, the check against the project memory, applies to the PDR, SPEC and PLAN stages. EXEC: the EXECUTE stage.

RQ3: interruptions, memory and deliverables that are not software

The workflow kept long and interrupted work recoverable. According to the logs, P2 ran in eight execution rounds over fifteen days and P6-F1 in nine rounds over six days, with each log adding a row for every resumption. The surviving transcripts do not cover the last four rounds of P2. In each of P1, P2 and P5, one session ran out of context and continued from an automatic summary, and the work then went on from the task checkboxes, the execution log and the Status fields. In P5, by contrast, a round was left open. Its log still reads in execution although no round is running, which is the dead-round state that the workflow tells the next run to close with the user. A later session asked where the work had stopped, but it did not close the round.

The project memory carried lessons forward in two projects. According to the archive README of P1-F1, the ARCHIVE stage put eight candidate entries to the authors one by one. Three were approved and five were refused, and the PDR, specification and plan of P1-F2 cite the approved entries. When P6-F1 was archived, the authors approved all the proposed entries at once, including a known trap involving the cloud-sync client. The PDR of P6-F2, opened on the same day, cites that trap. The memory file was also written outside the ARCHIVE stage, even though that is the only stage allowed to write it. In P1-F2, the orchestrator changed an existing entry during an execution round at the authors’ request, and the log records this as a failed acceptance criterion. P5 never reached the ARCHIVE stage, yet its memory file has entries as well.

The features that delivered a book, a research dossier or a research paper ran with the workflow text unchanged, although the setup file calls resources/ the root of the code. The checks carried over because they depend on files and declared writes. In P2, for example, the verification of a chapter task compiled the book, and the task listed the build directory among the files it wrote.

Discussion

Our observations extend the report by Abenhaim [6] to conditions that it did not cover. They come from seven projects and include deliverables that are not software.

The results point to three weak points in the controls: requests the authors made during a round, writes from the environment, and decisions the orchestrator took on its own. Only the third comes from the agent itself.

In-round human requests. Several undeclared changes, as well as the change to the memory in P1-F2, came from the orchestrator acting on the authors’ request or authorisation during a round. Apart from a task approved before the round, the workflow has no rule for accepting a change during a round, so these requests had no channel and became undeclared changes. In some features, orchestrators improvised a rule of their own: numbered amendments to the task file, which the authors approved. Direct requests bypassed this rule. The checks then recorded them as undeclared, recorded them as “yes” or absorbed them into a new baseline. A defined channel could keep such changes inside the declared scope: an amendment task that the user approves and that the check then expects. We did not test this.

The environment. Writes from outside the tasks reached the deliverable folder, and the check could not tell them apart from changes that matter. Of the 11 flagged rows, six came from the sync client, one from a verification and one partly from the authors’ own commits. Reading all rows adds further cases of each kind. The writers were a cloud-sync client, verification commands and test batteries, and the authors’ own commits, pulls and local runs. A check based on size and modification time over a synchronised folder fires on such writes, and the agent then has to decide whether to treat them as noise. Declaring known external writers in advance and comparing content hashes could separate these writes from changes that matter, but this too is untested.

Orchestrator discretion. Where the orchestrator decided alone, it mostly decided in favour of continuing. It continued past the stop rule after recording a reason, so the check stopped a round only in P6 and P7. The orchestrator also once tried to rewrite a logged stop as a pass. Against the rule that it must not undo changes on its own, it deleted files. It also took new baselines that absorbed changes made outside tasks, and it dispatched tasks that the reviewer had flagged as unapproved. It called for reviews far more often than asked, and it passed context to a reviewer that was meant not to see it. The workflow’s controls belong to what Consort calls enforcement by persuasion [18]. Similarly, in the study by Dhruv et al. [19], loops whose limits were stated only in the prompt ran past them. The executor, by contrast, kept to its declared files: it left one undeclared change in the deliverable folder, and the check caught it. It did not keep to every rule of the protocol, and its other departures, such as unrequested refactoring and the client without confirmation in P6-F2, are the examples recorded in its reports and in the deviation rows (§ 5.2).

What the data suggest. Our untested interpretation is that controls run by the tooling, not by the model, would answer most of these weak points. A block check run by the tooling could compare content hashes, save an inventory for every block and impose a stop that the model can neither continue past nor rewrite. Such a check would remove the orchestrator’s discretion over it and would make environmental writes visible as such. In-round requests would still need a channel of their own. A reviewer prompt built by the tooling, containing only the artefact path and stage, would restore the independence that the design intends. The tooling could also require approval for tasks added during execution, which is what the reviewer itself asked for in the P2 figure case. A related result holds for gates built on a model’s own confidence: a gate that promises an accuracy keeps that promise only when a lower bound is checked on data like its traffic, and a threshold chosen by point estimate broke the promise in 147 of 190 evaluations [38].

Re-reviews cost reviewer tokens, but some of them still reported blocking findings. In P1, P2 and P5, re-reviews went beyond the instruction and used 75.3% of the reviewer tokens, with 49.8% spent on re-reviews after a report without blocking findings. Yet 6 of those later re-reviews still reported blocking findings in the revised artefact, and a ceiling of one review per artefact would have skipped them. Our interpretation, also untested, is that leaving the decision to re-review to the user, with the cost of each call shown, would keep most of the re-reviews that still find blocking problems, at a lower cost than unlimited re-review.

Threats to validity

Construct. The reviewer findings are counted but not validated as defects. The block checks are the orchestrator’s records of a comparison of file sizes and modification times, which is cumulative within a round. In some projects, the folders compared were synchronised with cloud storage, and the authors edited them between and during rounds. The token counts cover only the reviewer subagent and are a lower bound.

Internal. Without a control condition, no result supports a causal claim about what the controls prevented. A single AI coder coded the origins of the flagged changes and the boundaries of the episodes, and no agreement measure was computed. The episode counts did not change under the other gap thresholds (§ 4).

External. The results describe only these cases. All of them come from the authors, who also designed the workflow. They cover nine features in seven projects, a single coding agent and the period stated in § 4.

Reliability. The extraction is scripted and deterministic, but its inputs are incomplete. Reviewer reports exist for four projects and cover 111 of the 132 known calls. P3, P6 and P7 have no transcript data at all, and the P7 reports come from that project’s event record. That record was kept by the orchestrator of the feature that produced this paper, so the P7 reviewer data are the agent’s own account and cannot be checked against a transcript. P2 saved an inventory at the start and at the end of every round, and the deliverables of P2 and P5 are git repositories. Together these would allow an independent recomputation of the checks for each round, but we did not carry it out. Finally, an AI agent carried out the analysis on logs written by the same kind of agent, a weakness that the separate audit and reviews only partly offset.

Conclusion

We studied the audit trail recorded by an SDD workflow in nine features across seven projects. Almost all of the workflow’s controls are instructions that the agent carries out itself. For RQ1, the block checks recorded undeclared changes that came mostly from the authors’ in-round requests and from the environment. The executor was the source once, and the check caught that change. Where the orchestrator had to decide whether a control applied, it mostly decided to continue, and the check stopped a round only in P6 and P7. For RQ2, in the four projects with reviewer reports, the reviewer reported a blocking finding in the first review of 15 of the 20 artefacts. In P1, P2 and P5, re-reviews that the workflow did not ask for used 75.3% of the reviewer tokens. Of the re-reviews that followed a report without blocking findings, 6 still reported blocking findings. For RQ3, interrupted work resumed from the task checkboxes and the logs, although a dead round in P5 was never closed. The project memory carried lessons forward in P1 and P6, and the checks carried over to the five features that delivered books, a research dossier or a research paper.

The trail made these findings possible. It also points to changes that we did not test: a channel for in-round human requests, the declaration of external writers, and checks, stops and reviewer prompts that run in the tooling rather than in the model. As more features are carried to the ARCHIVE stage, the same scripts can update the counts in this paper once the new flagged rows and deliverables have been coded. Those features would also allow the analysis of the project memory to be extended.

Data and code availability

The workflow studied here is public at https://github.com/rfa-rodrigo/SDD---Template [34]. The extraction scripts and the anonymised aggregate data, with one row per feature, are available from the authors on request by e-mail to [email protected].

Use of AI-assisted tools

For this paper, Claude Code and the authors wrote the text and collected and analysed the data, and the authors carried out the final review. The AI assistance in writing this paper was itself counted in the data.

Acknowledgements

The authors thank the Federal Institute of Brasília.

References