betaFeedback

WorksComputer Science

Annotrace: A Workbench for Step-Level Annotation of Agent Traces

Abstract

Every study of why LLM agents fail begins by labelling traces by hand, and every such study builds its own throwaway tool. Observability platforms record agent traces but offer no structured labelling; annotation tools label text but have no notion of a step, a delegation or a tool call, leaving it unclear what the smallest unit of execution to label should be. Annotrace reads six trace formats, including OpenTelemetry GenAI, LangSmith and Langfuse, and turns each run into an ordered, nested list of steps. The annotator writes their own labelling codebook; a frontier model proposes a starting set from a sample of their own traces, answers first, and is asked exactly the questions the human is asked, making annotation a joint human-AI process. Annotrace measures agreement between any two annotators, human or AI. On a multi-agent question-answering corpus it reproduces a previous manual effort's 235 annotation units exactly, and two models labelling the same traces agree 91% of the time at κ=0.82κ= 0.82.

Introduction and Problem

Understanding the behavior of LLM agents requires looking beyond whether a task ultimately succeeds or fails to

what happens during execution. Agent trajectories expose the intermediate decisions, tool interactions, and delegations through which an outcome is produced. A successful run may still contain incorrect or redundant steps, while a failure may originate much earlier than the final output (Figure 1). Consequently, recent work on agent diagnosis increasingly relies on manually inspecting and annotating execution traces: MAST derives failure modes from annotated trajectories [3]; Who&When annotates failure logs for agent and step-level attribution [9]; AgentError Taxonomy similarly studies where agents fail [10]; and AgentRx checks constraints against execution trajectories [1]. Yet such annotation is typically tied to a particular analysis objective and trace format, limiting reuse across studies.

One multi-agent run on HotpotQA showing the outcome and annotated execution steps

Figure 1. One multi-agent run on HotpotQA ([8]). It answers correctly, so a trace-level scheme calls it a success. Its steps record eight wasted search cycles and a verification the worker’s own reasoning contradicts. Indentation is delegation depth; the decisive fault sits inside a recovered cycle (Figure 2).

Existing tools address only parts of this problem. Observability platforms such as LangSmith, Langfuse, and Arize Phoenix record and display agent traces, with OpenTelemetry providing emerging conventions for their representation [6], but they are not designed for structured human annotation. Conversely, general-purpose annotation tools such as Label Studio [7], doccano [5], and INCEpTION [4] provide annotation workflows over documents or text spans. Agent trajectories are structurally different: they contain heterogeneous and potentially nested execution units, including decisions, delegations, tool calls, and responses. Moreover, different studies ask different questions of these steps and often construct their own annotation schemes and procedures. Trajectory annotation therefore requires not only preserving execution structure, but also flexibly defining what to annotate and supporting the annotator throughout this process.

We present Annotrace, a workbench for structured, step-level annotation of agent trajectories. Annotrace (1) converts heterogeneous trace formats into a unified nested representation, making individual execution steps directly annotatable; (2) allows users to define what to annotate through editable questions and labels rather than a fixed taxonomy; and (3) integrates LLMs into the annotation workflow to propose schemas and pre-annotate traces using the same questions presented to human annotators. Together, Annotrace provides an end-to-end workflow from raw agent traces to structured step-level annotations, while keeping the annotation

Copyright © 2027, Association for the Advancement of Artificial Intelligence (www.aaai.org). All rights reserved. objective and final labels under the user’s control.

Technical Details and Architecture

Adapters. An adapter turns one format into Step objects carrying named fields, an actor, a kind and a nesting depth (Table 1). Detection reads file content — each adapter looks for a marker no other format uses, such as run_type with dotted_order for LangSmith — so the user need not know what they were given. Granularity differs by format and is exposed rather than chosen. Where a worker’s reasoning is one text field of interleaved think, search, information and answer tags, the adapter recovers Thinking–Action–Response cycles (Figure 2). A step containing finer steps is left unselected, so nothing is labelled twice.

Annotrace workflow

Figure 2. Annotrace workflow. Top: adapters convert six trace formats into nested steps. Middle: a model proposes an annotation schema and answers the same questions as the human annotator. Bottom: adapters can recover and separately label Thinking–Action–Response cycles embedded in a single text field.

FormatStep kindsNested
OpenTelemetryinvoke_agent, chat, execute_toolyes
LangSmithchain, llm, tool, retrieveryes
Langfuseagent, generation, tool, spanyes
Chat messagesmessage, tool call, tool resultno
Who&Whenmessageno
Coordinatordecision, reply, cycle, outputyes

Table 1. The six adapters. Kinds are each format’s own vocabulary; a question can be restricted to any subset of them.

Schema as questions. A schema is a list of questions, each with a prompt, the labels the user typed, the step kinds it applies to, and optionally a condition on an earlier answer: ask “what went wrong?” only when “severity” is “Critical”. Conditions are authored in the interface, not a file. No taxonomy ships; the three included schemas are examples a user may delete.

The model in the loop. A model is used twice, both times asked the questions the human is asked, generated from the same object. It reads ten randomly sampled runs and proposes a starting schema (at most three questions, four labels each, with house rules), then answers a trace. Every answer records whether a person or the model produced it; a suggestion never overwrites a human decision, and editing one makes it the annotator’s. Any of 46 models may be chosen, so a cheap model can draft and an expensive one check.

Agreement. Two annotation sets are compared question by question, reporting raw agreement. Cohen’s κ\kappa and the steps they disagree on; either side may be the open session. κ\kappa is reported because raw agreement flatters an uneven schema.

Innovation and Significance

The schema comes from the user’s data. Existing tools ask for a label set before the annotator has read anything. Annotrace shows the traces first, then proposes labels from a sample of them. Given the one-line goal “which step caused the run to give a wrong answer”, a model proposed Was this step correct? Yes / No / Irrelevant with rules including “pick No only if you can see a clear mistake”. The proposal lands in editable fields and is never applied unasked.

Evidence. On a 40-run multi-agent corpus the coordinator adapter reproduced all 235 Thinking-Action-Response unit keys of a previous hand-built effort, on 40 of 40 runs, so earlier work still aligns. Two models labelling the same six runs agreed on 91% of 22 answers at κ=0.82\kappa= 0.82. A schema proposal takes about four seconds, annotating a run three to five, at a fraction of a cent per call. Renaming a label keeps and marks the answers that used the old name rather than discarding them. Figure 3 shows the annotation screen: a visitor loads a run, types one line saying what they are looking for, takes the proposed labels, and disagrees with the model on a step, in under two minutes.

The annotation screen

Figure 3. The annotation screen. The sidebar holds the run’s question and the agent’s answer, so both stay in view while the steps scroll. Colour marks the kind of step until a label is chosen, then marks the label. Steps still wanting an answer say so.

References

  1. [1]Barke, S.; Goyal, A.; Khare, A.; Singh, A.; Nath, S.; and Bansal, C. 2026. AgentRx: Diagnosing AI Agent Failures from Execution Trajectories. arXiv:2602.02475.arxiv.org/abs/2602.02475
  2. [3]Cemri, M.; Pan, M. Z.; Yang, S.; Agrawal, L. A.; Chopra, B.; Tiwari, R.; Keutzer, K.; Parameswaran, A.; Klein, D.; Ramchandran, K.; Zaharia, M.; Gonzalez, J. E.; and Stoica, I. 2025. Why Do Multi-Agent LLM Systems Fail? In Advances in Neural Information Processing Systems, Datasets and Benchmarks Track.arxiv.org/abs/2503.13657
  3. [4]Klie, J.-C.; Bugert, M.; Boullosa, B.; de Castilho, R. E.; and Gurevych, I. 2018. The INCEpTION Platform: Machine-Assisted and Knowledge-Oriented Interactive Annotation. In Proceedings of the 27th International Conference on Computational Linguistics: System Demonstrations, 5–9.
  4. [5]Nakayama, H.; Kubo, T.; Kamura, J.; Taniguchi, Y.; and Liang, X. 2018. doccano: Text Annotation Tool for Human. https://github.com/doccano/doccano. Accessed: 2026-09-18.
  5. [6]OpenTelemetry Authors. 2025. Semantic Conventions for Generative AI Systems. https://opentelemetry.io/docs/specs/semconv/gen-ai/. Accessed: 2026-09-18.
  6. [7]Tkachenko, M.; Malyuk, M.; Holmanyuk, A.; and Liubimov, N. 2020. Label Studio: Data Labeling Software. https://github.com/HumanSignal/label-studio. Accessed: 2026-09-18.
  7. [8]Yang, Z.; Qi, P.; Zhang, S.; Bengio, Y.; Cohen, W. W.; Salakhutdinov, R.; and Manning, C. D. 2018. HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answering. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing (EMNLP), 2369–2380.arxiv.org/abs/1809.09600
  8. [9]Zhang, S.; Yin, M.; Zhang, J.; Liu, J.; Han, Z.; Zhang, J.; Li, B.; Wang, C.; Wang, H.; Chen, Y.; and Wu, Q. 2025. Which Agent Causes Task Failures and When? On Automated Failure Attribution of LLM Multi-Agent Systems. In Proceedings of the 42nd International Conference on Machine Learning (ICML).arxiv.org/abs/2505.00212
  9. [10]Zhu, K.; Liu, Z.; Li, B.; Tian, M.; Yang, Y.; Zhang, J.; Han, P.; Xie, Q.; Cui, F.; Zhang, W.; Ma, X.; Yu, X.; Ramesh, G.; Wu, J.; Liu, Z.; Lu, P.; Zou, J.; and You, J. 2025. Where LLM Agents Fail and How They Can Learn From Failures. arXiv:2509.25370.arxiv.org/abs/2509.25370

Paper details

vlnc:W_k2njd5he
Version
v1 · latest
Published
Fresh — added to the Hub within the last 30 days
Added
License
CC BY 4.0
Usage

views · 0

downloads · 0

A view counts once per browser per day, a download each time the PDF is requested.

  • Computer Science

Not recorded for this paper

View statement
Looking for related papers…
Contents