betaFeedback

WorksComputer Science

Comparing Computer Use Behavior Across Multiple Levels of Abstraction

Abstract

Interaction traces record how people and agents use computers, and comparing them can reveal usability barriers, differences in expertise, and human–agent similarity. Yet traces are recorded as low-level, interface-specific actions, making comparisons difficult across tasks, interfaces, and input modalities. Existing approaches either compare these actions directly, reduce traces to aggregate outcomes, or summarize them at higher-level goals, obscuring where behavior differs. We present MAP (Multi-level Action Parsing), a hierarchical semantic representation and annotation pipeline that maps traces into a common vocabulary across linked levels of abstraction. MAP compares traces by sequence, recurring patterns, and effort, localizing differences by type and abstraction level. A technical evaluation across three computer use datasets shows high agreement with manual annotations and broad coverage of behavior. Applying MAP shows that sighted and blind users increasingly converge at higher levels despite navigation differences, while agents omit corrective, help-seeking, and recovery behaviors present in human traces.

Figure 1. MAP transforms raw interaction traces into multi-level semantic representations (center), enabling comparison of user behavior from low-level actions to high-level tasks. Given two traces, MAP supports comparisons of similarity between users (top) based on sequence alignment, effort allocation, and recurring behavioral patterns (bottom).

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

Conference acronym ’XX, Woodstock, NY

© 2026 Copyright held by the owner/author(s). Publication rights licensed to ACM. ACM ISBN 978-1-4503-XXXX-X/20/06

https://doi.org/XXXXXXX.XXXXXXX

CCS Concepts

• Human-centered computing → HCI theory, concepts and models.

ACM Reference Format: Ananya Gubbi Mohanbabu, Rosiana Natalie, Anhong Guo, and Amy Pavel. 2026. Comparing Computer Use Behavior Across Multiple Levels of Abstraction. In Proceedings of Make sure to enter the correct conference title from your rights confirmation email (Conference acronym ’XX). ACM, New York, NY, USA, 21 pages. https://doi.org/XXXXXXX.XXXXXXX

Introduction

Understanding how people use computers has long been a core goal of HCI [25]. A rich source of evidence is the interaction trace: a logged sequence of computer use actions (e.g., clicks and keypresses) together with their context (e.g., applications and interface elements) [25, 67]. Researchers use such traces to understand how an individual performed a task and to compare how people use computers — e.g., to reveal usability issues by identifying deviations from an ideal trace [49, 60], characterize interaction patterns of experts and novices [21], and surface web browsing strategies [13, 55].

Prior work makes such comparisons by holding the task, interface, and input modality constant to compare low-level actions directly [49, 55, 60], or by reducing the trace to aggregate measures such as completion time, action count, or success [5, 21, 22]. But, such work does not yet compare computer use behaviors that cut across tasks, interfaces, and input modalities – e.g. to compare across tasks to surface usability barriers, across modalities to locate accessibility gaps, or across interfaces to characterize general computer use expertise. Recent datasets of interaction traces record this kind of variation [17, 22, 44, 64], creating the opportunity to make such comparisons at scale.

But, it remains challenging to compare traces as they are recorded in a low-level, input- and interface-specific vocabulary. For example, a screen reader user may press Tab six times while a sighted user points to reach the same button, but their traces differ. Recent work thus describes traces at a higher level of abstraction by using LLMs to summarize low-level actions into goals (e.g., search for a product, fill in the shipping form) [59, 66], then aligns those descriptions to report where two traces cover the same steps [59]. However, as each trace is labeled independently the same goal may be described in different ways, such that it remains challenging to count or match goals across a corpus of traces (e.g., to identify where a group of users got stuck). Further, this level of abstraction is useful for comparing higher-level workflows but obscures differences we care about when comparing how people use computers: people can reach the same goal by different methods (e.g., search vs. faceted browsing), or follow the same method with different costs (e.g., navigating facets linearly vs. by heading). To meaningfully compare computer use behavior across heterogeneous interaction traces thus requires a representation with a shared vocabulary to aggregate labels across traces and multiple linked levels of abstraction so that we can recover how and at what level interaction traces differ.

We introduce MAP (Multi-level Action Parsing), a framework for comparing computer use behavior across multiple levels of abstraction and dimensions of similarity. MAP provides a hierarchical semantic representation of interaction traces across four connected levels, from input actions as Operations (L1), to their Functions & Consequences (L2), Methods (L3), and Goals (L4). We construct these levels through the following procedure: Operations (L1) are low-level actions common to interaction traces (e.g., pointing, clicking). Functions and Consequences (L2) are derived by unifying existing standards that describe how interfaces are implemented [2, 31] (e.g., “focus ⟨search box⟩”), Methods (L3) are derived by open-coding interaction traces and consolidating equivalent labels (e.g., “edit”, “search”), and Goals (L4) are task-dependent and thus open-vocabulary as in prior work (e.g., “Edit the file named final.txt”) [59, 66]. MAP’s vocabulary instantiates this procedure for desktop GUI computer use. To assign this vocabulary to interaction traces at scale, MAP applies labels deterministically when possible (e.g., L1 operations and L2 functions) and otherwise with an LLM with a fixed vocabulary (e.g., L3 methods), such that only task-dependent fields remain open-vocabulary. MAP’s shared representation thus makes it possible to identify where traces diverge.

To capture different ways in which behavior can be similar, MAP applies three complementary metrics: sequence similarity, which measures alignment in the order of interactions; pattern similarity, which captures recurring local interaction patterns; and effort similarity, which captures how actions and time are allocated across a workflow. Each metric can be computed at each abstraction level, enabling MAP to localize both the type and level of behavioral differences between interaction traces.

We validate MAP via a technical evaluation and an examination of its capacity to answer questions posed in prior research. Our technical evaluation compares MAP’s annotations against manual annotations of 2,342 actions across three computer use datasets spanning human and agent traces and sighted and screen reader interactions (A11y-CUA [22], AgentSynth [63], and AgentNet [65]). MAP agrees with manual annotation on 77.3–87.3% of actions per semantic field, and 99.2% of actions across the full corpus receive a defined label rather than unknown, indicating that the vocabularies cover observed behavior. We further characterize where the pipeline disagrees with annotators: most L3 errors reflect segmentation differences, and L2 errors rarely propagate to L3.

We use MAP to replicate and extend prior findings on how sighted users, blind users, and computer use agents complete the same tasks [10, 22, 46, 59]. Consistent with prior work, blind and sighted users differ sharply in low-level interactions, and agents differ from their corresponding human groups in how they execute tasks. MAP further localizes where these differences occur: the additional effort for blind users is concentrated in navigation rather than selection or typing, while the two groups largely converge on the same higher-level Methods. Agents likewise become more similar to humans at the Method level, but reproduce fewer of the corrective, help-seeking, and recovery behaviors found in human traces. We then discuss opportunities to apply MAP’s framework in the future to surface efficient strategies from expert traces to inform novice users on how to improve, to localize usability problems on interfaces across tasks and users, and to share a user’s computer use behavior without exposing their identity. We also discuss limitations of MAP and potential opportunities to generalize MAP beyond desktop GUIs in the future. Collectively, MAP enables researchers to compare computer use behavior across tasks, interfaces, and input modalities by localizing similarities and differences at multiple abstraction levels.

In summary, we contribute:

  • MAP, a hierarchical semantic representation of interaction traces that labels recorded actions across four levels of abstraction and an automatic pipeline that assigns these labels to interaction traces.

  • A decomposition of behavioral similarity into three dimensions — sequence, recurring patterns, and effort allocation — computed with standard measures over MAP labels at any level, so that behavioral differences can be localized by type and level of abstraction.

  • A technical evaluation and comparative analysis using MAP — replicating and extending prior work comparing blind and sighted computer users, and assessing differences between human and agent traces.

Related Work

Interaction traces capture how humans and agents carry out computer tasks, from individual inputs to higher-level goals. Prior work has recorded and represented traces in machine-readable formats, abstracted interaction traces into meaningful higher-level units, and compared executions with interaction traces. We build on these directions to create MAP, a shared semantic hierarchy that captures intermediate functional levels between inputs and goals, enabling consistent analysis across traces.

Recording Interaction Traces

An interaction trace records a sequence of low-level actions a person or agent performs while completing a computer task, often together with the interface state in which each action occurred. Such traces are used to study computer use behavior [22] and to train and evaluate computer use agents [17, 44, 58, 64]. Prior work has progressively enriched these records with information about the context of each action [4, 15, 16]. Early systems paired actions with UI states [16], while later datasets added screenshots and structured interface hierarchies [15]. More recent work standardizes which fields a UI log should record, including actions, UI elements and states, applications, and tasks, so logs from different collection tools can be combined for analysis and automation [4]. This standardizes what events are recorded, but not how different events that accomplish the same behavior are represented. For example, clicking a Save button and pressing Ctrl+S can be recorded using the same fields, but the data model does not itself represent that both actions accomplish save. MAP addresses this second problem by mapping different recorded actions to shared semantic labels based on what they accomplish.

OneTrace similarly brings interaction histories from different applications and users into a common trace structure so that they can be viewed and used together [8]. Users then manually group these low-level interaction traces they consider relevant to the same activity. MAP differs by automatically mapping heterogeneous actions to shared semantic labels based on what they accomplish and the Methods they contribute to. Thus, mouse pointing and repeated keyboard navigation can remain distinct low-level actions while being recognized as equivalent at a higher semantic level.

Interaction traces support learning, behavioral analysis, and agent evaluation and debugging [17, 18, 22, 24, 36, 44, 56, 58, 64]. Learned representations of screens and UI components support retrieval and action prediction [24, 36]; computer use datasets use traces to train agents, evaluate task completion, and study behavior [17, 22, 44, 58, 64]; and tools use traces to inspect agent behavior or identify failures [18, 56]. These analyses generally operate on recorded actions, interface states, or task outcomes, making direct comparison difficult when different inputs accomplish equivalent behavior. MAP preserves those low-level differences while additionally mapping them into shared Functions, Consequences, Methods, and Goals, allowing traces to be compared at the level where their behavior becomes equivalent.

Semantic Abstraction of Interaction Traces

Abstracting interaction traces makes it possible to recognize when different low-level actions accomplish the same thing and to compare behavior beyond exact input sequences. Task models provide an early precedent for representing computer use at multiple levels of abstraction. Hierarchical Task Analysis decomposes tasks into goals and subgoals [50], GOMS represents goals through methods and lower-level operators [30], and the Keystroke-Level Model represents execution through primitive operators [11]. These models are typically constructed top-down by researchers for a known task rather than inferred from recorded traces. MAP adopts this hierarchical structure but derives its labels from traces, connecting low-level Operations (L1) to what actions do and change (L2), Methods (L3), and task-level Goals (L4).

More recent work infers semantic structure directly from recorded traces. Early systems classified sequences of UI states into predefined procedure-like categories, such as searching or composing [16], corresponding to MAP’s Methods (L3). Like MAP, these categories are drawn from a fixed vocabulary. However, they represent only this procedure level rather than linking it to lower-level actions, their outcomes, and higher-level task Goals. LLM-based approaches instead infer open-ended descriptions: some extract executable multi-action units such as logging in or booking a flight [28], while others group low-level actions into intermediate subgoals and then an overall task goal [66]. Recent work similarly induces natural-language steps from human and agent traces and aligns them to compare how the same task is completed [59]. These approaches capture semantics similar to MAP’s Methods and Goals, but their labels are generated independently for each trace, so equivalent behavior may be phrased differently and must be matched afterward. They also do not represent MAP’s L2 distinction between what an action does, what it acts on, and what changes as a result. MAP instead uses shared vocabularies at L2 and L3 and links them to both the underlying Operations and task-dependent Goals, enabling equivalent behavior to be aggregated across traces while preserving how it was carried out.

Comparing User Workflows

Prior work compares low-level interaction traces directly. Clickstream visualizations reveal common page transitions [68], while sequential pattern mining identifies ordered action subsequences that recur across traces [38]. Other tools align traces around selected actions [33] or compare observed traces with supported task paths to identify inefficient behavior [19]. These approaches work well when traces share the same interface and action vocabulary, but equivalent behavior may appear different when users accomplish the same thing through different low-level actions.

Other work first transforms low-level traces into representations specific to a task or agent environment [12, 62]. Traces are compared using higher-level semantics by defining a semantic model for each benchmark website that maps GUI interactions to task-level actions and states [12]. This enables precise comparison within that environment, but the semantic actions must be defined separately for each website and do not directly provide a shared representation across different applications or interfaces. Tool-using agent traces can instead be aligned through tool calls and dependencies among their inputs, outputs, and artifacts [62]. This requires a shared tool environment rather than providing a common semantic representation of computer use behavior. MAP instead applies the same semantic representation across tasks, interfaces, and input modalities, allowing heterogeneous traces to be compared without defining environment-specific semantic actions or requiring shared tools.

MAP Taxonomy

To enable systematic comparison of computer use interaction traces across multiple users and tasks, we introduce MAP, a hierarchical labeling approach to semantically annotate traces from various domains including web browsing, system operations, media, document editing, and workflow applications. MAP represents interaction traces at four connected levels so behavior can be compared at different granularities. Operations (L1) capture the recorded input, Functions & Consequences (L2) capture what each action does and changes in the interface, Methods (L3) capture how users accomplish an immediate objective, and Goals (L4) capture the broader task outcome. The hierarchy builds on the decomposition used in GOMS [30], while adding interface-level semantics between L1 operators and L3 methods (Table 1 summarizes each level, refer to Appendix B for full taxonomy).

To support comparisons across users, tasks, and datasets, we aimed for the MAP taxonomy to provide coverage for interactions across diverse interfaces, generalize across applications and operating systems, and assign consistent labels to behaviors so recurring patterns can be aggregated within and across traces. MAP therefore annotates behavior from the recorded actions and their interface context, including the active application, interacted UI elements, and assistive technology when needed.

Operations (L1)

Operations preserve the low-level mouse or keyboard input used to interact with the interface, allowing MAP to compare how users physically carry out the same behavior before higher-level abstraction. MAP represents mouse input as scroll, primary click, double primary click, secondary click, point, and drag drop, and keyboard input as keypress or shortcut (Table 3). Because datasets record these inputs differently (e.g., mouse_press vs. mouse_click, or separate mouse_down and mouse_up events), they need to be mapped to this common L1 operation set. We also collapse consecutive point and scroll events, since their logging frequency varies across datasets and would otherwise reflect sampling differences rather than behavior. Cursor coordinates are retained, when available, as context for assigning higher-level labels.

Keyboard input requires additional normalization because the same raw key event may represent ordinary keypress or a shortcut, particularly for screen reader interaction. We therefore use a dictionary of approximately 400 known shortcuts, including screen reader commands, and their expected outcomes to deterministically distinguish shortcut from keypress operations (Table 2).

Functions & Consequences (L2)

L2 makes individual actions comparable by describing what each action does in the interface and what changes as a result. Each action is represented by its function and what the action acts on, followed by its consequence and the UI element where that change occurs (if applicable). For example, selecting a button may close a dialog box: select ⟨button⟩→\langle\text{button}\rangle\to close ⟨dialog box⟩\langle\text{dialog box}\rangle.

Functions describe how an L1 operation acts on its direct UI target. Mouse operations map deterministically to functions (Table 3): for example, primary click maps to select, while point maps to navigate. Keyboard operations depend on context because the same key can serve different functions. For example, Shift+h may type “H” in a text field or navigate to a heading when a screen reader is active. We therefore use the focused UI element and screen-reader state to deterministically distinguish typing, navigation, selection, and program commands (Table 2).

Consequences describe the observable change produced by an action and where that change occurs. In select ⟨button⟩→\langle\text{button}\rangle\to close ⟨dialog box⟩\langle\text{dialog box}\rangle, the button is acted on directly, while the dialog box is where the resulting change occurs. Separating an action from its

L#LevelDefinitionExamplesRelated Work
L1OperationsAtomic input actions executed through mouse or keyboard.primary click, scroll, keypress, shortcutKeystroke-Level Model [11]
L2Functions & ConsequencesCaptures what an operation does, which UI element it acts on, and the resulting change to the interface.select ⟨button⟩ → open ⟨dialog box⟩<br>navigate ⟨link⟩ → focus ⟨link⟩AT-SPI2 [2], UIA Control Patterns [31], WAI-ARIA, Keyboard shortcuts [1, 3]
L3Methods & UI ContainersCaptures how a sub-goal is carried out within an application. The same method may be realized through different sequences of operations and functions.edit ⟨paragraph⟩ in ⟨text document⟩<br>add ⟨chart⟩ in ⟨spreadsheet⟩GOMS [30]
L4GoalsCaptures the intended task outcome. A goal may span one or more applications.Export spreadsheet as PDF<br>Schedule a meeting[59, 66]

caption: The four semantic abstraction levels in MAP, with each level’s definition, example labels, and related work that informed its design. L1 represents individual input operations. L2 represents interaction semantics as Function ⟨Direct UI Target⟩ → Consequence ⟨Indirect UI Target⟩. L3 represents methods as Method ⟨Specific Entity⟩ in ⟨UI Container⟩, and L4 captures the goal those interactions accomplish.

:::

OperationIntentExamplesFunction
KeypressPrintable text entered in an editable UI elementa, 5, h in a text fieldType
UI navigationTab, ↓, PageDownNavigate
Activates or selects the focused UI elementEnter, Space on a buttonSelect
ShortcutPrintable text entered in an editable UI elementShift+h (for "H")Type
UI navigation, including screen-reader commandsShift+h (with screen reader ON), Ctrl+Tab, Alt+TabNavigate
Activates or selects the focused UI elementNVDA+Enter, VO+SpaceSelect
Invokes an application- or OS-level commandCtrl+S, Ctrl+C, F1-F12Program

Table 2. Mapping L1 keyboard operations to L2 functions. L1 keyboard operations are classified as keypress or shortcut based on the executed key(s) and known shortcut mappings, while their L2 function is determined from the interaction context.

L1 OperationsL1 Specific DetailsL2 Functions
Scroll(x1,y1)(x_1, y_1), (x2,y2)(x_2, y_2)Navigate
Primary click(x,y)(x, y)Select
Double primary click(x,y)(x, y)Select
Secondary click(x,y)(x, y)Select
Point(x,y)(x, y), (x2,y2)(x_2, y_2)Navigate
Drag drop(x1,y1)(x_1, y_1), (x2,y2)(x_2, y_2)Drag
ShortcutModifier + Non-ModifierType, Navigate, Select, Program
KeypressSingle KeyType, Navigate, Select

Table 3. Mapping from L1 Operations and specific details to their corresponding L2 Functions. Mouse operations map to fixed functions, while keyboard operations may map to different functions depending on the executed key or shortcut.

outcome distinguishes interactions that perform the same function on the same type of element but produce different interface changes. We derive fixed vocabularies for functions, consequences, and UI targets from existing interface and accessibility frameworks, including WAI-ARIA, AT-SPI, and UI Automation Control Patterns [1, 2, 3]. We consolidate semantically equivalent concepts across frameworks. For example, SelectionItemPattern.Select() [31], STATE_SELECTED [2], and aria-selected [2] map to select. An LLM assigns UI targets and consequences from these fixed vocabularies (Appendix B.1, B.2) using the interaction trace and preceding labels (Table 4), and it assigns Unknown when the trace provides insufficient evidence for any of the fixed labels in our vocabulary.

L#LabelInputInference
L1OperationsITRule-based
L2FunctionsITRule-based
Direct UI TargetsIT + Targets’ listLLM
ConsequencesIT + Consequences’ listLLM
Indirect UI TargetsIT + Targets’ listLLM
L3MethodsIT + Methods’ listLLM
Specific UI EntityITLLM
UI ContainersIT + UI Containers’ listLLM

Table 4. Overview of how MAP labels are inferred for actions in an interaction trace (IT). For each taxonomy level, we show the information provided as input and whether labels are assigned using deterministic rules or LLM inference from a fixed candidate label set.

L2 captures what occurred in the interface, but not why it contributed to the user’s task. For example, select ⟨check box⟩ could correspond to filtering products or completing a form. L3 captures this higher-level purpose.

Methods

L3 captures the higher-level procedure a sequence of interactions carries out, allowing traces with different low-level executions to be compared as the same method. It groups contiguous L2 interactions that contribute to one immediate objective into a Method, together with the task-relevant Specific Entity and the UI Container in which it occurs, denoted as Method ⟨Specific Entity⟩ in ⟨UI Container⟩. For example, one user may navigate through several menus to add page numbers while another uses a keyboard shortcut - their L2 sequences differ, but both are represented at L3 as add ⟨page numbers⟩ in ⟨text document⟩. L3 therefore makes the shared procedure comparable while L1 and L2 retain the differences in how it was executed and the intermediate outcomes observed respectively.

L3’s Method describes what procedure is being performed, Specific Entity identifies what it is being performed on, and UI Container captures the broader interface context. Methods and UI Containers use fixed vocabularies so equivalent procedures and contexts receive comparable labels across users, tasks, and datasets. Specific Entities remain open-vocabulary because they depend on the particular task, such as page numbers, homework.docx, or a named product. This separation keeps the fixed labels general: the same Method can apply to different task-specific entities, while UI Containers describe types of interfaces rather than particular applications. For example, add can describe adding different entities within a text document, without introducing a new Method for each entity.

We construct the Method vocabulary by open-coding a subset of interaction traces from a computer use dataset [22]. We first assigned descriptive labels to sampled interaction segments, then consolidated labels that described the same procedure and generalized labels that were specific to an application or task. This resulted in a fixed vocabulary of 14 Methods (refer to Appendix § 10.§ 3, § 10.§ 4 for L3 Methods and UI Containers vocabulary). The traces used for developing this vocabulary are not included in the evaluation set. We similarly define 16 UI Containers spanning applications, websites, documents, windows, and other content areas. For example, labels such as text document describe a type of working context rather than a particular program or file, allowing the same container label to be reused across datasets. Both vocabularies include Unknown when the interaction trace does not provide enough evidence to assign a predefined label confidently.

Goals

Goals provide the highest level of abstraction in MAP, grouping contiguous L3 Methods that jointly accomplish the same task-level objective. This corresponds to the level of semantic abstraction commonly inferred from interaction traces in prior work [29, 66]. Because Goals are task-dependent, they use an open vocabulary to accommodate diverse objectives across datasets.

MAP Similarity Metrics

The MAP taxonomy provides a common semantic representation of interaction traces, but comparing users also requires specifying what aspect of their behavior is similar. We capture three complementary dimensions: sequence similarity measures whether two traces progress through a task in a similar order, pattern similarity measures whether they exhibit similar recurring local behaviors, and effort similarity measures whether they allocate comparable time and actions to the same parts of a workflow. After annotating two traces with MAP, we compute each metric independently at L1–L3, allowing similarity to be examined both by behavioral dimension and level of abstraction.

MAP-Sequence Similarity

MAP-Sequence similarity captures whether two traces perform interactions in a similar order, distinguishing traces that contain similar interactions but arrange them differently. Prior work has used edit-distance-based methods to compare text [35] and interaction sequences in software tutorials [32, 43]. We similarly use Needleman–Wunsch global alignment [42] to identify which interactions correspond across the two complete traces and where they differ. At each level, identical labels receive a match score of +1, while mismatches and gaps receive a penalty of −1. We define sequence similarity as the proportion of positions in the resulting alignment whose labels match exactly. The alignment additionally identifies interactions that are substituted, added, or omitted between traces.

MAP-Pattern Similarity

MAP-Pattern similarity measures whether two MAP-labeled traces exhibit similar local interactions, even when their overall task paths differ (e.g., two traces that use hotkey shortcuts vs. point-and-click). This complements sequence similarity by capturing recurring behavioral tendencies. For each of MAP levels from L1-L3, we extract n-grams of lengths 1, 2, and 3 and compute cosine similarity between n-gram frequency vectors at each length, similar to bag-of-words metrics frequently used for text comparison [23]. We then compute a weighted average that rewards longer n-grams to capture more specific behavioral styles between the two traces. This metric can reveal similarity between user interaction traces within task and across tasks.

MAP-Effort Similarity

MAP-Effort similarity measures whether two users allocate comparable interaction effort to equivalent parts of a task: two traces may be semantically identical yet differ substantially in cost – e.g., formatting a document in 5 actions and 10 seconds versus 15 actions and 40 seconds. We characterize effort by action count and time, two standard measures of interaction efficiency in HCI [11], and compute similarity independently at each MAP level ll.

When computing the similarity between traces AA and BB, let SlS_l be the union of semantic labels appearing in either trace at level ll (e.g., an operation at L1 or method ⟨UI container⟩ at L3), and let al,sAa^A_{l,s} and tl,sAt^A_{l,s} (and likewise for BB) denote the number of actions and total time associated with label ss. For each label, we compute the squared log-ratios dl,sa=ln⁡2((al,sA+1)/(al,sB+1))d^a_{l,s} = \ln^2((a^A_{l,s}+1)/(a^B_{l,s}+1)) and dl,st=ln⁡2((tl,sA+1)/(tl,sB+1))d^t_{l,s} = \ln^2((t^A_{l,s}+1)/(t^B_{l,s}+1)), which capture proportional rather than absolute differences [52] and are symmetric in AA and BB. We combine the action and time differences for each label as effortl,s=(dl,sa+1)(dl,st+1)\mathrm{effort}_{l,s} = (d^{a}_{l,s} + 1)(d^{t}_{l,s} + 1), and sum across labels to obtain effortDiffl(A,B)=∑s∈Sleffortl,s\mathrm{effortDiff}_{l}(A, B) = \sum_{s \in S_l} \mathrm{effort}_{l,s}. We then map this dissimilarity to effortSiml(A,B)=100e−effortDiffl(A,B)/100\mathrm{effortSim}_{l}(A, B) = 100 e^{-\mathrm{effortDiff}_{l}(A,B)/100}. A difference in actions, time, or both lowers the score, and computing it at each level distinguishes traces that allocate effort differently across semantic labels.

MAP Pipeline

The MAP pipeline builds the hierarchy from low-level Operations (L1) to higher-level Goals (L4). It assigns labels deterministically when they can be derived directly from recorded inputs, and uses an LLM to infer labels that require interaction context.

MAP Annotation

MAP annotates traces hierarchically from L1 to L4, with each level using the recorded interaction trace and labels from preceding levels. Before annotation, dataset-specific action names are mapped to the common L1 Operations defined in § 3. Both L1 Operations and L2 Functions labels are assigned deterministically. The remaining L2 fields (Direct UI Target, Consequence, and Indirect UI Target) are selected from their fixed vocabularies using the surrounding interaction context (refer to Appendix § 9 for L2, L3, L4 prompts).

Annotation uses information available in the trace, including the active application or window, interacted or focused UI element, and preceding actions and labels. Screen-reader state is included when available and needed to interpret keyboard interaction. We exclude task descriptions, task outcomes, user metadata, and screenshots, allowing the same procedure to be applied across datasets with different recording setups. This differs from approaches that additionally use screenshots for segmentation or goal inference [59].

At L3, MAP divides the trace into segments: contiguous runs of interactions that carry out the same Method on the same Specific Entity within the same UI Container. The model jointly determines these boundaries and assigns the three L3 fields. A new segment begins when the Method, Specific Entity, or UI Container changes, or when the active interface changes. The full segmentation rules and L3 prompt are provided in Appendix § 9. At L4, contiguous L3 segments that contribute to the same task-level objective are grouped under a Goal. We annotate the non-deterministic semantic labels from L2–L4 using Claude Sonnet 5. Refer to supplementary materials for all prompts and full MAP vocabulary.

Pipeline Evaluation

We evaluate how accurately MAP annotates interaction traces and how well it generalizes across users, agents, tasks, and applications.

Datasets

We evaluate MAP on three computer use datasets: A11y-CUA [22], AgentSynth [63], and AgentNet [65]. A11y-CUA contains the same 60 tasks completed by 16 participants (8 sighted and 8 blind users) and by agents under a default condition (CUA-D) or screen reader condition (CUA-SR). Its timestamps and detailed interaction context support both annotation evaluation and the similarity analyses reported later. We use a subset of A11y-CUA to construct the L3 Method vocabulary. However, these traces are disjoint from the A11y-CUA traces used in our annotation evaluation. We additionally evaluate on AgentSynth and AgentNet to test whether MAP’s labels apply beyond the dataset used during vocabulary development. AgentSynth contains synthetic agent traces, while AgentNet contains human demonstrations; together they introduce applications and interaction patterns beyond those in A11y-CUA. Because neither provides the timing information needed for time-based effort similarity, we use all three datasets to evaluate annotation accuracy and vocabulary coverage, and A11y-CUA for the cross-user and agent similarity analyses.

5.2.2 Manual Annotation. We manually annotated 30 traces containing 2,342 actions: 20 from A11y-CUA (10 SU, 5 BLVU, and 5 Agent-Default), 5 from AgentSynth, and 5 from AgentNet. A11y-CUA traces were sampled randomly within each condition. For AgentSynth and AgentNet, we selected traces from five different applications in each dataset to evaluate MAP beyond the applications represented in A11y-CUA.

For each action, we annotated L2’s Direct UI Target, Consequence, and Indirect UI Target. We also divided each trace into L3 segments and annotated each segment’s Method, Specific Entity, and UI Container. We did not manually annotate L1 Operation or L2 Function, which are assigned deterministically. For A11y-CUA, annotators inspected the action trace together with the recorded video and audio, including screen-reader output when present. AgentSynth and AgentNet do not provide corresponding videos, so annotators instead used the interaction trace together with the textual observations and reasoning recorded with the actions.

Annotators followed a codebook specifying how to apply the L2 and L3 labels, including label definitions, and L3 segmentation rules (full codebook is provided in the Appendix § 10). Two researchers independently annotated three A11y-CUA pilot traces (249 actions; 1,245 labels across the five closed-vocabulary fields). We refined the annotation instructions and examples based on disagreements until reaching Cohen’s κ=0.79\kappa= 0.79, conventionally interpreted as substantial agreement [34]. One researcher then annotated the remaining 27 traces using the finalized codebook. Annotation took approximately 25 minutes per trace, or about 12 hours in total.

We then ran the complete MAP pipeline on the same 30 traces without access to the manual annotations. We compared its L2 and L3 annotations with the manual annotations to evaluate semantic labeling and L3 segmentation.

5.2.3 Results. Accuracy. MAP achieves generally high exact-match accuracy against manual annotations (Table 5). Across the 28 traces, mean per-trace accuracy is 81.8% (σ=14.7\sigma= 14.7) for Direct UI Target, 87.3% (σ=11.0\sigma= 11.0) for Consequence, and 77.3% (σ=18.2\sigma= 18.2) for Indirect UI Target. At L3, accuracy is 79.5% (σ=28.8\sigma= 28.8) for Method and 86.7% (σ=23.2\sigma= 23.2) for UI Container.

L3 scores are particularly sensitive to segmentation: when MAP and the annotator place a segment boundary differently, the Method or UI Container assigned to several neighboring actions can all count as mismatches even when they identify the same broader procedure. Across 136 inspected L3 errors, 38.2% are boundary shifts, 36.8% merge adjacent manual segments, and 18.4% split one manual segment; only 6.6% have matching boundaries but a different Method label. Thus, 93.4% of the inspected L3 errors concern how interactions are grouped rather than which Method describes an identically bounded segment. For example, a sequence that opens one browser, dismisses it, switches browsers, and then searches can

DatasetGroupTracesActions /TraceFunction*Direct UI TargetConsequenceIndirect UI TargetMethodUI Container
A11y-CUASU10 (534)53.4 ± 29.4100.079.7 ± 15.290.0 ± 6.773.5 ± 15.982.6 ± 22.493.0 ± 7.5
BLVU5 (324)64.8 ± 20.1100.070.2 ± 16.686.9 ± 12.179.4 ± 11.681.6 ± 29.398.7 ± 2.0
CUA-D5 (308)61.6 ± 63.3100.086.2 ± 4.879.2 ± 14.982.8 ± 7.493.0 ± 10.179.6 ± 29.4
AgentSynth—5 (978)195.6 ± 95.2100.091.0 ± 7.988.4 ± 15.685.0 ± 22.463.4 ± 39.880.3 ± 24.2
AgentNet—5 (198)39.6 ± 31.1100.084.2 ± 19.489.0 ± 7.269.6 ± 30.373.9 ± 41.775.4 ± 42.2
Average78.1 ± 72.0100.081.8 ± 14.787.3 ± 11.077.3 ± 18.279.5 ± 28.886.7 ± 23.2

caption: Agreement between MAP and manual annotations across 30 traces and 2,342 actions. Agreement values report mean percentage agreement across traces (± SD). Traces reports the number of traces, with total actions in parentheses, and Actions / Trace reports mean actions per trace (± SD). *Function, like Operation (L1), is assigned deterministically and is therefore 100% by construction. A11y-CUA groups complete the same tasks, whereas AgentSynth and AgentNet contain traces from different tasks and applications and are included to evaluate generalization.

:::

reasonably be divided into several procedures, while MAP groups the sequence into one search segment. These cases reflect differences in segmentation granularity more often than confusion between unrelated Methods.

L2 errors also do not usually determine the L3 result. L3 errors occur at nearly the same rate when the L2 tuple is incorrect (36.2%) as when it is correct (37.3%): 393 actions have an incorrect L2 annotation but a correct L3 annotation, while 642 have a correct L2 annotation but an incorrect L3 annotation. This indicates that L3 annotation is driven primarily by the surrounding sequence and where procedure boundaries are placed, rather than by any single L2 label.

Coverage. We separately examine whether the taxonomy contains labels applicable to the behaviors observed in each dataset, independent of whether MAP selects the correct label. Across A11y-CUA’s 540 traces and 60,974 actions, 99.2% of closed-vocabulary fields can be assigned a predefined label rather than Unknown, and 118 of the 126 taxonomy labels appear at least once (93.7%). The same predefined vocabularies cover 99.9% of fields in AgentSynth and 99.8% in AgentNet, indicating that behaviors in these datasets rarely require labels outside the taxonomy. AgentSynth and AgentNet instantiate smaller subsets of the vocabulary - 70/126 (55.6%) and 71/126 (56.3%), respectively - because they contain a narrower range of observed interaction behaviors than A11y-CUA. Across the three datasets, 120 of the 126 taxonomy labels appear at least once (95.2%).

Results

We demonstrate MAP by investigating two research questions from prior work. RQ1: How do sighted and blind users differ in how they complete tasks with computers? [9, 22, 46, 51, 53] Comparing these groups can characterize accessibility gaps in computer interaction and identify where interfaces create additional effort for blind users. RQ2: How closely do computer use agent workflows align with human workflows? [22, 59] This question is increasingly relevant as agents are used to simulate users for applications such as usability testing and design evaluation [26, 45]. Comparing agent and human traces can therefore show which aspects of human behavior agents reproduce and which they omit.

We investigate these questions using A11y-CUA [22], analyzing 30 tasks performed by 16 users (SU1–SU8, BU1–BU8) and two agents (CUA-D and CUA-SR). We compare within- and across-group behavior using MAP-Sequence, Pattern, and Effort similarity at L1–L4. We report pairwise similarity matrices and cluster participants and agents using K-means over the three similarity metrics at each level. For each clustering, we report the Silhouette score, where higher values indicate that traces are more similar within clusters and more distinct across clusters, to quantify how strongly each abstraction level separates behavioral groups.

How do blind and sighted users differ in how they complete tasks with computers?

Operations (L1): Blind and sighted users differ in how they interact with the interface. Prior work has shown that SUs and BLVUs differ in their input methods: sighted users primarily use mouse actions, whereas blind screen reader users use keyboard actions [9, 22]. MAP confirms this at L1. The two groups are separated across all three similarity metrics, especially for Effort and Pattern (Silhouette: Sequence=0.40, Effort=0.70, Pattern=0.77). SUs use point and primary click for 32.9% and 25.1% of their actions, respectively, while BLVUs interact entirely through keypresses and shortcuts. However, SUs are not only mouse users: keypresses still account for 28.0% of their actions, showing that they regularly combine mouse and keyboard input.

The similarity metrics further show that BLVUs use more similar recurring action patterns, but vary more in their overall action sequences. Their most common bigrams are keypress→keypress (70.9%) and keypress→shortcut (16.3%). SU traces, in contrast, reflect their use of both input methods: their most common bigrams include keypress→keypress (24.7%) and point→primary click (22.4%). BLVUs therefore have higher within-group Pattern similarity than SUs (93.8 vs. 86.2), but lower Sequence similarity (46.6 vs. 52.9). Overall, L1 mainly captures differences in how the two groups interact with the interface and at L2, we examine what those actions are doing.

Functions & Consequences (L2): Accessibility differences are concentrated in navigation. Prior observational studies show that screen reader users frequently use sequential and exploratory navigation to move through and find content, often using scanning and multiple keyboard navigation strategies [9, 10, 51]. MAP confirms this difference at L2 and localizes where it occurs. navigate accounts for 70.8% of BLVU actions, compared with 39.7% of SU actions. The difference is even clearer in Function→Consequence pairs: navigate→focus, where navigation moves focus to another interface element, accounts for 47.8% of BLVU actions but only 1.1% of SU actions. navigate→none, where a navigation command probes the interface but produces no recorded interface consequence, accounts for another 14.6% of BLVU actions and is nearly absent for SUs. The difference in effort is also concentrated in navigation: BLVUs use 9.72 actions on average per navigation instance compared with 1.30 for SUs, while differences for select (1.22 vs. 1.15) and type (5.80 vs. 6.26) are much smaller.

RQLevelPrior work findingConfirmed by MAPNew finding from MAP
RQ1L1SUs primarily use mouse input, whereas BLVUs use keyboard input [9, 22].YesBLVUs repeat similar keyboard-action patterns more consistently, but vary more in the overall order of their actions across tasks.
RQ1L2BLVUs frequently use sequential and exploratory navigation and use multiple navigation strategies [9, 10, 51].YesBLVUs require substantially more effort for navigation, while typing and selection are similar across groups.
RQ1L3BLVUs and SUs differ in how they formulate and reformulate search queries [46].YesBLVUs have longer search queries but SUs query more often per task. Additionally, despite these lower-level differences, SUs and BLVUs usually perform the same Methods to complete a task.
RQ1L4BLVUs adopt coping and recovery strategies when encountering accessibility barriers [9, 53].YesBLVUs more often add help-seeking goals, whereas SUs more often add verification and window-management goals.
RQ2L1Agents differ from humans in their low-level execution, and changing an agent’s access mode changes its interaction behavior [22, 59].YesMatching access mode does not guarantee human-like execution: CUA-SR uses far fewer distinct keyboard inputs than BLVUs, while CUA-D remains less similar to SUs.
RQ2L2Screen reader use increases agent actions, and agents often type long inputs where humans navigate through the interface [22].YesHumans interleave typing with navigation and screen reader management more often, whereas agents more often type uninterruptedly and then select UI targets.
RQ2L3Humans and agents can follow similar high-level task steps despite different low-level execution [59].YesCUA-D verifies more often than SUs, whereas SUs move and dismiss interface elements more often. Both agents perform Methods such as open and edit at similar rates to their corresponding human groups.
RQ2L4Humans and agents can pursue similar task goals despite differences in how they execute them [59].YesAgents usually reach the same primary task goal as humans but omit additional goals such as help-seeking, or closing extra windows and tabs.

Table 6. Summary of prior findings recovered by MAP and new findings revealed by its multi-level representation.

Looking across L1 and L2 using the MAP taxonomy, shows that the two groups can perform the same function using very different operations. For example, when navigating to a (list item), SUs use point in 99% of cases, whereas BLVUs use keypress (89%) or shortcut (11%). The reverse is also true: the same operation can serve different functions. Among SUs, 81.9% of keypresses are used to type, whereas among BLVUs, 75.6% are used to navigate. L2 therefore reveals differences that are hidden at L1. Although BLVUs all use only keyboard input, their within-group Effort similarity falls from 86.5 at L1 to 46.5 at L2 (see Figure 2), showing greater variation in the effort required to carry out functions such as navigation. Thus, the main difference at L2 is not simply that BLVUs use the keyboard, but that much more of their interaction is devoted to navigating and searching through the interface.

Figure 2. Pairwise Sequence, Effort, and Pattern Similarity among sighted users (SU), blind users (BU), CUA-D, and CUA-SR across MAP levels. Darker cells indicate higher similarity. Strong within-group blocks at L1 and L2 weaken at L3 and L4, showing that SU and BU differ more in low-level execution than in higher-level methods and goals. CUA-D and CUA-SR generally resemble SU and BU, respectively, at lower levels.

Methods (L3): Blind and sighted users converge on Methods, but effort differences remain. Prior work has shown that blind and sighted users can differ in how they search, including how they formulate and reformulate queries [46]. We observe similar differences: BLVU query bursts contain 17.3 characters on average compared with 9.9 for SUs, while SUs issue slightly more query bursts per task (2.5 vs. 2.2).

However, across the full set of tasks, the two groups become much more similar at L3. Across-group Sequence and Pattern similarity reach 68.8 and 72.5, close to the corresponding within-group similarities (SU: 72.0/76.8; BLVU: 67.5/71.4) (Figure 2). In 23 of 30 tasks, similarity between SUs and BLVUs increases from L1 to L3 by 42.3 points on average (σ=22.1\sigma= 22.1). Thus, actions that look very different at L1 often correspond to the same Methods at L3.

The groups can still differ substantially in how those Methods are carried out. For example, for the Method open, SU actions consist of 50% select and 40% navigate at L2, whereas BLVU actions consist of 9% select and 73% navigate. Effort also varies more among BLVUs: within-BLVU Effort similarity is 53.6, compared with 78.1 for SUs (Table 12). This shows that even when BLVUs perform similar Methods, they differ more from one another in how many lower-level actions those Methods require.

The groups do not always become more similar at L3. In 7 of 30 tasks, SU–BLVU similarity decreases from L1 to L3 because groups perform different Methods even when their lower-level actions look similar. In one task, for example, similarity decreases from 82.3 at L1 to 42.1 at L3 because SUs additionally perform dismiss and verify, Methods that are absent from the BLVU traces. These differences are difficult to see at L1, where the underlying clicks and keypresses may still look similar. L3 therefore shows both where different interactions accomplish the same Methods and where the groups actually follow different Methods.

Goals (L4): User groups converge on task goals, with differences in auxiliary goals. Prior work has shown that blind users adopt coping strategies when they encounter accessibility barriers [9, 53]. We observe evidence of these strategies at L4. In particular, BLVUs take on additional goals to seek help from online resources while completing tasks: 9 of the 22 goals observed among BLVUs involve help-seeking, including 5 that use a screen reader assistant. SUs also take on additional goals, but of a different kind: 8 of their 20 observed goals involve verification or window management. Thus, the differences between the groups at L4 are concentrated in these additional goals rather than in the main goals needed to complete the task. Otherwise, SU and BLVU traces largely converge at L4. Across-group Sequence and Pattern similarity reach 70.7 and 74.7, and both groups average about 1.6 goals per trace. Thus, despite differences in operations, navigation, and Methods at lower levels, the two groups generally work toward the same goals for the same tasks (Figure 2). Differences at L4 instead reflect additional goals along the way.

How closely do computer use agent workflows align with human workflows?

Prior work shows that humans and agents can follow similar high-level steps despite differences in how they execute them [59]. A11y-CUA further shows that changing an agent’s access mode substantially changes its behavior and performance [22]. We use MAP to connect these findings by examining where agents align with humans across abstraction levels. We compare CUA-D with SUs and CUA-SR with BLVUs, using the corresponding within-human similarity as a baseline.

Operations (L1): Agents mirror human access modality, but not execution. Prior A11y-CUA analysis showed that neither agent condition fully reproduces the interaction of its corresponding human group. CUA-D uses mouse and keyboard input like SUs but differs in its mix of actions, while CUA-SR uses keyboard input like BLVUs but differs in the specific keys and shortcuts it uses [22]. MAP shows how these differences appear when human and agent traces are represented using the same L1 operations. CUA-D is much less similar to SUs than SUs are to one another: its Pattern similarity with SUs is 42.8, compared with 86.2 within SUs. Although both use mouse and keyboard input, their recurring operation patterns differ. For example, CUA-D uses double primary click more often than SUs (8.8% vs. 4.5%) and does not use drag_drop at all (vs. 1.5% for SUs).

CUA-SR, in contrast, is much more similar to BLVUs at L1. Its Pattern similarity with BLVUs is 88.7, reaching 95% of the 93.8 similarity observed between BLVUs themselves. This high similarity reflects their shared reliance on keypress and shortcut operations. However, the specific keyboard inputs still differ: CUA-SR uses 102 distinct keystrokes and shortcuts, compared with 225 for BLVUs. Thus, L1 captures the shared access mode between agents and their corresponding human groups, while differences in their specific execution remain within those operations.

Functions & Consequences (L2): Screen reader use shows similar increases in actions for agents and humans. Agents make fewer corrections than humans. A11y-CUA found that screen reader use substantially increases agent actions [22]: CUA-SR performs 3.0× as many actions as CUA-D, closely matching the 2.9× increase from SUs to BLVUs. MAP shows where this increase occurs at L2. For agents, actions with no recorded interface consequence increase from 5.7% for CUA-D to 16.7% for CUA-SR; for humans, they similarly increase from 4.3% for SUs to 21.1% for BLVUs. Thus, much of the additional interaction introduced by screen reader use occurs in actions that do not immediately produce a recorded interface change.

A11y-CUA also observed that agents often type long URLs or commands to navigate or complete tasks, whereas humans more often navigate through the interface [22]. MAP makes this difference visible at L2. For CUA-D, type(text area)→text input(text area) accounts for 27.1% of its L2 patterns, compared with 10.4% for SUs, showing that the agent spends more of its interaction in uninterrupted typing. When typing is interrupted, 86% of CUA-D interruptions are followed by select and only 2% by navigate, compared with 44% and 47% for SUs. CUA-SR also almost never produces the mute consequence that BLVUs use to manage screen reader speech (0.02 vs. 3.26 per 100 actions). Thus, similar increases in total actions do not imply similar interaction behavior: humans more often interleave typing with navigation and screen reader management, whereas agents rely more on extended typing and direct selection. This distinction matters when agents are used to simulate users for interface evaluation [26, 45]: matching task completion or action count does not necessarily mean reproducing how humans interact with the interface.

Methods (L3): Humans and agents converge on Methods despite different low-level execution. Prior work finds that humans and agents can complete tasks using similar high-level steps even when their low-level actions differ [59]. MAP confirms this at L3. SU–CUA-D Pattern similarity rises from 42.8 at L1 to 73.5 at L3, reaching 96% of the within-SU similarity (76.8). BLVU–CUA-SR similarity shows the same pattern, reaching 70.7 at L3, or 99% of the within-BLVU similarity (71.4). In other words, by L3, each agent is almost as similar to its corresponding human group as humans within that group are to one another.

The L3 labels show where this similarity comes from. For CUA-D and SUs, open accounts for 36.8% and 38.4% of L3 Method instances, respectively, while edit accounts for 18.4% and 17.7%. Thus, although humans and agents differ substantially in the lower-level actions used to carry out tasks, they often perform the same L3 Methods at similar rates.

Goals (L4): Agents usually reach the same primary task goals as humans, but omit auxiliary goals. Prior work finds that humans and agents can pursue similar high-level task goals despite differences in lower-level execution [59]. MAP confirms this most clearly for CUA-D. Its Pattern similarity with SUs is 77.5, comparable to the within-SU similarity of 76.8, and 96.7% of CUA-D traces end in the most common final goal observed for humans on the same task. CUA-SR shows weaker alignment: 76.7% of its traces end in the most common human final goal, and it averages 1.17 goals per trace compared with about 1.5 for humans.

The L4 labels also reveal a difference that final task goals alone do not capture. Both agents use fewer distinct goals across their traces than humans (45 for CUA-D vs. 86 for SUs; 35 for CUA-SR vs. 88 for BLVUs). In particular, agents often omit auxiliary goals that appear in human traces, such as seeking help or closing extra windows and tabs. Thus, agents often match humans on the primary goal of the task, while differing in the additional goals humans pursue along the way.

Discussion

Our results show that the level at which interaction traces are compared matters: users and agents can differ substantially in their low-level actions while converging on similar methods and goals, and similar workflows can still differ in the effort required to carry them out. We discuss how MAP’s linked abstraction levels can support applications that build on these distinctions, how the representation may generalize beyond computer use, and limitations of the current approach.

Potential Applications of MAP

MAP can surface more efficient strategies from interaction traces. Our results suggest that MAP could support systems that help users discover easier ways to complete computer tasks. Prior work has recommended commands and keyboard shortcuts to improve efficiency [14, 37, 40, 41, 57]. MAP offers a way to find such alternatives directly from interaction traces by comparing users who complete the same L3 Method but use different lower-level actions.

For example, three screen reader users all performed convert ⟨file⟩\langle\text{file}\rangle in ⟨spreadsheet⟩\langle\text{spreadsheet}\rangle, but took very different paths. One user pressed F12 to open Save As directly and finished in 42 actions. Another opened the File menu with Alt+F and then pressed a, taking 70 actions. A third user moved through the menu item by item and took 100 actions, including 43 navigate→focusnavigate \to focus actions. Because all three users were carrying out the same Method, MAP makes it possible to identify the shorter interaction path and the specific shortcut that enabled it. We see a similar case for export ⟨webpage⟩\langle\text{webpage}\rangle in ⟨browser⟩\langle\text{browser}\rangle: one user opened the print dialog directly with Ctrl+P, while another reached it by navigating through browser controls.

This points to a broader use of semantic interaction traces: instead of recommending a shortcut in isolation, a system could first find users performing the same Method, identify which executions require fewer actions, and surface the strategy that produced that difference. Such recommendations would still need to account for context, since a trace alone cannot establish whether a shortcut is available, accessible, or appropriate in another user’s current interface state. Future systems could combine MAP with interface-state information and user preferences to decide when an observed strategy is actually useful to recommend.

MAP can flag potential usability issues from interaction traces. Our results also suggest that MAP could help identify parts of an interface where users repeatedly encounter difficulty. Prior work has used interaction time, repeated actions, path deviations, and corrective actions as signals of usability problems [6, 19, 27, 48]. MAP adds a way to connect these signals to the part of the task where the difficulty occurs. By comparing the time and number of actions required for the same L3 Method across users, it can flag unusually costly Methods and then use the lower levels to show which interactions contributed to that cost.

For example, in an A11y-CUA task that asked users to change a presentation to the Gallery theme, the format ⟨Gallery theme⟩\langle\text{Gallery theme}\rangle Method required unusually many actions for seven of nine blind users and five of nine sighted users (Figure 4). The interface presents themes in a horizontally scrolling gallery without a direct way to select Gallery by name. At L2, the flagged traces contain many repeated navigate→focusnavigate \to focus interactions as users move through theme options to reach the desired theme. MAP therefore identifies not only that this Method requires unusually high effort, but also that much of that effort comes from repeatedly navigating through the gallery.

Figure 4. Difficulty in changing a PowerPoint theme. Each point represents one user’s format Method. The horizontal axis shows actions relative to the most efficient successful user in the same group; the vertical axis shows time per action relative to the user’s usual pace. Points beyond the thresholds are long-winded or slow. Most difficulty comes from requiring many actions rather than performing them slowly.

More broadly, this could support usability analysis at scale. Rather than inspecting every trace manually, designers could use MAP to find Methods that repeatedly require high effort across users, inspect the lower-level interactions associated with those Methods, and prioritize them for further study or redesign. Future systems could also combine these signals with information about the interface itself to distinguish difficulty caused by the interface from differences in user strategy, and to suggest possible changes such as adding direct selection, shortcuts, or more accessible navigation. These flags should be treated as candidates rather than automatic diagnoses: unusual time or action counts can indicate a usability problem, but still require inspection or follow-up testing to understand why it occurred.

MAP may help reduce identifiable information when sharing interaction traces. Interaction traces are useful for studying computer use, but they can also reveal who produced them. Prior work shows that keyboard, mouse, and GUI behavior can identify individual users and distinguish humans from agents [7, 39, 54]. Our results suggest that the level at which a trace is represented may affect how much of this identifying behavior remains.

We test this by predicting which of 18 trace sources produced a trace using the features available at each MAP level, including label frequencies, action and time allocation, and similarity scores. Under leave-one-task-out evaluation, top-1 classification accuracy is 25.4% at L1 and rises to 43.3% at L2, showing that adding semantic information can initially make users more distinguishable. At L3, however, accuracy drops to 13.0%, closer to the 5.6% expected by random chance. This drop appears across trace sources: classification falls from 83.3% to 20.0% for agents, 32.5% to 8.3% for SUs, and 44.2% to 15.8% for BLVUs.

This suggests a possible trade-off when sharing interaction data. L2 retains differences in how users navigate and interact with interfaces, which can make individual behavior easier to recognize. L3 instead keeps the Methods users perform while removing more of the lower-level detail that distinguishes how each person performs them. Method-level traces may therefore support some forms of behavioral analysis while exposing less identifying information than lower-level traces. However, lower classification accuracy does not by itself provide a privacy guarantee. Future work should test this with stronger identification attacks, larger populations, and combinations with other potentially identifying information before using semantic abstraction as a privacy mechanism.

Generalizing Semantic Trace Abstraction Beyond Computer Use

MAP is designed for desktop GUI interaction, and its current vocabulary reflects that scope. For example, labels such as keypress, navigate, and open describe actions and outcomes that occur when people use desktop applications. Extending MAP to a different interaction setting would therefore require new labels. What may generalize is not the vocabulary itself, but the idea of linking multiple levels that describe how an action is performed, what it accomplishes, which method it contributes to, and which larger goal that method serves.

This distinction becomes clearer for interfaces outside desktop computer use. In virtual reality, L1 might include controller movements, gaze, or gestures; L2 could describe effects such as selecting, grabbing, or moving a virtual object; L3 could represent methods such as placing or assembling objects; and L4 could capture the larger task goal. In robotics demonstrations, L1 could contain low-level robot motions, L2 their effects on objects, L3 manipulation methods such as grasping or placing, and L4 the task being completed [61]. These domains would need different labels from MAP, but could retain the same links between execution, outcome, method, and goal.

Programming and CAD are closer to our current setting because they are often performed through desktop interfaces, but their domain-specific behavior is not fully represented by our current vocabulary. A programming extension, for example, may need to distinguish methods such as debugging, refactoring, and testing rather than treating them only as generic editing or navigation. CAD traces may similarly need methods that capture domain-specific design operations rather than only the GUI actions used to invoke them [20]. Extending MAP would therefore mean keeping the abstraction structure while replacing or expanding the vocabulary to capture what is meaningful in that domain.

PCA projections of SU, BU, CUA-D, and CUA-SR traces across Sequence, Effort, and Pattern Similarity and MAP abstraction levels

Figure 3. PCA projections of SU, BU, CUA-D, and CUA-SR traces across Sequence, Effort, and Pattern Similarity and MAP abstraction levels. SU and BU are strongly separated at L1 and L2 but increasingly converge at L3 and L4, suggesting that group differences are greater in low-level execution than in higher-level methods and goals. Each panel reports the kk-means (k=2k = 2) silhouette score and group-match count.

Our results suggest why this structure may be useful beyond desktop interaction. We repeatedly find that behaviors that look very different at a low level can accomplish the same Method or Goal, while important differences in effort remain visible in the lower levels. The same question arises in other domains: two people, or a person and a robot, may accomplish the same goal using very different actions. Comparing only the low-level actions can make equivalent strategies look unrelated, while comparing only the final goal hides how differently it was achieved. A multi-level representation preserves both. Future work could test whether the same design choices that were useful in MAP—separating actions from their consequences, linking lower-level behavior to Methods, and retaining effort at each level—also support meaningful comparisons in VR, robotics, and other interactive domains.

Limitations

MAP relies on information that is available across the interaction logs we study: keyboard and mouse actions and the available interface context around those actions [22, 63, 65]. It does not require signals that are only available in some datasets, such as screen reader speech, screenshots aligned to every action, or accessibility-tree state. This allows the same pipeline to be applied across different logging setups, but it also limits what MAP can infer. For example, a keypress may move screen reader focus even when that change is not recorded in the trace. Additional signals could help resolve such cases, and future work could test how much screenshots, accessibility-tree information, or screen reader output improve annotation accuracy.

L3 introduces another source of ambiguity because MAP must decide both where one Method ends and another begins and which Method those actions represent. For example, when a user opens the Save As dialog and then changes a file’s format, the action that opens Save As could be grouped with the preceding open Method or with the following convert Method. Moving this boundary changes which actions are grouped together and can therefore change the assigned Method. Specific Entities and L4 Goals are also open vocabulary: for example, two traces may describe the same goal using slightly different wording that must be mapped to the same concept. We canonicalize these labels, but some interactions can still reasonably be divided or described in more than one way. Our manual agreement results therefore show how consistently MAP follows our annotation scheme, while recognizing that another reasonable annotation may sometimes differ.

The L3 Method and UI Container vocabularies were developed from A11y-CUA and then evaluated on A11y-CUA, AgentSynth, and AgentNet. They cover nearly all interactions in these datasets, but those datasets represent only a subset of computer use. Other workflows may introduce behavior that our current labels do not distinguish well. For example, creative work may involve repeated experimentation without a clear sequence of Methods, recovery may alternate between diagnosing and fixing a problem, and users may interleave several goals rather than completing one before starting another. Applying MAP to these settings may therefore require adding or refining labels while keeping the same linked levels.

Finally, our similarity metrics capture three aspects of workflow behavior: the order of semantic labels, recurring patterns, and the effort spent at each level. They do not capture every meaningful difference between workflows. Prior work also compares loops, skipped steps, alternative paths [47], and how far a workflow progresses before failure [12]. Future work could extend MAP to represent errors and recovery, transitions between applications, task success, and consistency across repeated attempts.

Conclusion

We presented MAP, a hierarchical taxonomy for representing interaction traces across four levels of abstraction from input-level operations to high-level goals. MAP uses an automated annotation pipeline to assign MAP labels. The pipeline achieves strong agreement with human annotations and we demonstrate this on three datasets that record computer use interaction traces of humans and agents. MAP also provides three similarity metrics (sequence, pattern, and effort) that can be computed at any level or combined across abstraction levels. We compared how sighted users, blind users, and computer use agents complete tasks using the A11y-CUA dataset. We show that MAP replicates prior findings at lower abstraction levels while surfacing new findings at intermediate and higher abstraction levels. We demonstrated MAP’s use cases in improving novice users’ expertise in computer use tasks by deriving efficient strategies from expert users, identifying potential usability issues in GUIs, and identifying users’ behavioral fingerprints in computer use. Together, MAP’s hierarchical taxonomy and similarity metrics provide a unified framework for researchers analyzing and comparing interaction traces across multiple levels.

MAP Prompts

L2 Functions & Consequences

Infer L2 labels for each action in the UI interaction trace. L2 structure: Function, Direct UI Target → Consequence, Indirect UI Target.

Function shows what the operation did (already determined for each action in the input). Direct UI Target is the element on which the action occurs. Choose the most specific directly interacted-with element supported by the trace. For example, use cell rather than table and list item rather than list for Direct UI Target. Consequence is the direct result of the action. Indirect UI Target is the affected UI object on which change is observed because of the consequence of the action.

Examples: a "select" Function on a "list item" (Direct UI Target) representing a file, where the action opens that file, gives the Consequence "open" on Indirect UI Target "file". A "drag" Function on "list item" (Direct UI Target) representing a file, dropped into a folder, gives the Consequence "move" and Indirect UI Target "folder".

Instructions for determining the L2 output for each operation:

  • The direct and indirect target may be the same or different. For direct target, choose the specific UI element on which action occurs and not its container (indirect UI target).

  • For point actions, infer the direct and indirect UI target from the destination, using the following action only when needed. For keyboard actions, use the focused element as the Direct UI Target.

  • Use the given Function, lower-level labels, and raw action details (event, keys, application, window_title, target, focused_element, and nearby actions) to infer the remaining labels.

  • For each action, output only index, direct_ui_target, consequence, indirect_ui_target, rationale, confidence (0–1), and a short plain-language description.

  • Output a JSON array with exactly actionCount entries, one per action, sorted by index. Do not merge or omit actions.

L3 Methods

Infer a canonical segmentation and L3 semantic label for every action in the UI interaction trace. L3 structure: [Method, Specific Entity] in {UI Container}.

Your first task is segmentation. Divide the complete trace into contiguous task-level procedures, then assign exactly one L3 label to every procedure. All actions within a procedure must receive the exact same Method, Specific Entity, and UI Container. Start a new segment only when the task-level objective or completed outcome changes. Also segment when the active application changes on desktop or when the webpage changes in a browser. Do not create boundaries for changes in event type, menus or dialogs, navigation, verification, corrections, retries, pauses, or interaction technique. Setup, recovery, and confirmation actions inherit the procedure they support.

Reference example: use the following manually annotated trace as a reference for the intended L3 segmentation granularity.

Task: “Add the video cat.mp4 to family.pptx, both in Document > My-Home folder.”

  • Segment 1 - Navigate to the task folder. Open File Explorer and navigate through folders to My-Home. L3: [open, My-Home folder] in {file explorer}

  • Segment 2 - Copy the video. Select cat.mp4 and copy it to the clipboard. L3: [export, cat.mp4] in {file explorer}

  • Segment 3 - Open the presentation. Select family.pptx and open it, the presentation interface becomes active. L3: [open, family.pptx] in {presentation}

  • Segment 4 - Add the video to the presentation. Focus the presentation canvas and paste the copied video. L3: [add, cat.mp4] in {presentation}

Method represents the subgoal that an action contributes to, not what the individual action immediately does. For example, opening a font menu, choosing a size, and confirming it are all part of format text in a text document, rather than separate open, select, and confirm Methods. Specific Entity is the shortest grounded name of the task-relevant item on which the Method is performed. Use the visible or referenced name when available (e.g., My-Home); otherwise, use a concise descriptor. Specific Entity is open vocabulary. UI Container is the enclosing document or application type in which the procedure occurs. When multiple UI Container labels could apply, choose the container corresponding most directly to the content or workspace being interacted with (e.g., spreadsheet over website, text document over browser, file explorer over desktop). Choose the closest matching label from the allowed Method and UI Container lists, and use unknown when no listed label applies.

Instructions for determining the L3 output for each operation:

  • A reader should be able to read the three fields together as one coherent procedure: [Method, Specific Entity] in {UI Container} (e.g., “edit, Introduction, in text document”).

  • Segment the complete trace first, then assign one L3 label to each segment. Every action within

the same segment must receive identical method, specific_entity, and ui_container values.

  • Use lower-level labels and raw action details (event, keys, application, window_title, target, focused_element, and nearby actions) to infer segmentation and labels.

  • The L3 label should fit naturally with the preceding and following segments in the overall interaction flow.

  • For each action, output only index, method, specific_entity, ui_container, rationale, confidence (0–1), and a short plain-language description.

  • Output a JSON array with exactly actionCount entries, one per action, sorted by index. Do not merge or omit actions.

L4 Goals

Infer L4 semantic labels for actions in the UI interaction trace. Goal is a short, task-level phrase that names the user’s objective achieved by a group of L3 methods, describing the outcome of the high-level goals. A goal must normally contain multiple L3 steps rather than restating a single L3 Method. Instructions for determining the L4 output for each operation:

  • Use the lower-level labels and lower-level action details (event, keys, application, window_title, target, focused_element, and nearby actions) for grounding.

  • Group contiguous L3 steps that jointly serve the same objective.

  • Every L3 step must belong to exactly one Goal; groups cannot overlap or skip intervening steps.

  • For each output goal, include l3StepIndices (all covered L3 step indices, in order), rationale, and confidence (0–1).

MAP Taxonomy Vocabulary

Consequences (L2)

LabelDefinition
cancelStop or abandon a pending operation without applying or committing its intended changes.
deselectRemove an existing selection.
toggleSwitch a control between two states, such as on/off.
zoomChange the magnification of the current view.
cutRemove selected content and place it on the clipboard.
copyPlace selected content on the clipboard without removing it.
pasteInsert clipboard content at the current target.
deleteRemove content, an item, or an object.
undoReverse the most recent change.
redoReapply a previously undone change.
closeDismiss an open tab, menu, dialog, view, or window from current view.
maximizeExpand a window to fill the available screen area.
focusKeyboard/input focus moves to a target (e.g. clicking into a field, tabbing to a control) but no new UI surface appears.
saveSave the current document, file, or state.
createMake a new item or UI object.
select allSelect all applicable content or items in the current scope.
minimizeHide a window while keeping it open.
openA new UI surface becomes visible that wasn’t before, e.g. a menu drops down, a dialog appears, a panel expands.
highlightContent is selected or emphasised across a range, usually by shift+arrow or dragging.
moveAn object, window, or content is itself relocated spatially, by dragging.
text inputAdd or edit textual content.
autofillAutomatically populate one or more input fields with stored, suggested information, rather than entering the values manually.
hoverMove or hold the pointer over an element without activating it, whether or not a visual change appears.
panThe visible viewport shifts across a larger canvas or scrollable area, by scrolling.
confirmFinalizes and closes a pending choice or flow, e.g. clicking OK/submit on a dialog.
announceA bare Alt tapped on its own while a screen reader is running, where the screen reader intercepts the modifier press and re-speaks the current item, state, or status; ordinary navigation keys remain navigate or focus even when speech follows as a side effect.
muteA bare Ctrl tapped on its own while a screen reader is running, where the screen reader intercepts the modifier press and silences the speech currently being spoken.
noneAction had no effect, e.g. a lone modifier key press.
unknownNone of the other consequence options clearly fit this action.

Table 7. L2 Consequence vocabulary. Consequence identifies the immediate result of an action on the interface or application state. The vocabulary contains 28 labels plus unknown; all 28 labels occur in A11y-CUA.

UI Targets

LabelDefinition
buttonA clickable UI control (e.g. toolbar button) that triggers an action when selected.
radio buttonA single selectable option in a set of mutually exclusive choices.
linkA specific, identifiable clickable element (text, hyperlink, or image) that triggers navigation to a different location.
search boxAn input field for entering a query or URL to search with a webpage or application.
sliderA draggable control for setting a numeric value within a range.
date picker†A calendar widget for selecting a specific date.
color pickerA widget for selecting a color from a palette, spectrum, or input value.
checkboxA binary toggle that can be checked or unchecked independently of other options.
menu itemA single actionable option inside an open menu, dropdown, or context menu, e.g., File menu → Save As is the menu item.
formA group of related fields and controls used to enter or submit information.
tableA 2-dimensional structure with data organized into a grid of rows and columns.
cellA specific unit within a table, defined by a row and column position.
columnA vertical series of cells in a table.
rowA horizontal series of cells in a table.
windowA discrete container that displays an application’s contents in a resizable frame.
scrollbarA draggable control that moves the viewport through content that extends beyond the visible area.
webpageA page’s general content or background, e.g. clicking whitespace.
text areaAn editable field or area where the user can enter or modify written content other than search box.
imageA static photo, illustration, or graphic viewed or placed as content, e.g. a thumbnail or inserted picture.
videoA dynamic visual media element that can be played, paused, or scrubbed.
audio†A media element containing only sound content.
canvasAn editable workspace where visual content is drawn or manipulated, e.g. a chart canvas or drawing tool’s working area.
desktopThe main system workspace containing windows, icons, and other items.
listA grouped collection of related items or options.
list itemA single entry within a list (e.g. file list in a folder, icons in taskbar).
caretActive cursor position where text insertion happens.
commandA text or voice instruction issued to execute an action, e.g. via a command line.
clipboard†Temporary system storage used by cut, copy, and paste operations.
autofillAutomatically populate one or more input fields with suggested information by moving the caret, rather than entering the values manually.
unknownNone of the other direct UI target options clearly fit this action.
fileA UI item containing data, content, or executable information.
context menuA menu opened for a specific item or location, usually by right-clicking, containing actions for that context; unlike a regular menu, it is tied to the item or location where it was opened.
dialog boxA temporary modal that appears over the current interface to request input, confirm an action, or display a message before the user can continue.
item cardA bounded card element displaying details for a single item in a collection, feed, or grid layout.
folderA container used to organize files and subfolders.
applicationA desktop application excluding the browser.
noneNo UI object was affected, because the action produced no change in the interface; applies whenever the Consequence is none, mute, or announce. Distinct from unknown, which means an object was affected but could not be identified.

Table 8. Table 8: L2 Direct UI Target vocabulary. Direct UI Target identifies the interface element on which an action is performed. MAP assigns the most specific element supported by the trace (e.g., cell rather than table). The top section lists all Direct UI Target labels. The bottom section extends this vocabulary with additional labels that can be used only for Individual UI Targets. The Direct UI Target vocabulary contains 29 labels plus unknown; 26 of the 29 labels occur in A11y-CUA. † indicates labels not observed in A11y-CUA.

Methods

LabelDefinition
searchLocate a specific item or information by entering or submitting a query through a search feature.
exportSave a separate output copy of existing content to a destination; convert applies instead when changing the file format is itself the objective.
openActivate a specific content item, such as a file or document, so its contents become available for use; reaching a website through the browser is not open.
verifyExamine content of a UI item to check its information without modifying it, e.g. verifying product added to cart, changes to a document.
editChange only the textual contents of a UI item (e.g. changing text in a document, a filename, or form response) or change an existing numeric or slider-style setting of a UI item to a new value via an opened settings panel.
formatChange how existing textual and visual content is presented, without changing its substantive meaning, such as font, alignment, color, style, or layout.
moveAn object, window, or content is itself relocated spatially, by dragging.
dismissClose or minimize a window, transient popup, context menu, or confirmation without acting on it further.
convertTransform an existing file from one file format into another while preserving its underlying content.
zoom†Change magnification level of the content in current view.
checkout†Complete a purchase flow, e.g. confirming payment or submitting an order.
addAdd a brand-new object into a document via a ribbon or menu command or create a UI item via an “add” button.
filterApply a selection criterion on an already available data range or list to narrow it down.
unknownNone of the other method options clearly fit this action.

Table 9. L3 Method vocabulary. Method describes how a subgoal is carried out. We first open coded 53 Method terms during an annotation pilot, then consolidated them into 13 labels that were used consistently across independent annotation runs. The vocabulary contains 13 labels plus unknown; 11 of the 13 labels occur in A11y-CUA. †indicates labels not observed in A11y-CUA.

UI Containers

LabelDefinition
browserThe browser’s own interface, such as tabs, address bar, bookmarks, navigation controls, downloads, or browser settings; excludes the content of loaded webpages.
websiteThe content and interface of a loaded website or web app.
image editorAn application used to view, create, or edit images or graphics.
media playerAn application used to play or control audio or video media.
code editorAn application for writing, editing, and running source code.
canvas†An interactive area where visual content is drawn or manipulated.
spreadsheetA worksheet containing data organized into rows and columns.
text documentAn application to view or edit written text in a document.
email client†An interface for reading, composing, organizing, or managing email.
presentationAn application used to view, create, edit, or present slides in a slide-deck.
file explorerA system interface for browsing and managing files and folders.
desktopThe operating system workspace containing windows, icons, and system controls.
clockAn application for viewing time and managing alarms, timers, stopwatches, or world clocks.
settingsA system-level interface for changing preferences, permissions, accessibility options, or other configuration.
applicationA native desktop application whose interface doesn’t match any other listed UI container.
unknownNone of the other UI container options fits this action.

Table 10. L3 UI Container vocabulary. UI Container identifies the application area in which a subgoal is carried out, such as a dialog, menu, toolbar, or document. The vocabulary contains 15 labels plus unknown; 13 of the 15 labels occur in A11y-CUA. † indicates labels not observed in A11y-CUA.

C Within- and Across Group Similarities Across MAP Abstraction Levels

LevelSequenceEffortPattern
L10.400.700.77
L2a0.260.290.51
L2b0.210.270.39
L30.030.150.06
L40.060.100.06

Table 11. Silhouette scores for kk-means clustering (k=2k = 2) across MAP abstraction levels and similarity metrics. Higher scores indicate stronger separation between the two clusters; separation is strongest at L1 and decreases at higher abstraction levels.

MetricLevelWithin SUWithin BUSU–CUA-DSU–BUSU–CUA-SRCUA-D–BUCUA-D–CUA-SRBU–CUA-SR
SequenceL152.90 (3.31)46.60 (5.32)34.73 (3.23)12.79 (2.98)11.70 (1.96)13.77 (2.51)14.7342.93 (2.19)
SequenceL2a49.21 (2.15)36.88 (4.79)38.82 (2.20)21.17 (4.07)18.98 (1.22)16.37 (3.07)16.9227.21 (1.80)
SequenceL2b41.62 (2.04)30.43 (3.80)30.11 (2.45)16.39 (3.20)14.53 (1.14)15.88 (3.19)19.0522.58 (1.25)
SequenceL371.95 (6.12)67.48 (5.67)68.91 (5.15)68.77 (6.76)62.85 (3.70)64.38 (4.60)66.2261.42 (4.39)
SequenceL472.21 (6.71)73.76 (5.44)74.44 (6.58)70.66 (6.46)62.29 (4.66)76.46 (4.24)66.6764.44 (4.53)
EffortL172.76 (5.12)86.50 (4.15)25.57 (4.67)10.71 (4.03)3.15 (1.90)5.91 (1.03)0.9273.25 (4.02)
EffortL2a69.42 (2.53)46.49 (8.67)51.65 (2.80)38.84 (10.18)16.00 (2.80)22.80 (6.92)10.0227.87 (3.77)
EffortL2b55.23 (3.20)29.17 (7.10)24.41 (2.49)14.61 (5.35)5.08 (0.93)11.47 (3.59)6.7511.58 (2.90)
EffortL378.12 (4.02)53.57 (9.33)68.50 (2.13)59.91 (10.93)43.96 (3.87)54.31 (8.25)51.4445.54 (6.56)
EffortL464.80 (6.58)54.42 (9.80)62.23 (6.27)54.33 (10.07)42.91 (4.76)59.52 (8.21)50.5447.85 (6.59)
PatternL186.20 (3.32)93.82 (2.45)42.83 (4.51)33.15 (4.97)30.93 (4.59)38.83 (1.26)39.4488.69 (1.57)
PatternL2a72.07 (3.43)76.85 (4.23)51.12 (4.22)41.11 (3.08)33.91 (2.98)34.39 (2.30)33.0457.84 (2.81)
PatternL2b61.09 (3.30)58.82 (6.14)41.50 (4.68)27.65 (3.66)23.99 (3.42)32.38 (3.88)36.9744.16 (1.43)
PatternL376.76 (5.42)71.41 (5.35)73.54 (5.00)72.51 (6.26)72.16 (3.65)67.89 (5.15)71.8570.68 (3.76)
PatternL477.46 (6.09)76.39 (5.19)77.47 (4.69)74.74 (6.50)68.30 (4.03)78.79 (5.07)72.0570.55 (4.15)

Table 12. Within- and between-group Sequence, Effort, and Pattern Similarity across MAP abstraction levels. Values report mean similarity with standard deviation in parentheses. CUA-D–CUA-SR contains a single pair and therefore has no standard deviation.

References

  1. [1][n. d.]. Mac Keyboard Shortcuts. https://support.apple.com/en-us/102650
  2. [2]2023. Development guide for the accessibility infrastructure - WAI-ARIA Basics. https://gnome.pages.gitlab.gnome.org/at-spi2-core/devel-docs/
  3. [3]2025. Keyboard shortcuts in Windows. https://support.microsoft.com/en-us/windows/keyboard-shortcuts-in-windows-dcc61a57-8ff0-cffe-9796-cb970c75eec
  4. [4]Luka Babb and Jana-Rebecca Rehse. 2024. Process-related user interaction logs: state of the art, reference model, and object-centric implementation. Information Systems 124 (2024), 102386.DOI
  5. [5]Reyna Abhyankar, Qi Qi, and Yiying Zhang. 2026. Osworld-human: Benchmarking the efficiency of computer-use agents. Proceedings of Machine Learning and Systems 8 (2026), 482–494.arxiv.org/abs/2506.16042
  6. [6]David Akers, Matthew Simpson, Robin Jeffries, and Terry Winograd. 2009. Undo and erase events as indicators of usability problems. In Proceedings of the SIGCHI conference on human factors in computing systems. 659–668.
  7. [7]Kyle O Bailey, James K Ocolika, and Gilbert L Peterson. 2014. User identification and authentication using multi-modal behavioral biometrics. Computers & Security 43 (2014), 77–89.DOI
  8. [8]Alexandre Battut, Kevin Ratovo, and Michel Beaudouin-Lafon. 2025. OneTrace: Improving Event Recall and Coordination With Cross-Application Interaction Histories. International Journal of Human–Computer Interaction 41, 5 (2025), 3241–3258.DOI
  9. [9]Jeffrey P Bigham, Anna C Cavender, Jeremy T Brudvik, Jacob O Wobbrock, and Richard E Ladner. 2007. WebInSitu: a comparative analysis of blind and sighted browsing behavior. In Proceedings of the 9th International ACM SIGACCESS Conference on Computers and Accessibility. 51–58.
  10. [10]Yevgen Borodin, Jeffrey P Bigham, Glenn Dausch, and IV Ramakrishnan. 2010. More than meets the eye: a survey of screen-reader browsing strategies. In Proceedings of the 2010 international cross disciplinary conference on web accessibility (W4A). 1–10.
  11. [11]Stuart K Card, Thomas P Moran, and Allen Newell. 1980. The keystroke-level model for computer performance time with interactive systems. Commun. ACM 23, 7 (1980), 396–410.
  12. [12]Jiwan Chung, JiHyuk Byun, Vibha Vineet, and Seon Joo Kim. 2026. Where Did It Go Wrong? Process-Level Evaluation of Web Agents with Semantic State Tracking. arXiv preprint arXiv:2606.15673 (2026).arxiv.org/abs/2606.15673
  13. [13]Lillian Clark, I-Hsien Ting, Chris Kimble, Peter C Wright, and Daniel Kudenko. 2006. Combining Ethnographic and Clickstream Data to Identify User Web Browsing Strategies. Information Research: an international electronic journal 11, 2 (2006), paper–249.
  14. [14]Andy Cockburn, Carl Gutwin, Joey Scarr, and Sylvain Malacria. 2014. Supporting novice to expert transitions in user interfaces. ACM Computing Surveys (CSUR) 47, 2 (2014), 1–36.
  15. [15]Biplab Deka, Zifeng Huang, Chad Franzen, Joshua Hibschman, Daniel Afergan, Yang Li, Jeffrey Nichols, and Ranjitha Kumar. 2017. Rico: A mobile app dataset for building data-driven design applications. In Proceedings of the 30th annual ACM symposium on user interface software and technology. 845–854.
  16. [16]Biplab Deka, Zifeng Huang, and Ranjitha Kumar. 2016. ERICA: Interaction mining mobile apps. In Proceedings of the 29th annual symposium on user interface software and technology. 767–776.
  17. [17]Xiang Deng, Yu Gu, Boyuan Zheng, Shijie Chen, Sam Stevens, Boshi Wang, Huan Sun, and Yu Su. 2023. Mind2web: Towards a generalist agent for the web. Advances in Neural Information Processing Systems 36 (2023), 28091–28114.arxiv.org/abs/2306.06070
  18. [18]Michael Desmond, Ja Young Lee, Ibrahim Ibrahim, James M Johnson, Avirup Sil, Justin McNair, and Ruchir Puri. 2025. Agent trajectory explorer: Visualizing and providing feedback on agent trajectories. In Proceedings of the AAAI Conference on Artificial Intelligence, Vol. 39. 29634–29636.DOI
  19. [19]Matteo Filosa, Alexandra Plexousaki, Matteo Di Stadio, Francesco Bovi, Dario Benvenuti, Tiziana Catarci, and Marco Angelini. 2025. TraVIS: A User Trace Analyzer to Support User-Centered Design of Visual Analytics Solutions. IEEE Transactions on Visualization and Computer Graphics (2025).DOI
  20. [20]J Gopsill, C Snider, L Shi, H Ben, et al. 2016. Computer aided design user interaction as a sensor for monitoring engineers and the engineering design process. In DS 84: Proceedings of the DESIGN 2016 14th International Design Conference. 1707–1718.
  21. [21]Tovi Grossman and George Fitzmaurice. 2015. An investigation of metrics for the in situ detection of software expertise. Human–Computer Interaction 30, 1 (2015), 64–102.DOI
  22. [22]Ananya Gubbi Mohanbabu, Rosiana Natalie, Brandon Kim, Anhong Guo, and Amy Pavel. 2026. A11y-CUA Dataset: Characterizing the Accessibility Gap in Computer Use Agents. In Proceedings of the 2026 CHI Conference on Human Factors Zellig S Harris. 1954. Distributional structure. Word 10, 2–3 (1954), 146–162.
  23. [24]Zecheng He, Srinivas Sunkara, Xiaoxue Zang, Ying Xu, Lijuan Liu, Nevan Wichers, Gabriel Schubiner, Ruby Lee, and Jindong Chen. 2021. Actionbert: Leveraging user actions for semantic understanding of user interfaces. In Proceedings of the AAAI Conference on Artificial Intelligence, Vol. 35. 5931–5938.
  24. [25]David M Hilbert and David F Redmiles. 2000. Extracting usability information from user interface events. ACM Computing Surveys (CSUR) 32, 4 (2000), 384–421.
  25. [26]Steffen Holter, Eunyee Koh, Mustafa Doga Dogan, and Gromit Yeuk-Yin Chan. 2026. UXCascade: Scalable Usability Testing with Simulated User Agents. arXiv preprint arXiv:2601.15777 (2026).arxiv.org/abs/2601.15777
  26. [27]Jason I Hong, Jeffrey Heer, Sarah Waterson, and James A Landay. 2001. WebQuilt: A proxy-based approach to remote web usability testing. ACM Transactions on Information Systems 19, 3 (2001), 263–285.
  27. [28]Forrest Huang, Gang Li, Tao Li, and Yang Li. 2024. Automatic macro mining from interaction traces at scale. In Proceedings of the 2024 CHI Conference on Human Factors in Computing Systems. 1–16.
  28. [29]Yucheng Jiang, Zora Zhurov Wang, Ruishi Chen, and Divi Yang. 2026. Inducing Task Models from Computer-Use Traces. arXiv preprint arXiv:2608.20319 (2026).arxiv.org/abs/2608.20319
  29. [30]Bonnie E John and David E Kieras. 1996. The GOMS family of user interface analysis techniques: Comparison and contrast. ACM Transactions on Computer-Human Interaction (TOCHI) 3, 4 (1996), 320–351.DOI
  30. [31]Karl-Bridge-Microsoft. 2025. UI Automation Control Patterns Overview - Win32 apps. https://learn.microsoft.com/en-us/windows/win32/winauto/uiauto-controlpatternsoverview
  31. [32]Nicholas Kong, Tovi Grossman, Björn Hartmann, Maneesh Agrawala, and George Fitzmaurice. 2012. Delta: a tool for representing and comparing workflows. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems. 1027–1036.
  32. [33]Cedric Krause, Shivam Agarwal, Michael Burch, and Fabian Beck. 2023. Visually Abstracting Event Sequences as Double Trees Enriched with Category-Based Comparison. In Computer Graphics Forum, Vol. 42. Wiley Online Library, e14805.DOI
  33. [34]GG Landis JRKoch. 1977. The measurement of observer agreement for categorical data. Biometrics 33, 1 (1977), 159174.pubmed.ncbi.nlm.nih.gov/843571
  34. [35]V V Levenshtein. 1966. Binary codes capable or ‘correcting deletions, insertions, and reversals. In Soviet physics-doklady, Vol. 10. 707–710.
  35. [36]Toby Jia-Jun Li, Lindsay Powobksi, Tom Mitchell, and Brad A Myers. 2021. Screen2vec: Semantic embedding of gui screens and gui components. In Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems. 1–15.DOI
  36. [37]Wei Li, Justin Matejka, Tovi Grossman, Joseph A Konstan, and George Fitzmaurice. 2011. Design and evaluation of a command recommendation system for software applications. ACM Transactions on Computer-Human Interaction (TOCHI) 18, 2 (2011), 1–35.DOI
  37. [38]Zhicheng Liu, Yang Wang, Mira Dontcheva, Matthew Hoffman, Seth Walker, and Alan Wilson. 2016. Patterns and sequences: Interactive exploration of click-streams to understand common visitor paths. IEEE transactions on visualization and computer graphics 23, 1 (2016), 321–330.DOI
  38. [39]William Lugoolobi, Samuelle Marro, Jabez Magomere, Joss Wright, and Chris Russell. 2026. Known By Their Actions: Fingerprinting LLM Browser Agents via UI Traces. arXiv preprint arXiv:2605.14786 (2026).
  39. [40]Sylvain Malacria, Joey Scarr, Andy Cockburn, Carl Gutwin, and Tovi Grossman. 2013. Skillometers: Reflective widgets that motivate and help users to improve performance. In Proceedings of the 26th annual ACM symposium on User interface software and technology. 321–330.
  40. [41]Justin Matejka, Wei Li, Tovi Grossman, and George Fitzmaurice. 2009. CommunityCommands: command recommendations for software applications. In Proceedings of the 22nd annual ACM symposium on User interface software and technology. 193–202.
  41. [42]Saul B Needleman and Christian D Wunsch. 1970. A general method applicable to the search for similarities in the amino acid sequence of two proteins. Journal of molecular biology 48, 3 (1970), 443–453.
  42. [43]Amy Pavel, Floraine Berthoudoz, Björn Hartmann, and Maneesh Agrawala. 2013. Browsing and analyzing the command-level structure of large collections of image manipulation tutorials. Citeseer, Tech. Rep. (2013).
  43. [44]Christopher Rawles, Alice Li, Daniel Rodriguez, Oriana Riva, and Timothy Lillicrap. 2023. Androidworld: A large-scale dataset for android device control. Advances in Neural Information Processing Systems 36 (2023), 59708–59728.DOI
  44. [45]Tim Rieder, Marian Schneider, Mario Truss, Vitaly Tsaplin, Alina Rublea, Sinem Dere, Francisco Chicharro Sanz, Tobias Reiss, and Mustafa Doga Dogan. 2026. SimAB: Simulating A/B Tests with Persona-Conditioned AI Agents for Rapid Design Evaluation. arXiv preprint arXiv:2603.01024 (2026).arxiv.org/abs/2603.01024
  45. [46]Nuzhah Goodha Sahib, Anastasius Tombros, and Tony Stockman. 2012. A comparative analysis of the information-seeking behavior of visually impaired and sighted searchers. Journal of the American Society for Information Science and Technology 63, 2 (2012), 377–391.DOI
  46. [47]Clemens Schreiber, Amine Abbad-Andaloussi, Andrea Burattin, Andreas Oberweis, and Barbara Weber. 2025. Graph-based similarity measures for the structural comparison of process traces.
  47. [48]Inayatullah Shah. 2008. Event patterns as indicators of usability problems. Journal of King Saud University Computer and Information Sciences 20, 1 (2008), 31–43.DOI
  48. [49]Antonio C Siochi and Roger W Ehrlich. 1991. Computer analysis of user interfaces based on repetition in transcripts of user sessions. ACM Transactions on Information Systems (TOIS) 9, 4 (1991), 309–335.DOI
  49. [50]Neville A Stanton. 2006. Hierarchical task analysis: Developments, applications, and extensions. Applied ergonomics 37, 1 (2006), 55–79.pubmed.ncbi.nlm.nih.gov/16139236
  50. [51]Hironobu Takagi, Shin Saito, Kentarou Fukuda, and Chieko Asakawa. 2007. Analysis of navigability of Web applications for improving blind usability. ACM Transactions on Computer-Human Interaction (TOCHI) 14, 3 (2007), 13–es.DOI
  51. [52]Leo Törnqvist, Pentti Vartia, and Yrjö O Vartia. 1985. How should relative changes be measured? The American Statistician 39, 1 (1985), 43–46.
  52. [53]Markel Vigo and Simon Harper. 2013. Coping tactics employed by visually disabled users on the web. International Journal of Human-Computer Studies 71, 11 (2013), 1013–1025.DOI
  53. [54]Ethan Wang, Zubair Shafiq, and Yash Vekaria. 2026. FP-Agent: Fingerprinting AI Browsing Agents. arXiv preprint arXiv:2605.01247 (2026).arxiv.org/abs/2605.01247
  54. [55]Gang Wang, Xinyi Zhang, Shiliang Tang, Haitao Zheng, and Ben Y Zhao. 2016. Unsupervised clickstream clustering for user behavior analysis. In Proceedings of the 2016 CHI conference on human factors in computing systems. 225–236.DOI
  55. [56]Junjie Wang, Yawen Wang, Mengzhuo Chen, Xiaofei Xie, Chunyang Chen, Fangwen Mu, Zhe Liu, and Qing Wang. 2026. A survey for llm agent trajectory analysis: From failure attribution to enhancement.DOI
  56. [57]Xu Wang, Benjamin Lafreniere, and Tovi Grossman. 2018. Leveraging community-generated videos and command logs to classify and recommend software workflows. In Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems. 1–13.DOI
  57. [58]Xinyuan Wang, Bowen Wang, Dunjie Lu, Junlin Yang, Tianbao Xie, Junli Wang, Jiaqi Deng, Xiaole Guo, Yiheng Xu, Chen Wu, et al. 2026. Opencua: Open foundations for computer-use agents. Advances in Neural Information Processing Systems 38 (2026), 139756–139806.arxiv.org/abs/2508.09123
  58. [59]Zora Zhiruo Wang, Yijia Shao, Omar Shaikh, Daniel Fried, Graham Neubig, and Diyi Yang. 2025. How do ai agents do human work? comparing ai and human workflows across diverse occupations. arXiv preprint arXiv:2510.22780 (2025).
  59. [60]Sarah J Waterson, Jason I Hong, Tim Sohn, James A Landay, Jeffrey Heer, and Tara Matthews. 2002. What did they do? understanding clickstreams with the webquilt visualization system. In Proceedings of the Working Conference on Advanced Visual Interfaces. 94–102.DOI
  60. [61]Tim Welschmechold, Nichola Abdo, Christian Dornhege, and Wolfram Burgard. 2019. Combined task and action learning from human demonstrations for mobile manipulation applications. In 2019 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS). IEEE, 4317–4324.
  61. [62]Eden Wu, Sonia Castelo, Yourong Liu, Cláudio T Silva, and Juliana Freire. 2026. AgentTrails: Towards Trust and Reuse for Agentic Tasks. arXiv preprint arXiv:2607.18816 (2026).arxiv.org/abs/2607.18816
  62. [63]Jingxu Xie, Dylan Xu, Xuandong Zhao, and Dawn Song. 2026. Agentsynth: Scalable task generation for generalist computer-use agents. In International Conference on Learning Representations, Vol. 2026. 155149–155177.arxiv.org/abs/2506.14205
  63. [64]Tianbao Xie, Danyang Zhang, Jixuan Chen, Xiaochuan Li, Siheng Zou, Ruisheng Cao, Toh J Hua, Zhourjun Cheng, Dongchan Shin, Fangyu Lei, et al. 2024. Os-world: Benchmarking multimodal agents for open-ended tasks in real computer environments. Advances in Neural Information Processing Systems 37 (2024), 52040–52094.arxiv.org/abs/2404.07972
  64. [65]Yingxuan Yang, Huacan Chai, Shuai Shao, Yuanyi Song, Siyuan Qi, Renting Rui, and Weinan Zhang. 2026. Agentnet: Decentralized evolutionary coordination for llm-based multi-agent systems. Advances in Neural Information Processing Systems 38 (2026), 107309–107336.DOI
  65. [66]Guanghua Zhang, Mohamed Adel Nagui Ahmed, Zhiming Hu, and Andreas Bulling. 2025. Summact: Uncovering user intentions through interactive behaviour summarisation. In Proceedings of the 2025 CHI Conference on Human Factors in Computing Systems. 1–17.arxiv.org/abs/2410.08356
  66. [67]Guanghua Zhang, Matteo Bortolotto, Zhiming Hu, Lei Shi, Mihai Bâce, and Andreas Bulling. 2023. Exploring natural language processing methods for interactive behaviour modelling. In IFIP Conference on Human-Computer Interaction. Springer, 3–26.arxiv.org/abs/2303.16039
  67. [68]Jian Zhao, Zhicheng Liu, Mira Dontcheva, Aaron Hertzmann, and Alan Wilson. 2015. Matrixwave: Visual comparison of event sequence data. In Proceedings of the 33rd Annual ACM Conference on Human Factors in Computing Systems. 259–268.

Paper details

vlnc:W_c85x6z8q
Version
v1 · latest
Published
Fresh — added to the Hub within the last 30 days
Added
License
CC BY 4.0
Usage

views · 8

downloads · 1

A view counts once per browser per day, a download each time the PDF is requested.

  • Computer Science

Not recorded for this paper

View statement
Looking for related papers…
Contents