betaFeedback

WorksComputer Science

Controlling Proactive Multimodal AI: Separating Perception from Participation

Abstract

Proactive multimodal AI can watch a user's screen, listen to speech, track interaction, and decide whether to enter unfolding work. Little is known about how people want to control what such AI may perceive and how far it may participate. Informed by a 14-person Wizard-of-Oz study, we built SpotMe, a design probe that senses screen, speech, and interaction; chooses Silence, Ask, or Assist; and lets users separately configure what it may perceive, how far it may participate, and what types of support it may provide. In a 16-person study on a civic dashboard, participants first experienced proactive help, then inspected decisions to act or remain silent, and then configured SpotMe's perception and participation. We found that participants could keep all perception channels available while lowering participation to Ask or Silence; that exposing suppressed decisions made silence interpretable and contestable; and that seeing what SpotMe used led some to change how they clicked, highlighted, and spoke. We contribute an empirical account of inspecting and controlling proactive multimodal AI by decoupling perception from participation.

Separating perception from participation in proactive multimodal AI

Figure 1. Separating perception from participation in proactive multimodal AI. SpotMe lets users configure what it may perceive and, separately, how far it may participate—whether to stay silent, ask before helping, or assist directly. Example configurations show that users may keep perception broad while restricting participation, with an Audit that makes interventions and silent decisions inspectable.

CCS Concepts: • Human-centered computing → Empirical studies in HCI; Interactive systems and tools; • Computing methodologies → Artificial intelligence.

Authors’ Contact Information: Muawiz Feroze Khan, University of Illinois Urbana-Champaign, Urbana, Illinois, USA, [email protected]; Fatima Ali, University of Illinois Urbana-Champaign, Urbana, Illinois, USA, [email protected]; Yun Huang, University of Illinois Urbana-Champaign, Urbana, Illinois, USA, [email protected].

Additional Key Words and Phrases: multimodal large language models, proactive assistance, mixed-initiative interaction, human-grounded evaluation, dashboard exploration

Introduction

Proactive multimodal AI can watch a user’s screen, hear their speech, and read their input, then decide whether to enter work whose next step is still forming. Recent systems use such contextual signals to identify moments when assistance may be useful [10, 45, 53, 54]. In data exploration, they similarly monitor interaction context to offer guidance without waiting for a request [33, 72]. That participation is consequential during open-ended inquiry, where questions, interpretations, and next steps evolve in the interaction [67]: mistimed help can interrupt ongoing reasoning [33, 72], while task-level suggestions can steer what users notice or how they proceed [4]. Controlling such a system is therefore not only a matter of how often it speaks. Users may also need to bound what it may perceive and how far it may participate—axes that existing interfaces often fuse into a single “how proactive” choice.

Prior work has begun making proactive participation intelligible and configurable. Systems expose rationales, working context, or ongoing agent activity around visible interventions [34, 38, 53, 72], and others let users set how often or what kinds of support an agent may provide [32, 72]. Configurable sensing, separately, lets people bound what a system may capture. Yet these handles do not independently set perception versus initiative, and they rarely expose the system’s most common act: not intervening. Although recent systems represent silence or no-action as an explicit outcome, that decision is often internal or used for evaluation rather than shown to users with the evidence and rationale for withholding [9, 53, 70]. If silence is illegible, “don’t interrupt” can look like the system has stopped following—so people cannot keep it watching while locking its mouth. Little is known about how users want to control proactive multimodal AI along those axes during evolving inquiry, or how users draw on inspectable decisions, including decisions to remain silent, when bounding that control.

To investigate these questions, we used a three-phase approach in the context of open-ended exploration of a public civic-data dashboard on U.S. fatal police use of force. In Phase I, we asked how people experience proactive assistance during open-ended exploration and what they expect from such an assistant. We examined this through a formative Wizard-of-Oz study with 14 participants, which produced a working design vocabulary and initiative modes for the later probe, as well as expectations around making proactive decisions inspectable and configurable. In Phase II, we asked how these formative insights could be instantiated in a system that supports inspectable and configurable proactive participation. We then translated them into SpotMe, a Chrome-extension research prototype used as a design probe. SpotMe uses screen activity, think-aloud speech, and interaction traces to decide whether to choose Silence, Ask whether support is wanted, or Assist directly; makes visible interventions and selected suppressed decisions inspectable through an Audit; and lets users separately configure its perception, initiative, and support types. In Phase III, we asked three research questions: RQ1, how users judge proactive participation during evolving inquiry; RQ2, how users interpret, evaluate, and respond to decisions to intervene or remain silent when those decisions are made inspectable; and RQ3, how users configure perception, initiative, and support types and what considerations shape those choices. We examined this in a 16-participant study in which participants first experienced SpotMe’s proactive assistance, then inspected its interventions and suppressed decisions, and finally configured its future participation.

In summary, this work makes three contributions to proactive AI for evolving inquiry:

  • A conceptual decomposition of proactive AI control. Participants treated the control as two separable boundaries—what the system may perceive and how far it may participate—rather than as a single “how proactive”

continuum. Within that decomposition, we propose inspectable non-participation as a control option: users can examine not only why an AI intervened but why it withheld participation. Inspection supported evaluation and disagreement; changing what the system did next ran through subsequent configuration, not through inspection alone.

  • SpotMe, a probe that instantiates this relationship. SpotMe operationalizes independent controls over perception, initiative, and support type, together with an Audit of interventions and selected suppressed decisions. The artifact’s role is to make the conceptual decomposition available in interaction, not to introduce mixed-initiative, configurable sensing, or explanation as new mechanisms.

  • An empirical account of that relationship in use. In a 16-person probe study, participants could keep perception available while lowering initiative to Ask or Silence. Inspecting withheld decisions helped them interpret silence as a participation choice; some also adapted how they clicked, highlighted, or spoke so the system could read them—what we term legibility work. We report these as observations from a staged, think-aloud protocol, not as demonstrated long-term or naturalistic effects.

Related Works

Proactive AI Participation in Open-Ended Exploration

AI support for data exploration has often been user-invoked, including natural-language interfaces for analytical queries [24, 51, 57], LLM-based visualization authoring and correction [13, 17, 63], and dashboard assistants that respond to explicit requests [16]. Beyond this, data exploration systems have also explored mixed-initiative support [11, 25]. Mixed-initiative interaction treats initiative as shared between the user and system [2, 30, 31]. Within this broader model, proactive AI focuses on the system taking initiative: anticipating when assistance may be useful, initiating without an explicit user request, and targeting assistance toward the user’s current or emerging goals [7, 14, 26, 52].

Recent AI systems ground decisions about whether and how to participate in increasingly rich context about users and their activity. Proactive programming assistants use code and interaction state to identify support opportunities [10, 53], while other systems draw on conversational deliberation [46], desktop activity [47], working-memory estimates [54], or behavioral and physiological indicators of difficulty [45]. In data exploration, WebSeek uses interaction cues to decide when, what, and how to guide users [33], while ProactiveVA uses interaction behavior to anticipate analytical needs and initiate guidance [72]. These decisions are especially consequential during evolving inquiry, where users’ information needs and next steps may evolve as they interact with the data [67], and behaviors during sensemaking can resemble difficulty [40]. Initiative can therefore become task-level [4]: by interpreting a pattern or suggesting what to examine next, an AI may shape what users notice, how they interpret it, or where they explore next. We examine proactive participation during this kind of evolving inquiry through open-ended dashboard exploration, where users were not given a prescribed task or exploration path but developed their own questions and next steps while interacting with the data.

Making Proactive AI Participation Intelligible

Explanations and transparency have long been used to make AI systems more intelligible by revealing why an output occurred, what evidence informed it, or why one output was produced rather than an alternative [44]. Yet explanations do not necessarily improve judgment: they can increase trust or reliance without helping users verify AI behavior [5, 19]. and can also increase cognitive load [64, 66]. These limitations motivate explanations that support verification [23] and address users’ information needs rather than simply expose model internals [43].

Recent work has begun extending such intelligibility to proactive AI. Jain et al. co-designed explanations for actions taken by a proactive messaging agent [34], while proactive programming systems expose intervention rationales and working context [38] or make the agent’s ongoing activity and context visible [53]. Users found these forms of visibility useful, including concise rationales and previews in ProAIDE [38] and greater awareness of AI processes in Codellaborator [53]. Similar benefits appeared in data-focused systems: ProactiveVA users valued step-by-step visibility into the agent’s behavior [72], while QueryGenie users reported that its context-sensitive adaptations became more predictable with use [8]. Even so, other proactive systems explicitly reason about whether to contribute while still keeping that deliberation internal [46].

A distinct challenge arises when the participation decision results in silence as it produces no visible interaction for an explanation to attach to. Recent proactive systems increasingly represent no-action or silence as an explicit outcome, but these decisions are typically used internally or for evaluation rather than surfaced for users to inspect [9, 53, 70]. Therefore, little is known about how users make sense of proactive AI’s decisions not to act, or how exposing the evidence and rationale behind such decisions shapes their understanding of the system’s participation. SpotMe examines this gap by making both visible interventions and suppressed decisions inspectable.

Controlling Future Proactive AI Participation

Empirical work shows that decision control can improve outcomes that explanations alone do not reliably improve, underscoring that making AI behavior intelligible is not the same as letting users shape what it does next [66]. Human-centered AI and contestability likewise emphasize preserving human influence alongside automation [1, 28]. Even controls that do not causally change system behavior can increase users’ perceived control or power [50, 62]. Many proactive systems primarily give users ways to respond to a visible intervention. ProAIDE lets users apply, engage, dismiss, or ignore individual interventions [38], while Codellaborator supports immediate approval or redirection and adapts future intervention frequency from ignored suggestions rather than through an explicit user-set policy [53].

Some systems go further by giving users explicit configurations that shape future proactive behavior. Houde et al. let users configure whether a conversational agent participates reactively or proactively, how frequently it intervenes, and where its interventions appear [32]. In data exploration, ProactiveVA similarly lets users adjust intervention frequency and enable or disable support categories [72]; however, in both of these works, the controls do not allow users to independently configure what perception signals the system can use and how much initiative the system may take. Outside proactive AI, configurable sensing systems show that users can control what contextual information a system captures or uses [6, 42], showing that sensing itself can be made user-configurable.

Yet little is known about whether users treat what proactive AI may perceive and how far it may act as distinct boundaries, or how they configure these boundaries as their inquiry evolves, particularly as they inspect the system’s decisions to intervene or remain silent. We examine this distinction through separate controls over SpotMe’s perception, initiative, and support types.

Method Overview

We used a three-phase approach in the context of a public civic-data dashboard of U.S. fatal police use-of-force incidents (Figure 2). Phase I was formative: using a Wizard-of-Oz study, we examined participants’ experiences with proactive assistance during open-ended exploration and their expectations of such an assistant. Analysis of these interactions and interviews produced an empirically grounded design vocabulary and initiative modes for the later probe, and informed design goals around inspectability and control. Phase II translated these findings into SpotMe, a Chrome-extension research prototype used as a design probe. Phase III then examined how participants experienced SpotMe’s proactive assistance, inspected its rationales for decisions to act or remain silent, and configured its future behavior.

Method overview

Figure 2. Method overview. We used a three-phase approach in a public civic-data dashboard about U.S. police use-of-force incidents. Phase I used a formative Wizard-of-Oz study to derive a working design vocabulary, initiative modes, and expectations for inspectability and control. Phase II translated these findings into SpotMe, a proactive assistant and design probe. Phase III examined how participants experienced SpotMe, inspected its decisions to act or stay silent, and configured what it could perceive and how it could participate.

Ethical Considerations

Our study protocol was determined exempt by the University of Illinois, Urbana-Champaign Institutional Review Board (IRB). All participants reviewed the IRB-approved study information and provided informed consent before participation.

Before each session, participants reconfirmed consent for Zoom screen and audio recording and transcription; webcam use was optional, although some chose to keep their cameras on. Because the study involved exploration of a public dashboard concerning fatal police use of force, participants were reminded that they could skip any question, take a break, or stop participating at any time. Phase III was conducted on the researcher’s computer through Zoom remote control, so participants were not required to expose their personal desktops, files, or browsing activity during the study.

SpotMe used screenshots, think-aloud speech, and interaction activity as transient context during each session; this context was cleared before the next participant. Study recordings and research data were stored on university-managed cloud storage with access restricted to the research team. Before analysis, directly identifying information was removed from the transcripts, and analyses were conducted using anonymized transcripts, survey responses, and session logs. Demographic information such as age was reported in broad categorical ranges rather than exact values to further reduce the risk of re-identification.

Phase I: Formative Wizard-of-Oz Study

Study Procedure

Phase I was formative: we sought to understand participants’ experiences with proactive assistance during open-ended dashboard exploration and their expectations of such an assistant. We used an exploratory Wizard-of-Oz study to expose participants to situated proactive assistance and derive design goals, rather than to evaluate whether particular interventions were appropriate or effective. We recruited 14 participants (8 women, 6 men; ages 18–64) through purposive and snowball sampling, with backgrounds spanning academia, journalism, technology and information management, government, police crime analysis, and human rights and equity. Participants received $20/hour.

During 45–60-minute remote think-aloud sessions, participants shared their screens, explored the dashboard, and worked toward an insight they considered meaningful. Throughout their exploration, participants were told they were interacting with an AI assistant; in practice, a researcher serving as the Wizard monitored their shared screen, dashboard interactions, and think-aloud speech in real time, decided when to intervene, and sent messages through Zoom chat. The Wizard had familiarized themselves with the dashboard through prior cognitive walkthroughs and either composed interventions directly or, when useful, consulted GPT-4o to draft responses that were reviewed and edited before being sent [69]. Sessions ended with a semi-structured interview about participants’ reactions to, experience with, and expectations from the assistant.

Data and Analysis

Our data comprised full-session transcripts containing participants’ think-aloud and interview responses, timestamped Wizard chat logs, and session recordings. We organized the interaction analysis around Wizard interventions. For each intervention, we used the transcript and/or video to characterize the participant’s state immediately beforehand and their verbal or behavioral response afterward. We excluded user-initiated reactive exchanges, yielding 70 proactive Wizard episodes. Of these, 38 showed positive verbal or behavioral uptake of the support.

We distinguished what kind of contribution an intervention made from whether and how it should have entered. The eight support types concern the former. Because the later probe needed a working set of contributions it could offer, we derived those types primarily from the 38 episodes in which participants took the support up in context. Positive uptake was evidence that a form of assistance was recognizable and usable during unfolding inquiry, not that the Wizard’s action was optimal. The remaining 32 episodes were not discarded: they were more informative for initiative and control than for expanding the type vocabulary. Non-uptake often meant that a contribution of an already-identifiable kind was mistimed, unwelcome, or should have been asked or withheld, rather than that the Wizard had used an unrecognized kind of support. Coding types equally from rejected or ignored interventions would have conflated an unwelcome or mistimed contribution with a missing support type. Those episodes, together with the interviews, therefore informed the Silence/Ask/Assist modes and the inspectability and control goals, rather than serving as equal input to the type set. We treat the resulting eight types as an empirically grounded design vocabulary for this probe, not as a comprehensive taxonomy of proactive support.

One researcher developed initial codes by comparing the participant state, Wizard intervention, and subsequent response across these episodes. A second researcher reviewed the episode coding and type assignments, and both researchers iteratively refined the vocabulary through discussion. We analyzed two dimensions separately: (1) recurring user situations and the corresponding kinds of support the Wizard provided, and (2) the form that proactive initiative took in the interaction. This analysis produced eight working support types and three proactive initiative modes, which we report below. Separately, the same researchers thematically analyzed the semi-structured interview portions of the transcripts to identify broader expectations about proactive assistance.

Formative Findings and Design Goals

Recurring support situations. Across the positive-uptake episodes, we derived working support types. We grouped the corresponding forms of assistance into eight working types that became the probe’s design vocabulary: Orientation [15, 16, 29, 73], Interaction Repair [21, 58, 72], Dashboard Scope Disclosure [3, 16, 37], Data Meaning and Provenance [15, 16, 29, 56], Misreading Correction [72], Interpretation [16, 27, 29, 73], Exploration Expansion [68, 71, 72, 73], and Social Repair [3, 12, 18]. These types characterize the kind of user situation the assistance addressed, not the correctness or optimality of any single Wizard response, and not a complete space of proactive support. Table 1 grounds each type in a representative episode, showing the participant’s state before the intervention and the Wizard’s response.

Support categoryUser state beforeWizard intervention
Orientation: Navigating the dashboard to find relevant views, controls, or data entries.P8 tried to locate Illinois while the table was sorted by county names and described the state ordering as “jumbled.”Assist. Oriented P8 to the table’s current county-name sorting and directed them to sort by State instead.
Interaction Repair: Recovering from an unintended interface, filter, or navigation state.P9 encountered an unwanted demographic-filter state and was trying to return to the full view.Assist. Repaired the filter state by indicating that clicking the selected demographic again would clear it.
Dashboard Scope Disclosure: Clarifying what the dashboard can or cannot support.P4 repeatedly tried to select multiple counties for direct comparison.Assist. Disclosed that the dashboard did not support direct multi-county comparison.
Data Meaning & Provenance: Explaining labels, measures, sources, or data scope.P10 expressed uncertainty about the meaning of the “API” demographic label.Assist. Defined API as Asian or Pacific Islander.
Misreading Correction: Correcting a displayed value or chart reading the user appeared to misread.P12 read the chart as showing 70% of incidents involving Black people, while the displayed value was 35%.Assist. Corrected the displayed percentage from 70% to 35%.
Interpretation: Helping users make sense of visible values, patterns, or comparisons.P12 read fragmented values aloud while trying to make sense of the visualization.Ask. Offered interpretation support by asking whether help making sense of the data was wanted.
Exploration Expansion: Suggesting a relevant next view, filter, or comparison.P6 repeatedly explored Illinois but had not visited a relevant Illinois-specific view.Assist. Expanded exploration by suggesting the additional Illinois-specific view.
Social Repair: Repairing earlier assistance after miscommunication or unhelpful guidance.P2 reported that earlier navigation guidance was unhelpful after interpreting “top” as the top of the dashboard rather than the webpage.Assist. Repaired the earlier guidance by clarifying the location of the Data Dashboards control and the Race/Sex Breakdown view.

Table 1. Table 1. An empirically grounded design vocabulary for the probe, illustrated with representative positive-uptake episodes. Each row defines a working support type and grounds it in one episode from the 38-episode positive-uptake set. These types are not a comprehensive taxonomy of proactive support, and cases are illustrative rather than estimates of optimality.

Forms of proactive initiative. The Wizard also varied how proactive assistance entered the interaction. Wizard-initiated messages either Asked whether support was wanted or Assisted directly. Because the Wizard continuously followed the live interaction while intervening only at selected moments, non-intervention constituted a third state, Silence. Ask created an opportunity for participants to accept or decline support before it was provided. A declined Ask case illustrates the distinction: when P14 remarked “whatever that means” while reading “Fatal Encounters,” the Wizard asked whether clarification was wanted; P14 declined, responding, “No, I don’t need help. Thank you for asking.” Consistent with mixed-initiative accounts of systems deciding whether to defer, seek information, or act [2, 30], our formative results also yielded three modes of proactive initiative—Silence, Ask, and Assist—while explicit participant requests remained a separate reactive path. This informed DG1: Initiative Modes: support different modes of proactive involvement through Silence, Ask, and Assist.

Inspectability and user control. The post-session interviews surfaced expectations about how such proactive decisions should be made visible and configurable. Participants wanted to understand what the assistant had attended to and why it intervened. P10 said they would feel more confident if they were “able to see what SpotMe actually looked at” and understand why it suggested a particular action, while P6 questioned whether an intervention had been shaped by their voice, cursor movements, or both. These reflections informed DG2: Inspectable Decisions: make proactive decisions visible and understandable by exposing what informed an intervention and why the assistant chose to act. Participants also wanted influence over the assistant’s ongoing observation and involvement. P14 proposed being able to opt out when moving across tabs or screens, while P9 emphasized that desired involvement could change within the same session: “At certain points, you may want [it], at certain points, you may not.” These reflections informed DG3: User Control: let users control what the assistant can observe and how it may initiate support.

Phase II: Designing SpotMe

SpotMe instantiates the three design goals from Phase I: it supports multiple initiative modes, makes decisions to act or stay silent inspectable, and lets users bound what the assistant can perceive and how it may initiate support.

SpotMe ran as a Chrome extension connected to a FastAPI backend. To support a situated hypothesis of the user’s activity and objective, the extension captures a 30s rolling context window of perception signals: 10 sampled screenshots of dashboard activity, interaction logs [39, 59], and speech transcribed using Gemini 2.5 Flash-Lite. The backend combines this context window with page-specific dashboard knowledge and session memory, then passes the assembled context to GPT-5.4 to return a structured decision about whether and how SpotMe should participate.

To support near-real-time use while preserving overlapping context, SpotMe schedules an initiative evaluation every 15s, sending the preceding 30s window each time rather than only the 15s of newly captured activity [65]. We selected the 30s/10-screenshot context configuration through calibration against the positive-uptake episodes from Phase I, treating them as empirically observed opportunities for proactive support rather than ground-truth labels of the correct intervention. This configuration preserved opportunity recognition while keeping decision latency mostly compatible with the 15s evaluation boundary (M=13.2sM = 13.2\mathrm{s}, Mdn=13.2sMdn = 13.2\mathrm{s}); when a prior decision exceeded the interval, SpotMe skipped the stale boundary rather than queueing delayed interventions.

Across the rest of this section, we use a scenario adapted from positive-uptake episodes to illustrate how the design goals are instantiated in SpotMe. In the scenario, Mikayl explores the dashboard with SpotMe (Figure 3).

SpotMe’s inspectable and configurable initiative during dashboard exploration

Figure 3. SpotMe’s inspectable and configurable initiative during dashboard exploration. The figure illustrates an example scenario adapted from positive-uptake episodes in Phase I. (A) The user misreads a visible race breakdown value. (B) SpotMe directly corrects the misreading and provides an in-line rationale. (C) The audit panel exposes the perception signals and tradeoff behind the intervention. (D) The user disables interpretation support in the configuration panel; SpotMe’s eight support categories are organized into three interface groups for configuration. (E) A later interpretation opportunity is withheld and logged as a suppressed decision. The scenario is illustrative rather than a claim of typicality or optimality, or a complete taxonomy of proactive support.

Deciding When and How to Participate

SpotMe evaluates each window through a structured decision policy (Appendix § 12). The policy first routes direct user questions to a Reactive path; all other windows are treated as proactive decisions. For proactive decisions, it forms a concise situation summary using the available perception signals: what is happening, what the user appears to be focused on, and which signals matter. It then identifies whether any useful contribution is possible and, if so, classifies it using support categories from Phase I.

While exploring the race patterns in the dashboard, Mikayl says, “So 70% of these incidents involved Black people.” The visible chart, however, shows Black civilians as 35%. SpotMe hypothesizes that Mikayl is interpreting the race breakdown and may have misread a percentage, classifying the possible contribution as Misreading Correction.

If no specific contribution can be identified, SpotMe chooses Silence [53]. If a contribution is possible, the policy evaluates whether SpotMe should initiate using four considerations adapted from mixed-initiative reasoning: evidence strength (how well the context supports the situation and factual basis), expected benefit (what intervening could add), cost of silence (what may remain unresolved), and cost of intervention (the disruption or harm if SpotMe’s interpretation is wrong or unwanted)[30]. Guided by the same considerations, SpotMe now decides how to initiate: it uses Ask when it has identified a possible support opportunity but needs confirmation of the user’s situation before helping, and Assist when the support is sufficiently grounded.

Since Mikayl’s spoken percentage and the visible percentage clearly conflict, the evidence is strong, Ask is unnecessary, and choosing Silence could leave the mistaken percentage unresolved. Thus, SpotMe chooses Assist and displays: “...the Race chart shows Black civilians as 35%, not 70%.”

Making Decisions Inspectable: Audit Panel

SpotMe makes proactive interventions inspectable through the Audit panel (Figure 3C), which presents a user-facing account of the evidence and justification behind each decision rather than a trace of the model’s latent reasoning [43, 61]. Each audit entry summarizes what each perception signal captured, what SpotMe noticed, what it thought was happening, the support it considered, the tradeoff between staying silent and acting, and why it chose its initiative mode. Visible interventions also include a brief rationale in the main panel (Figure 3B).

Mikayl opens the audit entry for SpotMe’s correction. The entry shows that SpotMe connected three perception signals: the visible Race chart labeled Black civilians as 35%, Mikayl had spoken “70%,” and his cursor activity was near the relevant chart segment. Mikayl also sees SpotMe’s situation summary and the tradeoff behind intervening: staying silent could leave an incorrect percentage unresolved, while acting was low-risk because the evidence for a misreading was strong.

SpotMe also makes selected silent decisions inspectable through the Audit panel (Figure 3E), including cases where SpotMe found no support opportunity and cases where it identified a possible contribution but withheld it. To avoid repeated audit entries, SpotMe does not log successive evaluations that reflect the same unchanged context. For withheld decisions, the audit explains whether SpotMe stayed silent because the evidence, expected benefit, cost of acting, or user configuration did not justify a visible response.

Bounding Future Participation: Configure SpotMe Panel

SpotMe provides controls that let users bound its future participation along three dimensions. Perception controls determine which perception signals are available to SpotMe for future decisions. Initiative controls let users set an initiative ceiling that limits how far SpotMe may participate. We denote individual decision modes as Silence, Ask, and Assist, and the corresponding user-configured ceilings as [Silence], [Ask], and [Assist], named after the highest mode each permits:

  • [Assist] permits Silence, Ask, or Assist.

  • [Ask] permits Silence or Ask.

  • [Silence] permits only Silence.

Users can also turn individual support categories on or off.

At runtime, the frontend sends the current user configuration with each rolling context window. The backend first applies any perception configuration before model inference: disabled perception channels are excluded from the model input. The backend then sends the remaining context together with the user configuration to the model. SpotMe reasons within the enabled support types and initiative ceiling, withholding or reducing an intervention when a support type is disabled or its preferred initiative mode exceeds the ceiling. As a final guardrail, the backend independently enforces these limits on the returned decision and records both the preferred initiative mode and the final permitted mode.

Mikayl disables Interpretation support in SpotMe’s configuration panel. Later, while exploring Illinois, he says, “...I’m going to Pulaski because I’ve heard a lot about Pulaski rumors...” SpotMe identifies an Interpretation opportunity: the outside rumor could shape how he reads the county-level data. Although SpotMe would have preferred Assist to support a more cautious interpretation, this support is disabled, so it withholds the intervention and records Silence in the Audit panel.

Phase III: User Study

To examine how users experienced, inspected, and configured SpotMe’s proactive participation, we asked:

  1. RQ1. How do users judge proactive participation during evolving inquiry?

  1. RQ2. How do users interpret, evaluate, and respond to decisions to intervene or remain silent when those decisions are made inspectable?

  1. RQ3. How do users configure perception, initiative, and support types, and what considerations shape those choices?

Participants

We recruited 16 participants with varied professional and academic backgrounds through direct outreach, personal and professional networks, and snowball sampling. We purposively sought variation in participants’ technical backgrounds and prior experience with data, dashboards, and AI. Participants were required to be fluent in English and able to verbalize their thoughts while interacting with the dashboard and SpotMe. Participants were compensated $20/hour. Table 2 summarizes participant characteristics.

IDAgeGenderOccupation / FieldData/Dashboard UseAI FamiliarityAI Attitude
P125–34FAdmissions ProfessionalWeekly45
P245–54MProfessor of Graphic DesignYearly62
P325–34MCustomer Experience Manager (Public Safety Tech)Weekly65
P425–34MMicrobiology & Biochemistry ResearcherDaily56
P535–44MLaw Enforcement OfficerDaily67
P655–64FAssociate Director / Graduate InstructorMonthly54
P725–34MResearch Evaluation SpecialistDaily77
P835–44FMultimedia JournalistMonthly56
P925–34MAdjunct Instructor / MBA StudentWeekly42
P1045–54FAdministrative AideWeekly47
P1125–34FBusiness Analyst (Data / Analytics)Daily66
P1225–34MOffensive AI Security EngineerMonthly77
P1325–34MFinance LeadDaily34
P1425–34FProject Manager (Private Equity)Daily65
P1525–34FContent Creation (Social Media)Weekly45
P1625–34FPolitical Science ResearcherWeekly36

Table 2. Participant characteristics. Data/dashboard use summarizes how often participants reported working with or interpreting data, charts, or dashboards: Daily = daily or almost daily; Weekly = a few times a week; Monthly = a few times a month; Yearly = a few times a year. AI familiarity ranges from 1 (not at all familiar) to 7 (extremely familiar), and AI attitude ranges from 1 (very negative) to 7 (very positive).

Study Procedure

We conducted approximately 90-minute remote think-aloud sessions over Zoom. The dashboard and SpotMe ran on the researcher’s computer, with participants interacting through Zoom remote control. Before each session, SpotMe was reset to a common starting state: all perception channels enabled, all support categories enabled, and [Assist] as the initiative ceiling. As shown in Figure 4, sessions progressed through three stages: participants first experienced SpotMe’s proactive assistance, then explored the Audit panel, and finally configured SpotMe. Although the Audit and

Phase III study procedure

Figure 4. Phase III study procedure. Participants first experienced SpotMe’s proactive assistance, then explored the Audit panel to review proactive and suppressed decisions, and finally configured SpotMe. Participants then completed a post-study questionnaire followed by a semi-structured interview.

Configure SpotMe panels were present throughout the session, moderators introduced them only at their corresponding stages. We staged these features sequentially so participants could first form their own experience of SpotMe’s proactive behavior, then inspect how those decisions were made, and finally use that understanding when reasoning about how they wanted SpotMe to behave in the future.

During Stage 1 (5–10 min), participants explored the dashboard with SpotMe active and reflected on how they thought it was working and what they wanted to work differently. In this stage, participants could also ask SpotMe direct questions; later stages restricted reactive use to focus on inspecting and configuring proactive decisions. During Stage 2 (15–20 min), we introduced the Audit panel; participants reviewed proactive and suppressed decisions, compared SpotMe’s rationale with their own interpretation, and continued exploring with the Audit available. During Stage 3 (20–25 min), we introduced controls over perception, initiative, and support types. Participants predicted the effects of the controls and completed two change–observe–reflect cycles, testing one perception setting and one initiative setting; support-type settings remained available for participants to adjust at their discretion. Finally, they selected a configuration they imagined wanting if SpotMe were installed on their own computer. We treat that selection as situated configuration reasoning after a roughly 90-minute session on the researcher’s machine, not as a measure of stable preference. Participants then completed a post-study questionnaire (Appendix § 11) containing study-specific items on their experience with SpotMe, alongside adapted measures of trust from S-TIAS [49] and usability from UMUX-LITE [41], followed by a 20–25-minute semi-structured interview about how their experience changed across the session, their reactions to SpotMe’s rationale behind proactive and suppressed decisions, and the considerations shaping their configuration choices.

Data and Analysis

Our dataset comprised anonymized full-session transcripts of participants’ think-aloud interactions and semi-structured interviews, full-session Zoom recordings, SpotMe session logs of proactive, reactive, and suppressed decisions, post-study questionnaire responses, and participant background information collected during recruitment. Transcripts served as the primary material for a hybrid deductive–inductive thematic analysis [22]; logs and recordings were consulted when needed to verify the context surrounding an excerpt. One researcher conducted all sessions and the second co-conducted most. Analysis began during data collection with an in-depth review of two recently completed cases, using the research questions and study stages as an initial scaffold while also marking salient observations outside those categories. The primary researcher then coded all 16 transcripts in successive batches, while the second researcher reviewed the evolving codes and interpretations; regular discussions were used to refine the analysis, consider alternative readings, and revisit earlier cases.

Although coding initially followed the three study stages, we did not retain the stages as final analytic categories because later stages incorporated features introduced earlier, and participants often referred back to earlier interactions when reviewing the Audit and deciding how to configure SpotMe. We instead used within-case analysis to trace participants’ experiences across the full session and cross-case comparison to identify recurring patterns and meaningful contrasts. During writing, the research team reviewed the developing themes and returned to the transcripts to refine or qualify interpretations. Following HCI guidance that coder agreement is not a universal criterion of qualitative rigor [48], we used collaborative review rather than inter-rater reliability. Questionnaire responses were analyzed descriptively using means, medians, and standard deviations to contextualize the qualitative findings rather than support inferential claims.

Findings

Proactive Assistance Was Judged Relative to Users’ Unfolding Reasoning

Across the session, participants evaluated SpotMe’s proactive assistance not only in terms of how much help it provided, but also in relation to where they were in their own unfolding reasoning. When SpotMe’s participation aligned with their ongoing reasoning, participants described it as timely or validating: P5 appreciated receiving “a hint of what you probably... need at the point in time,” while P6 described its assistance as “helpful validation” that they were “on the right track in terms of my thinking.” At other moments, SpotMe entered before participants felt ready for assistance: P7 described feeling “pressure” and “self-consciousness” when SpotMe responded while they were still forming a thought, while P10 felt it was already “a step ahead of me,” responding “quicker than I think I’m done with my thoughts.” Conversely, SpotMe could also fall short of how far ahead participants wanted it to anticipate: P12 wanted it to think beyond the immediate next move—to the “next step to the next step.”

These moment-to-moment differences occurred alongside generally positive overall ratings: 15 of 16 participants rated SpotMe’s assistance as aligned with what they were trying to do at 5–7 (M=5.69M = 5.69, SD=0.79SD = 0.79; Mdn=6Mdn = 6), while 13 of 16 rated the statement that SpotMe generally intervened at appropriate moments at 5–7 (M=5.63M = 5.63, SD=1.20SD = 1.20; Mdn=6Mdn = 6). Thus, participants generally saw SpotMe as aligned with what they were trying to do, while their qualitative accounts reveal more variation in judgments about when its participation was appropriate.

Together, these accounts show that participants experienced SpotMe’s participation as aligned, premature, or insufficiently anticipatory relative to their own unfolding reasoning. Thus, participants’ judgments of proactivity were not reducible to wanting more or less assistance; they also reflected whether SpotMe’s judgment that a moment warranted participation corresponded with their own sense of where they were in their reasoning.

Inspecting SpotMe’s Decisions to Act or Stay Silent

Participants came to recognize silence as a deliberate participation decision. Before inspecting SpotMe’s Audit, some participants interpreted its silence ambiguously: they could not tell whether SpotMe had deliberately chosen not to participate or had simply not yet acted. P12 recalled interpreting one silent period as delay, wondering, “why is it taking so long? What is it even thinking about?” P14 similarly described initially reading silence as possible lag or as SpotMe waiting for a direct question, but after reviewing the Audit described SpotMe as having been continuously deciding whether to participate. P16 made the same distinction, moving from assuming that SpotMe was “just kind of waiting to intervene” to describing its silence as “more of, like, a calculated decision.” All 16 participants rated the statement that seeing decisions where SpotMe stayed silent helped them understand how it was making proactive decisions at 5–7 (M=6.44M = 6.44, SD=0.73SD = 0.73; Mdn=7Mdn = 7). Across these accounts, participants came to understand that staying silent could itself be a deliberate decision about whether to participate.

Understanding SpotMe’s decision did not necessarily mean agreeing with it. When participants inspected SpotMe’s stated rationale, they could consider both how SpotMe had interpreted the moment and whether its resulting decision to intervene or stay silent matched what they wanted. P11 had initially wanted SpotMe to confirm that they had successfully completed a step, but after inspecting the suppressed decision reconsidered that silence, describing not being “spoon fed” as “a good thing in retrospect.” P6 understood a similar rationale—that SpotMe stepped in when they appeared lost but “stays silent” once they had found what they needed—yet still felt that confirmation that they were “on the right spot” would have been more helpful. Agreement could also vary across decisions for the same participant: P3 accepted one suppressed decision but responded to another, “Don’t stay silent next time,” while P9 similarly considered one suppression appropriate but challenged another because assistance could have helped them reach an answer sooner. Participants could also agree with the decision while questioning part of its justification: P15 thought SpotMe correctly noticed their misuse of the word “rate” and was right to intervene, but disagreed with part of its explanation for why the intervention was warranted. All 16 participants rated the statement that SpotMe’s explanation helped them understand why it intervened at 5–7 (M=6.50M = 6.50, SD=0.73SD = 0.73; Mdn=7Mdn = 7). Together, these accounts show that understanding SpotMe’s stated rationale did not determine whether participants considered the resulting participation decision appropriate: they could understand its judgment and still agree, disagree, or question particular parts of its justification.

Evaluating a decision and changing future behavior were separate choices. Participants distinguished evaluating a specific participation decision from deciding whether that evaluation should shape SpotMe’s future behavior. P3 wanted to mark an Audit decision as “a good decision, or not a good decision,” and, after disagreeing with one suppressed decision, tell SpotMe “Don’t stay silent next time,” allowing that feedback to shape how SpotMe handled similar moments later. P12 wanted these choices kept separate: after noticing that an earlier negative comment appeared to influence a later suppressed decision, they did not want a reaction in one interaction to automatically shape SpotMe’s behavior in subsequent situations. Together, these contrasting preferences show that evaluating an individual participation decision and allowing that evaluation to carry forward into future behavior were distinct choices.

Seeing what SpotMe sensed did not necessarily explain how it inferred users’ states or needs. Even when participants could make sense of a proactive or suppressed decision, some still questioned how SpotMe moved from what it sensed to an inference about their goal, state, or need (P4, P5, P6, P7, P10). P5 articulated this distinction directly: mouse-and-keyboard information showed simply “the actions,” while they wanted to understand “how it’s predicting and interpreting behavior.” While reviewing an Audit entry in which SpotMe treated their behavior as evidence of possible confusion, P6 asked, “How would it know I’m confused?”, speculating whether the inference came from “how much time I am spending per click” or whether their cursor moved in a “calm manner and not being sporadic and all over the place.” This uncertainty was distinct from whether participants agreed with the resulting decision: in another case, P6 agreed that SpotMe should remain silent but was still unsure how it had inferred that they were already progressing without needing help. Thus, the missing information was not necessarily more visibility into what SpotMe sensed, but the connection between that evidence and the goal, state, or need SpotMe inferred when deciding whether to intervene or stay silent. These questions arose even though participants generally reported that the exposed information itself was easy to understand: 15 of 16 rated the statement that SpotMe’s rationale and Audit information was easy to make sense of at 5–7 (M=6.13M = 6.13, SD=0.89SD = 0.89; Mdn=6Mdn = 6). Together, these accounts distinguish understanding SpotMe’s presented evidence from understanding the inference built from it: participants could see what SpotMe observed and still want to know why those observations indicated that they were confused, progressing, or in need of help.

Controlling How SpotMe Perceived and Participated

Controlling SpotMe’s Initiative: Assist, Ask, or Silence

We refer to participants’ initiative settings by the maximum participation SpotMe was allowed: [Assist] allowed SpotMe to assist directly, ask first, or remain silent; [Ask] allowed it to ask first or remain silent; and [Silence] restricted it to remaining silent. Participants’ final preferences varied: seven selected [Assist] (P1, P2, P3, P5, P7, P8, P13), four selected [Ask] (P6, P10, P11, P12), and two selected [Silence] (P9, P15), while three others (P4, P14, P16) described context-dependent preferences that did not map cleanly onto a single setting. All 16 participants rated the statement that they understood how changing SpotMe’s allowed initiative would affect its behavior at 5–7 (M=6.19M = 6.19, SD=0.75SD = 0.75; Mdn=6Mdn = 6).

[Assist]: Allowing direct assistance did not mean wanting continuous intervention. Participants who preferred [Assist] gave SpotMe discretion to provide direct help when warranted, while still expecting it to ask or remain silent at other moments. For some, that discretion was especially valuable during unfamiliar work: P2 wanted SpotMe to “jump in” when unfamiliar data became overwhelming, while P3 wanted the “maximum amount” of help while the dashboard was still new. Others preferred [Assist] to avoid repeated approval: P7 thought responding to Ask each time under [Ask] could become “tedious and annoying”; with [Assist], they could instead ignore help they did not need. P8 similarly wanted SpotMe as an “active team player,” while emphasizing that they could “choose or not to listen” to its contributions. Importantly, allowing [Assist] did not imply wanting frequent direct intervention. P1 valued that SpotMe had not felt overwhelming and stayed quiet while they were making progress, while P13 said they would switch to [Ask] if it began “pinging me over and over again.” Preferences could also shift with context: P4 preferred [Assist] for unfamiliar topics but [Ask] when they felt confident or wanted to think something through themselves; P16 similarly preferred [Assist] when first encountering a dashboard or looking for something specific, and [Ask] after becoming familiar with it. Across these accounts, [Assist] represented permission for SpotMe to act directly when useful, not a preference for constant intervention: participants still expected SpotMe to exercise restraint, and some preferred a lower initiative setting as their familiarity or desire to reason independently increased.

[Ask]: Asking first preserved space for users’ own reasoning. Participants who preferred [Ask] wanted SpotMe to seek permission before providing help, preserving space for them to continue their own reasoning first. P6 emphasised this need to preserve their own thought process: “sometimes you do need to complete your thought process. You need your human mind to work.” P10 similarly wanted to “explore a little bit on myself” before receiving help and valued being asked first so they could decline information they did not want or need. P12 described this permission boundary directly: under [Ask], “I allowed it to intervene when I wanted it to.” P11 likewise preferred [Ask] because they wanted to “continue thinking for myself.” For example, P11 used later review of suppressed decisions as a reason for choosing [Ask]: knowing they could later review what SpotMe had considered “really helped me choose it” when moving from [Assist] to [Ask]. Across these accounts, [Ask] functioned not as a rejection of AI assistance, but as a boundary around when that assistance could enter the inquiry: participants could preserve room to reason independently while retaining the option to receive help when they were ready.

[Silence]: Restricting proactive entry shifted initiation back to the user. P9 selected [Silence] as their final default, specifically because they did not want SpotMe proactively entering their exploration. They expected to understand most of the dashboard themselves and said they would use SpotMe “as, like, a regular AI chatbot,” seeking help only when they chose to initiate it. P15 described a similar user-initiated boundary: SpotMe could continue observing and processing their activity under [Silence], but “I have to initiate when I need help.” They nevertheless imagined an exception to this preference: although they generally wanted to initiate help themselves, they still wanted SpotMe to step in if they were “completely off-base, completely wrong, completely misinterpreting the data” or “going in the complete wrong direction.” For P15, then, [Silence] was not an absolute boundary: the severity of a potential misreading could change when they considered proactive intervention appropriate. [Silence] could also support deferred review: P6 imagined using SpotMe “as a study tool,” working through a task without proactive intervention and then reviewing its suppressed decisions afterward. P11 similarly wanted SpotMe “on the back end” during discussions with teammates, then to return to the Audit afterward and inspect “what it was thinking.” Across these accounts, [Silence] shifted control over when assistance entered back to the user while still allowing SpotMe to remain available in the background, either for later user-initiated help or retrospective inspection.

Controlling What SpotMe Could Perceive

Before configuring SpotMe’s perception channels, participants had inspected Audit entries showing how Eyes, Ears, and Mouse+Keyboard contributed to its decisions. When considering whether to restrict a channel, participants reasoned about what context SpotMe would lose and whether they would need to communicate that context through another channel. This understanding was also reflected descriptively in the questionnaire: 15 of 16 participants rated the statement that they understood how changing what SpotMe could perceive would affect its behavior at 5–7 (M=6.13M = 6.13, SD=0.96SD = 0.96; Mdn=6Mdn = 6).

Some participants adapted how they communicated based on what SpotMe could perceive. Some participants deliberately adapted their behavior to produce signals they expected SpotMe to use. P12 highlighted text “on purpose” because “I know it looks for my clicks,” while P13 said they “started asking leading questions” once they came to expect SpotMe to respond to questions. Participants also described shifting important context to another channel when a perception was unavailable. P16 saw Mouse+Keyboard as unnecessary because, if something was important, they would “just say it out loud.” P1 similarly preferred turning Ears off if typed chat were available, favoring deliberate typed requests because continuous listening felt “a little less intentional.” Adapting to the available channels could also create additional interactional effort. With Ears off, P11 explained, “I would need to ... be more active in what I am pressing and what I am scrolling through. And be more intentional with it. So I think it’s a little bit ... tiring for me to think about.” With Ears available, they described making themselves understood as requiring “significantly less effort.” These cases suggest that perception controls can change how people make themselves legible, sometimes at extra effort.

Enabling a perception channel did not necessarily mean allowing all information from that channel to shape SpotMe’s decisions. For some participants, the on/off perception controls were too coarse because they wanted boundaries over when sensed information should influence SpotMe, not only whether a channel was available. P3 wanted an indicator showing when SpotMe treated something in their speech as important, making visible not only that Ears was enabled but when particular speech was shaping SpotMe’s attention. P15 wanted a finer-grained boundary within Ears: if their husband entered the room and they began discussing something personal while the dashboard remained open, they wanted SpotMe to “actively filter out that information and only focus on the things that are relevant.” Together, these accounts suggest that perception boundaries could concern not only whether SpotMe had access to a channel, but when and which information from that channel should contribute to its participation decisions.

Keeping All Perception Channels On Did Not Mean Wanting More Initiative. Nine participants kept all three perception channels available; among them, five selected [Assist], three [Ask], and one [Silence]. For P8, broad perception and initiative aligned: they wanted Eyes, Ears, and Mouse+Keyboard available together with [Assist], describing SpotMe as staying “in sync” as an “active participant” in the exploration. P10, however, kept all three perceptions available so SpotMe could “detect what I need,” while choosing [Ask] because they wanted to “explore a little bit on myself” before receiving direct help. P11 made this separation explicit across contexts: although their default combined all three perceptions with [Ask], when imagining working with teammates they would switch to [Silence] but “wouldn’t change anything else,” because they still wanted SpotMe to retain context for later inspection in the Audit. P15 preferred keeping all three perception channels available with [Silence], allowing SpotMe to keep “observing, digesting, processing everything” while they continued thinking on their own. When they later asked for help, they expected SpotMe to already know where help was needed and provide a direct answer without requiring additional explanation. Thus, keeping all three perception channels available did not necessarily mean wanting SpotMe to intervene more: participants could give SpotMe access to the same ongoing context while still restricting its initiative to [Ask] or [Silence].

7.3.3 Interpretation and Exploration Expansion Raised Concerns About Steering Inquiry. Among the eight support types used in the probe’s Phase I design vocabulary (§ 4; Table 1), most participants left the types enabled, but Interpretation and Exploration Expansion prompted particular concern because they could shape how users understood the data and what they attended to next. P9 turned off Interpretation because having an AI proactively “tell me what to think about these patterns” felt “kind of risky.” P11 wanted SpotMe to provide interpretation and suggest additional directions for exploration, yet described such guidance as a “double-edged sword”: an exploration suggestion could create “a certain agenda of looking at a certain state” or “a bias of sorts against one that I might not have gone for otherwise.” P11 nevertheless wanted this support, emphasizing that it remained their responsibility to “continue to dig deeper and not just take what SpotMe is saying at face value.” Across these accounts, judgments about support categories were not only about whether assistance was useful; Interpretation and Exploration Expansion also raised questions about how proactive guidance might steer users’ sensemaking by shaping what they concluded from the data or where they looked next.

Discussion & Design Implications

Across the session, participants evaluated SpotMe’s proactive assistance not only in terms of how much help it provided, but also in relation to where they were in their own unfolding reasoning (§ 7). Thus, judgments of proactivity were not reducible to wanting more or less assistance. Our results suggested that controlling such a system is not only a matter of how often it speaks; users may also need to bound what it may perceive, how far it may participate, and what kinds of assistance are legitimate if it does. These axes often fuse into a single “how proactive” choice in existing interfaces.

Perceive ×\times Participate ×\times Scope

Prior work has given users explicit control over future proactive behavior, such as whether and how frequently an agent intervenes, where interventions appear, or which support types it may provide [32, 72]. Configurable sensing, separately, lets people bound what a system may capture [42]. Mixed-initiative systems have long considered whether to act, defer, ask, or remain silent, typically as an internal coordination policy [2, 30]. The novelty is not any of these mechanisms alone. It is their relationship: a system that may continue perceiving while being prohibited from participating, with that non-participation remaining inspectable, and with the kinds of help it may give bounded independently of both.

In this paper, we treat control over context-aware proactive AI as three boundaries that need not move together: perception, or what information may enter the AI’s model of the user’s activity; participation, or what actions the AI may take on that model, including remaining silent; and scope, or what kinds of assistance are legitimate if it does participate.

The distinctive cell in this space is high perception with low participation: sensing without speaking. In our study, SpotMe let participants configure perception channels separately from its initiative setting. Of the nine participants who kept all three perception channels available in this session, four selected a lower initiative setting, [Ask] or [Silence]. These situated choices suggest that wanting less proactive participation did not necessarily mean wanting SpotMe to follow the ongoing activity less closely. P10 kept all three channels available so SpotMe could “detect what I need,” while choosing [Ask] in order to explore first. P15 kept SpotMe “observing, digesting, processing” under [Silence] so that later, user-initiated help would already have context. P11 would switch only participation when working with teammates and “wouldn’t change anything else.” Together, these accounts show that participants could treat perception and participation as distinct boundaries in this session, rather than that users generally want independent controls.

Participation, in this space, functioned as permission rather than volume. Participants who selected [Assist] still expected SpotMe to ask or remain silent at other moments (P1, P8, P13). Those who selected [Ask] preferred room to finish a thought before help entered (P6, P10, P12). Those who selected [Silence] returned initiation to the user while SpotMe could remain available as a watcher or for later inspection in the Audit (P9, P15, P6, P11). These were Stage 3 selections after a single session on the researcher’s computer, not stable installation preferences.

Scope is a third bound, with narrower evidence. Two participants (P9, P11) treated Interpretation and Exploration Expansion as a different kind of participation from interface assistance such as repairing a filter: these types could shape what they concluded from the data or where they looked next. P9 turned Interpretation off because having an AI proactively “tell me what to think about these patterns” felt “kind of risky,” while P11 described exploration suggestions as a “double-edged sword” that could create an “agenda” or “bias.” When initiative becomes task-level [4], entering can steer inquiry, so bounding participation is more consequential than managing interruption alone.

The perception boundary was also coarser than channel on or off. Two participants (P3, P15) wanted to know when particular speech was being treated as important, or to filter bystander talk while the dashboard remained open. Enabling a channel did not necessarily mean allowing all information from that channel to shape SpotMe’s decisions. SpotMe instantiates this three-boundary space through independent controls over perception, initiative, and support type, together with an Audit of interventions and suppressed decisions. It is one instantiation of the space rather than the contribution itself, and the eight support types remain a working vocabulary for this probe rather than a comprehensive taxonomy of scope.

Design Implication 1: Treat perception, participation, and scope as separable control boundaries. Proactive AI should not infer desired participation from the perception a user allows, or reduce perception because the user chooses lower initiative. Allowing the system to keep watching is likewise not permission for every kind of assistance.

Inspectable Silence Makes the Split Usable

Prior work has explored ways to make the rationale behind visible intelligent-system behavior available for inspection [34, 38, 44, 72]. A distinct challenge arises when the participation decision results in silence, which produces no visible interaction for an explanation to attach to [9, 53, 70]. SpotMe additionally let users inspect selected moments when it chose to remain silent. For participants, this made silence interpretable as the outcome of a participation decision rather than as delay, waiting, or the absence of system activity. Before inspecting the Audit, three participants (P12, P14, P16) interpreted silent periods as lag or as SpotMe waiting to be asked. After reviewing withheld decisions, they described silence as a calculated decision about whether to participate. All 16 participants rated seeing silent decisions as helping them understand how SpotMe was making proactive decisions. Understanding did not necessarily mean agreeing: five participants (P3, P6, P9, P11, P15) disagreed with a decision or with part of its justification. Once non-intervention became inspectable, “don’t interrupt” could be evaluated as a stance rather than guessed at as an absence. That inspectability is what makes keeping perception available while locking participation usable as a control, rather than indistinguishable from the system having stopped following.

Making a participation decision inspectable did not, however, make every inference behind it intelligible. Five participants (P4, P5, P6, P7, P10) could see what SpotMe sensed and still questioned how those observations became an inference about their goal, state, or need—for example, how mouse movement indicated confusion. This matters for proactive AI because decisions about whether to participate may depend on inferred states the user has never explicitly declared.

Inspection and disagreement also did not themselves change what SpotMe did next. Contestability, in the sense of remaining able to influence subsequent system behavior [1, 66], required a further step. The Audit provided a basis for evaluating decisions; prospective control ran through later configuration, as when P11 moved from [Assist] to [Ask] after reviewing suppressed decisions, or P9 turned off Interpretation. P3 wanted a per-decision “Don’t stay silent next time” control that the probe did not provide, while P12 objected to an earlier reaction appearing to shape a later withhold without an explicit request that it do so. Where we refer to contestability, we mean this chain—inspection, then disagreement, then a bound on subsequent behavior—not inspection or disagreement alone. Participants used inspectability while reasoning about subsequent configuration; because every participant reached the Configure stage after the Audit, we cannot claim that inspectability causes users to configure AI differently.

Design Implication 2: Make non-participation inspectable, and separate evaluating a decision from changing future policy. Systems should expose withheld moments with evidence and rationale, not only visible interventions, and should let users evaluate a particular proactive or silent decision without assuming that this judgment automatically alters future behavior. If users want that judgment to generalize, the transition should be explicit.

Legibility Work as a Cost of Inspectable Perception

Prior work on algorithmic folk theories shows that people adapt their behavior based on how they understand algorithmic systems [20, 35, 55], and explainable-AI research has similarly reported users intending to change their behavior to provide AI systems with better inputs [36]. In SPotME, participants could inspect the perception channels and evidence informing proactive and suppressed decisions rather than relying only on assumptions about what the system could observe. After they learned what SPotME perceived, two participants (P12, P13) adapted clicks or questions in session, and three (P1, P11, P16) described anticipated extra effort if a channel were unavailable. We describe this as legibility work: additional effort to make ongoing activity interpretable to an AI system. It was not a majority pattern (five of 16 participants), and we observed it after participants inspected the Audit, in a think-aloud protocol where speech was itself a perception channel. Inspectability can create conditions for this work; we do not claim that it induces it.

This can create a tension for inspectable proactive AI. Making the system’s perception more understandable can help users interpret its participation decisions, while also giving them a model of how to make their own behavior easier for the system to interpret. Greater visibility into which signals the system treats as important could intensify that tension [60]. In open-ended inquiry, such adaptations matter because clicks, speech, and attention are not merely inputs to the AI; they are also part of how users formulate questions, explore alternatives, and develop their own reasoning. P11, for example, anticipated becoming more active and intentional in their clicking and scrolling when Ears was unavailable, and described having to think about doing so as “a little bit ... tiring.”

Design Implication 3: Account for legibility work as a possible cost of inspectable perception. Designers should consider not only whether explanations help users understand proactive AI, but also whether exposing what the system perceives encourages users to reorganize their activity around those signals. Interfaces should minimize the effort required to make relevant context understandable, and should not treat such adaptation as a cost-free side effect of transparency.

Limitations

Our findings reflect situated experience with one probe rather than preferences that generalize. The eight support types were derived from Wizard behavior and encoded into SPotME; they are a working vocabulary for this probe, not a general taxonomy, and Phase III studies control under that vocabulary rather than validating the type set. Participants also had to think aloud while speech was itself a perception channel, giving SPotME richer evidence of their reasoning than everyday use. Some later used that speech when making themselves understandable to the system, so observations of legibility work remain bounded to this protocol. The study was a single staged session on the researcher’s computer: everyone experienced proactivity, then Audit, then controls, starting from all channels on and [Assist]. Configuration choices were therefore made after inspection by design. We cannot claim that inspectability causes users to configure AI differently, nor that these preferences would persist in longer-term or device-owned use. Police use-of-force data may also heighten concerns about interpretation and steering. Finally, the probe could not express some requested bounds—filtering within an enabled channel, exception-based initiative, or interactional roles—which we treat as open design questions rather than implied completeness of the present controls.

Conclusion

We investigated how proactive AI participation can be made inspectable and user-configurable during open-ended inquiry through SPotME, a design probe that makes interventions and selected suppressed decisions inspectable and lets users separately configure perception, initiative, and support types. When selected suppressed decisions were made inspectable, participants interpreted silence as a decision. In the subsequent configuration stage, they could keep sensing while lowering initiative; some used later inspection as a reason for that choice. Participants could also choose lower initiative while keeping perception available, allowing SPotME to continue following their activity while they retained greater control over when its assistance entered their inquiry. The study suggests inspectability can induce extra effort that participants undertook to make their ongoing activity understandable to the AI. Together, these findings recast proactive participation as more than a system judgment about when assistance may be useful: users may also want to understand and shape the terms under which an AI follows and enters their inquiry. These findings point toward proactive AI that makes both intervention and non-intervention inspectable while allowing users to separately configure what the system may perceive and how proactively it may participate.

Post-Study Questionnaire

Participants completed the following questionnaire after interacting with SpotMe. Unless otherwise specified, items used a 7-point response scale.

Experience with SpotMe. For Questions 1–15, participants responded from 1 = Strongly disagree to 7 = Strongly agree.

  1. SpotMe’s assistance was useful during my dashboard exploration.

  2. SpotMe’s assistance generally aligned with what I was trying to do.

  3. SpotMe generally intervened at appropriate moments.

  4. The amount of proactive assistance SpotMe provided was appropriate.

  5. SpotMe generally chose an appropriate way to respond—staying silent, asking first, or helping directly.

  6. I enjoyed the experience of interacting with SpotMe while exploring the dashboard.

  7. SpotMe’s assistance helped me feel more confident exploring and making sense of the dashboard.

  8. I would want to use a system like SpotMe for similar dashboard exploration in the future.

  9. SpotMe’s rationales helped me understand why it intervened.

  10. Seeing decisions where SpotMe stayed silent helped me understand how it was making proactive decisions.

  11. It was easy for me to make sense of SpotMe’s rationale and audit information.

  12. I understood how changing what SpotMe could perceive would affect its behavior.

  13. I understood how changing SpotMe’s allowed initiative would affect its behavior.

  14. The available controls let me express how I wanted SpotMe to behave.

  15. I felt in control of how SpotMe participated during my dashboard exploration.

Trust in SpotMe. Questions 16–18 were adapted from the three-item S-TIAS [49]. Participants responded from 1 = Not at all to 7 = Extremely.

  1. I am confident in SpotMe.

  2. SpotMe is reliable.

  3. I can trust SpotMe.

Usability. Questions 19–20 were adapted from UMUX-LITE [41]. Participants responded from 1 = Strongly disagree to 7 = Strongly agree.

  1. SpotMe’s capabilities meet my requirements.

  2. SpotMe is easy to use.

AI background.

  1. How would you rate your overall familiarity with using AI tools?

Response scale: 1 = Not at all familiar; 7 = Extremely familiar.

  1. Overall, how would you describe your attitude toward AI technologies?

Response scale: 1 = Very negative; 7 = Very positive.

Demographics.

(23) What is your age range?

  • Less than 18

  • 18–24

  • 25–34

  • 35–44

  • 45–54

  • 55–64

  • 65–74

  • 75+

  • Prefer not to answer

(24) What is your gender? Select all that apply.

  • Female

  • Male

  • Non-binary / Third Gender

  • Prefer not to answer

  • Other

(25) Where do you currently live?

City/state/country is enough. Please do not provide a full address.

(26) How do you describe your race and/or ethnicity? Select all that apply.

  • White

  • Hispanic, Latino, or Spanish origin

  • Black or African American

  • Asian

  • American Indian or Alaska Native

  • Middle Eastern or North African

  • Native Hawaiian or other Pacific Islander

  • Prefer not to answer

  • Other

Optional comment.

(27) Is there anything about SpotMe’s proactivity, reasoning, or controls that you would like to add before we discuss your responses?

Optional free-response question.

SpotMe Prompt

The following is the full prompt template used by SpotMe to decide whether and how to participate. Runtime values for the current interaction context, session memory, transcript, user configuration, and dashboard knowledge were inserted into the corresponding placeholders. The implementation-level codes L0_withhold, L1_offer, and L2_deliver correspond to the user-facing modes Silence, Ask, and Assist, respectively.

You are SpotMe, a proactive AI assistant embedded in the [Dashboard Name] public-data dashboard. Your primary task is to
    decide whether and how to initiate useful support during the user's exploration. A separate Reactive path handles
    direct requests to SpotMe.

RESPONSE STYLE
Use short, respectful, non-judgmental responses.
Describe dashboard information neutrally.

SOURCE LABELS
Use these labels exactly when referring to information that influenced a decision:
- Eyes: information from the available visual frames.
- Ears: information from the speech transcript.
- Mouse + keyboard: available interaction activity.
- Memory: relevant earlier context from this session.
- Dashboard knowledge: relevant information from the supplied [Dashboard Name] knowledge base.
- General knowledge: relevant information beyond the supplied [Dashboard name] knowledge base.
Eyes, Ears, and Mouse + keyboard are Perception.
Dashboard knowledge and General knowledge are Knowledge.
The speech transcript is machine-generated and may be imperfect.

CURRENT CONTEXT
TRIGGER METADATA:
{json.dumps(trigger_context, indent=2)}
RECENT SESSION MEMORY SUMMARY:
{json.dumps(session_memory_summary, indent=2)}
AUDIO TRANSCRIPT PAYLOAD:
{json.dumps(transcript_payload or {}, indent=2)}
USER CONFIGURATION:
{json.dumps(spotme_config, indent=2)}
DASHBOARD KNOWLEDGE BASE:
{dashboard_knowledge_base}

OVERLAPPING WINDOWS
Use Memory to distinguish new activity from content repeated across overlapping windows. If the current window only
    repeats previously handled content and contains no new request, action, or relevant change, do not treat the
    repetition as new activity.

STEP 1 — IS THIS REACTIVE?
Reactive assistance occurs when the user directly addresses SpotMe with a question or request.
If the current window only repeats a previously handled direct request and contains no new follow-up, it is not a new
    Reactive request.
An explicit address such as "SpotMe" or "Hey SpotMe" together with a question or request is strong evidence of a Reactive
    interaction.
If this is Reactive:
- identify the user's direct request and the context relevant to answering it;
- answer the request directly using the available Perception, Memory, Dashboard knowledge, and General knowledge;
- keep the response brief and grounded;
- if a grounded answer cannot be provided, briefly state what SpotMe cannot determine;
- then serialize the result into the JSON object and stop.
If this is not Reactive, continue to STEP 2.

STEP 2 — UNDERSTAND THE SITUATION
Situation means SpotMe's current understanding of what is happening and what the user appears to be trying to do.
First ask:
1. What is happening in the current interaction?
2. What is the user currently focused on or trying to do?
3. What available Perception, Memory, or Knowledge is relevant to understanding this?
Use those answers to form one concise Situation.
Then continue to STEP 3.

STEP 3 — IDENTIFY A USEFUL CONTRIBUTION
First ask:
What useful contribution, if any, could SpotMe make in the current Situation?
Use the following Support types as domain-specific scaffolding for answering this question:
- finding_way_around: help locate a dashboard feature, view, control, or destination.
- troubleshooting_issues: help recover from a stuck, failed, or misconfigured dashboard state.
- explaining_limits: explain a dashboard capability or limitation.
- explaining_data: help understand dashboard data, labels, sources, definitions, or caveats.
  • checking_misreadings: address a possible mismatch between the user's reading and displayed dashboard information.

  • interpreting_patterns: help make sense of a displayed measure, value, comparison, pattern, or trend in relation to relevant context.

  • suggesting_next_steps: suggest a relevant unexplored direction connected to the user's current or recent dashboard exploration.

  • social_repair: repair an earlier SpotMe response that was rejected, corrected, confusing, or unhelpful.

The Support types are categories, not an ordering.

Select the single Support type that best describes the useful contribution.

Possible intervention means one specific action SpotMe could take in the current Situation.

Use the Situation, selected Support type, and any relevant Memory or Knowledge to form one Possible intervention.

If no specific Possible intervention can be identified, no proactive assistance is needed. Continue directly to the JSON output.

Then ask:

Why this Possible intervention?

Explain what in the Situation led to this Possible intervention.

State the specific information from Perception, Memory, or Knowledge that informed the choice.

Then continue to STEP 4.

SHOULD I INITIATE?

Evaluate the Possible intervention through the following four questions, in order.

Evidence strength

Evidence strength means how strongly the available context supports the Situation and the factual basis needed for the Possible intervention.

Ask:

How strong is that support?

Expected benefit

Expected benefit means the concrete value the Possible intervention could add in the current Situation.

Ask:

What concrete benefit would this Possible intervention provide in the current Situation?

Cost of silence

Cost of silence means the consequence of not providing the Possible intervention in the current Situation.

Ask:

What is the consequence of staying silent here?

Cost of intervention

Cost of intervention means the consequence of providing the Possible intervention when SpotMe's understanding is wrong or the intervention is unwanted in the current Situation.

Ask:

What is the consequence of intervening here if that happens?

Now, and only now, ask:

Should I initiate?

Use the four answers together to make a contextual Yes or No judgment.

Then ask:

Why did I decide this?

Explain why SpotMe chose Yes or No.

State which of the four considerations mattered most and what specific information from Perception, Memory, or Knowledge most influenced the judgment.

If the answer is No, the preferred proactive initiative level is L0_withhold. Continue to STEP 6.

If the answer is Yes, continue to STEP 5.

HOW SHOULD I ACT?

Use the same Situation, Possible intervention, Evidence strength, Expected benefit, Cost of silence, and Cost of intervention.

The proactive initiative levels are:

  • L1_offer: ask whether the user wants the Possible intervention before providing it.

  • L2_deliver: provide the Possible intervention now.

Ask:

How should I act?

Choose L1_offer when a brief question can resolve a specific uncertainty that matters to providing the Possible intervention.

Choose L2_deliver when the information needed to provide the Possible intervention is already sufficiently grounded.

Then ask:

Why did I choose this action?

Explain what in the existing reasoning most influenced the choice of L1_offer or L2_deliver.

This is SpotMe's preferred proactive initiative level.
Continue to STEP 6.

STEP 6 — APPLY USER CONFIGURATION
Now apply USER CONFIGURATION to SpotMe's preferred proactive decision.
For proactive assistance:
- social_repair is permitted regardless of allowedSupportTypes.
- Otherwise, if supportType is not in allowedSupportTypes, set the final initiativeLevel to L0_withhold and continue to STEP 7.
- If the preferred level is L2_deliver and L2_deliver is allowed, use L2_deliver.
- If the preferred level is L2_deliver but L2_deliver is not allowed, use L1_offer if allowed; otherwise use L0_withhold.
- If the preferred level is L1_offer, use L1_offer if allowed; otherwise use L0_withhold.
- L0_withhold is always allowed.
The preferred proactive initiative level records SpotMe's judgment before these settings are applied.
The final initiativeLevel records what SpotMe is permitted to do.

STEP 7 — CONSTRUCT THE VISIBLE RESPONSE
Use the RESPONSE STYLE defined above.
For L0_withhold:
- message is empty.
For L1_offer:
- ask whether the user wants the Possible intervention before providing it.
For L2_deliver:
- provide the Possible intervention directly.

MAIN PANEL INLINE EXPLANATION
For a visible proactive response, create two short user-facing fields for the main side panel:
1. inlineObservation
A one-sentence plain-language description of the specific screen content, speech, interaction, or earlier session context that mattered to the decision.

2. inlineInitiativeRationale
A one-sentence plain-language explanation connecting that observation to the initiative decision.
Rules:
- Do not mention Eyes, Ears, Mouse + keyboard, Memory, Dashboard knowledge, or General knowledge.
- Do not mention L0, L1, L2, L0_withhold, L1_offer, or L2_deliver.
- Do not use internal field names.
- Use “you” language where natural.
- Keep each field to one sentence.
- For L1_offer, inlineInitiativeRationale should say that SpotMe asked first.
- For L2_deliver, inlineInitiativeRationale should say that SpotMe helped directly.
- For L0_withhold or assistancePath none, both fields must be empty.
- For reactive responses, both fields may be empty because reactive main-panel rationales are not shown.

SERIALIZE THE RESULT
knowledgeScope records the knowledge basis of the completed assistance:
- dashboard_grounded: the factual content is supported by the visible dashboard context and/or Dashboard knowledge, without General knowledge.
- external: the factual content relies on General knowledge without dashboard-grounded knowledge.
- mixed: the factual content relies on both dashboard-grounded information and General knowledge.
- none: no assistance was provided.
knowledgeScope is descriptive only and does not change the decision already reached.
Natural-language explanation fields use plain language. Do not expose internal field names or initiative codes. Describe the specific information and reasoning that mattered rather than listing source labels alone.

Backend field mapping:
- observation = answer to STEP 2 question 1
- hypothesizedIntent = answer to STEP 2 question 2
- signals, memoryUse, and knowledgeUse = information identified in STEP 2 question 3
- memoryUse = relevant Memory
- knowledgeUse = relevant Dashboard knowledge and General knowledge
- possibleInterventionRationale = answer to "Why this Possible intervention?"
- evidenceStrength = Evidence strength
- expectedBenefit = Expected benefit
- risksOfNotIntervening = Cost of silence
- risksOfIntervening = Cost of intervention
- shouldIntervene = final visible action after USER CONFIGURATION
- initiativeRationale = answer to "Why did I decide this?"
- preferredInitiativeLevel = SpotMe's preferred proactive level before configuration
- initiativeLevel = final level after configuration
- actionRationale = answer to "Why did I choose this action?"
- rationale = short inline rationale for the main side panel. It must not mention source labels or internal initiative
  codes.
- rationale = inlineInitiativeRationale
- inlineObservation = one-sentence user-facing "What I noticed" line for visible proactive responses.
- inlineInitiativeRationale = one-sentence user-facing "Why I stepped in" line for visible proactive responses.
- initiativeRationale must explain why SpotMe decided to initiate or stay silent using plain language, not internal
  initiative codes.
- actionRationale must explain why SpotMe asked first or directly helped using plain language, not internal initiative
  codes.

Important final visibility rule:
shouldIntervene reflects the final visible action after USER CONFIGURATION.
Set shouldIntervene to true only when:
- assistancePath is reactive; or
- final initiativeLevel is L1_offer; or
- final initiativeLevel is L2_deliver.
Set shouldIntervene to false when:
- assistancePath is none; or
- final initiativeLevel is L0_withhold.

Set assistancePath as follows:
- reactive: STEP 1 identified a direct request to SpotMe.
- none: no Possible intervention was identified.
- proactive: a Possible intervention was identified, including cases where it is ultimately withheld.

For Reactive:
- shouldIntervene = true;
- preferredInitiativeLevel = none;
- initiativeLevel = none;
- populate the Situation and source fields directly from the request;
- supportType = none;
- possibleIntervention = "";
- possibleInterventionRationale = "";
- proactive-only reasoning fields may be empty.

For no Possible intervention:
- assistancePath = none;
- shouldIntervene = false;
- supportType = none;
- knowledgeScope = none;
- preferredInitiativeLevel = L0_withhold;
- initiativeLevel = L0_withhold;
- message = "";
- rationale = "";
- inlineObservation = "";
- inlineInitiativeRationale = "";
- proactive reasoning fields may be empty.

For proactive L0_withhold:
- retain the completed proactive reasoning fields;
- shouldIntervene = false;
- message = "";
- rationale = "";

Return ONLY valid JSON using exactly this schema:
{
  "assistancePath": "none | proactive | reactive",
  "observation": "",
  "hypothesizedIntent": "",
  "signals": [],
  "memoryUse": "",
  "knowledgeUse": [],
  "possibleIntervention": "",
  "possibleInterventionRationale": "",
  "supportType": "finding_way_around | troubleshooting_issues | explaining_limits | explaining_data | checking_misreadings
    | interpreting_patterns | suggesting_next_steps | social_repair | none",
"knowledgeScope": "dashboard_grounded | external | mixed | none",
"evidenceStrength": "",
"expectedBenefit": "",
"risksOfNotIntervening": "",
"risksOfIntervening": "",
"shouldIntervene": false,
"initiativeRationale": "",
"preferredInitiativeLevel": "L0_withhold | L1_offer | L2_deliver | none",
"initiativeLevel": "L0_withhold | L1_offer | L2_deliver | none",
"actionRationale": "",
"message": "",
"rationale": "",
"inlineObservation": "",
"inlineInitiativeRationale": ""
}}

References

  1. [1]Kars Alfrink, Janus Keller, Gerd Kortuem, and Neelke Doorn. 2023. Contestable AI by Design: Towards a Framework: K. Alfrink et al. Minds and Machines 33, 4 (2023), 613–639.DOI
  2. [2]James E Allen, Curry I Guinn, and Eric Horvitz. 1999. Mixed-initiative interaction. IEEE Intelligent Systems and their Applications 14, 5 (1999), 14–23.DOI
  3. [3]Zahra Ashterkham, Mohit Jain, Q. Vera Xiao, and Justin D. Weisz. 2019. Resilient Chatbots: Repair Strategy Preferences for Conversational Breakdowns. In Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems. Association for Computing Machinery, 1–12. doi:10.1145/3290605.3300484
  4. [4]Sandeep Avula, Bogeum Choi, and Jaime Arguello. 2022. The Effects of System Initiative during Conversational Collaborative Search. Proceedings of the ACM on Human-Computer Interaction 6, CSCW1, Article 66 (2022), 30 pages. doi:10.1145/3512913DOI
  5. [5]Gagan Bansal, Tongshuang Wu, Joyce Zhou, Raymond Fok, Besmira Nushi, Ece Kamar, Marco Tulio Ribeiro, and Daniel Weld. 2021. Does the whole exceed its parts? the effect of ai explanations on complementary team performance. In Proceedings of the 2021 CHI conference on human factors in computing systems. 1–16.arxiv.org/abs/2006.14779
  6. [6]Carlos Bermejo Fernandez, Lik Hang Lee, Petterit Nurmi, and Pan Hui. 2021. Para: Privacy management and control in emerging iot ecosystems using augmented reality. In Proceedings of the 2021 international conference on multimodal interaction. 478–486.DOI
  7. [7]Caterina Bérubé, Marcia Nißen, Rasita Vinay, Alexa Geiger, Tobias Budig, Aashish Bhandari, Catherine Rachel Pe Benito, Nathan Ibarcena, Olivia Pistolese, Pan Li, et al. 2024. Proactive behavior in voice assistants: A systematic review and conceptual model. Computers in human behavior reports 14 (2024), 100411.DOI
  8. [8]Longfei Chen, Shenghan Gao, Shiwei Wang, Ken Lin, Yun Wang, Yan Lu, and Quan Li. 2026. When Systems Take Initiative: A Design Framework for Adaptive, Mixed-Initiative Database Querying. In Proceedings of the 2026 ACM Designing Interactive Systems Conference. Association for Computing Machinery, New York, NY, USA. doi:10.1145/3800645.3812906DOI
  9. [9]Tongbo Chen, Zhengxi Lu, Zhan Xu, Guocheng Shao, Shaohan Zhao, Fei Tang, Yong Du, Kaitao Song, Yizhou Liu, Yuchen Yan, Wenqi Zhang, Xu Tan, Weiming Lu, Jun Xiao, Yueting Zhuang, and Yongliang Shen. 2026. KnowU-Bench: Towards Interactive, Proactive, and Personalized Mobile Agent Evaluation. arXiv:2604.08455 [cs.AI]arxiv.org/abs/2604.08455
  10. [10]Valerie Chen, Alan Zhu, Sebastian Zhao, Hussein Mozannar, David Sontag, and Ameet Talwalkar. 2025. Need help? designing proactive ai assistants for programming. In Proceedings of the 2025 CHI Conference on Human Factors in Computing Systems. 1–18.arxiv.org/abs/2410.04596
  11. [11]Kristin Cook, Nick Cramer, David Israel, Michael Wolverton, Joe Bruce, Russ Burtner, and Alex Endert. 2015. Mixed-initiative visual analytics using task-driven recommendations. In 2015 IEEE conference on visual analytics science and technology (VAST). IEEE, 9–16.DOI
  12. [12]Andrea Cuadra, Shuran Li, Hansol Lee, Jason Cho, and Wendy Ju. 2021. My Bad! Repairing Intelligently Voice Assistant Errors Improves Interaction. Proceedings of the ACM on Human-Computer Interaction 5, CSCW1, Article 27 (2021), 24 pages. doi:10.1145/3449101DOI
  13. [13]Amit Kumar Das and Klaus Mueller. 2026. MisVisFix: An Interactive Dashboard for Detecting, Explaining, and Correcting Misleading Visualizations using Large Language Models. IEEE Transactions on Visualization and Computer Graphics 32, 1 (2026), 134–144. doi:10.1109/TVCG.2025.3633884DOI
  14. [14]Yang Deng, Lizi Liao, Wenqiang Lei, Grace Hui Yang, Wai Lam, and Tat-Seng Chua. 2025. Proactive conversational ai: A comprehensive survey of advancements and opportunities. ACM Transactions on Information Systems 43, 3 (2025), 1–45.DOI
  15. [15]Vaishali Dhanoa, Andreas Hinterreiter, Vanessa Fedik, Niklas Elmqvist, Eduard Gröller, and Marc Streit. 2024. D-Tour: Semi-Automatic Generation of Interactive Guided Tours for Visualization Dashboard Onboarding. IEEE Transactions on Visualization and Computer Graphics 31, 1 (2024), 721–731. doi:10.1109/TVCG.2024.3456347
  16. [16]Vaishali Dhanoa, Gabriela Molina León, Eve Hoggan, Eduard Gröller, Marc Streit, and Niklas Elmqvist. 2026. "Hey Dashboard!": Supporting Voice, Text, and Pointing Modalities in Dashboard Onboarding using Large Language Models. In Proceedings of the 2026 CHI Conference on Human Factors in Computing Systems. 1–15.DOI
  17. [17]Victor Dibra. 2023. DLA: A tool for automatic generation of grammar-agnostic visualizations and infographics using large language models. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 3: System Demonstrations). 113–126.
  18. [18]Hyo Jin Do, Ha-Kyung Kong, Pooja Telia, Jaewook Lee, and Brian P. Bailey. 2023. To Err is AI: Imperfect Interventions and Repair in a Conversational Agent Facilitating Group Chat Discussions. Proceedings of the ACM on Human-Computer Interaction 7, CSCW1, Article 99 (2023), 23 pages. doi:10.1145/3579532
  19. [19]Malin Eiband, Daniel Buschek, Alexander Kremer, and Heinrich Hussmann. 2019. The impact of placebo explanations on trust in intelligent systems. In Extended abstracts of the 2019 CHI conference on human factors in computing systems. 1–6.DOI
  20. [20]Motahhare Eslami, Aimee Rickman, Kristen Vaccaro, Amirhossein Alayesta, Andy Vuong, Karrie Karahalios, Kevin Hamilton, and Christian Sandvig. 2015. "I always assumed that I wasn’t really that close to [her]" Reasoning about Invisible Algorithms in News Feeds. In Proceedings of the 33rd annual ACM conference on human factors in computing systems. 153–162.DOI
  21. [21]Yingchaojie Feng, Xingbo Wang, Bo Pan, Kam Wai Wong, Yi Ren, Shi Liu, Zihan Yan, Yuxin Ma, Huamin Qu, and Wei Chen. 2023. Xnli: Explaining and diagnosing nli-based visual data analysis. IEEE Transactions on Visualization and Computer Graphics 30, 7 (2023), 3813–3827.
  22. [22]Jennifer Ferday and Eimear Muir-Cochrane. 2006. Demonstrating Rigor Using Thematic Analysis: A Hybrid Approach of Inductive and Deductive Coding and Theme Development. International Journal of Qualitative Methods 5, 1 (2006), 80–92. doi:10.1177/160940690600500107
  23. [23]Raymond Fok and Daniel S Weld. 2024. In search of verifiability: Explanations rarely enable complementary performance in AI-advised decision making. AI Magazine 45, 3 (2024), 317–332.arxiv.org/abs/2305.07722
  24. [24]Tong Gao, Mira Dontcheva, Eytan Adar, Zhicheng Liu, and Karrie G Karahalios. 2015. Datatone: Managing ambiguity in natural language interfaces for data visualization. In Proceedings of the 28th annual acm symposium on user interface software & technology. 489–500.DOI
  25. [25]David Gotz and Zhen Wen. 2009. Behavior-driven visualization recommendation. In Proceedings of the 14th international conference on Intelligent user interfaces. 315–324.DOI
  26. [26]Adam M Grant and Susan J Ashford. 2008. The dynamics of proactivity at work. Research in organizational behavior 28 (2008), 3–34.DOI
  27. [27]Yi Guo, Danqing Shi, Mingjuan Guo, Yanqiu Wu, Nan Cao, and Qing Chen. 2024. Talk2data: A natural language interface for exploratory visual analysis via question decomposition. ACM Transactions on Interactive Intelligent Systems 14, 2 (2024), 1–24.arxiv.org/abs/2107.14420
  28. [28]Jeffrey Heer. 2019. Agency plus automation: Designing artificial intelligence into interactive systems. Proceedings of the National Academy of Sciences 116, 6 (2019), 1844–1850.DOI
  29. [29]Md. Naimul Hoque and Nicole Sultanum. 2025. DashGuide: Authoring Interactive Dashboard Tours for Guiding Dashboard Users. Computer Graphics Forum 44, 3 (2025), e70107. doi:10.1111/cgf.70107DOI
  30. [30]Eric Horvitz. 1999. Principles of mixed-initiative user interfaces. In Proceedings of the SIGCHI conference on Human Factors in Computing Systems. 159–166.
  31. [31]Eric J Horvitz. 2007. Reflections on challenges and promises of mixed-initiative interaction. AI Magazine 28, 2 (2007), 3–3.
  32. [32]Stephanie Houde, Kristina Brimijoin, Michael Muller, Steven I. Ross, Dario Andres Silva Moran, Gabriel Enrique Gonzalez, Siya Kunde, Morgan A. Foreman, and Justin D. Weisz. 2025. Controlling AI Agent Participation in Group Conversations: A Human-Centered Approach. In Proceedings of the 30th International Conference on Intelligent User Interfaces. https://dl.acm.org/doi/full/10.1145/3708359.3712089DOI
  33. [33]Yanwei Huang and Arpit Naranecha. 2026. Facilitating Proactive Experience Design for Decision Making on the Web: A Design Probe with WebSeek. In Proceedings of the 2026 CHI Conference on Human Factors in Computing Systems. 1–20.arxiv.org/abs/2601.15100
  34. [34]Pranut Jain, Rosta Farzan, and Adam J Lee. 2023. Co-designing with users the explanations for a proactive auto-response messaging agent. Proceedings of the ACM on human-computer interaction 7, MHCI (2023), 1–23.DOI
  35. [35]Nadia Karizat, Dan Delmonaco, Motahhare Eslami, and Nazanin Andalibi. 2021. Algorithmic folk theories and identity: How TikTok users co-produce knowledge of identity and engage in algorithmic resistance. Proceedings of the ACM on human-computer interaction 5, CSCW2 (2021), 1–44.DOI
  36. [36]Sunnie SY Kim, Elizabeth Anne Watkins, Olga Russakovsky, Ruth Fong, and Andrés Monroy-Hernández. 2023. 'help me help the ai': Understanding how explainability can support human-ai interaction. In proceedings of the 2023 CHI conference on human factors in computing systems. 1–17.DOI
  37. [37]Philipp Kirschthaler, Martin Porcheron, and Joel E. Fischer. 2020. What Can I Say?: Effects of Discoverability in VUIs on Task Performance and User Experience. In Proceedings of the 2nd Conference on Conversational User Interfaces. Association for Computing Machinery, 1–9. doi:10.1145/3406119.3405755. 3406119DOI
  38. [38]Nadine Kuo, Agnia Sergeryuk, Valerie Chen, and Maliheh Izadi. 2026. Developer Interaction Patterns with Proactive AI: A Five-Day Field Study. In Proceedings of the 31st International Conference on Intelligent User Interfaces. Association for Computing Machinery, 349–362. doi:10.1145/3742413.3789148DOI
  39. [39]Michelle S Lam, Omar Shaikh, Hallie Xu, Alice Guo, Divi Yang, Jeffrey Heer, James A Landay, and Michael S Bernstein. 2025. Just-In-Time Objectives: A General Approach for Specialized AI Interactions. arXiv preprint arXiv:2510.14591 (2025).DOI
  40. [40]Sukwon Lee, Sung-Hee Kim, Ya-Hsin Hung, Heidi Lam, Youn-ah Kang, and Ji Soo Yi. 2015. How do people make sense of unfamiliar visualizations?: A grounded model of novice’s information visualization sensemaking. IEEE transactions on visualization and computer graphics 22, 1 (2015), 499–508.DOI
  41. [41]James R. Lewis, Brian S. Utesch, and Deborah E. Maher. 2013. UMUX-LITE: When There’s No Time for the SUS. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI ’13). Association for Computing Machinery, New York, NY, USA, 2099–2102. doi:10.1145/2470654.2481287DOI
  42. [42]BoRui Li, Jinda Yang, Apurva Jain, William Newberry, and Xing-Dong Yang. 2025. LiDArgus: Accommodating Diverse Privacy Preferences in LiDAR Sensing Through Configurable Sub-Zoning. In Proceedings of the 38th Annual ACM Symposium on User Interface Software and Technology. 1–23.DOI
  43. [43]Q Vera Liao, Daniel Gruen, and Sarah Miller. 2020. Questioning the AI: informal design practices for explainable AI user experiences. In Proceedings of the 2020 CHI conference on human factors in computing systems. 1–15.DOI
  44. [44]Brian Y Lim, Anind K Dey, and Daniel Avrahami. 2009. Why and why not explanations improve the intelligibility of context-aware intelligent systems. In Proceedings of the SIGCHI conference on human factors in computing systems. 2119–2128.DOI
  45. [45]Ailin Liu, Yesmine Karoui, Fiona Draxler, Frauke Kreuter, and Francesco Chiossi. 2026. Sensing what surveys miss: understanding and personalizing proactive LLM support by user modeling. In Proceedings of the 2026 CHI Conference on Human Factors in Computing Systems. 1–21.
  46. [46]Xingyu Bruce Liu, Shitao Fang, Weiyan Shi, Chien-Sheng Wu, Takeo Igarashi, and Xiang’Anthony Chen. 2025. Proactive conversational agents with inner thoughts. In Proceedings of the 2025 CHI Conference on Human Factors in Computing Systems. 1–19.arxiv.org/abs/2501.00383
  47. [47]Yaxi Lu, Shenzhi Yang, Cheng Qian, Guirong Chen, Qinyu Luo, Yesai Wu, Huadong Wang, Xin Cong, Zhong Zhang, Yankai Lin, et al. 2025. Proactive agent: Shifting llm agents from reactive responses to active assistance. In International Conference on Learning Representations, Vol. 2025. 47431–47457.arxiv.org/abs/2410.12361
  48. [48]Nora McDonald, Sarita Schoenebeck, and Andrea Forte. 2019. Reliability and inter-rater reliability in qualitative research: Norms and guidelines for CSCW and HCI practice. Proceedings of the ACM on human-computer interaction 3, CSCW (2019), 1–23.
  49. [49]Melanie J. McGrath, Oliver Lack, James Tisch, and Andreas Duenser. 2025. Measuring Trust in Artificial Intelligence: Validation of an Established Scale and Its Short Form. Frontiers in Artificial Intelligence 8 (2025), 1582880. doi:10.3389/frai.2025.1582880DOI
  50. [50]Jiwon Moon, Chien-Ming Huang, and Ziang Xiao. 2026. Understanding of Control in AI Reasoning for Human-AI Decision-Making. In Proceedings of the Extended Abstracts of the 2026 CHI Conference on Human Factors in Computing Systems. 1–9.DOI
  51. [51]Arpit Narechania, Arjun Srinivasan, and John Stasko. 2020. NL4DV: A toolkit for generating analytic specifications for data visualization from natural language queries. IEEE Transactions on Visualization and Computer Graphics 27, 2 (2020), 369–379.DOI
  52. [52]Zhenhui Peng, Yunhwan Kwon, Jiaan Lu, Ziming Wu, and Xiaojuan Ma. 2019. Design and evaluation of service robot’s proactivity in decision-making support process. In proceedings of the 2019 CHI conference on human factors in computing systems. 1–13.DOI
  53. [53]Kevin Pu, Daniel Lazaro, Ian Arawjo, Haijun Xia, Jiang Xiao, Tovi Grossman, and Yan Chen. 2025. Assistance or disruption? exploring and evaluating the design and trade-offs of proactive ai programming support. In Proceedings of the 2025 CHI conference on human factors in computing systems. 1–21.DOI
  54. [54]Kevin Pu, Ting Zhang, Naveen Sendhilnathan, Sebastian Freitag, Raj Sodhi, and Tanya R Jonker. 2025. Promemassist: Exploring timely proactive assistance through working memory modeling in multi-modal wearable devices. In Proceedings of the 38th Annual ACM Symposium on User Interface Software and Technology. 1–19.DOI
  55. [55]Emilee Rader and Rebecca Gray. 2015. Understanding user beliefs about algorithmic curation in the Facebook news feed. In Proceedings of the 33rd annual ACM conference on human factors in computing systems. 173–182.DOI
  56. [56]Eric D Ragan, Alex Endert, Jibonananda Sanyal, and Jian Chen. 2015. Characterizing provenance in visualization and data analysis: an organizational framework of provenance types and purposes. IEEE transactions on visualization and computer graphics 22, 1 (2015), 31–40.DOI
  57. [57]Vidya Setlur, Sarah E Battersby, Melanie Tory, Rich Gossweiler, and Angel X Chang. 2016. Eviza: A natural language interface for visual analysis. In Proceedings of the 29th annual symposium on user interface software and technology. 365–377.DOI
  58. [58]Vidya Setlur, Michael Correll, Arvind Satyanarayan, and Melanie Tory. 2023. Heuristics for supporting cooperative dashboard design. IEEE transactions on visualization and computer graphics 30, 1 (2023), 370–380.
  59. [59]Omar Shaikh, Shardul Sapkota, Shan Rizvi, Eric Horvitz, Joon Sung Park, Diyang Yang, and Michael S Bernstein. 2025. Creating general user models from computer use. In Proceedings of the 38th Annual ACM Symposium on User Interface Software and Technology. 1–23.arxiv.org/abs/2505.10831
  60. [60]Zhuyu Peng, Pei Chen, Yichen Cai, Ruoqing Lu, Zhaoqiang Jiang, Jiayang Li, Weitao You, and Lingyun Sun. 2026. Seeing Eye to Eye: Enabling Cognitive Alignment Through Shared First-Person Perspective in Human–AI Collaboration. In Proceedings of the 2026 CHI Conference on Human Factors in Computing Systems. 1–19.
  61. [61]Miles Turpin, Julian Michael, Ethan Perez, and Samuel Bowman. 2023. Language models don’t always say what they think: Unfaithful explanations in chain-of-thought prompting. Advances in Neural Information Processing Systems 36 (2023), 74952–74965.arxiv.org/abs/2305.04388
  62. [62]Kristen Vaccaro, Dylan Huang, Motahhare Eslami, Christian Sandvig, Kevin Hamilton, and Karrie Karahalios. 2018. The illusion of control: Placebo effects of control settings. In Proceedings of the 2018 CHI conference on human factors in computing systems. 1–13.DOI
  63. [63]Chenglong Wang, John Thompson, and Bongshin Lee. 2023. Data formulator: Ai-powered concept-driven visualization authoring. IEEE Transactions on Visualization and Computer Graphics 30, 1 (2023), 1128–1138.arxiv.org/abs/2309.10094
  64. [64]Xinru Wang and Ming Yin. 2021. Are explanations helpful? a comparative study of the effects of explanations in ai-assisted decision-making. In Proceedings of the 26th International Conference on Intelligent User Interfaces. 318–328.DOI
  65. [65]Yi Wen, Yu Zhang, Sriram Suresh, Zhicong Lu, Can Liu, and Meng Xia. 2026. InterFlow: Designing Unobtrusive AI to Empower Interviewers in Semi-Structured Interviews. In Proceedings of the 2026 CHI Conference on Human Factors in Computing Systems. 1–21.
  66. [66]Monika Westphal, Michael Vössing, Gerhard Satzger, Galit B Yom-Tov, and Anat Rafaeli. 2023. Decision control and explanations in human-AI collaboration: Improving user perceptions and compliance. Computers in Human Behavior 144 (2023), 107714.DOI
  67. [67]Ryen W White and Resa A Roth. 2009. Exploratory search: Beyond the query-response paradigm. Number 3. Morgan & Claypool Publishers.
  68. [68]Kanit Wongsuphasawat, Zening Qu, Dominik Moritz, Riley Chang, Felix Ouk, Anushka Anand, Jock D. Mackinlay, Bill Howe, and Jeffrey Heer. 2017. Voyager 2: Augmenting Visual Analysis with Partial View Specifications. In Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems. Association for Computing Machinery, 2648–2659. doi:10.1145/3025453.3025768DOI
  69. [69]Ruoxuan Yang, Yuwei Du, Hongyang Du, and Kaibin Huang. 2026. Mapping the Wizards’ Path: A Systematic Review of Wizard-of-Oz in HCI. In Proceedings of the 2026 CHI Conference on Human Factors in Computing Systems. 1–20.DOI
  70. [70]Shu Yang, Difei Xu, Jiaxin Pei, and Di Wang. 2026. ProACT: Towards Breakdown-Aware Proactive Agent in Multi-User Collaboration. arXiv:2607.03730 [cs.CL]arxiv.org/abs/2607.03730
  71. [71]Jian Zhao, Mingming Fan, and Mi Feng. 2022. ChartSeer: Interactive Steering Exploratory Visual Analysis With Machine Intelligence. IEEE Transactions on Visualization and Computer Graphics 28, 3 (2022), 1500–1513. doi:10.1109/TVCG.2020.3018724DOI
  72. [72]Yuheng Zhao, Xueli Shu, Liwen Fan, Lin Gao, Yu Zhang, and Siming Chen. 2025. Proactiva: Proactive visual analytics with llm-based ui agent. IEEE Transactions on Visualization and Computer Graphics (2025).arxiv.org/abs/2507.18165
  73. [73]Yuheng Zhao, Yixing Zhang, Yu Zhang, Xinyi Zhao, Junjie Wang, Zekai Shao, Cagatay Turkay, and Siming Chen. 2024. Leva: Using large language models to enhance visual analytics. IEEE transactions on visualization and computer graphics 31, 3 (2024), 1830–1847.DOI

Paper details

vlnc:W_dfswj4pa
Version
v1 · latest
Published
Fresh — added to the Hub within the last 30 days
Added
License
CC BY 4.0
Usage

views · 0

downloads · 0

A view counts once per browser per day, a download each time the PDF is requested.

  • Computer Science

Not recorded for this paper

View statement
Looking for related papers…
Contents