betaFeedback

WorksComputer Science

Collaborative Search Meets Large Language Models: An Empirical Study

Abstract

Collaborative search enables users to address complex information needs jointly. Advances in large language models (LLMs) enable autonomous agents to support collaborative information seeking within shared communication spaces. However, little is known about how these agents shape search experiences, behaviors, and outcomes compared with conventional search tools. We conduct a user study with 80 students working in 40 pairs. Each pair completes four collaborative search tasks under four scenarios: QAAgent, which answers queries using an LLM's parametric knowledge; CoSearchAgent, which uses multi-user conversational context to reformulate queries, retrieve web information, and generate source-grounded answers, while asking clarifying questions when needed; an in-chat search plugin that returns result lists; and manual Google Search. After each task, participants complete several questionnaires assessing their search experience and independently write a report; after all tasks, they participate in individual interviews. Results show that LLM-based agents reduce perceived workload, support collaboration, and reshape communication and evidence-use practices. They also encourage more independent paraphrasing, albeit at the cost of report richness. To the best of our knowledge, this is the first systematic empirical comparison of LLM-based agents and conventional tools for collaborative search across search experience, user behavior, and collaborative outcomes.

CCS Concepts

• Information systems → Collaborative search; Search engine architectures and scalability.

This work is licensed under a Creative Commons Attribution 4.0 International License.

SIGIR-AP 2026, Singapore, Singapore

© 2026 Copyright held by the owner/author(s).

ACM ISBN 979-8-4007-3033-7/2026/12

https://doi.org/10.1145/3848000.3850212

ACM Reference Format:

Peiyuan Gong, Jiamian Li, Erhan Zhang, Fengming Liang, Kexin Liu, Yue Chen, and Jiaxin Mao. 2026. Collaborative Search Meets Large Language Models: An Empirical Study. In Proceedings of the 2026 Annual International ACM SIGIR Conference on Research and Development in Information Retrieval in the Asia Pacific Region (SIGIR-AP 2026), December 13–15, 2026, Singapore, Singapore. ACM, New York, NY, USA, 10 pages. https://doi.org/10.1145/3848000.3850212

Introduction

Collaborative search enables users to jointly address complex information needs by searching for, evaluating, and sharing information [32, 33]. Compared with individual search, it allows collaborators to pool complementary knowledge, refine understanding through discussion, and synthesize diverse perspectives into a shared solution [2, 15, 28]. To support these activities, collaborative search systems typically provide shared workspaces, real-time communication, query histories, and tools for organizing and annotating information. Prior research suggests that collaborative search is widespread, with many users engaging in it at least weekly [23]. These findings underscore the importance of systems that support both information seeking and collaboration.

Early collaborative search systems typically took the form of dedicated standalone applications [2, 9, 11, 15, 24, 25, 28]. They commonly integrated communication and search interfaces, providing features such as shared query histories [9, 11, 15, 24, 25, 28], group notes and summaries [2, 11, 15, 25], and mechanisms for recommending or sharing results [11, 15, 24]. However, users often prefer coordinating through familiar instant messaging platforms while accessing conventional search tools separately because these platforms are familiar, convenient, and ubiquitous [23]. This preference motivates lightweight collaborative search systems that embed search functionality in instant messaging platforms and present results within multi-user conversations [3, 4, 5, 6]. Presenting retrieved information within the conversational interface reduces application switching and lets collaborators discuss and use results as part of their ongoing discourse.

Recent advances in large language models (LLMs) create opportunities to extend lightweight collaborative search beyond retrieving and presenting results. LLMs can interpret evolving conversational context, infer user intent [10, 20, 29], and interact through natural language [13, 21, 36]. These capabilities are valuable in multi-user settings, where information needs may span messages, evolve through discussion, or remain underspecified. By synthesizing conversations, LLMs can reformulate users’ information needs into effective queries, identify missing or ambiguous details, and ask clarifying questions before initiating or refining searches. Augmented with web retrieval, LLMs can synthesize source-grounded information, reduce unsupported claims, and cite supporting sources [18, 19, 37]. More broadly, LLMs may automate adaptive search assistance often implemented through the Wizard-of-Oz paradigm [3, 4, 5, 6, 7], making it more feasible to deploy and evaluate in real-world collaborative environments.

However, little research has systematically compared LLM agents with traditional search tools in collaborative search. It remains unclear how their richer, adaptive assistance affects search experience, shapes user behavior, and influences collaborative outcomes. Figure 1 contrasts a traditional search plugin with a QA agent. The search plugin delivers timely, source-grounded information, but may interrupt conversations and require users to evaluate and integrate results. The QA agent can synthesize information, adapt to context, and offer proactive support, yet may hallucinate or reduce engagement with original sources. These differences may affect how users formulate queries, verify and share information, divide tasks, coordinate decisions, and perceive workload and satisfaction. A systematic comparison is therefore needed across LLM agents, embedded search plugins, and standalone search engines.

Two lightweight collaborative search systems

Figure 1. Two lightweight collaborative search systems: (I) Search Plugin, which provides reliable sources but requires longer browsing; and (II) QAAgent, which offers convenient answers that speed up discussion but may leave users unsure.

To address this gap, we conduct a user study with 80 students organized into 40 pairs, systematically comparing two forms of LLM-based assistance with two conventional search tools. Specifically, QAAgent answers users’ questions directly using the LLM’s parametric knowledge, whereas CoSearchAgent [13] interprets the multi-user conversational context, rewrites queries, proactively asks clarifying questions, retrieves information from the web, and generates source-grounded answers. We compare these agents with two conventional search tools: SearchAgent, a lightweight plugin that returns ranked result lists within the group conversation, and manual Google search, through which collaborators search independently and share their findings in the conversation. Each pair completes four open-ended collaborative search tasks, one under each scenario. After each task, each participant completes several questionnaires assessing their search experience and then writes a brief search report. After completing all four tasks, they participate in individual interviews.

Our results reveal that LLM-based assistance reshapes collaborative search by reducing perceived task demands and strengthening teamwork. QAAgent delivers the broadest collaborative gains, improving mutual awareness, information sharing, and exchange efficiency. CoSearchAgent provides the best agent experience and reduces evidence inspection through synthesized, web-grounded answers. However, these experiential benefits do not necessarily improve report quality. In the plug-in setting with a fixed word limit, reports assembled from copied webpage content are of higher quality than those based on copied agent responses, as webpages provide richer information for selection and synthesis. This helps explain why Google Search and SearchAgent produce richer reports, whereas LLM-based agents encourage independent wording. Overall, automated retrieval and synthesis reduce effort, but latency and clarification can offset time savings, while direct answers raise concerns about completeness and trust. Effective agents should organize shared knowledge, connect evidence, guide task allocation, and keep proactive assistance under user control. The code, study scripts, and demo are available in our repository.

This work makes three contributions:

  • To our knowledge, this is the first systematic comparison of LLM-based agents and traditional tools for collaborative search, focusing on search experience, user behavior, and collaborative outcomes.

  • We study 80 participants in 40 pairs, each completing four tasks using two LLM-based systems, a conversational search plugin, and Google Search.

  • We reveal how LLM assistance reshapes search and collaboration, identifying key trade-offs and design implications for knowledge organization, verification, exploration, and proactive support.

Related Work

In this section, we review related work on collaborative search from two perspectives: system design and user behavior.

System Construction

Early collaborative search systems are dedicated applications integrating search with coordination support: CoSearch and SearchTogether support querying, communication, and activity awareness [2, 24]; CoSense and Coagmento offer workspaces and annotations [15, 25]; and ResultsSpace, Querium, and SearchX support recommendations and synthesis [9, 11, 28]. Collaborative ranking and allocation use interaction histories and relevance feedback to prioritize and distribute results among group members [12, 27, 35]. Despite reducing duplication and supporting evidence integration, dedicated systems require users to adopt unfamiliar interfaces and workflows. Users’ reliance on familiar communication tools and search engines [23] motivates lightweight, embedded support. Slack-based searchbots enable information requests, sharing, and discussion within ongoing group conversations [4, 5]. Wizard-of-Oz studies extend these bots toward proactive guidance through clarification, suggestions, and dialogue- and task-level initiative [3, 6, 7]. LLMs further automate and integrate these capabilities into conversational workflows: CoSearchAgent interprets multi-user conversations, retrieves web information, generates grounded answers, and asks clarification questions [13], enabling agents to participate in and guide collaborative search.

User Behavior Analysis

Collaborative search brings together searching, communication, and teamwork. Group members explain what they need, divide tasks, follow each other’s progress, and combine the evidence they find [16, 34]. Research in this area looks at how groups break down tasks, revise queries, discuss which results are useful, and bring information together [17, 32, 33]. Shared search histories and workspaces, along with cues about others’ activity, help members divide tasks and avoid repeating the same searches. Systems can also assign results based on each member’s role or expertise [24, 27, 34]. Communication helps groups coordinate, but too little, too much, or poorly timed communication can make coordination harder [14]. At the system level, conversational search lets users ask for information, discuss evidence, and involve agents in the search [4, 5]. Proactive agents can also clarify what users need and suggest where to search next [1, 6, 7]. However, it is still unclear whether LLM-based agents work better than traditional tools.

Many studies examine one feature or design at a time, rather than its combined effects on search experience, user behavior, and group results. To address this gap, we compare QAAgent, CoSearchAgent, SearchAgent, and Google Search in terms of workload, collaboration, communication, evidence use, and report quality.

LLM-Based Collaborative Search Agents

In this section, we introduce two LLM-based agents for collaborative search: QAAgent and CoSearchAgent [13].

QAAgent

We first propose QAAgent, a question-answering plugin that draws on an LLM’s pretrained knowledge [26, 31]. Specifically, given a user query qq, we construct a prompt PqaP_{qa} to guide the LLM in generating a relevant answer:

aqa=Pqa(LLM,q)(1)a_{qa} = P_{qa}(LLM, q) \tag*{(1)}

where aqaa_{qa} is generated directly by the LLM.

CoSearchAgent

We reimplement CoSearchAgent [13] to evaluate more sophisticated LLM-based support for collaborative search. It formulates search queries from ongoing discussions, retrieves relevant web evidence, and synthesizes this evidence into answers.

Conversational Query Understanding

Requests made during a conversation are often incomplete or context-dependent [22, 38]. This issue is especially important in collaborative search because a user’s request may refer to information introduced earlier by another group member. Let U={u1,u2,…,un}U = \{u_1, u_2, \ldots, u_n\} denote the multi-user conversation history and qq the current user query. We first use a query-rewriting prompt PrewriteP_{\mathrm{rewrite}} to resolve references and recover omitted context from the conversation, producing a self-contained query for web search:

(qrewrite,o)=Prewrite(LLM,U,q),(2)(q_{\mathrm{rewrite}}, o) = P_{\mathrm{rewrite}}(LLM, U, q), \tag*{(2)}

where qrewriteq_{\mathrm{rewrite}} is the rewritten query (qrewrite=qq_{\mathrm{rewrite}} = q if qq is self-contained), and oo indicates whether clarification is needed. When an ambiguity cannot be resolved from the conversation history, we use a clarification prompt PclarifyP_{\mathrm{clarify}} to generate a clarifying question:

qclarify=Pclarify(LLM,U,qrewrite).(3)q_{\mathrm{clarify}} = P_{\mathrm{clarify}}(LLM, U, q_{\mathrm{rewrite}}). \tag*{(3)}

After the user responds, CoSearchAgent incorporates the response into the conversation history and reassesses the query. It initiates a web search once no further clarification is required.

Web Evidence Extraction

CoSearchAgent submits the rewritten query qrewriteq_{\mathrm{rewrite}} to a search engine and obtains the top-mm results:

SERP=Search(qrewrite)={(ti,li,si)}i=1m,(4)SERP = Search(q_{\mathrm{rewrite}}) = \{(t_i, l_i, s_i)\}_{i=1}^{m}, \tag*{(4)}

where tit_i, lil_i, and sis_i denote the title, link, and search-engine snippet of result ii, respectively. For each result, the agent fetches the page content hih_i and uses an evidence-extraction prompt PextractP_{\mathrm{extract}} to identify evidence relevant to the rewritten query:

refi=Pextract(LLM,qrewrite,hi).(5)r e f_i = P_{\mathrm{extract}}(LLM, q_{\mathrm{rewrite}}, h_i). \tag*{(5)}

The agent then presents the retrieved sources with the extracted evidence in place of the original snippets:

SERPc={(ti,li,refi)}i=1m.(6)SERP_c = \{(t_i, l_i, r e f_i)\}_{i=1}^{m}. \tag*{(6)}

This representation retains access to the original sources while foregrounding the passages most relevant to the group’s information need.

Source-Grounded Answer Generation

Given the rewritten query qrewriteq_{\mathrm{rewrite}} and extracted evidence REF={ref1,ref2,…,refm}REF = \{r e f_1, r e f_2, \ldots, r e f_m\}, CoSearchAgent uses a retrieval-augmented generation prompt PragP_{\mathrm{rag}} to synthesize a source-grounded answer:

ac=Prag(LLM,qrewrite,REF).(7)a_c = P_{\mathrm{rag}}(LLM, q_{\mathrm{rewrite}}, REF). \tag*{(7)}

The answer aca_c includes citations that link its claims to the supporting sources. CoSearchAgent thus delegates query interpretation, retrieval, and answer synthesis while preserving inspectable evidence for verification and discussion.

User Study

To evaluate the effectiveness of LLM-based agents in collaborative search, we conduct a user study with 80 students organized into 40 pairs. The following subsections describe the research questions (Section 4.1), experimental procedure (Section 4.2), tasks (Section 4.3), search scenarios (Section 4.4), questionnaires (Section 4.5), and face-to-face interviews (Section 4.6).

Research Questions

We investigate how the use of LLM-based agents, as opposed to traditional search tools, influences collaborative search across three key dimensions:

RQ1. What differences in participants’ search experiences arise between LLM-based agent and traditional search scenarios?

RQ2. In what ways does the choice of search tools shape participants’ search strategies and collaboration behaviors?

RQ3. To what extent does the use of LLM-based agents influence the quality of collaboratively produced final reports?

Procedure

We recruit 80 students and form 40 participant pairs. Each pair completes four collaborative search tasks, one under each search scenario: Google Search, SearchAgent, QAAgent, and CoSearchAgent. We use two orthogonal 4×44 \times4 Latin squares to counterbalance task–scenario assignments and presentation order [8, 30]. Their superposition yields four base sequences that collectively cover all 16 task–scenario combinations. Four cyclic rotations of each base sequence produce 16 variants, ensuring that each combination appears once in each ordinal position. We assign 10 pairs to each base-sequence family and distribute them as evenly as possible across its four rotations. Thus, each pair experiences every search scenario once, and each task–scenario combination is completed by 10 pairs.

[Instructions] Before the experiment, moderators explain the procedure and demonstrate each search scenario. Pair members then briefly introduce themselves. For each task, pairs have up to 20 minutes to search, discuss, divide the work, develop a shared understanding, and plan their learning reports. Afterward, each participant independently writes a learning report of at least 500 words in Tencent Docs within 10 minutes and completes a post-task questionnaire. A break of up to 5 minutes is allowed between tasks. After all four tasks, each participant attends an individual face-to-face interview.

[Tools] The study uses two moderators, two computer workstations, and three rooms. The moderators conduct the briefing in a common room, after which the two participants of each pair work at separate computers in separate rooms. Slack provides a dedicated channel for communication and access to the assigned agents. Tencent Meeting records the sessions, Google Chrome supports web access when permitted by the assigned search scenario, and Tencent Docs supports the preparation and submission of individual learning reports.

[Limits] Participants must communicate with their partner only through the dedicated Slack channel and may not communicate verbally at any time during the tasks. They may not open software other than the specified experimental tools. Information seeking must follow the assigned search scenario: participants may search independently only in the Google Search scenario and may not switch to another search tool in the three agent scenarios. Although participants may discuss report structure during the collaborative session, each participant must write and submit the final learning report independently.

Tasks

In our user study, each pair completes four open-ended knowledge-exploration tasks covering AI painting, childhood autism, Romantic-period Western music, and air pollution. Each task presents a background scenario that establishes the participants’ roles and information needs, followed by a goal that requires the pair to investigate the topic collaboratively and each participant to submit an individual learning report. The tasks have no single correct answer and are designed to encourage participants to decompose their information needs, divide subtopics, exchange evidence, and synthesize what they learn. Table 1 summarizes the four tasks.

TaskBackgroundGoal
AI paintingTwo students explore how artificial intelligence is used in painting for an introductory course.Investigate AI painting collaboratively and write individual learning reports.
Childhood autismTwo volunteers help parents seeking information about possible autism in their two-year-old child.Investigate childhood autism collaboratively and write individual learning reports.
Romantic musicTwo concertgoers want to understand Western music from the Romantic period before attending a performance.Investigate Romantic-period music collaboratively and write individual learning reports.
Air pollutionTwo contestants prepare for an environmental knowledge competition concerning air pollution.Investigate air pollution collaboratively and write individual learning reports.

Table 1. Collaborative search tasks.

Scenarios

Each collaborative search task is paired with one of four search scenarios. All scenarios use Slack as the shared communication environment but differ in where participants search, how queries are formulated, and whether the returned information is synthesized by an LLM. The following three agents are embedded in Slack and can be invoked by typing “@name.” Figure 2 illustrates the four scenarios, which are presented to participants as follows:

Four search scenarios in the user study

Figure 2. Four search scenarios in the user study: (I) Google Search, used independently outside the conversation; (II) SearchAgent, integrating results into the conversation; (III) QAAgent, directly answering queries; and (IV) CoSearchAgent, interpreting multi-user conversations, retrieving relevant information, and providing tailored responses.

  • Google Search. Participants leave the Slack conversation to search independently with Google. They inspect the results themselves and return to Slack to share selected links, excerpts, or findings with their partner.

  • SearchAgent. Participants invoke SearchAgent within Slack. The agent retrieves and returns a list of web search results for participants to inspect and integrate into their ongoing conversation; it does not synthesize a direct answer.

  • QAAgent. Participants invoke QAAgent within Slack and receive a direct answer generated from the LLM’s parametric knowledge. The agent does not retrieve web results or provide a search-result list.

  • CoSearchAgent. Participants invoke CoSearchAgent within Slack. The agent interprets the multi-user conversation, rewrites user queries, retrieves relevant web information, and returns a context-aware response with inspectable results and source references, while proactively asking clarifying questions when needed.

Questionnaires

We administer three seven-point (1→71 \to7) questionnaires assessing workload, collaboration experience, and agent experience. Participants complete the workload and collaboration-experience questionnaires after each task and the agent-experience questionnaire after each agent scenario. Questionnaires are detailed below:

  • Workload. Table 2 lists the six items administered after every task to assess mental demand, physical demand, temporal pressure, perceived performance, effort, and frustration.

QuestionScore Range
[W1] How much mental activity, such as thinking, judging, decision-making, and searching, was required?High → Low
[W2] How much physical activity, such as clicking and browsing, was required?High → Low
[W3] How rushed or relaxed was the task pace?Rushed → Relaxed
[W4] How well did you complete the task?Failure → Perfect
[W5] How much effort did you expend?Much → Little
[W6] To what extent did you feel uneasy, discouraged, frustrated, or irritated?Much → Little

Table 2. Workload Questionnaire

Interview

  • Collaboration experience. Table 3 presents the five items administered after every task to assess mutual awareness, information sharing, division of labor, and communication continuity.

  • Agent experience. After each of the three agent scenarios, participants complete the seven items reported in Table 4. These items assess information usefulness and completeness, time savings, workload reduction, support for collaboration, information reliability, and overall satisfaction.

QuestionScore Range
[C1] During the task, I knew what my partner was doing.Disagree → Agree
[C2] During the task, my partner knew what I was doing.Disagree → Agree
[C3] It was easy to share information with my partner.Disagree → Agree
[C4] It was easy to divide the work with my partner.Disagree → Agree
[C5] Communication with my partner remained smooth and uninterrupted.Disagree → Agree

Table 3. Collaboration Experience Questionnaire

QuestionScore Range
[A1] The information provided by the agent was useful.Disagree → Agree
[A2] The information provided by the agent was comprehensive.Disagree → Agree
[A3] Overall, the agent saved us time.Disagree → Agree
[A4] Overall, the agent reduced our workload.Disagree → Agree
[A5] Overall, the agent facilitated our collaboration.Disagree → Agree
[A6] The information provided by the agent was reliable.Disagree → Agree
[A7] Overall, how satisfied were you with the agent?Dissatisfied → Satisfied

Table 4. Agent Experience Questionnaire

Once all four collaborative search tasks are complete, each participant takes part individually in a face-to-face, semi-structured interview. Through open-ended questions, the interviews elicit detailed reflections that fixed-response questionnaires cannot capture. Participants are asked to (1) rank the four search scenarios from most to least preferred and describe their experience with each; (2) discuss the advantages and disadvantages of QAAgent relative to the other scenarios; (3) discuss the advantages and disadvantages of CoSearchAgent relative to the other scenarios; and (4) suggest additional capabilities that an AI assistant should provide for collaborative multi-user search.

Results

We organize the results into three parts. Section 5.1 examines participants’ search experience through the workload, collaboration-experience, and agent-experience questionnaires. Section 5.2 then checks user behavior through communication activity, evidence use, and task-completion time. Section 5.3 measures report qualities from various perspectives.

Search Experience (RQ1)

Workload

To evaluate workload, Figure 3 compares four scenarios across six dimensions, with higher scores indicating more favorable experiences. QAAgent achieves the highest ratings for mental demand (W1=4.38W1 = 4.38), physical demand (W2=5.04W2 = 5.04), temporal demand (W3=5.50W3 = 5.50), and effort (W5=4.46W5 = 4.46), whereas CoSearchAgent leads in perceived performance (W4=5.03W4 = 5.03) and frustration (W6=5.59W6 = 5.59). Both LLM-based agents significantly outperform Google Search on W1, W2, W3, and W5, while QAAgent outperforms SearchAgent on W1–W3, W4 and W6 show no significant pairwise differences, indicating that workload benefits of LLM-based assistance concern reduced task demands and effort rather than perceived performance or frustration.

Mean workload ratings across scenarios

Figure 3. Mean workload ratings across scenarios. * and † indicate significant differences from Google and SearchAgent, respectively (p<.05p < .05).

Collaboration Experience

To assess collaboration experience, Figure 4 compares mutual awareness, information sharing, division of labor, and communication continuity across the four scenarios. QAAgent receives the highest mean rating on all five items (5.83–6.41) and significantly outperforms Google Search on C1–C5, SearchAgent on C1–C3, and CoSearchAgent on C1. CoSearchAgent also significantly exceeds Google Search on C1, C2, C3, and C5. These results show that LLM-based assistance generally improves collaboration relative to conventional user-led search, with QAAgent producing the broadest gains, particularly in mutual awareness and information sharing.

Mean collaboration-experience ratings across scenarios

Figure 4. Mean collaboration-experience ratings across scenarios. * and † indicate significant differences from Google and SearchAgent, respectively (p<.05p < .05).

Agent Experience

To examine participants’ evaluations of the three systems, Figure 5 compares usefulness, completeness, time savings, workload reduction, collaboration support, reliability, and overall satisfaction. CoSearchAgent ranks highest on five of seven items: usefulness (A1 = 5.45), completeness (A2 = 5.21), time savings (A3 = 5.70), workload reduction (A4 = 5.81), and overall satisfaction (A7 = 5.48). It significantly outperforms QAAgent on completeness, SearchAgent on workload reduction, and both agents on overall satisfaction. QAAgent ranks highest on collaboration support (A5 = 5.64), while SearchAgent ranks highest on reliability (A6 = 5.51) and significantly outperforms QAAgent. Overall, CoSearchAgent provides the strongest experience, while QAAgent’s lower completeness and reliability highlight the limitations of direct answer generation without web grounding.

Mean agent-experience ratings across scenarios

Figure 5. Mean agent-experience ratings across scenarios. * indicates a significant difference from SearchAgent (p<.05p < .05).

User Behavior (RQ2)

5.2.1 Communication Activity. To compare communication activity across search-support scenarios, we exclude agent-generated messages and analyze human-authored turns and agent invocations. Participants produce 39.23 turns per task with QAAgent, versus 30.60 with SearchAgent, 28.90 with CoSearchAgent, and 28.13 with Google Search (Figure 6a). QAAgent is invoked most frequently (25.63 times per task), followed by CoSearchAgent (16.80) and SearchAgent (11.48) (Figure 6b). Direct-answer assistance thus prompts repeated information requests and discussion, whereas communication under CoSearchAgent resembles that with conventional search tools.

Communication behavior across scenarios

Figure 6. Communication behavior across scenarios: (a) conversation turns and (b) explicit agent invocations.

5.2.2 Evidence Use. To examine how search scenarios shape evidence use, we compare URL sharing and result-list interactions across scenarios with external sources. Participants share 11.80 URLs per task with Google Search, versus 2.08 with SearchAgent and 1.03 with CoSearchAgent (Figure 7a); QAAgent is excluded because it provides no external sources. This pattern suggests a shift from manually sharing sources found outside Slack to discussing evidence already available in the channel. Participants also click fewer results with CoSearchAgent than with SearchAgent (13.18 versus 23.20) and perform fewer result-page navigation actions (25.53 versus 30.48; Figure 7b). Thus, CoSearchAgent enables participants to begin with a synthesized, web-grounded answer and selectively inspect its sources for further detail or verification.

Evidence-use behavior

Figure 7. Evidence-use behavior: (a) URLs shared and (b) result-list actions; percentages show reductions from SearchAgent to CoSearchAgent.

5.2.3 Temporal Behavior. To measure how search support shapes the temporal organization of collaborative search, we assess interaction duration, defined as the time until partners cease exchanging messages, and the proportion of human-authored messages receiving a partner response within 60 seconds. Mean interaction duration is 16.05 minutes with Google Search, 17.51 with SearchAgent, 19.28 with QAAgent, and 20.48 with CoSearchAgent (Figure 8a). Agent-assisted scenarios therefore sustain interaction longer than Google Search. The 60-second response rate is highest for QAAgent (66.8%), followed by SearchAgent (61.4%), Google Search (58.9%), and CoSearchAgent (51.8%) (Figure 8b). QAAgent thus supports more rapid exchanges, whereas CoSearchAgent combines longer interaction with slower turn-taking, consistent with its clarification, retrieval, and synthesis stages.

Temporal behavior across scenarios

Figure 8. Temporal behavior across scenarios: (a) task-completion time and (b) 60-second partner-response rate.

Report Quality (RQ3)

We assess 320 reports using four measures. Each report is evaluated twice by GPT-5.6-Sol using the task description and report text as input, with participant identities omitted. Richness (1–7) measures

whether a report contains sufficient relevant and specific information. For example, a rich report on air pollution describes major pollutants, their sources, health effects, and solutions. Synthesis (1–7) measures whether the report connects these facts into a clear explanation instead of simply listing them. Independence is the percentage of 30-character text sequences that do not also appear in another pair’s report on the same task; a higher value indicates less reuse of shared or source-like wording but does not by itself prove plagiarism or originality. Finally, gap is the absolute difference in overall quality scores between the two participants in the same pair, with a lower value indicating more consistent report quality.

Table 5 shows that Google Search and SearchAgent produce richer reports (4.56 and 4.48) than QAAgent and CoSearchAgent (4.21 and 4.24), partly because response-length limits on the lightweight agents in multi-party conversations make copied agent responses less informative than webpage content. Independence exhibits the reverse pattern: QAAgent ranks highest (99.45%), followed by CoSearchAgent (83.84%), Google Search (60.77%), and SearchAgent (58.57%). Both LLM-based agents significantly outperform the two conventional scenarios on this measure (p<.001p < .001). Among the three scenarios that provide participants with access to browsable web evidence, CoSearchAgent records the highest independence score. Pair gaps remain similar (0.50–0.67), with QAAgent exhibiting the smallest gap. Overall, direct access to result pages helps participants include more factual detail but also leads to greater reuse of source wording. QAAgent best supports independent expression, whereas CoSearchAgent provides the strongest balance between factual richness and independence.

ScenarioRichness ↑\uparrowSynthesis ↑\uparrowIndependence ↑\uparrowGap ↓\downarrow
Google Search4.564.6460.77%<u>0.54</u>
SearchAgent<u>4.48</u><u>4.62</u>58.57%0.67
QAAgent4.214.4899.45%†0.50
CoSearchAgent4.244.44<u>83.84%</u>†0.64

Table 5. Final-report quality. Bold: overall or web-evidence best; underline: second overall. †: p<.001p < .001 vs. Google Search and SearchAgent.

Discussion

We interpret quantitative results using post-study interviews. We group participants’ comments into recurring themes, and the number in parentheses indicates the number of unique participants whose responses are assigned to that theme.

Agent Trade-offs

Search Efficiency

Compared with Google Search and SearchAgent, LLM-based agents reduce effort in locating and synthesizing information. QAAgent users value answers, efficiency (50), speed (16), and concision (14), matching favorable workload ratings. CoSearchAgent reduces manual inspection by combining synthesis with evidence. Yet lower effort does not shorten sessions. Google Search remains fastest, while users find CoSearchAgent slow (23) and its clarification process burdensome (13); QAAgent also demands prompt reformulation (17). Thus, LLM-based assistance improves efficiency by transferring search and synthesis to agents, but latency and interaction overhead limit time savings.

Information Quality

LLM-based agents simplify information consumption but raise completeness and trust concerns. These are strongest for QAAgent, which lacks source evidence: users report missing sources and low confidence (30), insufficient content (26), broad or generic explanations (20), and repetition (18). CoSearchAgent mitigates them by pairing summaries with inspectable links, valued by 52; 18 highlight reliability. Yet web grounding cannot guarantee quality: questionable sources (8), weakly related links (7), and irrelevant or misunderstood answers (11) persist. SearchAgent therefore retains the highest reliability rating, underscoring that effective LLM assistance requires both synthesis and reliable verification paths.

Collaborative Interaction

Embedding LLM-based agents in shared conversations strengthens collaboration because partners see queries and answers. QAAgent leads in mutual awareness, information sharing, work division, and communication continuity; use also turns direct answers into shared discussion resources. CoSearchAgent’s clarification questions (12) and context-sensitive interaction (9) help teams refine ambiguous needs. Yet benefits depend on accurate, economical interaction. CoSearchAgent may misread context (8), over-clarify (13), or provide excessive content (8), while QAAgent demands careful follow-up phrasing. Thus, LLM agents improve coordination only when interventions match group intent without disrupting conversation.

Future Needs

Knowledge Organization

Most future requests focus on organizing the shared search process. Participants request summaries of conversations or search results (14), outlines or mind maps (13), content marking (7), and duplicate-question handling (5). These requests suggest that users need support in maintaining a shared information space, not merely answering queries. A useful agent should preserve key findings, connect related evidence, and support revisiting earlier decisions, extending it into a lightweight memory and synthesis layer for collaborative search.

Search Guidance

Participants also request help deciding what to search next, including refined search suggestions (10), proactively selectable directions (8), user-specified sources (5), and extraction of key content from linked pages (4). Some also request task decomposition and work allocation (3), as well as proactive participation in conversations (3). These requests support guided exploration: the agent should identify gaps, propose alternatives, and distribute work while leaving final decisions to collaborators. Moreover, reliable web extraction and synthesis are essential, as poor retrieval can weaken otherwise useful guidance.

Interaction Control

Future assistance must be easy to invoke, inspect, and control. Participants request one-click invocation, copying, and link sharing (9), clearer colors and citations (7), multimodal support for images, video, audio, and files (7), and shared files or click histories (3). Interview feedback also warns against slow responses, excessive content, and poorly timed interventions. Proactivity should therefore remain controllable: users should be able to trigger, dismiss, revise, or defer summaries and suggestions. These requests imply a collaborative workspace with persistent context, transparent evidence, multimodal interaction, and lightweight coordination that preserves user control.

Conclusion

In this work, we compare LLM-based and conventional support for lightweight collaborative search through a study of 80 participants in 40 pairs. Results indicate that LLM assistance generally reduces perceived task demands and improves collaboration, with effects varying by assistance type. QAAgent supports the most active, responsive conversations and receives the highest collaboration ratings, while CoSearchAgent offers the strongest overall experience and reduces result inspection through source-grounded synthesis. However, users remain concerned about latency, incomplete answers, source credibility, and excessive or poorly timed interventions. Conventional tools produce richer reports than the LLM-based agents, partly because webpages provide more detail than length-limited agent responses. LLM-based reports show less wording overlap across reports. Among scenarios with browsable Web evidence, CoSearchAgent combines the highest report independence with factual richness close to that of the other scenarios. Report quality is also similarly consistent within pairs across scenarios. These findings suggest that collaborative search agents should help teams make sense of shared evidence while leaving users in control of the assistance they receive.

Acknowledgments

This research was sponsored by the Natural Science Foundation of China (62572475, 61902209, 62377044), Intelligent Social Governance Platform, Major Innovation Planning Interdisciplinary Platform for the "Double-First Class" Initiative, Renmin University of China, the Research Funds of Renmin University of China (22XNKJ15), and Beijing Nova Program.

References

References

  1. [1]Mohammad Alianne­jadi, Leif Azzopardi, Hamed Zamani, Evangelos Kanoulas, Paul Thomas, and Nick Craswell. 2021. Analysing mixed initiatives and search strategies during conversational search. In Proceedings of the 30th ACM international conference on information & knowledge management. 16–26.
  2. [2]Saleema Amershi and Meredith Ringel Morris. 2008. CoSearch: a system for co-located collaborative web search. In Proceedings of the SIGCHI conference on human factors in computing systems. 1647–1656.
  3. [3]Sandeep Avula and Jaime Arguello. 2020. Wizard of oz interface to study system initiative for conversational search. In Proceedings of the 2020 conference on human information interaction and retrieval. 447–451.DOI
  4. [4]Sandeep Avula, Jaime Arguello, Robert Capra, Jordan Dodson, Yuhui Huang, and Filip Radlinski. 2019. Embedding search into a conversational platform to support collaborative search. In Proceedings of the 2019 Conference on Human Information Interaction and Retrieval. 15–23.DOI
  5. [5]Sandeep Avula, Gordon Chadwick, Jaime Arguello, and Robert Capra. 2018. Searchbots: User engagement with chatbots during collaborative search. In Proceedings of the 2018 conference on human information interaction & retrieval. 52–61.
  6. [6]Sandeep Avula, Bogeum Choi, and Jaime Arguello. 2022. The effects of system initiative during conversational collaborative search. Proceedings of the ACM on Human-Computer Interaction 6, CSCW1 (2022), 1–30.DOI
  7. [7]Sandeep Avula, Bogeum Choi, and Jaime Arguello. 2023. Why and When: Understanding System Initiative during Conversational Collaborative Search. arXiv preprint arXiv:2303.13484 (2023).arxiv.org/abs/2303.13484
  8. [8]James V Bradley. 1958. Complete counterbalancing of immediate sequential effects in a Latin square design. J. Amer. Statist. Assoc. 53, 282 (1958), 525–528.
  9. [9]Robert Capra, Annie T Chen, Katie Hawthorne, and Jaime Arguello. 2012. ResultsSpace: An experimental collaborative search environment. Proceedings of the American Society for Information Science and Technology 49, 1 (2012), 1–4.
  10. [10]Shiqi Chen, Yiran Zhao, Jinghan Zhang, J Chen, Siyang Gao, Pengfei Liu, Junxian He, et al. 2023. Felm: Benchmarking factuality evaluation of large language models. arXiv preprint arXiv:2310.00741 (2023).arxiv.org/abs/2310.00741
  11. [11]Abdigani Diriye and Gene Saltonchinsky. 2012. Quantum: a session-based collaborative search system. In European Conference on Information Retrieval. Springer, 583–584.DOI
  12. [12]Jill Freyne, Barry Smyth, Maurice Coyle, Evelyn Balfe, and Peter Briggs. 2004. Further experiments on collaborative ranking in community-based web search. Artificial Intelligence Review 21, 3 (2004), 229–252.DOI
  13. [13]Peiyuan Gong, Jiamian Li, and Jiaxin Mao. 2024. Cosearchagent: a lightweight collaborative search agent with large language models. In Proceedings of the 47th International ACM SIGIR Conference on Research and Development in Information Retrieval. 2729–2733.arxiv.org/abs/2402.06360
  14. [14]Roberto González-Ibáñez, Muge Haseki, and Chirag Shah. 2013. Let’s search together, but not too close! An analysis of communication and performance in collaborative information seeking. Information Processing & Management 49, 5 (2013), 1165–1179.
  15. [15]Roberto González-Ibáñez and Chirag Shah. 2011. Coagmento: A system for supporting collaborative information seeking. proceedings of the American Society for Information Science and Technology 48, 1 (2011), 1–4.
  16. [16]Morten Hertzum. 2008. Collaborative information seeking: The combined activity of information seeking and collaborative grounding. Information Processing & Management 44, 2 (2008), 957–962.DOI
  17. [17]Morten Hertzum and Preben Hansen. 2019. Empirical studies of collaborative information seeking: A review of methodological issues. Journal of documentation 75, 1 (2019), 140–163.DOI
  18. [18]Zhengbao Jiang, Frank Xu, Luyu Gao, Zhiqing Sun, Qian Liu, Jane Dwivedi-Yu, Yiming Yang, Jamie Callan, and Graham Neubig. 2023. Active retrieval augmented generation. In Proceedings of the 2023 conference on empirical methods in natural language processing. 7969–7992.
  19. [19]Jiajie Jin, Yutao Zhu, Zhicheng Dou, Guanting Dong, Xinyu Yang, Chenghao Zhang, Tong Zhao, Zhao Yang, and Ji-Rong Wen. 2025. Flashrag: A modular toolkit for efficient retrieval-augmented generation research. In Companion Proceedings of the ACM on Web Conference 2025. 737–740.
  20. [20]Guohao Luo, Hasan Abed Al Kader Hammoud, Hani Itani, Dmitrii Khizbullin, and Bernard Ghanem. 2023. Camel: Communicative agents for "mind" exploration of large language model society. arXiv preprint arXiv:2303.17760 (2023).
  21. [21]Manqing Mao, Paishun Ting, Yijian Xiang, Mingyang Xu, Julia Chen, and Jianzhe Lin. 2024. Multi-user chat assistant (MUCA): a framework using LLMS to facilitate group conversations. arXiv preprint arXiv:2401.04883 (2024).
  22. [22]Fengran Mo, Kelong Mao, Yutao Zhu, Yihong Wu, Kaiyu Huang, and Jian-Yun Nie. 2023. ConvGQR: Generative query reformulation for conversational search. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). 4998–5012.
  23. [23]Meredith Ringel Morris. 2013. Collaborative search revisited. In Proceedings of the 2013 conference on Computer supported cooperative work. 1181–1192.DOI
  24. [24]Meredith Ringel Morris and Eric Horvitz. 2007. SearchTogether: an interface for collaborative web search. In Proceedings of the 20th annual ACM symposium on User interface software and technology. 3–12.
  25. [25]Sharoda Paul and Meredith Ringel Morris. 2009. CoSense: enhancing sensemaking for collaborative web search. In Proceedings of CHI 2009.
  26. [26]Fabio Petroni, Tim Rocktäschel, Sebastian Riedel, Patrick Lewis, Alexander H Miller, and Sebastian Riedel. 2019. Language models as knowledge bases? arXiv preprint arXiv:1909.01066 (2019).
  27. [27]Jeremy Pickens, Gene Golovchinsky, Chirag Shah, Pernilla Qvarfordt, and Maribeth Back. 2008. Algorithmic mediation for collaborative exploratory search. In Proceedings of the 31st annual international ACM SIGIR conference on Research and development in information retrieval. 315–322.
  28. [28]Shandrugha Rikarnо Putra, Felipe Moraes, and Claudia Hauff. 2018. Searchx: Empowering collaborative search research. In The 41st International ACM SIGIR Conference on Research & Development in Information Retrieval. 1265–1268.
  29. [29]Cheng Qian, Bingxiang He, Zhong Zhuang, Jia Deng, Yujia Qin, Xin Cong, Zhong Zhang, Jie Zhou, Yankai Lin, Zhiyuan Liu, et al. 2024. Tell me more! towards implicit user intention understanding of language model driven agents. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). 1088–1113.
  30. [30]Hayne W Reeve. 1997. Counterbalancing and other uses of repeated-measures Latin-square designs: Analyses and interpretations. Journal of Experimental Child Psychology 64, 1 (1997), 137–158.
  31. [31]Adam Roberts, Colin Raffel, and Noam Shazeer. 2020. How much knowledge can you pack into the parameters of a language model? In Proceedings of the 2020 conference on empirical methods in natural language processing (EMNLP). 5418–5426.arxiv.org/abs/2002.08910
  32. [32]Chirag Shah. 2010. Collaborative information seeking: A literature review. (2010).DOI
  33. [33]Chirag Shah. 2014. Collaborative information seeking. Journal of the Association for Information Science and Technology 65, 2 (2014), 215–236.
  34. [34]Chirag Shah and Gary Marchionini. 2010. Awareness in collaborative information seeking. Journal of the American Society for Information Science and Technology 61, 10 (2010), 1970–1986.DOI
  35. [35]Laure Soulier, Lynda Tamine, and Wahiba Bahsoun. 2013. A collaborative document ranking model for a multi-faceted search. In Asia Information Retrieval Symposium. Springer, 109–120.DOI
  36. [36]Chao-Hong Tan, Jia-Chen Gu, and Zhen-Hua Ling. 2023. Is ChatGPT a good multi-party conversation solver? In Findings of the association for computational linguistics: EMNLP 2023. 4905–4915.
  37. [37]Shi-Qi Yan, Jia-Chen Gu, Yun Zhu, and Zhen-Hua Ling. 2024. Corrective retrieval augmented generation. (2024).arxiv.org/abs/2401.15884
  38. [38]Fanghua Ye, Meng Fang, Shenghui Li, and Emine Yilmaz. 2023. Enhancing conversational search: Large language model-aided informative query rewriting. In Findings of the Association for Computational Linguistics: EMNLP 2023. 5985–6006.

Paper details

vlnc:W_gbqffxed
Version
v1 · latest
Published
Fresh — added to the Hub within the last 30 days
Added
License
CC BY 4.0
Usage

views · 0

downloads · 0

A view counts once per browser per day, a download each time the PDF is requested.

  • Computer Science

Not recorded for this paper

View statement
Looking for related papers…
Contents