WorksComputer Science
HiPaper: A Human-AI Collaborative System for Academic Literature Triage and Scholarly Sensemaking
Abstract
Scholarly sensemaking requires researchers to iteratively retrieve, collect, and synthesize literature as their understanding evolves. However, existing scholarly tools provide only partial support for this process. Recent artificial intelligence (AI)-based systems generally rely on opaque submit-and-wait pipelines that it users' ability to inspect, revise, and validate intermediate outputs. In this study, we present HiPaper, a human-AI collaborative literature review system informed by a formative study with academic experts. HiPaper organizes scholarly sensemaking as a three-stage interactive workflow spanning Retrieval, Collection, andSynthesis, centralized on the human-in-the-loop paradigm that exposes intermediate AI-generated content for user inspection and intervention through a persistent workspace called Knowledge Pocket. Through this workflow, users can iteratively refine search intent, curate and organize papers with AI-enriched metadata and clustering, and conduct evidence-traceable synthesis over selected literature. Our user study with 14 Ph.D. students showed that HiPaper efficiently supports scholarly sensemaking by enabling human-AI collaboration in both familiar and unfamiliar domains. Furthermore, the findings highlight how staged interactions with intermediate AI-generated content successfully scaffold user interactions throughout the sensemaking process.
Introduction
Scholarly sensemaking is a foundational research activity that helps scholars understand prior work, identify knowledge gaps, and develop new directions, primarily through literature review [59, 60]. Researchers generally begin with vague ideas, refine their queries as they learn, compare findings across papers, and gradually construct an evolving understanding of a topic. Rather than consisting solely of paper retrieval, scholarly sensemaking unfolds as an iterative process of retrieval, collection, and synthesis [40], in which researchers refine queries, organize information, compare evidence, and build an evolving understanding of a topic.
Despite its importance, existing scholarly tools only partially support literature review as an iterative exploration process. Traditional scholarly retrieval systems, such as Google Scholar, primarily employ keyword-based paper retrieval and offer limited support for follow-up processes like comparing findings across papers. Recent generative artificial intelligence (GenAI)-based systems aim to automate the review process end-to-end via a “submit-and-wait” interaction model. Users specify an initial goal and passively wait while the system executes an opaque agentic pipeline, which can take up to 198 seconds to process one document [3]. As a result, they create a fundamental mismatch with scholarly sensemaking, which develops through iterative refinement of search goals, evidence interpretation, and emerging understanding.
This mismatch is not merely a usability issue. When retrieval, collection, and synthesis are hidden inside an automated pipeline, users have limited opportunities to inspect intermediate outputs, revise search directions, or intervene before downstream synthesis is produced. Prior research suggests that such low-transparency interactions can frustrate users [20] and may weaken their ability to critically assess evidence [5, 6, 38]. These limitations motivate a human-in-the-loop (HITL) approach to artificial intelligence (AI)-assisted scholarly sensemaking. Instead of hiding intermediate AI-generated content, such systems should expose it for user inspection and support actions such as accepting, rejecting, reorganizing, and building upon partial results across multiple stages of work. In this way, user steering can remain tightly coupled with AI assistance and better align with the iterative nature of scholarly sensemaking [26].
To ground this design space, we conducted a formative study with academic experts. Our findings revealed recurring concerns about the automated nature of current systems, including reduced control and the risk of hallucinated outputs. They also pointed to a three-stage review strategy: widespread literature retrieval, AI-assisted literature collection, and synthesizing evidence through focused cross-paper comparisons. These findings suggest that Human-AI collaboration for academic literature triage and sensemaking should be organized as a staged interaction workflow that supports sustained user control over intermediate stages.
Building upon these findings, we present HiPAPER, a human-AI collaborative literature review system designed as a multi-stage HITL interactive workflow spanning Retrieval, Collection, and Synthesis, as illustrated in Figure 1. HiPAPER provides intermediate AI-generated content at each stage for user inspection and intervention. In Retrieval stage, users draft search intent and refine it through AI-generated query suggestions. During Collection, the system organizes retrieved papers into clusters and extracts paper structures, supporting user intervention through a dedicated structural workspace, termed “Knowledge Pocket”, to store and organize selected papers. In Synthesis, users ask paper-grounded questions over curated papers, and the system generates grounded insights that can be further probed through follow-up questions and validation.

Figure 1. HiPAPER: A system with three-stage human-AI collaborative workflow. Retrieval (Left): User express research intent, and the AI expands query terms via keyword suggestions; users decide to accept or refuse. Collection (Middle): AI organizes retrieved papers into clusters and extracts paper structures; users explore and curate evidence within a dedicated Knowledge Pocket workspace. Synthesis (Right): Users ask comparison questions over curated papers; AI generates grounded insights; users validate and iterate through follow-up questions.
HiPAPER was evaluated through a user study with 14 Ph.D. students. Our findings show that HiPAPER efficiently supports scholarly sensemaking by enabling human-AI collaboration across Retrieval, Collection, and Synthesis. The study further illustrates how the system supports a staged human–AI collaborative workflow across both familiar and unfamiliar knowledge domains, enabling users to engage continuously with intermediate AI-generated content throughout the sensemaking process.
Our main contributions are summarized as follows:
We conducted a formative study to understand workflows in scholarly sensemaking, identifying a three-stage strategy and associated design needs for human-AI collaboration.
We present HiPAPER, a human-AI collaborative system that combines a three-stage interaction workflow with Knowledge Pocket, a persistent workspace for organizing intermediate outputs to support user inspection, curation, and validation throughout literature review.
Through a user study with 14 Ph.D. students, we showed that HiPAPER efficiently supports scholarly sensemaking by enabling users to inspect, curate, and validate AI-generated intermediate outputs.
Related Work
Scholarly sensemaking encompasses the iterative process of literature triage [18, 61] and synthesis [30, 39] across Retrieval, Collection, and Synthesis.
As a primary entry for literature retrieval, traditional academic search engines [13, 29], such as Google Scholar [21], Semantic Scholar [52], and DBLP [31], still rely on keyword matching as a core mechanism, offering limited support for the complex cognitive activities involved in the literature review process [28]. With the advancement of large language models (LLMs) [27, 58, 65, 69], recent research [23, 64] is exploring integrating agent-based capabilities into scientific discovery pipelines to transcend the limitations of traditional retrieval paradigms [15]. Although these approaches represent an advance over conventional search tools in terms of automation and reasoning, their underlying retrieval stage still largely relies on traditional search engines, leaving the fundamental limitations of keyword-based retrieval largely unaddressed.
The collection stage [37, 63] refers to the phase in which researchers use reference management tools [68] such as EndNote [16] and Zotero [70] to collect, filter, and organize the acquired literature after an initial retrieval, progressively building a more tractable and research-relevant corpus. Prior work [17, 67] has explored the development of intelligent systems to enhance both efficiency and automation in literature collection. Nevertheless, these approaches remain predominantly oriented toward collection and management efficiency, providing limited support for incremental sensemaking processes that emerge throughout the course of a literature review.
The synthesis stage [51, 56] refers to the process by which researchers integrate, interpret, and critically reflect on the collected literature, aiming to construct a coherent understanding of a given research domain. Related work [32, 53, 55] has explored methods such as visual organization to support researchers in progressively forming [33] and evolving a holistic conceptual grasp of the field throughout the synthesis process. However, such approaches are highly dependent on individual researchers’ expertise and manual intervention, posing significant challenges in terms of both operational complexity [14] and time investment [49]. Recently, GenAI tools [46, 47] such as Elicit [62], SciSpace [25], and Research Rabbit [48] have further expanded the modes of support available for literature synthesis. Nevertheless, these systems exhibit several critical shortcomings [7], including limited process transparency, pronounced susceptibility to hallucination, and inadequate mechanisms for meaningful user intervention.
Relevance research [4, 24] indicates that more effective system design should reside in the collaborative zone between “human-directed” and “fully automated” approaches. Yet most existing tools remain confined to one of these two extremes, either manual retrieval [2, 32] or AI-based generation [9, 66], and consequently fail to support the full pipeline of the literature review process. There is a notable absence of systems that span the entire literature review workflow while substantially embodying the principles of HITL.
Formative Study
We began by investigating how computer science researchers leverage AI-assisted tools for scholarly sensemaking, their typical workflows for this process, and the primary challenges they encounter. Consequently, these insights informed our design objectives.
Methods
Prototype: To effectively elicit insights from the experts, we developed an initial prototype to demonstrate the proposed interaction workflow. Drawing upon standard literature review practices and our own domain expertise, we implemented a paper retrieval interface, a dedicated view for displaying paper metadata, and an AI-powered conversational interface to support question-answering. This version of the system is shown in Appendix B.
Participants: Our formative study recruited five researchers at various career stages within the computer science domain. The cohort included two Ph.D. students (P1, P2), two postdoctoral fellows (P3, P4), and one Assistant Professor (P5). All participants possessed extensive research experience.
Procedures: Each participant was first asked to provide informed consent for the study. Subsequently, we conducted a semi-structured interview via online video conferencing software. Participants were asked a series of open-ended questions, such as: “How do you utilize AI-assisted tools when conducting literature reviews?”, “Please
describe your typical literature review workflow.”, “What challenges do you face when using AI-assisted tools, and how do you resolve them?”, “When writing a review, what is the primary difficulty you encounter in integrating scattered literature into a coherent narrative?”, and “Overall, what would your ideal human-AI collaborative research assistant look like?”
Data Analysis: We used inductive thematic analysis to examine participants’ practices, needs, and challenges. Two authors independently reviewed the interview transcripts and generated initial codes. They then met to discuss discrepancies, iteratively refine the codebook, and consolidate the codes into higher-level themes.
Findings and Design Objectives
Our findings are organized around three scholarly sensemaking stages, identified through thematic analysis. Based on the insights associated with each theme, we derive a corresponding design objective (DO) to inform the refinement of HiPAper.
Retrieval: Intent Articulation, Search Quality, and Researcher Control
Participants consistently raised concerns about the relevance and quality of AI-assisted search results. Current tools were seen as relying too heavily on keyword matching or superficial similarity, often producing mismatched results. As P5 noted, “search for literature based on keyword matching or keyword similarity... quality is uneven.” P2 similarly described “It’s not what I want... strange literature,” while P1 observed that only “two or three of them will be what I’m really looking for.” Participants also emphasized that retrieval should remain human-directed rather than fully automated. P4 stated that “humans [should] control which articles... are included in the analysis.” P2 likewise argued that “the supervisor should dominate the research direction,” and that researchers must “adjust and control the AI’s direction in real time.” P1 warned that otherwise, “the work that [the AI] delivers may not be really what you want... or even in the wrong direction.”
Taken together, these findings suggest that researchers need retrieval support that better captures search intent, improves relevance, and preserves human control during iterative search.
DO-R1: The system should better capture users’ search intent and support iterative refinement of retrieval results for improved relevance and coverage.
DO-R2: The system should facilitate a HITL retrieval workflow by enabling researchers to inspect, review, and steer AI-assisted search outputs in real time.
Collection: Efficient Triage, Metadata Support, and Lightweight Organization
Participants consistently viewed literature triage as a human-directed process that AI should support rather than replace the human role in it. Researchers preferred screening papers manually using metadata such as titles, venues, keywords, and citations. As P4 noted, “I’ll scan through all the titles... such as citations.” Participants also described lightweight evaluation and categorization practices during early triage. P3 explained this as “make a very preliminary classification... A, B, C, D [marks],” and suggested that AI should assist rather than take over: “AI can provide a recommended score.”
At the same time, participants highlighted the inefficiency of fragmented workflows and favored systems that preserve human agency while reducing manual overhead. P5 described “cross-platform writing manually... very difficult,” and proposed “manual selection... categorizes, organizes, etc by AI.” After viewing the prototype, participants further emphasized the value of high-signal metadata and concise summaries during triage. P3 preferred interfaces that “show the journal, its tier, and its citation count... the abstract is not the first requirement.” P4 requested “one or two highly condensed sentences,” and P1 noted that adding papers “one by one... [is] kind of annoying.”
Taken together, these findings suggest the need for a unified collection space that supports human-directed paper triage through rich metadata, concise AI summaries, and lightweight organization.
DO-C1: The system should support efficient, human-directed triage by presenting rich metadata and concise AI summaries that help researchers rapidly assess paper relevance and quality.
DO-C2: The system should support lightweight organization of collected papers within a unified workspace, reducing cross-platform context switching while preserving user control over literature selection.
Synthesis: Trust, Verification, Relationships, and Evidence-Grounded Sensemaking
A major concern raised by participants was the trustworthiness of AI-assisted literature review tools, especially during later-stage interpretation and synthesis. Participants emphasized that although AI can improve efficiency, its outputs must be carefully verified before they can be incorporated into real research workflows. For example, P1 noted, “I need to check both the author and the title... make sure they are correct.” Participants also expressed concerns about the lack of transparency in how AI systems retrieve and prioritize information. As P3 explained, “it may have certain preferences and overlook some important information.” These findings suggest that trust in AI-assisted literature review depends heavily on transparency and verifiability. Researchers need support for inspecting how results are generated and for validating AI outputs against source materials.
Participants raised strong concerns about the trustworthiness of AI-assisted literature review, especially during interpretation and synthesis. Although AI could improve efficiency, its outputs were seen as requiring careful verification. As P1 noted, “I need to check both the author and the title... make sure they are correct.” Participants also worried about limited transparency in how AI retrieves and prioritizes information. As P3 explained, “it may have certain preferences and overlook some important information.” These findings suggest that synthesis support should be transparent, inspectable, and grounded in source materials. Participants also emphasized the need to understand both the thematic structure of a field and its development over time. For instance, participant P1 underscored the importance of “see[ing] the citation relationship between papers... connections between papers across different time periods,” while P5 stressed the value of “creating a roadmap.”
After viewing the prototype, participants further expressed a need for deeper synthesis support. P2 wanted to “understand the motivation behind researching these common problems.” P5 asked to trace answers to “which paper and which specific section provides the evidence,” and P4 emphasized “multiple classification systems... whether Paper A solves the limitations of Paper B.”
In summary, participants need synthesis support to verify AI outputs, examine inter-paper relationships, and generate grounded interpretations with clear evidence traceability.
DO-S1: The system should support transparent, source-grounded validation of AI outputs, enabling researchers to inspect, verify, and trust the generated results during synthesis.
DO-S2: The system should provide multi-dimensional sensemaking support, allowing researchers to dynamically categorize literature, examine relationships between papers, and synthesize information with clear traceability to source materials.
DO-S3: The system should support focused cross-paper inquiry over user-selected papers, enabling researchers to iteratively ask questions, compare evidence, and refine emerging interpretations.
HiPaper
Based on the identified DOs from the formative study, we present HiPaper, a human-AI collaborative system designed based on the HITL paradigm to support scholarly sensemaking, featuring three-stage LLM-powered functionalities for researchers. This section will demonstrate how HiPaper assists researchers in improving literature triage and sensemaking through three iterative stages: Retrieval, Collection, and Synthesis. The technical backbone supporting these functionalities is described in Appendix A.
Retrieval
To support users in expressing and refining their research intent (DO-R1) while preserving researcher control over AI-assisted retrieval (DO-R2), HiPaper provides an interactive retrieval workflow that combines AI assistance with user oversight.
HiPaper supports retrieval as an interactive process rather than a one-shot search action. Users can start with an initial topic or query and progressively refine their search direction as they receive results. This design addresses participants’ concerns that existing retrieval tools often fail to adequately capture what they are actually looking for, especially when search intent is broad, evolving, or difficult to describe with only a few keywords.
To improve retrieval relevance and coverage, HiPaper incorporates AI support to interpret user intent beyond surface-level keyword matching. At the same time, the system does not fully automate retrieval decisions. Instead, users remain actively involved in reviewing returned papers, identifying mismatches, and steering the search toward more relevant directions. By allowing researchers to inspect and iteratively adjust the retrieval process, HiPaper supports a HITL workflow that better aligns retrieval outcomes with users’ evolving goals.
This design addresses DO-R1 by supporting iterative articulation and refinement of search intent, and DO-R2 by enabling users to inspect, review, and steer AI-assisted retrieval outputs in real time.
Collection
The collection stage is designed to support efficient, human-directed triage of retrieved papers and lightweight organization of selected literature in a unified workspace, addressing DO-C1 and DO-C2.
AI Cluster (DO-C2). As illustrated in Panel B of Figure 2, the collection stage provides a cluster space that helps users organize retrieved papers into a structured topical landscape. Instead of relying solely on a flat ranked list, HiPaper groups papers into hierarchical clusters composed of root-level themes and more specific sub-clusters. This interactive design allows users to explore the literature at varying levels of granularity, inspect topical branches, and iteratively narrow their focus as their interests emerge. To support responsive interaction, cluster labels and related metadata are generated through an offline LLM-assisted preprocessing pipeline as described in section A.
Figure 2. HiPAPER retrieval and collection stages interface is organized into five panels. (A) Search Panel: Users enter a query; the AI expands the query with keyword suggestions, which users can Accept or Refuse via dedicated buttons; (B)Cluster & Pocket Panel: Retrieved papers are grouped into topic clusters (e.g., Human Computer Interaction, Machine Learning); users can organize curated papers into named Pockets (e.g., Pocket 1, Pocket 2) and trigger Deep Analysis on selected collections; (C) Paper List Panel: each paper entry displays its title, authors, venue, year, conference tier, relevance score, and two AI-generated summaries — Background and Methodology — enabling rapid triage without opening the full paper; (D) Action Panel: Each paper row with three UI buttons including Add to Pocket, Similar paper, and Dislike. (E) Knowledge Pocket Panel: Users continuously select papers from clusters and manage curated paper collections across multiple named Pockets.
AI-Enriched Metadata (DO-C1). As shown in Panel C of Figure 2, HiPaper provides a set of enriched metadata cues to reduce users’ effort in evaluating the quality and relevance of papers during collection. These cues include venue, publication year, relevance score, paper type, and citation count, which together support rapid triage and help users decide whether a paper should be collected for further inspection.
Under the Abstract column in Figure 2, the system presents two AI-generated summaries by default: a concise description of the research background and a summary of the method. These summaries allow users to quickly assess a paper’s focus and contribution before deciding whether to examine its full abstract.
Paper Actions (DO-C1, DO-C2). HiPaper provides a dedicated quick-actions area for paper triage and curation (Panel D). After reviewing a paper’s metadata, users can perform three primary actions. The first is Add to Pocket, which is anticipated to be the most frequent action. Following Fitts’ law [34], this button has the largest visual area; clicking it gives immediate visual confirmation and adds the paper to the active Knowledge Pocket, as shown in Panel E of Figure 2. To minimize cognitive burden, Knowledge Pockets are purposely designed to be limited to three, aligning with research on working memory capacity [11]. Each pocket uses a hierarchical, cluster-based organization to clarify the structure of the collected papers.
The second action, Similar Paper, opens a panel showing the top 10 related papers, allowing users to examine metadata and add papers to a pocket. The third action, Dislike, collapses and grays out the paper entry, reducing visual salience to help focus on promising papers. After collecting a sufficient set, users click the Deep Analysis button to proceed to the Synthesis stage.
Synthesis
The synthesis stage aims to support transparent verification, multi-dimensional sensemaking, and focused cross-paper inquiry over user-selected literature, addressing DO-S1, DO-S2, and DO-S3.
Knowledge Pocket as a Bridge (DO-S3): As shown in Figure 3, the Knowledge Pocket serves as a bridge between Collection and Synthesis. Papers in a Knowledge Pocket can be carried directly into the synthesis workspace and added to the chat panel (Panel H), enabling grounded analysis. By preserving user selections in a persistent workspace, the Knowledge Pocket supports a smooth transition from triage to deeper synthesis. A dashboard view of the knowledge pocket is provided in Panel G.
Figure 3. HiPAPER’s synthesis interface, organized into four panels. (E) Knowledge Pocket Panel: Users manage curated paper collections across multiple named Pockets. Papers are organized by cluster, with checkboxes for selection, and options to Add to Chat; (F) AI Insight Dashboard: Displays an automatically generated visual summary of relationships (e.g., Different colored lines between nodes) among selected papers, including conceptual links such as Intersection, Interpretation, Expansion, Abstraction, Reification, and Analogy, helping users identify structural connections across literature; (G) Pocket Dashboard: analyzes papers in the Knowledge Pocket and presents a visual summary of structural connections among them. (H) Evidence-Traceable Chat Panel: Users trigger AI-generated synthesis actions, including Summarize, Compare, Timeline, Research Gaps, and Method on selected papers. The AI produces grounded, citation-based responses with inline references (e.g., [1, 2]) traceable to specific papers. Users can further continue iterating via free-form questions in the input field.
AI Insight for Paper Relation Graph (DO-S1, DO-S2): Also shown in Figure 3, the synthesis stage provides an AI Insight Relation Graph to help users understand relationships among the literature they have collected (Panel F). Rather than requiring users to manually compare papers in sequence, HiPaper employed AI to get the potential inter-paper relationships based on the relationships among papers grounded on the prior research [12], which contain relationships as Intersection, Interpretation, Expansion, Abstraction, Reification, Analogy, Substitution, or No Meaningful Relationship.
To support an interpretable and interactive synthesis process, the graph is generated progressively in a streaming manner, enabling users to observe how the relational structure emerges over time. When users click an edge, the system synchronizes the selected relationship with the relationship panel and presents an explanation of the inferred connection. Supporting evidence is also provided to help users validate the AI-generated interpretation. In addition, users can add papers represented in the graph directly to the chat panel for further discussion and analysis, creating a tighter loop between visual sensemaking and conversational synthesis.
Evidence-Traceable AI Chat (DO-S3, DO-S1): To improve the robustness and trustworthiness of AI-assisted analysis, HiPaper incorporates a scoped-retrieval chat mechanism in the synthesis stage. Rather than generating responses over the entire corpus, the system constrains the AI’s reasoning to papers that users explicitly add from the Knowledge Pocket and the paper relationship graph into the chat panel. This design helps keep the conversation grounded in user-selected literature and reduces the risk of irrelevant or unsupported responses. To further support verification, each AI response is accompanied by citations linked to the underlying paper content. When users hover over a citation marker, the system reveals the corresponding evidence, along with condensed paper metadata, in a floating window, enabling users to quickly trace and validate the source basis of the AI’s answer.
User Evaluation
To evaluate HiPaper, we conducted a user study involving 14 Ph.D. students with backgrounds in computer science. Our primary research objectives were twofold: (1) to assess the overall usability of the system, and (2) to investigate how HiPaper supports researchers when navigating literature in both familiar and unfamiliar domains.
Study Design. Informed by the DOs derived from formative study, HiPaper is engineered to support researchers in exploring both familiar and unfamiliar research domains. To evaluate its efficacy, capability, and usability, we employed a within-subjects study design with counterbalanced task ordering. Participants were instructed to complete a rapid literature search task under two distinct conditions: one focusing on a topic within their existing area of expertise, and another exploring a research domain that was new to them.
Participants. This study recruited 14 participants from a single institution. All participants were doctoral students pursuing a Ph.D. degree and reported general research interests in computing-related fields. The participants comprised 10 men and 4 women, aged 25 to 30 years (, ). Because the study used a counterbalanced within-subjects design, participants were evenly assigned to one of two task orders, with 7 completing the unfamiliar-topic task first and 7 completing the familiar-topic task first. The study received ethical approval from the review board of [Institution Name] (reference: [IRB Number]).
Experiment Task. Because literature review inherently involves tracing idea origins, evaluating evidence, and identifying schools of thought [59], we used it as the evaluation context for HiPaper. Based on our formative findings and prior work [44], we hypothesized that literature review behaviors would differ between familiar and unfamiliar topics. To examine this, we designed a controlled task with an identical structure across conditions, varying only topic familiarity. In a pre-study questionnaire, participants rated candidate topics on a 5-point Likert scale. For each participant, a familiar topic () and an unfamiliar topic () were selected such that . We also ensured that each topic returned more than 300 papers in direct search.
Specifically, the task is that participants were asked to identify three key themes in the assigned topic, select two to three representative papers for each theme with justifications, and briefly describe relationships among the themes within 25 minutes. To avoid forcing meaningless interaction, participants could stop once they felt they had completed the task. Notably, our goal was to analyze interaction patterns rather than review quality, which is difficult to assess objectively due to highly individualized review strategies. Finally, besides the time limit, we imposed no additional constraints on how participants used HiPaper.
Procedure. After registration, participants provided informed consent. 1) Pre-questionnaire: Participants reported demographics and topic expertise, which were used to assign tasks. 2) Pre-study Interview: Participants described their usual workflows and prior use of AI-based tools. 3) Tutorial Session: Participants watched a pre-recorded video introducing the main features of HiPaper. 4) Task One: To counterbalance order effects, half of the participants first completed a literature review task in a familiar domain and the other half in an unfamiliar domain. 5) Post-questionnaire One: Participants completed the NASA Task Load Index (NASA-TLX) and a customized User Experience Questionnaire (UEQ). 6) Task Two: Participants then completed a second task in the alternate domain. 7) Post-questionnaire Two: Participants completed the NASA-TLX, customized UEQ, and System Usability Scale (SUS). 8) Post-study Interview: Participants took part in a semi-structured interview about their experience using the system. 9) End of Study: Participants received a supermarket voucher worth approximately 13 USD as compensation for their time.
Data Collection. We collected three types of data to evaluate HiPaper, including quantitative measures, qualitative feedback, and system interaction logs. Together, these data captured usability, perceived support for the literature review, and participants’ actual behaviors during use.
Quantitative Data. We used standardized and customized instruments to assess participants’ usability, workload, and user experience with the system. General Usability was measured using the SUS [8]. Task Workload was assessed using the NASA-TLX [22]. UEQ was measured with a customized multidimensional questionnaire on a 7-point Likert scale. The questionnaire covered four constructs, including Perceived Usefulness (4 items, adapted from the Technology Acceptance Model [35]), Search and Discovery Effectiveness (3 items, based on Marchionini’s Exploratory Search Theory [36]), Organization and Sensemaking Support (3 items, grounded in Sensemaking Theory [43, 50]), and Trust in AI Features (7 items, developed from Human-AI Interaction guidelines [4, 54]).
Qualitative Data. We conducted semi-structured interviews before and after participants interacted with HiPAPER to understand both their prior literature review practices and their experiences with the system. The pre-study interviews examined participants’ existing workflows for literature review and scholarly sensemaking. The post-study interviews focused on their direct interactions with HiPAPER, with attention to their perceptions, strategies, and overall evaluation of the platform.
User Behavior Logs. We also collected objective behavioral data through the built-in logging mechanism of HiPAPER. The system recorded all user interactions, which were grouped into three stages. These included the Retrieval stage, such as search button clicks and interactions with AI features in the search panel, the Collection stage, such as interactions with Smart Clusters, paper management actions, and AI summarization features, and the Synthesis stage, such as actions involving the AI insight panel, AI Chat, and the Knowledge Pocket.
Behavior Log Process
Phase Progression Over Session Time. To characterize participants’ progression through the sensemaking process, we mapped logged events to three high-level phases: Retrieval, Collection, and Synthesis. For each session, time was normalized from 0% to 100% based on session start and end timestamps, allowing sessions of different durations to be compared on a common scale. We then estimated the proportion of sessions in each phase at each point of normalized session progress, which provides a dwell-weighted view of how activity phases evolve throughout a task session.
Action-to-Action Transition Patterns. To capture participants’ interaction strategies, we analyzed action-to-action transitions within each session. Since the number of logs across the two task conditions is uneven, we therefore use the possibility analysis rather than the raw event counts. We constructed transition matrices for familiar-topic and unfamiliar-topic sessions, converted them to conditional probabilities of the form , and compared the two conditions using the difference . This representation highlights which next-step actions were relatively more likely in one condition than the other, independent of overall differences in event volume. Detailed processing steps, event mappings, and hyperparameters are provided in Appendix C.
Results
This section reports behavioral and quantitative results regarding the familiar and unfamiliar topic conditions.
Behavioral Differences Between Familiar and Unfamiliar Topics
Phase Progression Over Session Time
As illustrated in the Figure 4, the temporal distribution of the three states presents distinct patterns between the two types of tasks. Retrieval: While accounting for a relatively small proportion overall, its distribution varied. In Familiar tasks, retrieval was weak and scattered, appearing only occasionally in the middle and later stages. However, in Unfamiliar tasks, retrieval was more intensive, forming distinct bands particularly during the 10%-40% and 50%-70% stages of the task progress. Collection: Participants under familiar topics demonstrated a typical front-focusing pattern. The collection held an absolute dominant position with a major peak, followed by a smooth, monotonic decrease until it almost completely disappeared at the end. In contrast, when participants were working on the unfamiliar topic, the Collection behavior persisted throughout the task. Although there were early peaks, this state maintained a high density and sharp fluctuations. Synthesis: For the participants under a familiar topic, Synthesis progressively increased starting from the middle stage and completely dominated the final stage of the task. However, when participants were under unfamiliar topics, Synthesis appeared fragmented. It emerged early in the task (at roughly 20% progress) but fluctuated, resulting in a zigzag pattern that was highly entangled with the Collection curve throughout the middle and later stages.

Figure 4. Phase Progression Over Task Time.
Action-to-Action Transition Patterns
As shown in Figure 5, Positive values indicate transitions that are more frequent in Familiar tasks, while negative values indicate transitions more frequent in Unfamiliar tasks. a) Retrieval: Participants under Unfamiliar tasks display a higher frequency of consecutive Retrieval actions (-0.16) compared to Familiar tasks. b) Collection: Within the collection stage, Unfamiliar tasks are characterized by a higher frequency of repeated “Cluster” actions (-0.36). In contrast, Familiar tasks show a higher transition probability from “Cluster” to “Action Add To Pocket” (+0.21). c) Synthesis: The most pronounced differences occur within the synthesis stage. When participants are working on the Familiar, they exhibit a substantially higher probability of repeating the “AI Insight Add To Chat” action (+0.56). However, when they are under unfamiliar tasks, a higher transition rate from
Figure 5. Action-to-Action Transition Heatmap.
“AI Insight Add To Chat” to “AI Insight Node Click” (-0.44), as well as repeated “AI Insight Node Click” actions (-0.37).
Quantitative Results
Perceived Workload: A paired-samples t-test was conducted to evaluate the impact of task familiarity on users’ perceived workload. The results indicated no statistically significant difference in the Overall NASA-TLX score between the Familiar (, ) and Unfamiliar (, ) conditions, , . Similarly, no significant differences were found across the six individual subscales.
User Experience: Paired-samples t-tests were conducted to compare user experience metrics between the familiar and unfamiliar task conditions. The analysis revealed a statistically significant difference in Search & Discovery; participants rated the system’s support significantly higher in the unfamiliar condition (, ) compared to the familiar condition (, ), , , Cohen’s . No significant differences were observed for the remaining user experience dimensions. For Perceived Usefulness, no significant difference was found (, ) between the familiar (, ) and unfamiliar (, ) conditions. Similarly, Organization Support showed no significant difference (, ) between the familiar (, ) and unfamiliar (, ) conditions. For Trust in AI, scores did not differ significantly (, ) between the familiar (, ) and unfamiliar (, ) tasks. Finally, User Agency also yielded no significant difference (, ) when comparing the familiar (, ) and unfamiliar (, ) conditions. Additionally, the overall system usability, measured by the System Usability Scale (SUS) across all participants, achieved a mean score of 74.11 (). Overall System Usability: Independent of the task conditions, the overall system usability was evaluated using SUS. Across all 14 participants, the system achieved a mean SUS score of 74.11 ( = 11.66).
Discussion
Our findings indicate that AI-assisted scholarly sensemaking functions as an iterative, revisitable process rather than a rigid, linear progression from query to resolution. Both behavioral traces and interview data revealed that participants alternated continuously between the Retrieval, Collection, and Synthesis stages, particularly when navigating unfamiliar domains. Rather than indicating inefficiency, this non-linear workflow is a fundamental characteristic of AI-supported sensemaking; intermediate AI outputs frequently redefined participants’ subsequent search strategies, relevance criteria, and overall task conceptualization.
To contextualize these quantitative and behavioral outcomes, we leverage participants’ post-study interview feedback to elucidate how they utilized the system across both task conditions. Based on these insights, we derive four key implications for designing and evaluating interactive AI systems for scholarly sensemaking. Systems may be more effective if they foreground different control surfaces depending on epistemic need, rather than offering uniform assistance across all review contexts.
Topic Familiarity Shaped Interaction Strategy
The clearest pattern in our results is the contrast between familiar and unfamiliar tasks. In the phase progression analysis, familiar tasks showed a front-loaded Collection pattern followed by a relatively smooth transition into Synthesis. In contrast, unfamiliar tasks showed more sustained Retrieval activity and a more prolonged alternation between Collection and Synthesis. The customized UEQ results also suggested only Search & Discovery was rated significantly higher in the unfamiliar condition.
Participants’ interviews help explain this difference. Several described unfamiliar-topic work as requiring a broad orientation before focused selection. For example, one participant explained that, in an unfamiliar domain, they first needed to understand “what kinds of content exist at the level of broad modules” before moving to more specific areas, whereas in a familiar domain they could “give more keywords from the beginning to find the subtopic [I] want (P4)”. Another participant similarly noted that the system was “more helpful for the unfamiliar one, because it gives you an overview, while for familiar topics they instead wanted to “go deeper into the papers (P3)”. This interpretation is also echoed by participants who explicitly said that the system helped them “quickly understand a new field (P5)”, but was less useful when they already had a strong mental model of the topic.
These findings suggest that the same interface mechanisms served different functions depending on prior knowledge. In familiar domains, participants used HiPAPER more as a targeted triage and comparison tool. In unfamiliar domains, they relied on it more as an orientation scaffold. This complements prior critiques of one-size-fits-all literature tools by showing that the utility of retrieval, clustering, and synthesis aids is not fixed, but depends in part on the user’s familiarity with the topic and resulting sensemaking needs [32, 53, 55]. For interactive AI systems, this suggests that there is no single best interaction mode for AI-assisted scholarly sensemaking.
Literature Review with AI is Revisitable rather than Strictly Linear
The findings of the user behavioral suggests that AI-supported scholarly sensemaking is iterative rather than one-pass. Especially for unfamiliar topics, participants moved repeatedly between retrieval, collection, and synthesis, with early synthesis often prompting further search and regrouping instead of ending the task. This resonates with classic accounts of sensemaking as iterative and evolving rather than sequential [59, 60].
Interviews reinforced this pattern. One participant described using the analysis view to first understand unfamiliar concepts and then returning to the retrieval stage: “when I don’t know the concept ... I go to the analysis page ... use the AI to understand the definition and related concepts. Based on this understanding, I go back ... to find the paper” (P7). Another participant described a similar progression from overview to targeted follow-up: they first used class-level summaries “to build a macro-level understanding, and then, when I wanted more detail about differences or relationships, I used AI chat to ask further questions (P4).” These accounts align with the observed zigzag pattern in unfamiliar-topic sessions, where partial synthesis appeared to generate new insights rather than simply conclusions.
This suggests that the key design challenge is not only supporting each stage, but also enabling movement across stages without losing context. In HiPAPER, artifacts such as clusters, selected papers, summaries, and pocketed papers persisted across phases and could be reused, helping participants build on partial understanding rather than restarting. This addresses a limitation in many existing systems, which tend to support retrieval, collection, or synthesis in relative isolation [7, 32, 68]. We therefore argue that AI interfaces for scholarly sensemaking should support cross-phase revisitation rather than enforce a linear workflow.
User-Curated Evidence Scopes Support Grounding and Control
The transition heatmap further suggests that familiar tasks were more often moved from Cluster to Add to Pocket, whereas unfamiliar tasks were more likely to remain in repeated exploration. This move represents a concrete mechanism of interpretive control, situating the system within the collaborative zone between user direction and automation [4, 24]. Furthermore, this behavior extends prior research on collection tools [17, 67, 68] by demonstrating that user curation is not merely an organizational task but an epistemically vital grounding mechanism for AI-supported synthesis.
Interview data help explain why this happened. Participants frequently described the Knowledge Pocket as a way to turn candidate papers into a manageable evidence set. One participant said that using the three pockets to classify papers by different subtopics was “very useful to manage the papers and also make comparisons between these pockets (P1)”. Another participant explained “I used one pocket at a time for questions focused on a single topic, but added papers from multiple pockets when the question required comparison across topics (P12)”. A third participant emphasized that paper selection itself should remain human-directed: “paper selection needs to be led by humans (P14).” These comments suggest that pocketing was not just a storage action. It marked a transition from open exploration toward bounded synthesis.
This is suggested in HiPAPER, Knowledge Pockets were more than a bookmarking feature. Participants used them to assemble subsets of papers around a theme, compare papers within a set, and limit downstream analysis to papers they had explicitly selected. In this sense, pockets served as user-curated evidence scopes that shaped what the system could analyze at any given time. For design, this implies that systems should not only help users find papers, but also help them commit to a scoped subset that can be reused in downstream reasoning. In our data, this commitment appeared earlier and more directly in familiar-topic work, but the mechanism itself was valuable in both conditions.
Intermediate Artifacts Function as Control Surfaces, not just Outputs
Throughout the study, we observed participants treated AI-generated outputs not as static answers but as control surfaces to steer subsequent actions. This approach addresses critiques of GenAI transparency and hallucination [7, 46, 47] by facilitating meaningful user intervention. As P5 highlighted, manipulating these intermediate artifacts reduced manual labor while enhancing agency. By ensuring outputs remain inspectable and actionable, the system supports mixed-initiative interaction [4, 24], where AI complements scholarly judgment through human redirection and reinterpretation.
This instrumental approach clarifies why participants derived substantial value from the system, even while acknowledging the AI’s inherent imperfections. Summaries, for instance, were predominantly deployed as triage instruments rather than substitutes for comprehensive reading. P4 utilized them to “make a preliminary judgment on whether a paper was actually worth a deeper dive,” while P1 found that reviewing a summarized methodology was much easier than reading full abstracts” for rapid literature filtering. Furthermore, when navigating unfamiliar domains, clustering provided a crucial high-level topography of the research landscape. Participant P4 noted that examining “broad categories first helped me quickly establish a macro framework before drilling down into specific sub-topics.”
The relation graph likewise supported quick exploration of inter-paper connections. Participants used it after reviewing summaries or insights to ask follow-up questions and better understand relationships across papers. Participant P1 detailed: after reviewing initial summaries to find relevant topics, I would heavily rely on the chat AI to ask questions... to have a general understanding in this field.” Participant P4 corroborated: I would simply ask the chat to expand on the AI’s initial insights.”
The key contribution of HiPAPER is therefore not just integrating retrieval, clustering, and AI-chat, but enabling AI-generated and user-curated artifacts to persist across phases for continued reuse. More broadly, this suggests that interactive AI systems should treat outputs not only as answers, but as manipulable objects that support ongoing user control.
Overall Usability was Promising, Long-Term Effects Remain Exploration
The system’s SUS score of 74.11 indicates reasonable usability, with participants like P4 praising the “smooth and clear” cohesive workflow. By consolidating search, management, and synthesis into a single human–AI pipeline, the system addresses the fragmentation typically observed in literature review processes [2, 32, 68]. As P11 noted, this integration streamlines real-world workflows; however, our findings emphasize that such consolidation must be paired with the ability to inspect source documents and preserve scoped context to be fully effective.
At the same time, participants pointed out some disappointing design flaws. P5 expressed that “the most frustrating part was not being able to jump directly to the PDF”. Others reported unclear graph references, such as “general labels in insight panel, like paper A and paper B” or “difficulty preserving the state of scoped context when reopening pockets”. These frictions help explain why overall usability was positive but not yet excellent.
Additionally, because scholarly sensemaking is rarely a discrete, single-session effort, successful comprehension often requires the longitudinal tracking of a specific research topic. Consequently, the results of this study should be viewed as a preliminary exploration of the short-term effects of introducing a novel human-AI interaction paradigm for scholarly sensemaking. This also leaves open an important gap relative to prior systems work, which has often evaluated literature tools through bounded tasks rather than sustained research practice [9, 66]. The long-term impacts and sustained efficacy of this interaction paradigm remain a critical area for future longitudinal observation.
Limitations
There are several limitations to this study. The first limitation concerns the scope of the dataset; due to resource constraints, this study primarily collected literature published over the past five years, focusing on well-recognized venues within the HCI, Computer Graphics, and AI communities. Additionally, owing to copyright restrictions imposed by certain journals and conferences that lack Open Access policies, we restricted the literature data available in HiPAPER to the abstract level. A second limitation is that both the formative and evaluation studies were conducted with participants possessing strong computer-related backgrounds. Consequently, the comprehensive evaluation of HiPAPER among non-expert end-users remains an area for future exploration. The third limitation involves the loss of three sets of log data. Although we employed probability testing to mitigate this effect, we acknowledge that the missing data presents a limitation for analyzing user behavior logs. The last limitation lies in the study design. Because HiPAPER adopts a HITL design, it is difficult to establish a fully fair comparison with either traditional scholarly retrieval tools, such as Google Scholar, or fully automated AI research tools, such as Deep Research. On the one hand, comparisons with conventional retrieval systems are complicated by the AI-assisted features embedded in HiPAPER. On the other hand, comparisons with fully automated AI research tools are also imperfect, because the human involvement required by our system makes the final outcomes dependent on individual users. Moreover, the current study did not focus on assessing the quality of the final research outcomes themselves. We therefore acknowledge that direct comparative evaluation falls outside the scope of the present work and should be addressed in future long-term research.
Conclusion
In this study, we aimed to address the low transparency and fragmented workflow of AI-based scholarly sensemaking. Building on empirical findings from a formative study, we developed HiPaper, a three-stage human-AI interaction workflow that transforms opaque LLM pipelines into manipulable interaction objects, exposing intermediate artifacts (e.g., paper selection, paper clusters, and grounded insights) for user inspection, curation, and validation across Retrieval, Collection, and Synthesis. Our user study with 14 Ph.D. students demonstrated that participants used the HiPaper system differently in familiar and unfamiliar domains. Overall, HiPaper’s three-stage AI-assisted workflow was perceived as an efficient and effective approach to scholarly sensemaking. Based on these findings, we identify several directions for future research: 1) Longitudinal studies are needed to examine real-world adoption over time. 2) Improve transparency, grounding, and trust in AI-generated synthesis. 3) Extending beyond computer science may help generalize HiPaper’s design principles to broader domains.
A Workflow of HiPaper Backbone
Figure A.1 illustrates the overview of our backend system, revealing the implementation of each backend module that supports human-AI collaboration in academic literature triage and sensemaking.
Data Fetch and Enrich
We curate our corpus with 131,715 papers from 39 high-impact computer science conference and journal venues that primarily cover HCI, AI, Graphics, and Databases, with papers published from 2021 onward, as listed in Table 2 (Appendix D). We first use the DBLP API [31] to fetch papers for each venue-year pair and extract core metadata to yield a reliable and de-duplicated paper set. We then query OpenAlex [45], arXiv [10], OpenReview [42], and paper landing pages to enrich complementary metadata such as abstracts, PDF links, and references with lightweight quality checks. All fields are normalized into a unified schema, as illustrated in Table 3 (Appendix D).
Retrieval
Traditional academic search often relies on keyword matching, which works when users know the exact terminology. However, it can miss semantically relevant papers phrased differently. To enhance retrieval performance, we adopt Semantic Retrieval to represent queries and papers in a shared embedding space, enabling semantic retrieval. Specifically, We encode each paper’s title and abstract with a state-of-the-art text embedding model jina-embeddings-v5-text-nano [1] as . It produces a 768-dimensional dense vector, which we store in Qdrant [57] for efficient vector similarity search. At retrieval time, the user query is encoded with the same model to obtain , and papers are ranked by cosine similarity, . The system returns the top- papers, with default , enabling retrieval of semantically similar work even without exact keyword overlap. Since users are often more familiar with keyword-style search, their initial queries are usually non-specific for semantic retrieval. We therefore add an LLM-based Query Refinement module that interprets user intent and rewrites the input into a more descriptive academic query. The prompt for query refinement is provided in Appendix E.
Collection
To support users to rapidly select preferred papers, our system enriches papers with a hierarchical research taxonomy, concise background and method summaries, and a paper type label. We propose a three-stage LLM pipeline with human verification to address the inadequacy of existing taxonomies for this setting, since curated systems (e.g., ACM CCS [19]) evolve slowly and miss emerging terms, while automated taxonomies [45] yield uneven hierarchies, overlapping categories, and unstable boundaries. i) In the generation stage, the model assigns each paper a taxonomy from its title and abstract, including one research field and method topics. ii) In the reflection stage, the model revises duplicated concepts, hierarchy errors, and inconsistent granularity in its own outputs. We use Qwen-3-4B-Instruct [65] for these two stages to support efficient large-scale processing. iii) In the combination stage, GPT-5.4 [41] merges candidate taxonomies, resolves inconsistent depths, and normalizes semantically similar categories. Human experts then review and revise the final hierarchy. In addition, We prompt LLMs to provide further information to help user capture the main idea of papers by briefly summarizing papers’ backgrounds and methods and inferring paper types. Detailed prompts are provided in Appendix E.
Synthesis
After collecting the preferred papers into the knowledge pockets, our system conducts Paper Relationship Generation and provides Evidence-Traceable QA to assist users in the synthesis step of the literature review.
Streaming Paper Relationship Generation. To help users capture relationships between papers, we implement a Streaming Paper Relationship Generation that incrementally extracts semantic edges from paper titles and abstracts. Given selected paper metadata, the system constructs a relation-extraction prompt based on a predefined relation schema [12]. The LLM then streams its output, while the system accumulates partial responses, parses complete relation edges as they become available, and renders them immediately. The front end updates the graph state online, progressively adding newly generated nodes and links. The prompt is provided in Appendix E.
Evidence-Traceable QA. Accurate interpretation of scientific literature requires precise and verifiable statements. To reduce the risk of LLM hallucination, we ground responses in evidence extracted from the selected paper abstracts and add a final verification step. The evidence-traceable QA first identifies the source statements that are most relevant to the user’s question via Evidence Extraction. These statements are then used as the only support for Answer Drafting, and citation markers link each claim to its source passage. A final Answer Reflection prompt checks whether each claim is grounded in the cited evidence and removes or revises unsupported content. This design improves provenance and reduces unsupported claims. The prompt is provided in Appendix E.
Prototype
In this section, Figures B.1–B.5 present the core workflow of the system, including query input with AI-generated search suggestions, exploration of retrieved papers, paper recommendation, collection of relevant papers into knowledge pockets, and knowledge analysis supported by a LLM assistant. Together, these prototypes demonstrate the intended interaction flow and key functional components of the system.
HiPaper Behavior Log Processing Details
This appendix provides the detailed preprocessing and computation procedures used to derive the two behavioral visualizations reported in the main paper: Phase Progression Over Session Time and Action-to-Action Transition Heatmap.
Data Preparation
Interaction logs were stored as JSON files. Each session was assigned to one of two task conditions, Familiar or Unfamiliar, based on the study design. For all analyses, events within each session were sorted in ascending order by timestamp_ms.
Figure A.1. Backend workflow of HiPAPER. The system moves from data fetching to retrieval, paper collection, and synthesis. (a) Papers from selected venues and years are fetched from metadata sources, normalized into a unified record format, and encoded from title and abstract for vector indexing. (b) A user’s rough query is rewritten into a retrieval-oriented academic query to retrieve semantically similar papers from the vector database. (c) Retrieved papers are collected as structured records and enriched with LLM-generated metadata. Users can also add similar papers through similarity-based browsing. (d) Selected papers are organized into Knowledge Pockets for streaming paper relationship graph construction and evidence-traceable QA.
The two behavioral analyses used slightly different preprocessing steps because they captured different aspects of interaction. The Phase Progression Over Session Time analysis preserved the temporal structure of the full session, including events that did not map to one of the three analytical phases. In contrast, the Action-to-Action Transition analysis focused on meaningful action-to-action behavior and therefore excluded page-level navigation events before constructing transition matrices.
Phase Progression Over Session Time
The Phase Progression analysis summarizes how participants distributed activity across the high-level phases of Retrieval, Collection, and Synthesis over the session of a task.
Session Normalization
For each session, we identified the first and last event timestamps, denoted by and . To make sessions of different durations comparable, we normalized each session onto a common progress scale from 0% to 100%. For each event , its relative session position was computed as
Thus, 0% corresponds to the beginning of the session and 100% to the end, regardless of the session’s absolute duration.
Event-to-Phase Mapping
Each event was mapped to one of four labels using a predefined phase mapping function: Retrieval, Collection, Synthesis, or None. The first three labels correspond to the high-level analytical phases used in HiPAPER. Events that did not belong to any of these three categories were assigned to None. As a result, some portions of a session were intentionally left outside the three focal phases. The mapping details from the raw log to phase please refer to Table 1.
| Raw event type | Phase label | Heatmap label | Note |
search_query | Retrieval | Retrieval | User issues a search query to retrieve candidate papers or information in search page. |
header_search | Retrieval | Retrieval | User issues a search query to retrieve candidate papers or information in the header of explore page. |
action_add_to_pocket | Collection | Action Add To Pocket | User saves a paper or item to the pocket for later consideration. |
action_dislike | Collection | Dislike | User marks a paper or item as irrelevant or undesirable. |
action_remove_from_pocket | Collection | Action Remove From Pocket | User removes a previously saved paper or item from the pocket. |
action_similar | Collection | Similar | User requests or opens similar items related to the current paper. |
click | Collection | Cluster | User clicks a clustered paper/item representation to inspect it. |
filter_change | Collection | Filter | User changes filtering criteria to refine the visible set of papers/items. |
ai_insight_add_to_chat | Synthesis | AI Insight Add To Chat | User sends an AI-generated insight or suggested content into the chat workspace. |
ai_insight_node_click | Synthesis | AI Insight Node Click | User clicks an AI insight node to inspect or expand synthesized information. |
chat_llm_response | Synthesis | Chat LLM Response | System-generated language-model response shown in the chat workspace. |
chat_user_message | Synthesis | Chat User Message | User submits a message or prompt in the chat workspace. |
Table 1. Event mapping used across the two behavioral analyses. The phase-progression analysis mapped raw events to high-level sensemaking phases, while the transition analysis used human-readable action labels for visualization.
Grid-Based Phase Assignment
To derive a comparable session-level representation, we sampled each normalized session on a uniform grid of 401 points spanning 0% to 100% progress. For each grid point , we identified the event interval covering that point. Operationally, event was treated as owning the interval from to , and the phase label of event was assigned to the full interval.
This procedure yielded, for each session, a binary indicator vector for each phase. At each grid point, the indicator takes value 1 if the session was in that phase at that point in normalized progress, and 0 otherwise.

Figure B.1. Prototype Retrieval Workspace.

Figure B.2. Prototype Collection Workspace.
Aggregation Across Sessions
Within each condition, we averaged the binary indicator vectors across all sessions. Therefore, for each phase and each grid point, the resulting value represents the proportion of sessions in that condition that were occupying that phase at that point in normalized progress.
This representation is dwell-weighted rather than event-count-weighted: if participants spent a long duration in a phase, that phase occupies a wider region of the plot even if relatively few events occurred during that time.
Smoothing and Interpretation
To reduce visual noise, the aggregated phase curves were smoothed using a Gaussian filter with grid bins, corresponding to approximately 0.75% of normalized session progress. Smoothed values were clipped to the interval .
An important consequence of the None category is that the phase proportions at a given progress point do not necessarily sum to 1.0. Specifically, when sessions are in events that do not map to Retrieval, Collection, or Synthesis, those sessions do not contribute to any of the three plotted phase proportions.

Figure B.3. Prototype Similar-Paper Recommendation in the Collection Workspace.

Figure B.4. Prototype Synthesis Workspace for Paper Analysis.
Action-to-Action Transition analysis
The transition analysis captures local action-to-action movement patterns within sessions and compares these patterns across the Familiar and Unfamiliar task conditions.
C.3.1 Preprocessing. Before constructing transitions, all events associate with page-level interaction were excluded (such as event type=page change, label= to_Collection). These page-change events were not treated as meaningful interaction actions for this analysis. In addition, the event types search_query and header_search were merged
were merged into a single retrieval category to avoid splitting semantically equivalent search actions across multiple labels.
C.3.2 Transition Extraction. Within each (, ), events were sorted by . Each event was then paired with the immediately following event in the same session, producing a set of ordered transitions of the form (, ). Each transition inherited the condition label of its session.
C.3.3 Transition Event Definitions. For the transition analysis, rows with category == "page" were excluded before transition extraction. In addition, header_search and search_query were merged

Figure B.5. Prototype Synthesis Interface with Graph-Based Analysis and AI Assistant.
into a single search label. All other event types retained their original action labels. Table 1 lists the action labels used in the transition matrix.
Transition Count Matrices
We grouped transitions by (condition, event_type, next_event) and counted their frequency. These counts were then pivoted into square transition matrices, one for each condition. Missing cells were filled with 0. To ensure direct comparability, both matrices used the same row and column ordering. Event labels were ordered first by broader stage grouping (Retrieval → Collection → Synthesis → Other) and then alphabetically within each group.
Row Normalization
Because the two conditions contained different total numbers of transitions, direct comparison of raw counts would be influenced by exposure volume. To control for this, each transition matrix was row-normalized to obtain conditional probabilities:
Equivalently, each row estimates the probability , that is, the probability that action occurs next given that the current action is .
If a row had zero total outgoing transitions in a condition, no conditional distribution could be computed for that action. In such cases, the row was treated as undefined (NaN) and masked in the heatmap visualization.
Difference Matrix
To compare conditions, we computed the difference between the two row-normalized matrices:
Each cell therefore indicates how much more likely a given next action was after current action in familiar-topic sessions than in unfamiliar-topic sessions. Positive values indicate transitions favored by the Familiar condition, whereas negative values indicate transitions favored by the Unfamiliar condition.
Because each row of a row-normalized transition matrix sums to 1, each row of the difference matrix sums to 0, except for rows that are undefined due to missing outgoing transitions in one of the conditions.
Interpretation
This probability-difference formulation emphasizes relative transition tendencies rather than raw interaction volume. For example, a negative value for a self-transition such as Cluster → Cluster indicates that, given a cluster action has just occurred, participants in the Unfamiliar condition were more likely than those in the Familiar condition to immediately perform another cluster action. This interpretation is independent of the total number of transitions observed in each condition.
Summary of Analytical Differences
The two behavioral visualizations differ in both their unit of analysis and their normalization strategy.
Phase Progression Over Session Time uses normalized session progress as the unit of analysis and preserves the temporal structure of the full session. It shows when participants were more likely to be in Retrieval, Collection, or Synthesis over the course of a session.
Action-to-Action Transition Probability Heatmap use event-to-event transitions as the unit of analysis. They show how the conditional probability of moving from one action to another differs between the Familiar and Unfamiliar conditions.
Together, these two analyses provide complementary perspectives on behavior: the Phase Progression Over Task Time captures macro-level temporal progression, while the transition-probability view captures micro-level interaction strategy.
Datasets
In our HiPAPER, we collected 131,715 papers published in 38 selected venues from 2021 to 2025. Table 2 summarizes the 38 publication venues. For each venue, we report its abbreviation, full name, number of records, publication type, and commonly used venue quality indicators, including CORE, CCF, and JCR. The selected venues primarily cover major research areas relevant to our system, including HCI, AI, graphics, and data management.
| Abbreviation | Venue Name | #Record | Domain | CORE | CCF | JCR |
| AAAI | AAAI Conference on Artificial Intelligence | 11416 | Conference | A* | A | – |
| AAMAS | International Conference on Autonomous Agents and Multiagent Systems | 2076 | Conference | A* | B | – |
| ACL | Annual Meeting of the Association for Computational Linguistics | 8374 | Conference | A* | A | – |
| CHI | ACM Conference on Human Factors in Computing Systems | 4612 | Conference | A* | A | – |
| CSCW | ACM Conference on Computer-Supported Cooperative Work and Social Computing | 781 | Conference | A | A | – |
| CVPR | IEEE/CVF Conference on Computer Vision and Pattern Recognition | 11674 | Conference | A* | A | – |
| ECCV | European Conference on Computer Vision | 4034 | Conference | A* | B | – |
| EMNLP | Conference on Empirical Methods in Natural Language Processing | 9230 | Conference | A* | B | – |
| HCI | Human–Computer Interaction | 104 | Journal | – | – | Q1 |
| ICCV | IEEE/CVF International Conference on Computer Vision | 4264 | Conference | A* | A | – |
| ICDE | IEEE International Conference on Data Engineering | 1916 | Conference | A* | A | – |
| ICLR | International Conference on Learning Representations | 9489 | Conference | A* | A | – |
| ICML | International Conference on Machine Learning | 10186 | Conference | A* | A | – |
| IEEE VR | IEEE Conference on Virtual Reality and 3D User Interfaces | 439 | Conference | A* | A | – |
| IJCAI | International Joint Conference on Artificial Intelligence | 4757 | Conference | A* | B | – |
| IJHCI | International Journal of Human-Computer Interaction | 2612 | Journal | – | B | Q1 |
| IJHCS | International Journal of Human-Computer Studies | 597 | Journal | – | A | Q1 |
| ISMAR | IEEE International Symposium on Mixed and Augmented Reality | 565 | Conference | A | B | – |
| ISMAR-Adjunct | IEEE International Symposium on Mixed and Augmented Reality Adjunct | 973 | Conference | – | – | – |
| JMLR | Journal of Machine Learning Research | 619 | Journal | – | A | Q1 |
| KDD | ACM SIGKDD Conference on Knowledge Discovery and Data Mining | 2926 | Conference | A* | A | – |
| MM | ACM International Conference on Multimedia | 2752 | Conference | A* | A | – |
| NeurIPS | Conference on Neural Information Processing Systems | 13255 | Conference | A* | A | – |
| SIGGRAPH | ACM SIGGRAPH Conference | 1112 | Conference | A* | A | – |
| SIGGRAPH-ASIA | ACM SIGGRAPH Conference in Asia | 1053 | Conference | A* | A | – |
| SIGIR | International ACM SIGIR Conference on Research and Development in Information Retrieval | 2179 | Conference | A* | A | – |
| SIGMOD | ACM SIGMOD International Conference on Management of Data | 1157 | Conference | A* | A | – |
| TKDE | IEEE Transactions on Knowledge and Data Engineering | 1206 | Journal | – | A | Q1 |
| TMM | IEEE Transactions on Multimedia | 3071 | Journal | – | A | Q1 |
| TNNLS | IEEE Transactions on Neural Networks and Learning Systems | 4708 | Journal | – | B | Q1 |
| TOCHI | ACM Transactions on Computer-Human Interaction | 317 | Journal | – | A | Q1 |
| TOG | ACM Transactions on Graphics | 1068 | Journal | – | A | Q1 |
| TOIS | ACM Transactions on Information Systems | 552 | Journal | – | A | Q1 |
| TVCG | IEEE Transactions on Visualization and Computer Graphics | 184 | Journal | – | A | Q1 |
| UIST | ACM Symposium on User Interface Software and Technology | 1034 | Conference | A* | A | – |
| VLDB | International Conference on Very Large Data Bases | 1924 | Conference | A* | A | – |
| VRW | IEEE Conference on Virtual Reality Workshops | 1045 | Conference | – | C | – |
| WWW | The Web Conference | 3454 | Conference | A* | A | – |
Table 2. Summary of Venues in the Database of HiPaper Systems.
Table 3 presents the schema of the paper records in our dataset. Each record is structured as a unified representation that integrates bibliographic metadata, content fields, and derived analytical attributes. The schema includes standard identifiers such as DOI, title, authors, venue, and year, as well as access information like PDF URLs and abstracts. In addition, we incorporate LLM-generated annotations, including high-level paper categories, concise TL;DR summaries of background and method, and hierarchical taxonomy labels. Citation-related statistics, such as yearly citation counts and percentile scores, are also included to support impact-aware analysis. This structured schema enables consistent indexing, retrieval, and synthesis over papers.
| Key | Value Type | Example | Meaning |
| doi | str | "10.1609/AAAI.V36I1.12345" | DOI of the paper. |
| title | str | "Anonymized Research Title on Learning Systems" | Paper title. |
| authors | list[str] | ["Author A", "Author B", "Author C"] | Ordered list of authors. |
| venue | str | "AAAI" | Venue abbreviation. |
| year | int / str | 2022 | Publication year. |
| pdf_url | str | "https://publisher.org/article/view/12345/12000" | URL to the paper PDF. |
| abstract | str | "This paper studies an anonymized problem..." | Abstract text of the paper. |
| paper_category | str | "Methodology" | High-level paper category. |
| TL;DR (Background) | str | This paper address an anonymized problem in... | Short background summary. |
| TL;DR (Method) | str | This paper proposed an anonymized method... | Short method summary. |
| citation_percentile | float | 0.49 | Citation percentile score. |
| cited_by_year | list[dict] | [{"year": 2024, "cited_by_count": 1}] | Citation counts grouped by year. |
| reference | list[str] | ["https://openalex.org/W123...", ...] | Referenced works or their identifiers. |
| llm_taxonomy | dict | {"Level 1": "Machine Learning", "Level 2": ["Topic X"]} | LLM-generated taxonomy labels. |
Table 3. Schema Summary of Paper Records (Examples are Anonymized).
Prompts
This section summarizes the prompts used across our HiPAPER. They are organized by functional stage in our system, including Retrieval, Collection, and Synthesis.
Retrieval Prompts
In this section, we illustrate prompts used to convert raw user information needs into retrieval-friendly academic queries. These prompts improve semantic alignment between underspecified user requests and the language commonly used in paper titles and abstracts, thereby supporting more accurate vector-based paper retrieval.
Collection Prompts
In this section, we enrich paper metadata via LLMs to enable a structured collection space. It supports hierarchical taxonomy assignment, taxonomy normalization, taxonomy merging, paper background and method summarization, and coarse paper-type identification, enabling consistent indexing and collection-level organization.
Synthesis Prompts
In this section, we prompt LLMs to synthesize information across multiple papers. They support paper relationship graph construction and validation, evidence extraction for question answering, answer drafting from selected evidence, and answer reflection for improving factual support, relevance, and clarity.
Query Refinement Prompt
System Prompt: You convert rough user search topics into compact academic retrieval queries for semantic vector search. Produce a query that looks natural in research paper titles, abstracts, or author keywords, while staying faithful to the user’s intent.
Guidelines.
Preserve the original intent.
Use natural academic phrasing to improve semantic alignment.
Do not over-specify broad or underspecified queries with invented tasks, methods, domains, datasets, benchmarks, or constraints.
Output only one query.
Maximum 30 words.
User Prompt:
<raw_query>
Level 1 Cluster Generation
System Prompt: Classify the paper into exactly one existing subfield taxonomy. You must choose from the provided subfield list only. Do not create, rewrite, or paraphrase taxonomy names. Return JSON only in this format {"subfield": "<subfield>"}.
User Prompt:
Paper title <title>
Paper abstract <abstract>
Existing subfield taxonomy <subfield_choices_json_array>
Rules
Output exactly one subfield taxonomy.
The output must be exactly one item from Existing subfield taxonomy.
The selected subfield must be the best fit for the paper.
Level 2 Cluster Generation
System Prompt: Select exactly <output_count> <target_desc> taxonomy items for the paper. Strongly prefer reusing existing <target_desc> taxonomy from the provided list. Only create a new taxonomy when none of the existing options is reasonably relevant. Return JSON only in this format {"<target_key>": ["taxonomy1", "taxonomy2", "..."]}.
User Prompt:
Paper title <title>
Paper abstract <abstract>
Latest selected <parent_key> <parent_value>
<extra_context> Existing <target_key> taxonomy <target_choices_json_array>
Rules
Output exactly <output_count> unique <target_key> items.
All <target_key> items must align with the selected <parent_key>.
Reuse existing taxonomy whenever possible, including partially related items.
Create a new taxonomy item only if no existing item is reasonably relevant.
New taxonomy must be broad, reusable, and without abbreviations.
Each taxonomy item must be at most <max_taxonomy_words> words.
<extra_generation_rules>
Previous failed output (fix issues and return valid JSON only): <last_failed_response>
Taxonomy Reflection
System Prompt: You are a strict taxonomy normalizer for <level_key>. Bias strongly toward reusing existing taxonomy terms. Choose covered=true by default when there is reasonable semantic overlap. Return JSON only: {"covered": true, "taxonomy": "<existing_taxonomy>"} or {"covered": false}.
User Prompt:
Paper title
<title>Paper abstract
<abstract>Candidate new
<level_key>taxonomy<candidate>Existing
<level_key>taxonomy list<choices_json_array>Rules
Prefer covered=true whenever any existing taxonomy is semantically close, broader, narrower, or partially overlapping.
Set covered=false only if the candidate is clearly unrelated to all existing taxonomy items.
If covered=true, taxonomy must be exactly one item from the existing taxonomy list.
Avoid creating new taxonomy unless absolutely necessary.
Previous failed output (fix issues and return valid JSON only):
<last_failed_response>
Taxonomy Combination
System Prompt: You merge semantically equivalent or highly overlapping taxonomy items into a unified set. Strongly prefer reusing existing taxonomy names. Do not create redundant or overly specific variants. Return JSON only in this format {"merged_taxonomy": ["taxonomy1", "taxonomy2", "..."]}.
User Prompt:
Input taxonomy list
<taxonomy_list>Existing canonical taxonomy
<existing_taxonomy_list>Rules
Merge items with identical or highly similar meanings.
Prefer a single representative term for each semantic group.
Reuse existing taxonomy whenever possible.
Avoid introducing new taxonomy unless necessary to resolve ambiguity.
Ensure output taxonomy is minimal and non-redundant.
Each taxonomy item must be concise and generalizable.
Paper Background and Method Summarization
System Prompt: Write a TLDR in exactly two sentences. Sentence 1: research background/problem context. Sentence 2: method/approach. Each sentence must be <= 50 words. Return JSON only: {"tldr": {"background": "sentence1", "method": "sentence2"}}.
User Prompt:
Paper title
<title>Paper abstract
<abstract>Rules
Output exactly 2 sentences in order.
Sentence 1 must describe background/problem context only.
Sentence 2 must describe method/approach only.
Each sentence must be at most
<max_words_per_sentence>words.Keep wording concise and specific.
Paper Type Infer
System Prompt:
You are a strict research paper category classifier. Given ONLY the title and abstract, determine the PRIMARY contribution category of the paper.
Definitions.
Methodology. The paper proposes a novel model, algorithm, technical framework, or system. Experiments or datasets may be included, but the core contribution is a new technical method.
Benchmark. The paper primarily introduces a new dataset, benchmark, evaluation protocol, or leaderboard. The main contribution is evaluation infrastructure rather than a new technical method.
Survey. The paper systematically reviews, summarizes, or categorizes existing literature. It does NOT introduce a novel algorithm or technical framework.
Position Paper. The paper presents a conceptual argument, research agenda, or vision without proposing a concrete technical method or systematic survey.
Workshop Proposal. The paper proposes organizing a workshop or community event.
Others. Anything that does not clearly fit the above categories.
Respond ONLY with a JSON object of the form: {"paper_category": "allowed_categories"}.
User Prompt:
Title
<title>Abstract
<abstract>
Paper Relationship Graph Generation
System Prompt:
You are an expert academic researcher and literature review assistant. Your task is to analyze a provided list of academic papers and generate a structured map of the relationships among them.
To do this, you will use the “Entities and Relations” framework and the typology of academic relationships developed by Shayan Doroudi [12].
Typology:
Intersection: Paper A and Paper B share common entities (concepts) or relations that are highly relevant to each other.
Interpretation: Paper A provides a new lens or framework to interpret the findings of Paper B.
Expansion: Paper B provides a missing link that expands the scope of Paper A (e.g., Paper A shows X causes Y; Paper B shows Y causes Z, expanding A to link X to Z).
Abstraction: Paper B provides a generalized, theoretical framework that subsumes the specific, concrete findings of A.
Reification: Paper B provides a concrete, specific example or historical instance of the abstract theory in A.
Analogy: Papers A and B operate in different domains but share a deep structural similarity or underlying abstraction.
Substitution: Paper A and Paper B have the exact same goal, but substitute one specific method for another.
No_meaningful_relationship: The papers may share a trivial keyword, but they operate in entirely different contexts with no structural, causal, or thematic overlap worth noting.
Rules:
Use source PaperId/targetPaperId exactly from provided ids.
confidence must be numeric in [0, 1].
evidence must be an array of short strings.
Prefer no_meaningful_relationship instead of forcing weak links.
Keep graph sparse and meaningful.
Do not invent papers, fields, or IDs.
<In-context learning cases>
User Prompt:
Task query
<query_or_default>
Papers
Paper 1: <paper_1_id>; <paper_1_title>; <paper_1_venue>; <paper_1_year>; <paper_1_abstract>.
Paper 2: <paper_2_id>; <paper_2_title>; <paper_2_venue>; <paper_2_year>; <paper_2_abstract>.
<additional_papers_block>
Return JSON only.
Paper Relationship Graph Validation
System Prompt:
You are a strict reviewer for an academic paper relationship graph, using the Doroudi (2023) typology. You receive the same paper list plus a draft graph. Your task is to review each edge and produce a corrected graph.
Output must be VALID JSON only, with this exact shape:
{
"summary": "string",
"edgesReviewed": [
{
"sourcePaperId": "string",
"targetPaperId": "string",
"relationType": "one allowed label",
"confidence": 0.0,
"reason": "short rationale referencing specific entities/relations",
"evidence": ["short evidence snippet"],
"validatorAction": "kept|relabeled|removed"
}
]
}
Rules:
- Keep only defensible edges.
- You may relabel or remove weak or incorrect edges.
- Use validatorAction to indicate review decision.
- If relationType is no_meaningful_relationship, validatorAction should usually be removed.
- confidence must be numeric in [0, 1].
- Do not add IDs not present in the provided papers.
- JSON only. No markdown.
User Prompt:
Task query: <query_or_default>
Papers
Paper 1: <paper_1_id>; <paper_1_title>; <paper_1_venue>; <paper_1_year>; <paper_1_abstract>.
Paper 2: <paper_2_id>; <paper_2_title>; <paper_2_venue>; <paper_2_year>; <paper_2_abstract>.
<additional_papers_block>
Draft graph JSON to review
<draft_graph_json>
Return JSON only.
Evidence Extract
System Prompt:
You are a research extraction agent. Given a set of papers and a question, extract the most relevant evidence sentences from each paper’s abstract. Output ONLY a plain list, one line per evidence. Each line must start with the paper tag followed by a colon and the evidence sentence. Example format:
[1]: Evidence sentence from paper 1.
[2]: Evidence sentence from paper 2.
[1]: Another evidence from paper 1.
Do not add any headers, bullets, markdown formatting, or extra text.
User Prompt:
Question
<question>
Papers
<paper_contexts_joined>
Answer Drafting
System Prompt: You are a research assistant. Based on the extracted evidence below, write a well-structured answer to the user’s question. Use inline citation tags like [1, 2] to reference specific papers. Only cite papers that are directly relevant to your answer – do NOT cite every paper. Do NOT include a References section at the end – it will be appended automatically. Be concise and accurate.
User Prompt:
Question
<question>
Extracted evidence
<extracted_evidence>
Answer Reflection
System Prompt: You are an answer reflection agent. Review the drafted answer against the user question and the extracted evidence. Check whether the answer is accurate, complete, concise, and fully supported by the cited evidence. Remove unsupported claims, fix misinterpretations, and improve relevance and structure. Preserve inline citation tags like [1, 2]. Return JSON only in this format {"answer": "revised_answer"}.
User Prompt:
Question
<question>
Extracted evidence
<extracted_evidence>
Draft answer
<draft_answer>
Rules
Ensure every substantive claim is supported by the extracted evidence.
Remove claims that are unsupported, weakly supported, or irrelevant to the question.
Keep only citations that directly support the associated statement.
Improve clarity, logical flow, and conciseness.
Do not add a references section.
Preserve the original meaning unless correction is required by the evidence.
Previous failed output (fix issues and return valid JSON only)
<last_failed_response>
References
- [1]Jina AI. 2024. jina-embeddings-v5: General-purpose text embeddings. https://jina.ai/embeddings/
- [2]Sabina Yasmin Alfarquee, Sufia Sultana, Richa Rastogi, and Zakira Jabeen. 2023. Integrating literature with technology and use of digital tools: Impact on learning outcomes. World 13, 1 (2023).
- [3]Doaa Allabadi, Kyle Bradbury, and Jordan M Malof. 2025. Improving and Evaluating Open Deep Research Agents. arXiv preprint arXiv:2505.10152 (2025).
- [4]Saleema Amershi, Dan Weld, Mihaela Vorvoreanu, Adam Fourney, Besmira Nushi, Penny Collisson, Jina Suh, Shamsi Iqbal, Paul N Bennett, Kori Inkpen, et al. 2019. Guidelines for human-AI interaction. In Proceedings of the 2019 chi conference on human factors in computing systems. 1–13.DOI
- [5]Joanna Bergström, Jarrod Knibbe, Henning Pohl, and Kasper Hornbæk. 2022. Sense of agency and user experience: Is there a link? ACM Transactions on Computer-Human Interaction (TOCHI) 29, 4 (2022), 1–22.DOI
- [6]Timothy W Bickmore and Rosalind W Picard. 2005. Establishing and maintaining long-term human-computer relationships. ACM Transactions on Computer-Human Interaction (TOCHI) 12, 2 (2005), 293–327.
- [7]Kyle Bittle and Omar El-Gayar. 2025. Generative AI and academic integrity in higher education: A systematic review and research agenda. Information 16, 4 (2025), 296.DOI
- [8]John Brooke et al. 1996. SUS-A quick and dirty usability scale. Usability evaluation in industry 189, 194 (1996), 4–7.DOI
- [9]Yudong Cai, Qiyue Deng, Ting Lv, Wan Zhang, and Yi Zhou. 2025. Impact of GPT on the Academic Ecosystem. Science & Education 34, 2 (2025), 913–931.
- [10]Cornell University. 2000. arXiv e-Print archive. https://arxiv.org.
- [11]Nelson Cowan. 2010. The mystical memory four: How is working memory capacity limited, and why? Current directions in psychological science 19, 1 (2010), 51–57.pubmed.ncbi.nlm.nih.gov/20445769
- [12]Shayan Doroudi. 2023. What is a related work? A typology of relationships in research literature. Synthese 201, 1 (2023), 24.DOI
- [13]Steffen Eger, Yong Cao, Jennifer D’Souza, Andreas Geiger, Christian Greisenger, Stephanie Gross, Yufang Hou, Brigitte Krenn, Anne Lauscher, Yizhi Li, et al. 2025. Transforming science with large language models: A survey on ai-assisted scientific discovery, experimentation, content generation, and evaluation. arXiv preprint arXiv:2502.05151 (2025).
- [14]WH ElMaraghy and RJ Urbanic. 2004. Assessment of manufacturing operational complexity. CIRP annals 53, 1 (2004), 401–406.
- [15]David Ellis. 1998. Paradigms and research traditions in information retrieval research. Information Services and Use 18, 4 (1998), 225–241.
- [16]EndNote. 1989. EndNote. https://endnote.com/.
- [17]Tresna Maulana Fahrudin, Nobuo Funabiki, Komang Candra Brata, Inzali Naing, Soe Thandar Aung, Amri Muhaimin, and Dwi Arman Prasetya. 2025. An improved reference paper collection system using web scraping with three enhancements. Future Internet 17, 5 (2025), 195.
- [18]Rayees Farooq. 2017. A framework for identifying research gap in social sciences: Evidence from the past. IUP Journal of Management Research 16, 4 (2017), 66–75.
- [19]Association for Computing Machinery. 2012. ACM Computing Classification System. https://dl.acm.org/ccs
- [20]Ulrich Gnewuch, Stefan Morana, Marc TP Adam, and Alexander Maedche. 2022. Opposing Effects of Response Time in Human–Chatbot Interaction: U. Gnewuch et al. Business & Information Systems Engineering 64, 6 (2022), 773–791.
- [21]Google. 2004. Google Scholar. https://scholar.google.com/.
- [22]Sandra G Hart. 2006. NASA-task load index (NASA-TLX); 20 years later. In Proceedings of the human factors and ergonomics society annual meeting, Vol. 50. Sage publications Sage CA: Los Angeles, CA, 904–908.DOI
- [23]Yichen He, Guhua Huang, Peiyuan Feng, Yuan Lin, Yuchen Zhang, Hang Li, et al. 2025. Pasa: An llm agent for comprehensive academic paper search. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). 11663–11679.arxiv.org/abs/2501.10120
- [24]Eric Horvitz. 1999. Principles of mixed-initiative user interfaces. In Proceedings of the SIGCHI conference on Human Factors in Computing Systems. 159–166.
- [25]Siddhant Jain, Aashesh Kumar, Trinita Roy, Kartik Shinde, Goutham Vignesh, and Rohan Tondulkar. 2024. SciSpace Literature Review: Harnessing AI for Effortless Scientific Discovery. In Advances in Information Retrieval, Nazli Goharian, Nicola Tonellotto, Yuhan He, Aldo Lipani, Graham McDonald, Craig Macdonald, and Iadh Ounis (Eds.). Springer Nature Switzerland, Cham, 256–260.DOI
- [26]Pascal Jansen. 2025. Human-in-the-Loop Optimization for Inclusive Design: Balancing Automation and Designer Expertise. arXiv preprint arXiv:2505.08375 (2025).arxiv.org/abs/2505.08375
- [27]Zhuohang Jiang, Pangjing Wu, Ziran Liang, Peter Q Chen, Xu Yuan, Ye Jia, Jiancheng Tu, Chen Li, Peter HF Ng, and Qing Li. 2025. Hibench: Benchmarking llms capability on hierarchical structure reasoning. In Proceedings of the 31st ACM SIGKDD Conference on Knowledge Discovery and Data Mining V. 2. 5505–5515.arxiv.org/abs/2503.00912
- [28]Katy Jordan and Sally Po Tsai. 2025. Ranking 'by relevance' in academic literature searches: Prevalence, definitions, and implications. Postdigital Science and Education 7, 4 (2025), 1092–1106.DOI
- [29]Celina Kacperski, Mona Bielig, Mykola Makhortykh, Maryna Sydorova, and Roberto Ulloa. 2023. Examining bias perpetuation in academic search engines: An algorithm audit of Google and Semantic Scholar. arXiv preprint arXiv:2311.09669 (2023).
- [30]Jeffrey W Knopf. 2006. Doing a literature review. PS: Political Science & Politics 39, 1 (2006), 127–132.DOI
- [31]Michael Ley. 2002. The DBLP computer science bibliography: Evolution, research issues, perspectives. In International symposium on string processing and information retrieval. Springer, 1–10.
- [32]Xiancheng Liu, Shuang Zhao, Li Li, Tian Yu, Tu Yu, Yi Wang, Zhiyi Ye, Changjun Hou, Yan Xu, Shan Liu, and Guixue Wang. 2022. Frontier and hot topics in electrochemical nanoscience sensing technology based on CiteSpace bibliometric analysis. Biosensors and Bioelectronics 201 (2022), 113932.DOI
- [33]Donghyoek Ma, Hanbae Jeong, Jon Ryuh Lee, and Seok-Hyung Bae. 2025. Garden of Papers: Finding, Reading, and Organizing Research Papers in a Visual, Integrated, and Flexible Workspace. In Proceedings of the 38th Annual ACM Symposium on User Interface Software and Technology. 1–15.
- [34]I Scott MacKenzie and William Buxton. 1992. Extending Fitts’ law to two-dimensional tasks. In Proceedings of the SIGCHI conference on Human factors in computing systems. 219–226.
- [35]Nikola Marangunić and Andrina Granić. 2015. Technology acceptance model: a literature review from 1986 to 2013. Universal access in the information society 14, 1 (2015), 81–95.
- [36]Gary Marchionini. 2006. Exploratory search: from finding to understanding. Commun. ACM 49, 4 (2006), 41–46.
- [37]Joanne V Mei, J Richard Alexander, Barbara W Adam, and W Harry Hannon. 2001. Use of filter paper for the collection and analysis of human whole blood specimens. The journal of nutrition 131, 5 (2001), 1631S–1636S.
- [38]Sriraam Natarajan, Saurabh Mathur 0002, Sahil Sidheekh, Wolfgang Stammer, and Kristian Kersting. 2025. Human-in-the-loop or AI-in-the-loop? Automate or Collaborate? AAAI (2025). https://ojs.aaai.org/index.php/AAAI/article/view/35083/37238arxiv.org/abs/2412.14232
- [39]John D Nicholson, Peter LaPlaca, Ahmed Al-Abdin, Richard Breese, and Zaheer Khan. 2018. What do introduction sections tell us about the intent of scholarly work: A contribution on contributions. Industrial Marketing Management 73 (2018), 206–219.DOI
- [40]Alison Nightingale. 2009. A guide to systematic literature reviews. Surgery (Oxford) 27, 9 (2009), 381–384. doi:10.1016/j.mpsur.2009.07.005 Determining surgical efficacy.DOI
- [41]OpenAI. 2025. GPT-5 System Card. https://openai.com
- [42]OpenReview. 2018. OpenReview. https://openreview.net.
- [43]Peter Pirolli and Stuart Card. 2005. The sensemaking process and leverage points for analyst technology as identified through cognitive task analysis. In Proceedings of international conference on intelligence analysis, Vol. 5. McLean, VA, USA, 2–4.
- [44]Sheila Pontis and Ann Blandford. 2016. Understanding “influence”: An empirical test of the D ata-F rame T heory of semsampling. Journal of the Association for Information Science and Technology 67, 4 (2016), 841–858.DOI
- [45]Jason Priem, Heather Piwowar, and Richard Orr. 2022. OpenAlex: A fully-open index of scholarly works, authors, venues, institutions, and concepts. arXiv preprint arXiv:2205.01833 (2022).arxiv.org/abs/2205.01833
- [46]Chandan K Reddy and Parshin Shojaee. 2025. Towards scientific discovery with generative ai: Progress, opportunities, and challenges. In Proceedings of the AAAI conference on artificial intelligence, Vol. 39. 28601–28609.arxiv.org/abs/2412.11427
- [47]Shuo Ren, Can Xie, Pu Jian, Zhenjiang Ren, Chunlin Leng, and Jiajun Zhang. 2025. Towards scientific intelligence: A survey of llm-based scientific agents. arXiv preprint arXiv:2503.24047 (2025).
- [48]Research Rabbit. 2021. Research Rabbit. https://www.researchrabbit.ai/.
- [49]Leonardo Rivera and Frank Chen. 2007. Measuring the impact of Lean tools on the cost–time investment of a product using cost–time profiles. Robotics and Computer-Integrated Manufacturing 23, 6 (2007), 684–689.DOI
- [50]Daniel M Russell, Mark J Stefik, Peter Pirolli, and Stuart K Card. 1993. The cost structure of sensemaking. In Proceedings of the INTERACT’93 and CHI’93 conference on Human factors in computing systems. 269–276.DOI
- [51]Kara Schick-Makaroff, Marjorie MacDonald, Marilyn Plummer, Judy Burgess, and Wendy Neander. 2016. What synthesis methodology should I use? A review and analysis of approaches to research synthesis. AIMS public health 3, 1 (2016), 172.
- [52]Semantic Scholar. 2015. Semantic Scholar. https://www.semanticscholar.org/.
- [53]Syed Hamad Hassan Shah, Shen Lei, Muhammad Ali, Dmitrii Doronin, and Syed Talib Hussain. 2020. Prosumption: bibliometric analysis using HistCite and VOSviewer. Kybernetes 49, 3 (2020), 1020–1045.
- [54]Ben Shneiderman. 2020. Human-centered artificial intelligence: Reliable, safe & trustworthy. International Journal of Human–Computer Interaction 36, 6 (2020), 495–504.arxiv.org/abs/2002.04087
- [55]Sangho Suh, Bryan Min, Srishti Palani, and Haijun Xia. 2023. Sensecape: Enabling multilevel exploration and sensemaking with large language models. In Proceedings of the 36th annual ACM symposium on user interface software and technology. 1–18.DOI
- [56]Harsh Suri and David Clarke. 2009. Advancements in research synthesis methods: From a methodologically inclusive perspective. Review of Educational Research 79, 1 (2009), 395–430.DOI
- [57]Qdrant Team. 2023. Qdrant: Vector Database for the Next Generation of AI Applications. https://qdrant.tech
- [58]Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. 2023. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971 (2023).arxiv.org/abs/2302.13971
- [59]Victoria Uren, Kuurang Buckingham Shum, Michelle Bachler, and Gangmin Li. 2006. Sensemaking tools for understanding research literatures: Design, implementation and user evaluation. International journal of human–computer studies 64, 5 (2006), 420–445.
- [60]Christine R Cheuk, Bonnie Cheuk, Louisa Lam, and Dave Snowden. 2024. Sense-making, sensemaking and sense-making—A systematic review and meta-synthesis of literature in information science and education: An Annual Review of Information Science and Technology (ARIST) paper. J. Assoc. Inf. Sci. Technol. 76, 1 (Dec. 2024), 3–97. doi:10.1002/asi.24866DOI
- [61]Bert van Wee and David Banister. 2024. Literature review papers: the search and selection process. Journal of Decision Systems 33, 4 (2024), 559–565.DOI
- [62]Sharon Whitfield and Melissa A. Hofmann. 2023. Elicit: AI literature review research assistant. Public Services Quarterly 19, 3 (2023), 201–207. arXiv:https://doi.org/10.1080/15228959.2023.2224125 doi:10.1080/15228959.2023.2224125
- [63]Adam B Wilcox, Kathleen D Gallagher, Bernadette Boden-Albala, and Suzanne R Bakken. 2012. Research data collection methods: from paper to tablet computers. Medical care 50 (2012), S68–S73.DOI
- [64]Yutaro Yamada, Robert Tjarko Lange, Cong Lu, Shengran Hu, Chris Lu, Jakob Foerster, Jeff Clune, and David Ha. 2025. The ai scientist-v2: Workshop-level automated scientific discovery via agentic tree search. arXiv preprint arXiv:2504.08066 (2025).arxiv.org/abs/2504.08066
- [65]An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chenguang Huang, Chenxu Lv, et al. 2025. Qwen3 technical report. arXiv preprint arXiv:2505.09388 (2025).arxiv.org/abs/2505.09388
- [66]Runlong Ye, Patrick Lee, Matthew Warona, Oliver Huang, and Carolina Nobre. 2025. ScholarMate: A Mixed-Initiative Tool for Qualitative Knowledge Work and Information Sensemaking. In Adjunct Proceedings of the 4th Annual Symposium on Human–Computer Interaction for Work (CHIWORK ’25 Adjunct). Association for Computing Machinery, New York, NY, USA, Article 7, 7 pages. doi:10.1145/3707640.3731913
- [67]Runhua Zhang, Yang Ouyang, Leixian Shen, Yuying Tang, Xiaojuan Ma, Huamin Qu, and Xian Xu. 2025. PaperBridge: Crafting Research Narratives through Human-AI Co-Exploration. In Proceedings of the 38th Annual ACM Symposium on User Interface Software and Technology. 1–21.DOI
- [68]Yingting Zhang. 2012. Comparison of select reference management tools. Medical reference services quarterly 31, 1 (2012), 45–60.DOI
- [69]Wayne Xin Zhao, Kun Zhou, Junyi Li, Tianyi Tang, Xiaolei Wang, Yupeng Hou, Yingqian Min, Beichen Zhang, Junjie Zhang, Zican Dong, et al. 2023. A survey of large language models. arXiv preprint arXiv:2303.18223 1, 2 (2023), 1–124.DOI
- [70]Zotero. 2006. Zotero. https://www.zotero.org/.



