Memory-Aware Multi-Agent Architecture with LLM-based Orchestration for Conversational Recommendation

Introduction

Conversational recommender systems aim to iteratively infer user preferences through interaction, as intent is rarely explicit and instead evolves across dialogue turns. However, most existing approaches treat sessions in isolation, discarding preference signals once interactions end, which prevents systems from leveraging accumulated user context across interactions. This limits their ability to model long-term intent and maintain coherence over extended dialogues [9, 8].

Recent advances in large language models (LLMs) have enabled new approaches to conversational recommendation, including multi-agent architectures where components specialize in tasks such as intent understanding, retrieval, and response generation [5]. While effective for intent interpretation, these approaches typically operate as loosely integrated components, without a unified system-level design that governs how information flows across interaction, retrieval, and ranking stages [8]. As a result, there is limited understanding of how these components interact in multi-turn and cross-session scenarios, particularly in architectures where LLMs act as orchestration layers.

This work presents a memory-aware multi-agent architecture in which an LLM-based supervisor coordinates interaction, integrates long-term memory into query construction, and guides recommendation decisions across turns.

The described approach is based on a multi-agent framework in which an LLM-based supervisor coordinates interactions, integrates long-term memory into query construction, and guides recommendation decisions across turns. The architecture explicitly incorporates cross-session memory as a first-class component, enabling the system to leverage past interactions when the user’s current intent is not clearly specified.

Our main contributions are:

• A memory-aware multi-agent architecture for conversational recommendation with LLM-based orchestration.

• An empirical analysis of the impact of orchestration and cross-session memory on recommendation effectiveness and interaction dynamics in multi-turn settings.

• The identification of a gap between retrieval and exposure in conversational recommendation.

– The Semantic Consistency Score (SCS) as a complementary metric for evaluating semantic alignment.

Experimental results indicate that memory reduces a substantial portion of the performance gap in underspecified scenarios, while conversational orchestration leads to higher acceptance rates. Additionally, LLM-based reranking enhances semantic alignment and diversity, although not always improving traditional ranking metrics. These findings highlight the importance of considering system-level interactions when designing conversational recommender systems.

Related Works

Conversational recommender systems (CRS) iteratively refine user preferences through dialogue [9, 22]. However, most approaches operate at the session level, without persistent memory across interactions, limiting their ability to model long-term user intent [8].

Large Language Models (LLMs) have been increasingly adopted in recommendation pipelines, for example as zero-shot rankers or conversational agents [6, 4]. While these approaches improve flexibility and semantic understanding, they typically treat LLMs as standalone components and do not address system-level integration with retrieval and memory.

Recent work on LLM-based agents emphasizes memory for long-term reasoning and adaptation, including approaches such as Generative Agents [16], MemoryBank [24], and Reflexion [20]. However, these mechanisms are not directly designed for recommendation pipelines, where memory must influence retrieval and ranking decisions.

In parallel, LLMs have been explored as coordination layers for multi-step reasoning and tool use [19], highlighting their potential for orchestrating complex pipelines. However, prior work typically studies memory, retrieval, ranking, or orchestration as separate mechanisms, without analyzing how cross-session memory should act as a first-class signal that influences query construction, routing, and recommendation decisions across turns. To the best of our knowledge, no prior work jointly investigates long-term memory, LLM-based orchestration, and hybrid retrieval-reranking interactions at the system level for conversational recommendation. Our work addresses this gap through a unified architecture and a controlled empirical analysis of how these components shape multi-turn recommendation behavior.

Architecture

System Architecture

We propose a multi-agent architecture that integrates LLM-based agents with classical recommendation algorithms, orchestrated through a structured pipeline.

Each agent fulfills a specific role, such as intent refinement, routing, or recommendation, while sharing access to memory and retrieval components. This design enforces a clear separation of responsibilities and enables consistent behavior across multi-turn interactions.

As illustrated in Figure 1, a Supervisor Agent acts as the central orchestrator between the user and internal system components. It maintains control over the interaction flow across stages including intent refinement, query construction, routing, and response generation, coordinating the transition between preference elicitation and recommendation. Memory is accessed prior to orchestration, enriching the user query with cross-session context before routing decisions are made. The architecture is implemented using LangGraph, enabling explicit control over execution flow and state management across agents, consistent with LLM-based coordination approaches [19].

User Modeling and Interaction

Supervisory Agent The Supervisor acts as the central coordination layer, responsible for managing interaction flow and refining user queries based on conversational and memory context. It integrates information from the current dialogue and memory retrieval to guide downstream components.

Rather than operating as a generic router, it structures the interaction by eliciting missing preferences and updating query representations across turns, enabling the system to progressively align recommendations with user intent, particularly in underspecified scenarios.

In this role, it coordinates interaction and query refinement while guiding the behavior of downstream components. Its design is inspired by recent work on LLM-based control and self-improvement mechanisms [19, 20].

The decision process is driven by the interaction state, retrieved candidates, and dialogue history, allowing the system to balance preference elicitation and recommendation generation across turns.

Conversational Agent The Conversational Agent is responsible for eliciting user preferences through targeted follow-up questions. As shown in Figure 1, it operates under the Supervisor, generating structured interactions that reduce ambiguity in user intent and narrow the candidate space. Its behaviour is guided by the current conversation state and retrieved memory context, enabling efficient preference refinement across turns [16].

Recommendation Engine

The recommendation engine, whose processing flow is illustrated in Figure 1, is implemented as a modular pipeline combining multiple retrieval and ranking strategies. It integrates:

Semantic Retrieval Dense representations from Sentence-BERT are used for similarity search between natural-language queries and items, as they capture semantic intent beyond lexical overlap [17]. Item embeddings are indexed with FAISS to support efficient approximate nearest-neighbor retrieval over the catalog [11]. We apply Maximal Marginal Relevance (MMR) to balance relevance and diversity, reducing redundancy in candidates retrieved by semantic similarity alone [2].

Collaborative Filtering We adopt Alternating Least Squares (ALS), a well-established implicit-feedback matrix factorization method, to capture latent user–item interaction patterns that complement textual and semantic signals [7].

Content-Based Filtering A content-based module uses TF-IDF representations extracted from textual features such as product descriptions and reviews, providing a lexical signal complementary to dense semantic retrieval [13].

Ensemble Strategy and LLM-based Reranking Results from the retrieval components are combined through a weighted ensemble, allowing heterogeneous collaborative, lexical, and semantic signals to contribute to candidate generation [18]. The candidate set is then refined by an LLM-based reranker, which operates only over retrieved items and prioritizes semantic relevance and contextual coherence rather than expanding the retrieval space [6].

Memory Orchestrator

The memory module provides cross-session context by storing and retrieving user-related information accumulated over previous interactions. Memory is accessed prior to orchestration, allowing retrieved signals to directly influence query construction and routing decisions.

Information is organized into structured nodes representing user preferences, constraints, and contextual signals. Updates follow an append-only strategy, where new nodes reference previous state for the same subject, forming a linked sequence of states over time while maintaining a single active representation.

This design allows the system to preserve and reuse user context across interactions, supporting more consistent recommendations in underspecified scenarios. It is conceptually aligned with recent approaches to memory-augmented LLM systems [24, 20], where external memory is used to extend context beyond individual interactions.

Model Set Context

1. [2025-03-04]. User is preparing for a Senior Software Engineer interview at Zomato.

2. [2025-03-04]. User has an interview at Zomato on 2025-03-05.

3. [2025-03-04]. User wants a consolidated set of Java and Spring Boot interview questions with answers for a Senior Software Engineer role at Zomato.

4. [2025-03-04]. User is preparing for a Senior Software Engineer interview at Atlassian.

5. [2025-03-04]. User has 5.5 years of experience in Java and Spring Boot, currently working as an Analyst at Deutsche Bank.

6. [2025-03-04]. User wants a complete set of Java and Spring Boot interview questions and answers from basic to advanced levels.

However, as user preferences evolve, previously stored signals may introduce noise, indicating that memory effectiveness depends on the stability of user intent.

Experimental Methodology

Dataset and Preprocessing

Experiments are conducted on the Amazon Reviews 2023 Electronics dataset [15, 14], which provides both structured interaction data and rich textual content, enabling query-driven and conversational recommendation scenarios.

To support controlled evaluation of LLM-intensive components, a subset of the 1,000 most active users is constructed (January 2004–May 2023), yielding 19,678 unique items and 54,521 interactions (sparsity: 99.72%), with a median of 54 interactions per user. Each record includes user_id, ASIN, timestamps, ratings, review text, and product metadata. Recommendation and evaluation rely on a product-centric view, focusing on items with available product metadata.

This subset is chosen to enable controlled analysis of architectural effects under computationally intensive settings, including multi-turn interactions and cross-session memory. While limited in scale, it allows isolating the impact of design choices such as orchestration and memory integration.

Experiments are conducted in a controlled environment to enable systematic comparison of components, rather than providing absolute performance estimates.

Given the limited scale of the resulting subset, results should be interpreted mainly through relative comparisons across configurations, as statistical power for ranking metrics such as Hit@K [21] and NDCG@K [10] is constrained and collaborative filtering effectiveness is reduced by sparse user neighborhoods.

Offline Evaluation of the Recommendation Pipeline

We adopt a Leave-Last-3-Out protocol with temporal consistency and synthetically generated queries from review text. Five models of increasing complexity are evaluated: Random, Popularity, Semantic retrieval (Sentence-BERT), Prototype (semantic + ALS + content-based), and Prototype + LLM (LLM-based reranking). These configurations cover standard recommender system families, while the hybrid prototype combines ALS-based collaborative filtering with sparse TF-IDF and dense Sentence-BERT retrieval [18, 7].

Ranking performance is measured using Hit@K and NDCG@K, with intralist diversity assessed through ILS@K following Ziegler et al. [26]. To complement these metrics in terms of semantic alignment under sparse settings [12], we introduce the Semantic Consistency Score (SCS), an LLM-based evaluation signal that assesses whether recommended items are semantically consistent with the inferred user intent.

Formally, for a user $u$ with query $q_u$ and recommended list $\mathcal{R}_u = \{r_1,\ldots,r_K\}$, an LLM evaluator returns a binary judgment $c_i \in\{0,1\}$ per item, where $c_i = 1$ if $r_i$ satisfies the constraints expressed in $q_u$. The per-user and global scores are defined as:

where $K$ denotes the recommendation list size and $\mathcal{U}$ denotes the evaluated user set, with $\lvert\mathcal{U}\rvert$ representing the number of users.

The evaluator operates with temperature 0.0 for deterministic judgments. Unclear classifications are treated as consistent to avoid penalizing ambiguous cases, which may slightly bias scores upward. As an LLM-based metric, SCS should therefore be interpreted as a complementary signal rather than a standalone evaluation measure.

Intent-based Conversational Evaluation

Conversational intent recovery is evaluated on 998 users with at least six interactions. For each user, one interacted item is held out as the target and removed from the history. A Claude 3 Haiku proxy user seeks the held-out item without naming it, progressively refining the request for up to six turns. No temporal split is enforced, as the goal is intent recovery rather than next-item prediction.

Cumulative $\mathit{Hit}@T$ extends the standard $\mathit{Hit}@K$ metric to multi-turn recommendation by measuring whether the target item has been retrieved at least once up to interaction turn $T$. Together with $\mathit{MRR}$ (Mean Reciprocal Rank) [21] computed over the accumulated ranked lists across turns, it serves as the primary evaluation metric. Results are reported under two views: internal (target retrieved by the pipeline) and shown (target displayed to the user).

Supervisor Evaluation

A three-tier ablation is conducted: Baseline issues recommendations from a single-shot query; Conversational adds a supervisor capable of requesting clarification; Conv. w/ Memory further activates the Memory Orchestrator to persist preferences across turns. All tiers share the same retrieval engine and evaluation partition.

Following established simulation practice [8, 3], a proxy user is initialized with the category, attributes, and price of a held-out target but may not name it, disclosing requirements organically [25]. The engine retrieves 10 items; the top 3 are presented. An item is accepted if it matches category and features, and its price falls within 30% of the target.

Performance is assessed along two dimensions. Recommendation quality is captured by Hit Rate (target in top-10 retrieved), Acceptance Rate (any item satisfying the latent-match rule), Lift (absolute acceptance gain over Baseline), and Conversions (sessions where an initial rejection recovers to acceptance). Dialogue efficiency is measured by Average Turns and Recommendation Latency (mean turn at which the first ranked list is issued).

Cross Session Memory Evaluation

To evaluate the effect of cross-session long-term memory on recommendation quality, we conducted a two-session experiment using 30 synthetic user profiles with gradually revealed preferences. In Session 1, a multi-turn dialogue was simulated and the extracted preferences were stored in a long-term memory module. In Session 2, the same vague query was issued under two conditions: Condition A (without memory) and Condition B (with memory from Session 1 restored), with all other system components fixed. Recommendation quality was evaluated using a deterministic LLM-as-a-judge protocol [23], which performed pairwise comparisons between conditions based on the user’s revealed preferences and Session 1 dialogue, assigning an alignment score (0–10) per list and a winner. We report win rate (B over A) and mean alignment score per condition.

Results

Offline Evaluation

Figure 2 reports Hit@10, NDCG@10, ILS@10, and Semantic Consistency Score (SCS) across all pipeline stages. The popularity baseline achieves the highest ranking accuracy (Hit@10 = 0.0167, NDCG@10 = 0.0037), reflecting its bias toward frequently interacted items, but yields lower semantic consistency (SCS = 0.785).

Semantic and hybrid approaches trade ranking accuracy for diversity and intent alignment. The proposed hybrid architecture improves both diversity (ILS@10 = 0.167) and semantic consistency (SCS = 0.930) over Semantic and Popularity baselines. Adding LLM-based reranking further increases diversity (ILS@10 = 0.197) and semantic consistency (SCS = 0.954), with a slight reduction in ranking accuracy, highlighting a trade-off between exact-match retrieval and semantic alignment.

Figure 3 provides a multi-dimensional comparison highlighting the trade-offs between accuracy, diversity, and semantic consistency across models.

Ablation Study. The ablation results are presented in Figure 4. Removing the content-based component leads to the largest drop in ranking performance, particularly in Hit@10 and NDCG@10, indicating its central role in item relevance. Removing collaborative filtering has limited impact, suggesting a complementary contribution. Disabling the LLM reranker produces minimal changes in Hit@10 but reduces semantic consistency, confirming its role in improving alignment rather than retrieval accuracy.

Intent-based evaluation

Table 1 reports cumulative Hit@$T$ under both evaluation views. By turn 6, the target appears in 74.75% of sessions under the internal view and 62.63% under the shown view, with MRR reaching 0.2836 and 0.2954, respectively.

$T$	Internal Hit@T	Shown Hit@T
1	0.4639	0.2836
2	0.6303	0.4459
3	0.6804	0.5331
4	0.7124	0.5731
5	0.7295	0.6022
6	0.7475	0.6263

Performance improves rapidly in early turns and then plateaus, indicating that most intent is captured within the first interactions. The gap between internal and shown views reveals an exposure bottleneck, where relevant items are retrieved but not surfaced to the user.

Supervisor Evaluation

Table 2 presents the ablation study across three configurations. The conversational tier recovers 25.3% of Baseline failures into acceptances through dialogue alone, raising overall Acceptance Rate from 37.6% to 57.9% (+20.3 pp). Adding long-term memory further increases acceptance to 66.5% (+28.9 pp), with 31.2% of Baseline failures recovered, suggesting that cross-session context compounds the gains of in-session clarification in the evaluated setting. Notably, Acceptance Rate (66.5%) exceeds Hit Rate (51.6%) in Conv. w/ Memory, indicating that the reranker, guided by conversational context, frequently surfaces adequate alternatives even when the exact target is never retrieved.

	Hit Rate	Acceptance	Lift	Conversions
Baseline	50.7	37.6	—	—
Conversational	51.1	57.9	+20.3	25.3
Conv. w/ Memory	51.6	66.5	+28.9	31.2

Efficiency metrics (Table 3) show that successful sessions converge quickly (1.44 and 1.71 turns), while failures require longer interactions (2.98 vs. 7.59), reflecting early commitment and subsequent recovery behavior

	Turns (Overall)	Turns (Success)	Turns (Failure)	Rec. Latency
Baseline	1.00	1.00	1.00	1.00
Conversational	2.09	1.44	2.98	1.16
Conv. w/ Memory	3.68	1.71	7.59	1.02

Cross Session Memory Evaluation

The inclusion of cross-session memory indicated a consistent improvement in preference recovery when handling vague queries. As illustrated in Figure 5, recommendations generated under Condition B (with memory restored) achieved a higher average alignment score ($\mu= 5.60$) compared to the baseline Condition A ($\mu= 4.07$).

Furthermore, pairwise comparisons between the two conditions highlight the robust performance of the memory-augmented system. Condition B out-performed the baseline in 57% of the evaluated cases, while Condition A was preferred in only 33% of cases, with the remaining 10% resulting in ties. These findings indicate that integrating cross-session memory consistently enhances the system’s ability to tailor recommendations to implicit user needs.

Discussions

Offline and Intent-based Evaluations

Popularity-based methods achieve higher Hit@K and NDCG@K but exhibit lower semantic consistency, reflecting their sensitivity to interaction frequency rather than user intent, a well-documented effect of popularity bias [1].

To address this, the Semantic Consistency Score (SCS) complements traditional ranking metrics by measuring alignment between recommendations and inferred user intent. While standard metrics emphasize exact matches and interaction frequency, SCS provides a semantic perspective that is particularly relevant in conversational settings, where user preferences are often underspecified. In our experiments, this metric reveals differences across methods that are not reflected in ranking-based evaluation alone.

Semantic and hybrid approaches improve diversity and alignment, indicating that representation-based retrieval better captures intent beyond historical interactions [17, 5]. LLM-based reranking follows prior observations on LLM rankers by improving semantic coherence despite limited Hit@K gains [6]. In our setting, this trade-off extends to conversational recommendation, where reranking interacts with orchestration and memory and is most appropriate when semantic alignment and diversity justify its computational cost.

Intent-based evaluation provides additional insight into interaction dynamics. Most user intent is recovered within the first turns before stabilizing. A consistent gap between retrieval and exposure indicates that recommendation quality depends not only on retrieval effectiveness, but also on how candidates are filtered, ranked, and presented during interaction.

These results should be interpreted in the context of the evaluation setting, where ground truth reflects historical behavior rather than the full space of user preferences. As a result, optimizing for exact matches may not align with capturing evolving intent, highlighting a central trade-off in conversational recommendation. Given the controlled and relatively small-scale setup, performance differences should be interpreted as indicative trends rather than statistically robust estimates.

Conversational Orchestration

The supervisor has a clear impact on interaction quality. Introducing a clarification-capable supervisor increases acceptance rates by up to 28.9 percentage points, enabling structured preference elicitation under underspecified intent.

Orchestration shapes exploration across turns by iteratively refining user preferences and updating query representations, allowing progressive alignment with intent rather than relying on a single-step prediction. These effects are reinforced by LLM-based reranking, which leverages accumulated conversational context to surface relevant alternatives.

Efficiency results reveal a trade-off: successful sessions converge quickly, while failures require longer interactions. This reflects early commitment when intent is clear, followed by deeper exploration under ambiguity. Overall, orchestration plays a central role in both preference elicitation and coordination of information throughout the interaction.

Cross-Session Memory

Cross-session memory improves performance under underspecified queries, increasing semantic alignment and outperforming the no-memory condition. This aligns with prior work on memory-augmented LLM systems[24, 16], where persistent memory enables more personalized and context-aware behavior across interactions.

Its effectiveness, however, depends on how stored preferences interact with current intent. As preferences evolve, previously stored signals may become less informative, leading to misalignment, particularly when outdated or overly generic signals are emphasized.

Rather than acting as a uniformly beneficial component, memory functions as an additional signal whose impact depends on its integration with interaction context and downstream decisions. When properly aligned, it consistently improves handling of underspecified queries, reinforcing its role in conversational recommendation.

Limitations

This work relies on a controlled evaluation setting designed to isolate architectural effects. The use of a reduced dataset and simulated interactions enables systematic comparison, but limits representation of real-world variability. Simulated users may not capture the diversity and evolution of real user intent, affecting observed interaction dynamics. Additionally, the limited dataset scale constrains variability in user profiles, and the magnitude of reported gains may differ under larger and more heterogeneous populations. The results should therefore be interpreted as evidence of system-level patterns rather than definitive performance estimates. Extending evaluation to larger-scale and real-world settings remains necessary to assess robustness and generalization.

Conclusion

This work presents a memory-aware multi-agent architecture for conversational recommendation, integrating LLM-based orchestration, long-term memory, and a hybrid retrieval pipeline. By coordinating preference elicitation, query construction, and recommendation generation, the approach highlights how system-level interactions shape conversational effectiveness.

Across offline, conversational, and cross-session evaluations, the results indicate that orchestration and memory play complementary roles. Supervisory control improves interaction under underspecified intent, while memory enhances personalization when aligned with interaction context. A consistent gap between retrieved and displayed items reveals an exposure bottleneck, indicating that recommendation quality depends not only on retrieval but also on how candidates are presented.

These findings should be interpreted as evidence of system-level effects in a controlled experimental setting, suggesting that effective conversational recommendation requires the joint design of orchestration, memory, and retrieval-ranking components.

Future work includes the development of adaptive memory mechanisms to address preference drift, as well as improved strategies for candidate selection and presentation. In addition, incorporating structured profiling signals represents a promising direction for informing early-stage interaction and improving robustness in cold-start and underspecified scenarios. Extending the evaluation to larger-scale and real-world settings remains essential to assess robustness under more diverse interaction patterns and user behaviors.

Acknowledgments This work was supported by Instituto de Ciência e Tecnologia Itaú and Eldorado Research Institute. We acknowledge the use of OpenAI tools for reference discovery and text refinement. Any opinions, findings, conclusions or recommendations expressed in this material are those of the authors and do not necessarily reflect the views of Itaú Unibanco and Instituto de Ciência e Tecnologia Itaú. This document is not and does not constitute or intend to constitute investment advice or any investment service. It is not and should not be deemed to be an offer to purchase or sell, or a solicitation of an offer to purchase or sell, or a recommendation to purchase or sell any securities or other financial instruments. In addition, all data used in this study comply with the Brazilian General Data Protection Law.

This preprint has not undergone any post-submission improvements or corrections. The Version of Record of this contribution is published in Intelligent Systems, and is available online at https://doi.org/10.1007/978-3-032-39892-5_25.