Bridging Exploration and Persona: Information-Directed Dueling for Test-Time Agent Personalization

Introduction

Large language model (LLM)-based agents support tasks ranging from creative writing to coding assistance [10, 15]. Their behavior, including response style and persona, is strongly influenced by the instructions supplied at inference time. Prompt design and automatic instruction optimization improve task-level behavior [34, 43]; personalization extends this objective toward responses suited to an individual user [27].

A natural control interface is the agent’s persona: a prompt-level description of the behavior and style the agent should adopt. Persona selection differs from task-level prompt optimization in three ways. First, personalization is contextual: the persona most useful for one request may be less suitable for another (e.g., a “creative collaborator” for brainstorming versus a “rigorous critic” for editing) [27, 39]. Second, user preferences can be implicit and noisy: interaction histories provide partial evidence about relevant tendencies [25]. Third, candidate persona descriptions can be semantically dense. Closely related descriptions (e.g., “stern tutor” and “strict mentor”) may induce meaningfully different responses despite high embedding similarity.

These properties motivate test-time adaptation (TTA) over a fixed persona pool. Prompt-level adaptation preserves a shared backbone while incorporating feedback about the current request. PersonaAgent [39] uses expressive linguistic feedback, while preference feedback reduces an interaction to choosing between two candidate responses, a supervision format also used in RLHF [20].

Recent approaches such as POHF [17] and PDO [36] apply bandit exploration to preference-driven prompt optimization. In the persona pools evaluated here, their UCB- and Thompson-sampling-based components revisit challengers more often and reach lower aggregate performance than our information-directed rule (§ 4).

Building on information-directed and dueling-bandit methods, we develop ID-TAP (Information-Directed Dueling for Test-Time Agent Personalization), a scalable challenger-selection framework that uses pairwise preferences as its online signal. Its technical focus is the joint design required by black-box personalization over large persona pools: query-conditioned utility estimation, lightweight hypothetical BT updates, and transfer across related queries. Specifically, a query-conditioned neural utility model estimates utility gaps, while a lightweight statistical explorer evaluates hypothetical comparison outcomes through an IDS-inspired entropy-reduction proxy. This separation supports efficient challenger scoring with one neural update per observed comparison, and similarity-weighted warm starts reuse preference evidence across queries. Under a fixed comparison budget, the resulting method attains the strongest aggregate result among the evaluated baselines.

Our contributions are summarized as follows:

Scalable Challenger Acquisition. We operationalize an IDS-inspired acquisition criterion for large discrete persona pools by combining neural utility gaps with a lightweight BT entropy-reduction proxy, enabling efficient scoring across candidate challengers.

Cross-Query Adaptation. We introduce a similarity-weighted warm start that reuses comparison evidence across related requests while retaining query-conditioned utility prediction.

Empirical Evaluation. Across seven benchmarks and two offline preference simulators, ID-TAP achieves the strongest aggregate result among the compared methods under the same ten-round budget. On the aggregate generation-task trajectory, it overtakes the strongest compared baseline after approximately 4 simulated-feedback rounds, suggesting favorable feedback-round efficiency in this setting.

Problem Formulation

We formalize persona selection as a contextual, query-level problem with an offline construction phase and an online test-time adaptation phase.

Persona and Contextual Space

For a personalized generation task, a persona $p$ is a prompt-level description that summarizes relevant traits, stylistic habits, and task preferences inferred from the user’s historical interaction context $C$. The offline procedure constructs a finite persona space $\mathcal{P}(C) = \{p_1, \ldots, p_K\}$ from this history. The candidates capture different emphases that remain grounded in $C$. Because requests from the same user can differ in intent and complexity, their utilities are query-dependent rather than governed by a single universally best candidate.

Offline–Online Framework

We use the following offline–online protocol:

Offline Phase (Persona Induction). Historical context $C$ is used to construct the shared candidate set $\mathcal{P}(C)$. The candidate strings are fixed during online adaptation and can be reused across the user’s subsequent queries.

Online Phase (Query-Specific Adaptation). At test time, the agent receives a query $q$. Let $u(q,p)$ denote the unknown expected utility of the response generated under persona $p$. The query-conditioned optimal candidate is

The neural score $f_{\theta}(q,p)$ introduced in § 3 estimates this latent utility from observed comparisons.

Test-Time Adaptation with Preference Feedback

ID-TAP learns estimates of the latent utility online while keeping the finite candidate personas fixed. We assume that user preferences are locally stable over short interaction horizons. Test-time adaptation therefore operates within the pre-constructed persona space $\mathcal{P}(C)$, with the historical context $C$ fixed throughout the session.

To identify a strong persona from lightweight interaction, we adopt a dueling-bandit interface in which the agent learns from pairwise preference feedback. At interaction step $t$, the algorithm selects a champion persona and a challenger persona, then generates the corresponding response pair $(r^{t}_{1}, r^{t}_{2})$. The champion exploits the neural model’s current estimate, whereas the challenger is chosen by the IDS-inspired acquisition rule in § 3. The user returns a binary comparison $y_t = \mathbb{I}\{r^{t}_{1} \succ r^{t}_{2}\}$, which updates both utility estimates and pairwise statistics. Given an interaction budget $T$, the objective is to return a persona $\hat{p}^{q}_{T}$ with high query-conditioned utility from these binary comparisons.

Methodology

ID-TAP uses an IDS-inspired acquisition strategy for neural dueling bandits in a personalized, test-time interaction setting. Before test time, we assume that the historical context set $C$ and the corresponding persona space $\mathcal{P}(C)$ have been constructed offline. Inspired by Information-Directed Sampling (IDS), our method balances exploitation and exploration by selecting comparison pairs that are both performance-relevant and uncertainty-reducing. Specifically, the challenger is chosen by relating its estimated utility gap to a plug-in entropy-reduction proxy computed from the BT explorer.

Algorithm Overview

At test time, a user sequentially queries an AI agent with personalized requests. For each query $q$, the agent interacts with the user for a fixed budget of $T$ rounds, collecting pairwise feedback to adapt persona selection to the query-conditioned preference. The neural parameters persist across a user’s queries; the BT state is initialized by Algorithm 2 and then updated locally. For the first query, the BT counts are zero and its pairwise probabilities are neutral. We use a fixed candidate-index rule to resolve exact ties in champion or challenger scores.

We model the utility function with a neural network. At each round, the agent first selects an exploitation arm

corresponding to the currently estimated best persona.

To efficiently explore alternative personas and reduce uncertainty, the second arm $p_{t,2}^{q}$ is selected using an information-directed criterion. Concretely, we choose the challenger to minimize the acquisition ratio:

where $\widehat{\Delta}_{t}(p) = f_{\theta}(q,p_{t,1}^{q}) - f_{\theta}(q,p)$ is the estimated utility gap, $\widetilde{g}_{t}$ is the BT-based entropy-reduction proxy defined below, and $\epsilon_{g} > 0$ prevents division by zero. The ratio favors challengers with a small predicted utility loss and a large estimated reduction in ranking uncertainty.

The user then provides preference feedback in the form of a binary comparison outcome. All collected pairwise interactions across queries and rounds are aggregated and used to update the neural utility model. After round $T$, the highest-scoring persona $\widehat{p}_{T}^{q}$ conditions the final generated response.

IDS-Inspired Challenger Selection

Building on information-directed methods for bandit and preference feedback [26, 13, 12, 22], ID-TAP adapts the information-ratio principle to persona-level pair selection. We fix $p_{t,1}^{q}$ as the current greedy champion and select a challenger $p_{t,2}^{q}$ by minimizing the acquisition ratio.

Plug-in Entropy-Reduction Proxy. The BT solver produces strengths $\widehat{\theta}_{t}^{q}$, which we map to the normalized plug-in score distribution $\widetilde{\pi}_{t}^{q}(k) = \operatorname{softmax}(\widehat{\theta}_{t}^{q}/\tau)_{k}$. For a champion–challenger pair

Algorithm 1 ID-TAP
1: Initialize Neural network $f_{\theta}$, persona space $\mathcal{P}(\mathcal{C})$
2: for $q = 1,2,\ldots,N$ do
3:   Initialize $(\mathbf{W}^{q},\mathbf{N}^{q},\mathbf{P}^{q})$ by Algorithm 2; set $(\Delta\mathbf{W}^{q},\Delta\mathbf{N}^{q}) \leftarrow (0,0)$
4:   for $t = 1,2,\ldots,T$ do
5:     $p_{t,1}^{q} \leftarrow \arg\max_{p \in \mathcal{P}(\mathcal{C})} f_{\theta}(q,p)$
6:     $p_{t,2}^{q} \leftarrow \arg\min_{p \in \mathcal{P}(\mathcal{C})\setminus\{p_{t,1}^{q}\}} \frac{\widehat{\Delta}_{t}(p)^{2}}{\max\{\widetilde{g}_{t}(p_{t,1}^{q},p),\epsilon_{g}\}}$
7:     Generate response pair $\{r_{t,1}^{q},r_{t,2}^{q}\}$
8:     Observe preference feedback $y_{t}^{q} \leftarrow \mathbb{I}\{r_{t,1}^{q} \succ r_{t,2}^{q}\}$
9:     Train $f_{\theta}$ with $D_{T}^{q} = \bigcup_{k<q}D_{T}^{k} \cup \{(q,p_{s,1}^{q},p_{s,2}^{q},y_{s}^{q})\}_{s\le t}$
10:    Update $(\mathbf{W}^{q},\mathbf{N}^{q},\mathbf{P}^{q})$ and local increments $(\Delta\mathbf{W}^{q},\Delta\mathbf{N}^{q})$
11:  end for
12:  Store $(q,e_{q},\Delta\mathbf{W}^{q},\Delta\mathbf{N}^{q})$; output $\widehat{p}_{T}^{q} \leftarrow \arg\max_{p} f_{\theta}(q,p)$
13: end for

$(p^q_{t,1}, p)$, we define

where $H(\tilde{\pi})=-\sum_k\tilde{\pi}(k)\log\tilde{\pi}(k)$, $\hat{P}_t$ is the BT predictive probability, and $\tilde{\pi}^{q,(y)}_t$ is obtained by adding hypothetical outcome $y$ and re-estimating the BT strengths. This IDS-inspired plug-in score ranks pairs by their predicted reduction in the dispersion of normalized BT strengths. Reproducible implementation details are provided in Appendix A.

Dual-System Architecture for Real-Time Efficiency

Equation (4) evaluates two hypothetical outcomes for each candidate, yielding $2(K-1)$ BT re-estimations per interaction round. The lightweight BT solver makes this candidate-wide scoring practical at large $K$, while the neural utility model performs one global update after each observed comparison. We therefore decouple global utility learning from local uncertainty estimation:

• System 1: Neural Utility Model ($f_\theta$) This system serves as the shared utility estimator. It maps queries and persona embeddings to utility scores $s_p$ and provides the gap estimates $\hat{\Delta}_t(p)$ from the interaction history $D$ accumulated across queries. It is updated once per interaction round after observing feedback.

• System 2: Statistical Explorer (BT-MLE) This system acts as the query-local explorer. It maintains a lightweight Bradley–Terry (BT) model using comparison counts $\{w_{ij},n_{ij}\}$. Its minorization–maximization (MM) solver can simulate the $2(K-1)$ hypothetical outcomes and recompute the normalized strength distribution efficiently.

Pairwise Probabilities. The BT explorer applies symmetric Beta smoothing to observed pairs and assigns neutral probability $P_{ij}=0.5$ to unobserved pairs. Appendix A gives the count updates, probability estimator, and MM solver.

Design Motivation. The dual-system design combines complementary estimation properties. System 1 (the neural utility model $f_\theta$) is trained with pairwise BCE, corresponding to the negative BTL log-likelihood used by System 2 (BT-MLE). System 1 selects the champion and estimates utility gaps through shared representations, while System 2 supplies the challenger-side entropy-reduction proxy from pairwise outcomes. Consequently, $\Psi_t(p)=\hat{\Delta}_t(p)^2/\max\{\tilde{g}_t(p^q_{t,1},p),\epsilon_g\}$ selects challengers that are both promising and informative. Existing neural-dueling, Bradley–Terry estimation, and classical M-estimation analyses motivate these components [33, 11, 32, 29]; our empirical study evaluates their coupled use with cross-query warm starts.

Cross-Query Knowledge Transfer. For a new query $q$, ID-TAP warm-starts the BT system from the query-local count increments $(\Delta\mathbf{W}^{(k)},\Delta\mathbf{N}^{(k)})$ collected on earlier queries. Let $b_k$ be the raw similarity score and normalize it as

The history stores only the counts generated locally on each query $q_k$, so each local increment enters any later query’s initialization at most once. The normalized weights cap the total inherited count mass at one query-equivalent. The inherited counts are converted to $\mathbf{P}_0^q$ using Eq. (8), the neural model separately learns from the interaction history $D$.

Experiments

Experimental Setup

Configuration. We adopt DeepSeek-V3.2 [6] as the backbone LLM for the agent and Qwen3-Embedding-4B [40] as our embedding model. The main experiments use $K = 400$ candidate personas per user. For each user, every experimental configuration is run independently three times, and the arithmetic mean of those runs is treated as that user’s result. Dataset-level means, standard deviations, and standard errors are computed across these per-user results. Geometric statistics are computed for each user’s persona pool before aggregation. $N_{\mathrm{eval}}$ denotes query-level evaluation instances, whereas $n_{\mathrm{user}}$ denotes sampled users in the auxiliary sensitivity and runtime analyses.

Datasets. Following PersonaAgent, we evaluate on LaMP (Salemi et al., [27]). We also include LongLaMP (Kumar et al., [14]), PrefEval (Zhao et al., [41]), and UltraChat (Ding et al., [7]), covering short- and long-form personalized generation and conversational settings. Dataset details and statistics are provided in Appendix § 11.

Baselines and Shared Evaluation Protocol

We compare ID-TAP against a generative personalization baseline and two online preference-learning baselines.

Generative Baseline. PersonaAgent [39] performs personalization by synthesizing user-specific information into prompts.

Online Learning Baselines. Since our setting involves iterative preference learning, we compare against two dueling-bandit acquisition rules: POHF, which uses a neural UCB-style rule, and the PDO sampling component, i.e., Double Thompson Sampling (D-TS) [35, 36]. Full PDO combines sampling with prompt mutation and pruning. We instantiate its D-TS sampling component on the shared fixed persona pool, yielding a matched comparison of pair selection under the same search space and feedback budget.

Shared Evaluation Protocol. PersonaAgent follows its reported configuration. All bandit methods share the same candidate personas, response generator, binary-feedback interface, and ten-round budget for each query. Within this common protocol, PDO Sampling represents its acquisition component and ID-TAP represents the proposed acquisition design.

Preference Feedback Mechanisms

ID-TAP’s online update consumes one binary outcome for each displayed response pair. In deployment, the user supplies this outcome after viewing the pair. For repeatable offline experiments, the held-out target instantiates a proxy user that returns the same type of outcome after each algorithm-selected comparison. Following POHF’s offline interaction protocol [17], persona construction, response generation, and arm selection use the history, query, and observed binary outcomes; after pair selection, the proxy user uses the held-out target to produce the next outcome. This protocol reproduces test-time information flow while standardizing the preference source across methods for $T = 10$ rounds.

For final W.R. evaluation, each method’s final response is paired with a naive response: an answer generated by the same backbone directly from the query and user history or context, without persona selection or test-time preference optimization. The pair order is randomized before evaluation, and W.R. is the percentage of comparisons in which the method response is preferred to the naive response.

ROUGE-L-Based Simulation. For each candidate response $r^{q}_{t,i}$, the proxy user computes the reference-grounded utility $s^{q}_{t,i} = \operatorname{ROUGE\mathchar-L}(r^{q}_{t,i}, r^{q}_{t})$. Following the offline preference-simulation protocol of POHF {citep}[17]`, we sample the binary outcome using the Bradley–Terry–Luce model:

Reference-Grounded Holistic Preference Proxy. An LLM judge receives the query, a user-specific reference, and an anonymized response pair. For every comparison, the two responses are randomly assigned to the Response 1 and Response 2 positions, and the judge’s selection is mapped back to the corresponding personas. The judge selects the response that better realizes the intended output as a whole, jointly considering task fulfillment, semantic intent, organization, coherence, and the tone and style expressed by the reference. The reference supplies a task- and user-conditioned example, and the judge balances semantic realization with organization and style. We refer to this signal as a reference-grounded holistic preference proxy. In both settings, the optimizer’s adaptation signal is the resulting binary comparison, while the proxy user retains the reference and its derived score.

Main Results and Analysis

Overall Performance. Table 1 summarizes final performance across the seven datasets. Under ROUGE-L-based preference simulation, R.L. values are normalized per query by the maximum value among the evaluated methods; under LLM-based preference simulation, W.R. measures how often the method’s final response is preferred to the naive response and is reported as mean±SEM across users. ID-TAP is numerically best in 13 of the 14 dataset–metric cells and has the strongest average in both columns; the exception is LongLaMP-2 W.R., where PersonaAgent is higher. The raw ROUGE-L table yields the same first-place ordering for ID-TAP on all seven datasets, a cross-dataset average of 0.2228 versus 0.2068 for the strongest baseline. Appendix § 11.§ 4 reports unnormalized ROUGE-L as mean$\pm$SEM together with evaluation-set sizes.

Dataset	PersonaAgent R.L.	PersonaAgent W.R.	POHF R.L.	POHF W.R.	PDO Sampling R.L.	PDO Sampling W.R.	ID-TAP R.L.	ID-TAP W.R.
LaMP-4	0.607	<u>70.6±8.6</u>	0.659	68.3±6.7	<u>0.660</u>	69.4±7.5	0.781	73.0±5.5
LaMP-5	0.723	59.7±1.8	0.822	67.5±1.0	0.817	68.7±1.4	0.924	77.2±0.4
LongLaMP-2	<u>0.887</u>	88.8±1.3	0.880	79.5±1.3	0.865	83.2±0.7	0.939	<u>84.7±2.0</u>
LongLaMP-3	0.882	<u>82.3±3.0</u>	<u>0.914</u>	80.3±2.8	0.912	77.2±3.5	0.964	85.7±2.1
LongLaMP-4	0.864	<u>85.2±1.3</u>	0.868	<u>84.9±1.6</u>	0.872	84.0±0.7	0.962	87.9±0.2
PrefEval	0.774	59.2±4.6	<u>0.943</u>	<u>81.7±5.8</u>	0.886	83.3±3.0	0.956	85.8±1.7
UltraChat	0.369	86.7±2.2	<u>0.875</u>	90.0±1.7	0.845	89.2±2.9	0.907	90.8±1.7
Average	0.729	76.1	<u>0.852</u>	78.9	0.837	<u>79.3</u>	0.919	83.6

Query-Level Performance on Generation Tasks. The left panel of Figure 2 shows query-wise ROUGE-L averaged over the five generation tasks LaMP-4, LaMP-5, LongLaMP-2, LongLaMP-3, and LongLaMP-4. Separate plots for every task, including PrefEval and UltraChat, are provided in Appendix § 16.§ 1.

Feedback-Round Efficiency and Dynamics. We analyze the optimization dynamics under ROUGE-L-based preference simulation in the right panel of Figure 2. The plot illustrates iteration-level performance averaged across the five LaMP and LongLaMP generation tasks shown in the figure. ID-TAP exceeds the compared baselines after approximately 4 simulated-feedback rounds on this aggregate curve, indicating faster optimization under the shared evaluation protocol. Detailed iteration-level trajectories for all seven tasks, including PrefEval and UltraChat, are provided in Appendix § 16.§ 2.

Matched Component Ablations

All variants independently produce their own test-time trajectories under the same persona pool, query order, comparison budget, model configuration, and evaluation protocol. Random Challenger retains the neural champion, both updates, cross-query states, and neural final selector, changing only the challenger to a uniform non-champion draw. ID-TAP-NH reinitializes the neural model and neutralizes the BT statistics for every query, whereas ID-TAP-NBT retains neural history but neutralizes BT statistics; hence NH versus NBT isolates neural-history transfer and NBT versus Full isolates the BT warm start. Random Response and Random Acquisition + Neural Final are additional diagnostics of random data collection and output selection.

Method	LongLaMP-3	LongLaMP-4	PrefEval
Random Response	0.601	0.497	0.448
Random Acquisition + Neural Final	0.615	0.516	0.507
ID-TAP-NH	0.764	0.647	0.629
ID-TAP-NBT	0.774	0.657	0.629
Random Challenger	0.778	0.644	0.618
Full ID-TAP	0.783	0.664	0.629

Full ID-TAP exceeds Random Challenger by 0.005, 0.020, and 0.011, supporting a modest contribution from information-directed challenger selection. Along the history progression NH$\to$NBT$\to$Full, neural history adds 0.010 on both LongLaMP tasks, and the BT warm start adds a further 0.009 and 0.007; PrefEval remains unchanged. Thus, both history mechanisms contribute on the sequential LongLaMP tasks. The wider gaps from the random-acquisition controls reflect champion-centered collection and learned final selection.

Persona-Construction and Density Analysis. Keyword-conditioned construction produces denser persona spaces and lower challenger exploration for POHF and PDO Sampling across the four evaluated datasets, while ID-TAP’s exploration rate changes by at most 1.5 percentage points. Appendix § 12 reports the construction, exploration, performance, and sensitivity results.

Robustness to Preference and Evaluation Models. ID-TAP is numerically highest across the reported matched-model, cross-model, and BERTScore configurations, providing complementary evidence across automated preference sources, evaluators, and metrics. Appendix § 13 provides the complete tables and protocol.

Runtime. Here, $K$ denotes the number of candidate personas in each user’s persona space. With $K = 100$ and $n_{\mathrm{user}} = 40$ sampled users per dataset, ID-TAP’s algorithmic update requires 0.065–0.571 seconds per round, compared with 0.95–3.15 seconds of end-to-end wall-clock latency. The algorithmic component accounts for approximately 7–22% of total latency. Full timing statistics and the measurement protocol are provided in Appendix § 14.

Related Work

Personalized LLM Agents. Personalized LLMs condition outputs on individual preferences, traits, or context [16, 44]. Retrieval-based methods expose relevant user history at inference time [25, 42], while memory and prompt-based approaches maintain or synthesize user-specific context [39, 18]. LaMP standardizes evaluation across personalized language tasks [27]. Recent work increasingly studies lightweight adaptation at inference time [16, 23]; ID-TAP contributes a pairwise-feedback method for selecting among fixed persona prompts.

Feedback Mechanisms and Test-Time Adaptation. Aligning LLM behaviors with user intent relies heavily on feedback-driven optimization. The canonical paradigm, Reinforcement Learning from Human Feedback (RLHF) [4, 45], typically employs supervised fine-tuning, reward modeling [31], and policy optimization via PPO [28, 20]. Direct Preference Optimization (DPO) [24] and RLAIF [2] streamline this process through direct preference learning or AI supervision. Test-Time Adaptation (TTA) further enables online refinement through iterative feedback loops [19, 30]; reasoning systems use self-correction or tree search [37, 21], while personalized prompt- and decoding-level methods adapt behavior with fixed backbone LLM weights [17, 23].

Exploration in Dueling Bandits. Dueling bandits learn from pairwise preferences [38], and contextual variants allow preferences to depend on the query [8, 3]. Relevant analyses cover structured stochastic preferences and neural utility models [3, 33]; confidence-based linear bandit methods provide additional background for contextual exploration [1]. Practical strategies include Thompson sampling [5], Double Thompson Sampling (D-TS) [35], and posterior sampling for LLM exploration [9]. For LLM prompt optimization, POHF [17] uses neural UCB-style acquisition, and PDO [36] combines D-TS with top-performer-guided mutation and pruning. Our PDO Sampling baseline instantiates D-TS in the common fixed persona pool and feedback budget. T-POP [23] operates during token-level decoding with white-box model access and addresses a complementary setting.

IDS balances estimated utility gaps against information gain [26] and has been studied in linear partial monitoring and dueling-feedback settings [13, 12, 22]. These works provide general information-directed objectives or analyses under structured models. ID-TAP builds on these foundations and focuses on the systems question posed by black-box LLM personalization: scoring a large, LLM-generated persona pool, separating query-conditioned neural utility learning from lightweight hypothetical BT updates, and transferring comparison evidence across related queries. The resulting contribution is a scalable, cross-query acquisition architecture for persona-level test-time adaptation.

Conclusion

We presented ID-TAP, an IDS-inspired framework for test-time persona selection. It combines query-conditioned neural utility gaps, a lightweight BT entropy-reduction proxy, and similarity-weighted warm starts across related queries. Across seven user-conditioned tasks with automated proxy feedback, ID-TAP achieves the highest aggregate mean among the evaluated baselines under a fixed comparison budget. Future work will evaluate the interface with end users and develop finite-sample analysis of the neural/BT procedure.

AI USE STATEMENT

In preparing this manuscript, generative AI tools were used to assist with LaTeX formatting, scientific-figure design and implementation, and editorial review. These tools did not generate or alter the reported experimental measurements. The authors reviewed all AI-assisted changes and take responsibility for the final text, claims, and artifacts.

REPRODUCIBILITY STATEMENT

The main paper specifies the problem formulation, ID-TAP procedure, and evaluation protocol. The appendix provides the BT-MLE entropy-proxy implementation, complete prompts, data splits, hyperparameters, persona-space construction analyses, robustness experiments, and timing protocol needed to interpret and reproduce the reported results. The public code artifact is available at https://github.com/s1412/ID-TAP.

REFERENCES

Implementation Details

Cross-Query BT Warm Start

Algorithm 2 gives the complete warm-start procedure corresponding to the normalized transfer rule in Section 3. Its history contains query-local count increments, and the normalized weights provide at most one query-equivalent of inherited mass.

Algorithm 2 Cross-Query BT Probability Inheritance

1. Input: Current embedding $e_q$; local history $\mathcal{H} = \{(q_k, e_{q_k}, \Delta\mathbf{W}^{(k)}, \Delta\mathbf{N}^{(k)})\}_{k<q}$

2. Output: Initialized $(\mathbf{W}^q, \mathbf{N}^q, \mathbf{P}^q)$

3. if $\mathcal{H} = \emptyset$ then

4. &nbsp;&nbsp;&nbsp;&nbsp;return $\mathbf{W}^q, \mathbf{N}^q \leftarrow0, \mathbf{P}^q \leftarrow0.5$

5. end if

6. $b_k \leftarrow\sigma\left(\beta_{\mathrm{sim}}\left(\operatorname{CosineSim}(e_q,e_{q_k})-\mu_{\mathrm{sim}}\right)\right)$ for every $k<q$

7. $Z_q \leftarrow\max\{1,\sum_{k<q} b_k\}$; $\mathbf{W}^q,\mathbf{N}^q \leftarrow0$

8. for each $(q_k,e_{q_k},\Delta\mathbf{W}^{(k)},\Delta\mathbf{N}^{(k)})\in\mathcal{H}$ do

9. &nbsp;&nbsp;&nbsp;&nbsp;$a_k \leftarrow b_k/Z_q$

10. &nbsp;&nbsp;&nbsp;&nbsp;$\mathbf{W}^q \leftarrow\mathbf{W}^q+a_k\Delta\mathbf{W}^{(k)}$, $\mathbf{N}^q \leftarrow\mathbf{N}^q+a_k\Delta\mathbf{N}^{(k)}$

11. end for

12. $\mathbf{P}^q_{ij} \leftarrow\dfrac{\widetilde{w}_{ij}+\alpha}{\widetilde{n}_{ij}+2\alpha}$

13. return $(\mathbf{W}^q,\mathbf{N}^q,\mathbf{P}^q)$

Maintained Statistics and Smoothed Pairwise Probabilities

For each persona pair $(p_i,p_j)$, we maintain (i) directed win counts $w_{ij}$ and (ii) symmetric comparison counts $n_{ij}=n_{ji}$. After each observed duel, counts are updated as:

Observed pairs receive the symmetric Beta-smoothed estimate

where $\alpha=0.7$. Unobserved pairs use $P_{ij}=0.5$, preserving neutral pairwise evidence until a direct or inherited comparison becomes available.

BT-MLE Strength Estimation via MM Iterations

Given the current count matrices, we estimate BT strengths $\hat{\theta}$ by an MM-style iterative solver over positive parameters $\pi_i=\exp(\theta_i)$. Let

and

Each MM iteration updates

for candidates with positive incident comparison mass. Candidates awaiting their first observed or inherited comparison retain the common neutral initialization. The updated strengths are then normalized (sum rescaling) and converted as $\hat{\theta}_i=\log(\pi_i+\epsilon)$. For numerical stability we use $\epsilon=10^{-8}$ and clip probabilities to $[\epsilon,1-\epsilon]$.

Normalized Strength Distribution and Entropy

From $\hat{\theta}$, we define the normalized plug-in score distribution $\tilde{\pi}_t(k) = \operatorname{softmax}(\hat{\theta}_t/\tau)_k$ for the fixed query $q$ and use it to rank hypothetical updates. Unobserved pairs enter the solver with neutral pairwise evidence. Candidates awaiting their first comparison receive neutral initial strengths, and one softmax over all $K$ strengths ensures $\sum_k \tilde{\pi}_t(k) = 1$. We then compute

where we use the natural logarithm throughout.

Outcome Simulation and Entropy-Reduction Proxy

For a champion–challenger pair $(c, d)$, outcome probability under BT is

where $y = 1$ means champion wins.

To evaluate the proxy, we simulate both hypothetical outcomes $y \in\{0, 1\}$:

1. update $(w, n)$ with the hypothetical result;

2. re-estimate BT strengths by Eq. (11);

3. recompute the normalized strength distribution $\tilde{\pi}_t^{(y)}$ and its entropy $H(\tilde{\pi}_t^{(y)})$.

Thus, the expected post-update entropy is

and $\tilde{g}_t(c, d) = \max\{0, H(\tilde{\pi}_t) - \mathbb{E}[H_{\mathrm{after}}]\}$ is the plug-in entropy-reduction proxy used by the acquisition rule. Clipping at zero removes tiny negative values caused by floating-point error.

Reproducible Procedure (Pseudo-code)

Algorithm 3 BT-MLE Entropy-Reduction Proxy for a Candidate Pair

1. Input: champion $c$, challenger $d$, counts $(w, n)$, smoothing $\alpha$

2. Estimate $\hat{\theta}$ from $(w, n)$ via MM updates (Eq. (11))

3. Compute $\tilde{\pi}_t = \operatorname{softmax}(\hat{\theta}/\tau)$ and $H_{\mathrm{before}} = H(\tilde{\pi}_t)$

4. Compute $p_{\mathrm{win}} = P(y = 1 \mid c, d)$ using Eq. (13)

5. for $y \in\{1, 0\}$ do

6. Simulate outcome $y$ and obtain hypothetical counts $(w^{(y)}, n^{(y)})$

7. Re-estimate $\hat{\theta}^{(y)}$ from $(w^{(y)}, n^{(y)})$

8. Compute $\tilde{\pi}_t^{(y)}$ and entropy $H_y = H(\tilde{\pi}_t^{(y)})$

9. end for

10. $\mathbb{E}[H_{\mathrm{after}}] \leftarrow p_{\mathrm{win}}H_1 + (1 - p_{\mathrm{win}})H_0$

11. return $\tilde{g}_t(c, d) = \max\{0, H_{\mathrm{before}} - \mathbb{E}[H_{\mathrm{after}}]\}$

Complexity. For each candidate challenger, the proxy performs two hypothetical BT re-estimations, one for each outcome. With a fixed number of MM iterations, an acquisition round therefore requires $2(K - 1)$ BT re-estimations.

Experiment Details

Dataset

LaMP LaMP evaluates personalization from limited user history across several language tasks. We use its two generative tasks:

• LaMP-4 (Personalized News Headline Generation): Given an article, generate a headline consistent with the user’s historical headline style.

• LaMP-5 (Personalized Scholarly Title Generation): Given an abstract, generate a paper title consistent with the user’s publication history.

LongLaMP LongLaMP extends personalized evaluation to longer-form generation. We use three tasks:

• LongLaMP-2 (Personalized Abstract Generation): Generate an abstract from a title and keywords in the user’s academic style.

• LongLaMP-3 (Personalized Product Review Generation): Generate a review from a product description, rating, and review summary in the user’s style.

• LongLaMP-4 (Personalized Topic Generation): Expand a topic summary into a Reddit post consistent with the user’s history.

PrefEval PrefEval is a multi-turn dialogue dataset designed for evaluating preference following. We adapt it by treating the dialogue history as persona context $P(C)$. The final user query is used as the input $q$, and the corresponding assistant response serves as the held-out reference. The agent is expected to answer $q$ while remaining consistent with the inferred user persona.

UltraChat UltraChat contains multi-turn human–AI conversations. We condition on the dialogue history and predict the user’s next query $Q$, treating next-turn prediction as a user-conditioned generation task distinct from assistant-response generation.

Prompt Templates

This section presents all prompt templates used in ID-TAP, including persona construction, response generation, and evaluation.

Persona Summarization Prompt. The persona space $\mathcal{P}(C)$ is constructed offline from the user’s historical context $C$. We first generate a base persona summary using the following prompt:

Persona Summarization Prompt
=== SYSTEM ===
You are an advanced profile analysis and summarization model. Your goal is to generate
    a coherent, SPECIFIC, and psychologically insightful user persona summary.

=== USER ===
Based on the user’s historical data provided below, generate an informative and
    coherent user persona summary.

### Task Instruction: {instruction}

### User History Data: {history_context}

### Instructions:
1. CORE ANCHORING: Identify 2-3 SPECIFIC, CONCRETE details from the history
2. BEHAVIORAL PATTERNS: Focus on personality traits, interests, tone, communication
    style
3. SPECIFICITY OVER ABSTRACTION: Use CONCRETE language, avoid vague terms

Please provide the final user persona summary (one paragraph, 150-300 words).

Persona Rephrasing Prompt. To construct a diverse persona space of size $K$, we iteratively rephrase the base summary. Each variant $p_i$ is generated by rephrasing the previous variant $p_{i-1}$ with bounded variation. To prevent semantic drift, we periodically reset to the original history context every $R = 15$ iterations.

Persona Rephrasing Prompt

=== SYSTEM ===
You are an advanced language model specialized in balanced rephrasing of personal
profiles. Your task is to rewrite a user persona summary with a SLIGHTLY DIFFERENT
perspective or emphasis.

=== USER ===
Rewrite the following user persona summary with a SLIGHTLY DIFFERENT perspective:

{previous_persona}

### STRICT Constraints:
- PRESERVE CORE ANCHORS: Keep 2–3 specific concrete details from the original
- BOUNDED VARIATION: Change emphasis or perspective, but the person should still be
  recognizable
- NO EXTREME SHIFTS: Do NOT flip personality traits dramatically

Response Generation Prompt. The following prompt template is used to generate persona-conditioned responses:

Response Generation Prompt

### Task Instruction: {instruction}

### Current Task Details: {query}

### Personality and Style Description: {persona_summary}

### Important Words from history: {keywords}

### History context: {history_entries}

### Output Requirement:
Complete the task according to the Task Instruction and Current Task Details.
    Prioritize task accuracy and relevance while using the Personality and Style
    Description to guide tone. Provide only the required response below:

LLM-as-Judge Evaluation Prompt. For pairwise preference evaluation, we employ an LLM-as-Judge approach. During online adaptation, the pair contains the two persona-conditioned responses selected for that round. For final W.R. evaluation, the pair contains the method’s final response and the naive response generated directly from the query and user history or context, without persona selection or test-time preference optimization. Before each judge call, the two responses are randomly assigned to the Response 1 and Response 2 positions; the returned label is then mapped back to the corresponding source.

LLM-as-Judge System Role

You are an expert evaluator comparing two candidate responses for the same personalized
task. The user-specific reference exemplifies the intended content and
realization. Decide which response better fulfills the intended output as a whole.
Jointly consider task fulfillment, semantic fidelity, organization and coherence,
and the tone and writing style expressed by the reference, treating surface
phrase overlap as secondary to the overall realization. Respond with ONLY a single
digit: 1 if Response 1 is better, or 0 if Response 2 is better.

Comparison Prompt

Current Query: {query}

User-Specific Reference: {ground_truth}

Response 1: {response_1}

Response 2: {response_2}

Which response better realizes the intended output overall? Reply with only 1 (Response 1 is better) or 0 (Response 2 is better). If they are equivalent overall, return 0 by default. Please provide the final result (1 or 0 only) below:

Data Splitting for LaMP Datasets

For LaMP-based datasets (LaMP-4, LaMP-5, LongLaMP-2, LongLaMP-3, LongLaMP-4), each user’s historical profile is split into two parts: (1) history context used for persona generation, and (2) input/output pairs used as test queries with held-out reference responses.

Splitting Rules. Let $n$ denote the number of profile entries for a user. The splitting follows a threshold-based strategy:

• If $n < T_{\mathrm{small}}$: The first $H_{\mathrm{small}}$ entries form the history context; the remaining entries become input/output pairs.

• If $T_{\mathrm{small}} \le n < T_{\mathrm{large}}$: The first $H_{\mathrm{large}}$ entries form the history context; the remaining entries become input/output pairs.

• If $n \ge T_{\mathrm{large}}$: The first $H_{\mathrm{large}}$ entries form the history context; entries from position $H_{\mathrm{large}}+1$ to $H_{\mathrm{large}} + (N_{\mathrm{IO}} - 1)$ become input/output pairs (capped at $N_{\mathrm{IO}} - 1$ additional pairs).

Parameter	Symbol	Value
Small profile threshold	$T_{\mathrm{small}}$	10
Large profile threshold	$T_{\mathrm{large}}$	20
History context count (small)	$H_{\mathrm{small}}$	5
History context count (large)	$H_{\mathrm{large}}$	10
Total I/O pairs limit	$N_{\mathrm{IO}}$	10

Query and Output Extraction. For each dataset, queries and held-out reference outputs are extracted from profile entries as follows:

• LaMP-4 (News Headline): Query = article text; Output = headline title

• LaMP-5 (Paper Title): Query = abstract; Output = paper title

• LongLaMP-2 (Abstract Generation): Query = paper title; Output = abstract

• LongLaMP-3 (Product Review): Query = product description + rating; Output = review text

• LongLaMP-4 (Reddit Post): Query = topic summary; Output = post content

The history context supports persona summarization and construction of $\mathcal{P}(C)$. Each input defines a test query, while its held-out output conditions the offline proxy user and evaluation. During adaptation, the proxy user retains this reference and ID-TAP receives the resulting binary comparison.

Unnormalized ROUGE-L Statistics

For each user, raw (unnormalized) ROUGE-L is averaged over three independent runs. Table 4 reports the mean and SEM across these per-user results. $N_{\mathrm{eval}}$ denotes the number of query-level evaluation instances contributing to the user-level aggregates. In our logging notation, lamp8/lamp9/lamp10 correspond to LongLaMP-2/LongLaMP-3/LongLaMP-4.

Dataset	$N_{\mathrm{eval}}$	PersonaAgent	POHF	PDO Samp.	ID-TAP
LaMP-4	363	0.1700±0.007846	0.1790±0.007499	0.1768±0.007085	0.1973±0.006559
LaMP-5	400	0.3924±0.009460	0.4385±0.008671	0.4346±0.008551	0.4927±0.008513
LongLaMP-2	400	0.1729±0.002141	0.1716±0.002119	0.1678±0.001918	0.1831±0.002042
LongLaMP-3	230	0.1322±0.001532	0.1391±0.001712	0.1378±0.001251	0.1456±0.001562
LongLaMP-4	326	0.1217±0.001576	0.1234±0.001806	0.1234±0.001547	0.1367±0.001853
PrefEval	40	0.1700±0.005887	0.2103±0.006989	0.1943±0.004623	0.2131±0.007093
UltraChat	40	0.0758±0.005492	0.1859±0.009107	0.1815±0.009301	0.1909±0.008171
Average	–	0.1764	0.2068	0.2023	0.2228

ID-TAP has the highest raw ROUGE-L mean on all seven datasets, with a cross-dataset average of 0.2228 compared with 0.2068 for the strongest baseline. These unnormalized values provide a scale-preserving counterpart to the normalized main results.

Persona Space Quality Analysis

To characterize the generated persona spaces, Table 5 reports five geometric metrics for all seven benchmarks. The analysis covers 180 users, with 400 candidate personas per user. Here, $N_{\mathrm{user}}$ is 20 for each LaMP/LongLaMP dataset and 40 for PrefEval and UltraChat.

Dataset	$N_{\mathrm{user}}$	$\bar{\mu}$	$\bar{\sigma}$	$p_{10}$	$p_{90}$	Near-dup $\ge0.95$ (%)	Eff. Arm (%)	Silhouette ($K=20$)
LaMP-4	20	0.672	0.090	0.553	0.779	0.041	97.6	0.104
LaMP-5	20	0.694	0.081	0.590	0.790	0.094	95.3	0.125
LongLaMP-2	20	0.701	0.080	0.599	0.794	0.092	95.0	0.121
LongLaMP-3	20	0.657	0.093	0.537	0.769	0.018	98.2	0.107
LongLaMP-4	20	0.686	0.084	0.577	0.787	0.037	97.5	0.105
PrefEval	40	0.716	0.083	0.607	0.814	0.190	94.4	0.098
UltraChat	40	0.715	0.087	0.600	0.819	0.158	94.5	0.097

Density & Spread. The mean pairwise cosine similarity $\bar{\mu}$ ranges from 0.657 (LongLaMP-3) to 0.716 (PrefEval), with standard deviations $\bar{\sigma}$ between 0.080 and 0.093. The 10th–90th percentile interval ($p_{10}$–$p_{90}$) spans roughly 0.22 in each dataset (e.g., [0.537, 0.769] for LongLaMP-3). These statistics describe moderately dense but non-saturated embedding spaces.

Redundancy. The near-duplicate rate (fraction of pairs with cosine similarity $\ge0.95$) stays below 0.20%, and the effective-arm ratio—the percentage of candidates whose nearest-neighbor similarity is below 0.95—ranges from 94.4% to 98.2%. Under this threshold, the pools contain little near-duplicate redundancy while retaining many distinct candidate embeddings.

Geometric Structure. K-means silhouette scores at $K=20$ range from 0.097 (UltraChat) to 0.125 (LaMP-5), indicating weak separation into discrete clusters. Together with the similarity statistics, these scores characterize the candidate pools as dense and weakly clustered; the acquisition experiments measure the corresponding method-level behavior.

Hyperparameters

This section details all hyperparameters used in ID-TAP, organized into four categories: neural network architecture, training configuration, algorithm-specific parameters, and LLM settings. Table 6 provides a comprehensive summary.

Category	Parameter	Symbol	Value
Neural Network	Input dimension	$d_{\mathrm{in}}$	$2 \times d_{\mathrm{emb}}$
Neural Network	Hidden dimension	$d_h$	1024
Neural Network	Number of hidden layers	$L$	1
Neural Network	Activation function	$\sigma(\cdot)$	GELU
Neural Network	Dropout rate	$p_{\mathrm{drop}}$	0.05
Training	Learning rate	$\eta$	$5 \times10^{-4}$
Training	Optimizer	–	AdamW
Training	Weight decay	$\lambda$	$10^{-3}$
Training	Batch size	$B$	4
Training	Update epochs	–	100
Training	Gradient clipping	–	1.0
Training	Learning-rate schedule	–	Cosine to $0.5\eta$
ID-TAP Algorithm	BT smoothing	$\alpha$	0.7
ID-TAP Algorithm	Acquisition floor	$\epsilon_g$	$10^{-8}$
ID-TAP Algorithm	Softmax temperature	$\tau$	4.5
ID-TAP Algorithm	Sigmoid steepness	$\beta_{\mathrm{sim}}$	6
ID-TAP Algorithm	Sigmoid midpoint	$\mu_{\mathrm{sim}}$	0.8
LLM Settings	Model	–	DeepSeek-V3.2
LLM Settings	Temperature (generation)	–	0.6
LLM Settings	Temperature (judge)	–	0.0
LLM Settings	Max output tokens	–	1024
LLM Settings	Frequency penalty	–	0.5
LLM Settings	Presence penalty	–	0.5

Neural Network Architecture. The utility function $f_{\theta}(q,p)$ is parameterized by a single-hidden-layer feedforward neural network. The input consists of the concatenation of query embedding $\phi(q) \in\mathbb{R}^{d_{\mathrm{emb}}}$ and persona embedding $\phi(p) \in\mathbb{R}^{d_{\mathrm{emb}}}$, where $d_{\mathrm{emb}}=1024$ for standard mode and $d_{\mathrm{emb}}=2048$ for contextual mode. We employ GELU activation for its smooth gradient properties and use a dropout rate of 0.05 for regularization. Weights are initialized using Kaiming normal initialization to ensure proper gradient flow at initialization.

Training Configuration. The neural network is trained using the AdamW optimizer with a learning rate of $5 \times10^{-4}$ and weight decay of $10^{-3}$. We use a cosine annealing learning rate scheduler that decays the learning rate to 50% of its initial value over the training epochs. Each update consists of 100 epochs with a batch size of 4, and gradient clipping with a maximum norm of 1.0 is applied to prevent gradient explosion during early training stages when comparison data may be noisy.

ID-TAP Algorithm Parameters. The acquisition ratio in § 3 uses a numerical floor $\epsilon_g = 10^{-8}$. The BT explorer uses smoothing $\alpha= 0.7$ and softmax temperature $\tau= 4.5$ for the normalized strength distribution. Cross-query transfer uses $\beta_{\mathrm{sim}} = 6$ and $\mu_{\mathrm{sim}} = 0.8$ in the normalized similarity weights.

LLM Configuration. The main configuration uses DeepSeek-V3.2 for response generation and for the LLM-based preference proxy and final evaluation; the ROUGE-L-based simulator follows the scoring rule in § 4.3. Appendix § 13 varies the preference and evaluation models. Generation uses temperature 0.6, whereas LLM-as-Judge evaluation uses temperature 0.0. The maximum output length is 1024 tokens, with frequency and presence penalties of 0.5.

Persona-Construction and Density Analysis

Construction and Exploration Diagnostic

For each dataset, we compare iterative rephrasing (IR) with keyword-conditioned construction (KW) using a common candidate budget, response generator, embedding model, interaction budget, and evaluation protocol. IR follows the persona rephrasing procedure in Section B.2, including the reset interval $R = 15$. KW extracts preference and style keywords from the same user history, assigns keyword combinations to candidate arms, and generates a persona conditioned on each combination. Both constructions use Qwen3-Embedding-4B representations before online optimization.

For method $m$, let $\mathcal{H}_{t-1}^{q,m} = \{p_{1,2}^{q,m}, \ldots, p_{t-1,2}^{q,m}\}$ denote the challengers selected before round $t$. The Arm-2 exploration rate is

The first round is excluded because its challenger is necessarily unseen. A repeated challenger is counted as exploitation, and the same tie-handling implementation is used for all methods.

Dataset	$\bar{\mu}$ (IR)	$\bar{\mu}$ (KW)	$\Delta\bar{\mu}$
LaMP-4	0.672	0.720	+0.048
LongLaMP-4	0.686	0.695	+0.009
PrefEval	0.716	0.830	+0.114
UltraChat	0.715	0.834	+0.119

Dataset	POHF	PDO Samp.	ID-TAP
LaMP-4	24.1	25.7	73.5
LongLaMP-4	25.3	28.1	72.2
PrefEval	65.0	64.0	95.3
UltraChat	67.3	66.8	94.8

Dataset	POHF	PDO Samp.	ID-TAP
LaMP-4	16.6 (−7.5)	17.6 (−8.1)	74.8 (+1.3)
LongLaMP-4	16.5 (−8.8)	20.3 (−7.8)	71.9 (−0.3)
PrefEval	61.0 (−4.0)	62.5 (−1.5)	94.4 (−0.9)
UltraChat	64.8 (−2.5)	60.5 (−6.3)	93.3 (−1.5)

Across all four datasets, KW increases mean persona similarity and is accompanied by lower exploration for POHF and PDO Sampling. ID-TAP’s exploration rate changes by at most 1.5 percentage points between the two constructions. Together, these results show an association between denser persona spaces and reduced conventional exploration, alongside comparatively stable exploration by ID-TAP.

Exploration–Performance Association

Dataset group	Mean POHF exploration	Mean ID-TAP margin
LaMP-4 and LongLaMP-4	16.55%	+0.0160
PrefEval and UltraChat	62.90%	+0.0065

The datasets with lower POHF exploration under KW also have the larger reported ID-TAP margin. Together with the within-dataset IR–KW comparison, this descriptive pattern aligns lower conventional exploration with a larger performance margin for ID-TAP.

Sensitivity to Persona-Space Size and Construction

In the following tables, $n_{\mathrm{user}} = 40$ denotes sampled users per dataset. These experiments use $K = 100$ candidates, whereas the main experiments use $K = 400$.

Dataset	ID-TAP	POHF	PDO Samp.	PersonaAgent
UltraChat	0.2430	0.2291	0.2213	0.1057
PrefEval	0.2587	0.2498	0.2430	0.1651
LaMP-4	0.2262	0.1900	0.2001	0.1529
LongLaMP-4	0.1392	0.1310	0.1320	0.1230

Dataset	ID-TAP	POHF	PDO Samp.	PersonaAgent
UltraChat	0.2565	0.2441	0.2496	0.1090
PrefEval	0.2614	0.2556	0.2527	0.1647
LaMP-4	0.2495	0.2201	0.2172	0.1554
LongLaMP-4	0.1388	0.1356	0.1350	0.1253

At $K = 100$, ID-TAP achieves the highest mean ROUGE-L under both candidate-space constructions, extending the main $K = 400$ result to a smaller persona pool and an alternative construction procedure.

Robustness to Preference and Evaluation Models

We evaluate three settings: GPT-4o for both preference generation and final evaluation, GPT-4o for preference generation with DeepSeek-V3.2 for final evaluation, and BERTScore as an evaluation metric independent of the optimizer’s preference signal. Each setting follows the main protocol for candidate pools, interaction budgets, response generation, prompts, and aggregation, while varying the specified preference or evaluation component.

Dataset	PersonaAgent	POHF	PDO Samp.	ID-TAP
LaMP-4	73.2	80.2	80.4	81.9
UltraChat	81.7	86.7	85.8	89.1

Dataset	PersonaAgent	POHF	PDO Samp.	ID-TAP
LaMP-4	68.1	75.0	77.2	78.2
UltraChat	64.2	85.0	82.5	89.2

Dataset	PersonaAgent	POHF	PDO Samp.	ID-TAP
UltraChat	0.838	0.890	0.889	0.892
PrefEval	0.847	0.885	0.887	0.888
LaMP-4	0.862	0.870	0.872	0.876
LongLaMP-4	0.812	0.814	0.812	0.818

ID-TAP achieves the highest reported mean in every configuration. The matched GPT-4o setting changes the automated preference source and final evaluator together, the GPT-4o/DeepSeek-V3.2 setting separates those roles, and BERTScore evaluates responses independently of the preference-optimization signal. The resulting ordering is consistent across the three evaluation configurations.

Runtime and Timing Protocol

Timing measurements were collected on a workstation equipped with a single NVIDIA GeForce RTX 4090 GPU using the model-serving backend, batch behavior, and concurrency configuration of the main experiments. Here, $K$ denotes the number of candidate personas in each user’s persona space, and $n_{\mathrm{user}}$ denotes the number of sampled users per dataset. The measurements use $K = 100$ and $n_{\mathrm{user}} = 40$. Each user’s timing is averaged over three independent runs, and the tables report the mean, standard deviation, and SEM across these per-user averages. Wall-clock time covers a complete interaction round, including response generation and evaluation; update time isolates persona selection, BT-MLE estimation, neural-model updating, and entropy-proxy computation.

Dataset	Mean	Std.	SEM
UltraChat	1.1142	0.4211	0.0666
PrefEval	0.9513	0.1745	0.0276
LaMP-4	2.5674	0.9069	0.1434
LongLaMP-4	3.1499	0.9750	0.1542

Dataset	Mean	Std.	SEM
UltraChat	0.0964	0.0189	0.0030
PrefEval	0.0651	0.0157	0.0025
LaMP-4	0.5709	0.1459	0.0231
LongLaMP-4	0.3172	0.0839	0.0133

Algorithmic updates require 0.065–0.571 seconds per round, compared with 0.95–3.15 seconds end to end. Based on the reported means, the algorithmic component accounts for approximately 7–22% of total latency, with model serving dominating the measured wall-clock time.

Persona Examples

LaMP-4

Persona A

This user approaches political coverage as a strategic observer, with a particular focus on the mechanics of the 2016 Republican primary. Their analytical lens is drawn to the concrete maneuvers and positional conflicts between figures like Donald Trump, Ted Cruz, and Jeb Bush. They consistently select content detailing the “neck-and-neck battle” in Iowa or the logistical specifics of a vice-presidential search, prioritizing articles that dissect how candidates compete. Their neutral tone and preference for headline framing evince as consequential clashes—a “fiery debate” or a “growing feud”—reveal a perspective that treats politics as a complex game. The core interest remains in tracking concrete developments, such as poll results and campaign strategies that signal who might “knock out” rivals.

Persona B

This user approaches political campaigns as intricate systems to be reverse-engineered, with a focus on how discrete operational choices cascade into winning or losing outcomes. Their detailed study of the 2016 Republican primary, particularly the delegate calculations that defined the Trump–Cruz dynamic in Iowa, serves as a foundational case. They treat strategic events—like Marco Rubio’s decision to decline a vice-presidential spot—not as political drama but as pivotal system inputs that reroute a campaign’s entire pathway. Polling shifts and tactical proposals are analyzed as interconnected variables within a dynamic model.

Persona C

The user is a politically observant individual with a concentrated focus on the 2016 Republican presidential primary’s competitive landscape. Their interest centers on key figures like Donald Trump, Ted Cruz, Jeb Bush, and Marco Rubio, with attention to pivotal moments such as Trump’s vice-presidential search involving Mike Pence and the “neck-and-neck battle” in Iowa polls. They consistently favor content analyzing electoral mechanics and candidate positioning—tracking how a Trump sweep could “knock out” rivals in Florida and Ohio. Their chosen material frames events through a lens of conflict and consequence, highlighting a “growing feud,” a “fiery debate,” or campaigns “scrambling to make up ground.”

LongLaMP-4

Persona A

This user is a systematic thinker who values logical consistency, clear hierarchies, and functional design across diverse domains. They critique Minecraft’s development for lacking direction, advocating for a structured “wood–stone–iron–diamond” material system over random additions like the “bizarre endgame” and poorly implemented potions. Their analytical approach extends to physics, explaining complex concepts like 4-vectors by simplifying them into core principles. They exhibit strong, evidence-backed opinions, defending financial markets as providing liquidity and downward price pressure, while dismissing overly broad definitions of art. Their communication is assertive and detailed, often using personal comparisons—like favoring a $700 HP laptop over a $1500 MacBook—to ground arguments in pragmatic cost-benefit analysis. This pattern reveals a persona deeply invested in deconstructing systems, prioritizing structured understanding and practical utility over abstract or sentimental reasoning.

Persona B

This thinker approaches systems with a forensic eye for functional hierarchy and practical coherence. Their satisfaction comes from mapping how components interlock to produce reliable outcomes, whether in game design’s reward loops or technology’s price-to-performance ratios. They defend Minecraft’s clean “wood–stone–iron–diamond” progression as logically sound architecture, critiquing later potion systems as messy deviations. Their analysis of consumer tech, such as comparing a $700 HP laptop to a $1500 MacBook, seeks tangible benchmarks over brand prestige. Communication is direct and dismantling: they prefer explaining complex phenomena—like the constancy of light via 4-vectors or financial instruments as liquidity tools—through grounded mechanics. For them, rigor is the removal of sentimental abstraction to reveal the operative skeleton beneath.

More Experimental Results

Figure 5 reports the individual-dataset results underlying the aggregate view in the main paper. Figure 6 shows how performance evolves within the fixed comparison budget and provides dataset-specific detail beyond the aggregate curve.

Detailed Performance Comparison Across Datasets

Figure 5 compares ID-TAP across three benchmark categories: (1) LaMP, covering personalized headline and scholarly-title generation; (2) LongLaMP, covering personalized abstract, review, and topic generation; and (3) multi-turn dialogue, covering UltraChat and PrefEval.

Detailed Iteration-Level Trajectories

Figure 6 presents the iteration-level trajectory for each benchmark under automated proxy feedback.