betaFeedback

WorksPhysics and Astronomy

Time-ordered free energy in correlated quantum systems: An agentic approach

Abstract

How much work can an agent extract from a temporal sequence of quantum states when it can only operate online under causal constraints—deciding which energy extraction method to use with knowledge of what it has observed before? Here, we study this problem in the context of quantum state sequences that are potentially non-Markovian—generated by some underlying hidden Markov machine that the agent cannot observe. Using techniques from dynamic programming and computational mechanics, we present a method to identify the provably optimal agent strategy, with time complexity that scales linearly with sequence length. This motivates us to introduce the maximum work such agents can extract—time-ordered free energy(TOFE)—as a fundamental measure of free energy available in a temporally correlated quantum system subject to causal considerations.

Introduction

While free energy is typically discussed in the context of a state out of equilibrium at a fixed time, it also arises in temporal sequences of states. In such settings, extracting this free energy as useful work typically requires an agent that remembers past states and adapts its future actions accordingly. Consider an agent foraging in an environment that emits a sequence of bits alternating between 0 and 1. Each system appears maximally mixed to an agent with no memory and thus contains no free energy. However, an agent that tracks the previous bit can extract up to kBTln⁡2k_{\mathrm{B}}T \ln2 of free energy at ambient temperature TT [1]. Classically, these ideas motivated the development of information ratchets. Operationally, they model an organism that is foraging for food or an edge device seeking a power source [2, 3, 4]. Conceptually, they underpin our understanding of free energy in temporal correlations and the thermodynamics of learning and prediction [5, 6, 7, 8, 9, 10, 11].

How do such ideas extend to quantum environments? Here, we consider an agent with access to a stream of LL quantum systems. The systems are temporally correlated in that they are generated by a hidden Markov model (HMM) whose states are hidden from the agent. The theoretically optimal strategy is to retain all systems and interact coherently with them at once [12, 13, 14, 15, 16, 17]. However, such strategies require quantum memory that scales without bound with the length of the data stream. This conflicts with the perspective of an autonomous machine harvesting energy online and presents an immense practical challenge with current technology [18, 19].

Consequently, we consider online agents bound by an arrow of time: they must process the sequence of quantum states and extract energy sequentially, with actions at time tt depending only on past observations, thereby forbidding postponement of extraction until the end of the protocol. We ask how much energy an agent can harvest under such causality constraints and how this maximum is achieved. This leads us to the time-ordered free energy (TOFE) of a sequence of quantum states, defined as the maximum free energy available to a causally constrained agent.

The TOFE may at first appear intractable to compute. Unlike classical bits, which can only be measured in a single basis, quantum systems allow many ways to harvest energy. Thus, an agent’s possible action sequences scale exponentially with LL, even when limited to finitely many actions per time step. Remarkably, we develop a dynamic programming (DP) algorithm that optimizes agent actions in time linear in LL for any fixed set of actions. We use this to answer a previously open question in sequential energy harvesting from quantum systems [20], showing that the best strategy is not necessarily to maximize the energy gain at each time step.

Framework—Consider a source that emits a quantum system QtQ_t in state σQt(xt)\sigma_{Q_t}^{(x_t)} at the discrete time step tt, where each state is identified by a label x∈Xx \in\mathcal{X}. We model this source as a hidden Markov model (HMM), denoted by M=(S,(σ(x))x∈X,(T(x))x∈X,μ0)\mathcal{M} = (\mathcal{S},(\sigma^{(x)})_{x\in\mathcal{X}},(\mathsf{T}^{(x)})_{x\in\mathcal{X}},\mu_0), where S\mathcal{S} represents the set of latent states and μ0\mu_0 defines their initial distribution. The dynamics of the system are governed by the elements of the transition matrix, Ts,s′(x)≡Pr⁡(St=s′,Xt=x∣St−1=s)\mathsf{T}_{s,s'}^{(x)} \equiv\Pr(S_t=s',X_t=x\mid S_{t-1}=s), which dictate the probability of transitioning from a latent state ss to s′s' while simultaneously emitting the quantum state σ(x)\sigma^{(x)}. Over a sequence of LL time steps, the process emits the multi-time quantum state ρ(1:L)\rho^{(1:L)}, expressed as:

ρ(1:L)=∑x1:L∈XLPr⁡(x1:L)⨂t=1LσQt(xt).(1)\rho^{(1:L)} = \sum_{x_{1:L}\in\mathcal{X}^{L}} \Pr(x_{1:L}) \bigotimes_{t=1}^{L} \sigma_{Q_t}^{(x_t)} . \tag*{(1)}

A simple example of an HMM source, known as the perturbed coin, can be found in Figure 1(a).

Perturbed coin process and causal network

Figure 1. (a) The perturbed coin process is an example of a quantum hidden state source. It possesses two latent states ss and s′s' (nodes). The directed edges describe its dynamics: an edge from ss to s′s' labeled p:σp:\sigma indicates that a source in state ss has probability pp to transition to s′s' and emit state σ\sigma. (b) A causal network depicting the interaction between the hidden quantum process (top, red) and the sequential agent (bottom, blue). The agent does not have access to sts_t and thus can only select an extraction method AtA_t based on its current memory state Kt−1K_{t-1}. The extracted work WtW_t acts as an observable, allowing the agent to update its memory to KtK_t.

We take on the role of the agent, who is given access to QtQ_t at each time step tt. We assume the agent lacks a quantum memory that persists between time steps, so it is restricted to performing local operations sequentially. It must obey causality: its actions at any time step tt can depend only on actions and observations at times t′<tt' < t. For an agent constrained by locality and causality, the maximum expected extractable work defines the TOFE, FTO(L)\mathcal{F}_{\mathrm{TO}}^{(L)}.

We adopt the standard resource-theoretic framework for thermodynamics, in which an agent extracts work from a sequence of quantum systems QtQ_t and stores it in an ideal battery BtB_t. Modeled as a semiclassical weight, the battery ensures that the extracted work is defined as the change in its energy. At each time step tt, the agent is restricted to thermal operations mediated by a memoryless thermal reservoir RR, which is initialized in the thermal state γ=Z−1e−βHR\gamma= Z^{-1}e^{-\beta H_R} with partition function Z=tr⁡(e−βHR)Z = \operatorname{tr}(e^{-\beta H_R}) at inverse temperature β=(kBT)−1\beta= (k_{\mathrm{B}}T)^{-1}. These widely shared assumptions establish fundamental thermodynamic benchmarks [14, 16, 21, 22, 23, 24, 25, 26, 27]. We define the action space as A={Wρ}\mathcal{A} = \{W_\rho\}, parametrized by target states ρ\rho, where WρW_\rho is tailored to extract all non-equilibrium free energy from ρ\rho [20, 28, 29]. Here, we assume that the Hamiltonian governing QtQ_t is energy-degenerate, and revisit the non-degenerate case in Appendix A.

The most general means for an agent to harvest work sequentially is via a causal strategy Γ\Gamma, a systematic rule for choosing which action at∈Aa_t \in\mathcal{A} to apply based on the prior history ht−1=(a1,w1,…,at−1,wt−1)h_{t-1} = (a_1,w_1,\ldots,a_{t-1},w_{t-1}). The TOFE over LL time steps is then the maximum extracted work when optimized over all causal strategies:

FTO(1:L)(ρ(1:L)):=max⁡ΓEHL∣Γ[∑t=1LWt],(2)\mathcal{F}_{\mathrm{TO}}^{(1:L)}(\rho^{(1:L)}) := \max_{\Gamma}\mathbb{E}_{H_L\mid\Gamma}\left[\sum_{t=1}^{L} W_t\right], \tag*{(2)}

where the expectation is taken over HL∣ΓH_L\mid\Gamma, the random variable governing hLh_L when the agent executes Γ\Gamma.

Naïvely, executing such an optimal strategy could require the agent to store the entire history hth_t. In Appendix C1, we show that this is unnecessary. Instead, an agent can store a suitable belief state

ηt:=[Pr⁡(St=s∣ht)]s∈S,(3)\boldsymbol{\eta}_t := \left[\Pr(S_t=s\mid\mathbf{h}_t)\right]_{s\in\mathcal{S}}, \tag*{(3)}

representing its subjective belief of which latent state the source is in, conditioned on its action-observation history. After each round, the agent updates this belief based on the observed work value using Bayesian inference [30, 31]. Specifically, upon observing a work value wtw_t, the agent’s belief state is updated according to the Bayesian update rule τ\tau,

ηt=τ(ηt−1,at,wt)\boldsymbol{\eta}_t = \tau(\boldsymbol{\eta}_{t-1},a_t,w_t)
∝∑xt∈XPr⁡(Wt=wt∣xt,at,ηt−1)ηt−1T(xt),(4)\propto\sum_{x_t\in\mathcal{X}} \Pr(W_t=w_t\mid x_t,a_t,\boldsymbol{\eta}_{t-1})\boldsymbol{\eta}_{t-1}\mathbf{T}^{(x_t)}, \tag*{(4)}

where proportionality ensures normalization [20, 32] and T(xt)\mathbf{T}^{(x_t)} is the labeled transition matrix of the HMM. In Appendix J, we show that updates can always be done near-reversibly, resulting in a negligible contribution to heat dissipation.

Consequently, the history-dependent policy Γ\Gamma can be replaced by a belief-dependent policy Λ\Lambda that maps any belief state to an allowed action [33, 34]. In the following, we will refer to Λ\Lambda simply as the policy. At time tt, an agent in the memory state ηt−1\boldsymbol{\eta}_{t-1} executes an action at=Λ(ηt−1)∈Aa_t=\Lambda(\boldsymbol{\eta}_{t-1})\in\mathcal{A} to extract free energy from σQt\sigma_{Q_t}, storing it in the battery BtB_t. The extracted work wtw_t corresponds to the measurement result for BtB_t in the energy eigenbasis [35, 36]. Following this, the agent updates its memory to ηt=τ(ηt−1,at,wt)\boldsymbol{\eta}_t=\tau(\boldsymbol{\eta}_{t-1},a_t,w_t). This protocol is illustrated in Figure 1(b), where the random variables St,Xt,At,Kt,WtS_t,X_t,A_t,K_t,W_t take the realizations st,xt,at,ηt,wts_t,x_t,a_t,\eta_t,w_t, respectively. Consequently, determining the TOFE and its optimal harvesting strategy reduces to finding the optimal policy Λ\Lambda.

Policy optimization—Recall that the average work extracted by a protocol designed for a state ρ\rho when acting on a state σ\sigma is given by [28]:

β⟨Wext⟩=[D(σQ∥γQ)−D(σQ∥ρ)].(5)\beta\langle W_{\mathrm{ext}}\rangle= \left[D(\sigma_Q\|\gamma_Q)-D(\sigma_Q\|\rho)\right]. \tag*{(5)}

Here, γQ=ZQ−1e−βHQ\gamma_Q=Z_Q^{-1}e^{-\beta H_Q} is the thermal state of QQ and D(σ∥ρ)=tr⁡(σlog⁡σ)−tr⁡(σlog⁡ρ)D(\sigma\|\rho)=\operatorname{tr}(\sigma\log\sigma)-\operatorname{tr}(\sigma\log\rho) is the quantum relative entropy. The first term in Eq. (5) represents the non-equilibrium free energy available in σ\sigma, while the second represents the dissipated heat due to expectation mismatch that vanishes if σ=ρ\sigma=\rho. Since our agent lacks knowledge of σ\sigma, the expected work extracted by taking action aa when the agent is in a belief state η\eta, ⟨W(η,a)⟩\langle W(\eta,a)\rangle, can be expressed as

β⟨W(η,a)⟩:=D(ξη∥γQ)−D(ξη∥ρa),(6)\beta\langle W(\eta,a)\rangle:= D(\xi_\eta\|\gamma_Q)-D(\xi_\eta\|\rho_a), \tag*{(6)}

where ξη:=∑x∈XPr⁡(x∣η)σ(x)\xi_\eta:=\sum_{x\in\mathcal{X}}\Pr(x|\eta)\sigma^{(x)} is the agent’s expected quantum state, and ρa\rho_a is the target state of action aa.

A possible policy is the “greedy” or local-optimizing (LO) policy, which would choose an action that maximizes the immediate work extracted. Based on Eq. (6), this LO policy would be tailored to the current expected state, ρa=ξη\rho_a=\xi_\eta, thereby eliminating the dissipative term and maximizing the immediate expected work [20]. However, strategically sacrificing immediate work by choosing a “mismatched” protocol may yield greater predictive information for subsequent steps, leading to higher cumulative work extraction. The dissipation term D(ξη∥ρa)D(\xi_\eta\|\rho_a) can thus be interpreted as the price of information—a trade-off between maximizing immediate reward and gaining knowledge for future benefit.

Optimizing over all histories scales exponentially with the sequence length LL. Instead, treating belief states KtK_t as sufficient statistics enables recursive DP. For a fixed action set, the optimal value functions obtained from progressively finer discretizations of the continuous belief space converge to the optimal value function of the underlying continuous-state problem [37, 38]. We therefore formulate the resulting finite-state problem as a dynamic program, whose time complexity scales linearly with LL.

Theorem 1. For a given set of belief states K\mathcal{K} and an action set A\mathcal{A}, there exists a dynamic programming algorithm to determine the optimal classical-causal policy Λ∗\Lambda^* in time O(L)O(L) that maximizes the average cumulative work extracted over LL time steps. Furthermore, if the agent’s initial belief matches the initial distribution of the underlying HMM, this maximized cumulative work is mathematically equivalent to the TOFE, FTO(1:L)\mathcal{F}_{\mathrm{TO}}^{(1:L)} in Eq. (2).

We detail the algorithm (Alg. 1) and prove its optimality in Appendix § 1.3, alongside the formal equivalence derivation in Appendix C. To ensure computational tractability at each step tt, Theorem 3 (Appendix § 1.6) proves it is sufficient to search only over the set of possible eigenbases rather than the full space of density matrices.

Thermodynamic hierarchy—In the asymptotic limit, the TOFE rate is defined using Eq. (2) as

References

  1. [1]Leo Szilard. On the decrease of entropy in a thermodynamic system by the intervention of intelligent beings. Behavioral Science, 9(4):301–310, 1964.pubmed.ncbi.nlm.nih.gov/5888785
  2. [2]Susanne Still, David A Sivak, Anthony J Bell, and Gavin E Crooks. Thermodynamics of prediction. Physical review letters, 109(12):120604, 2012.pubmed.ncbi.nlm.nih.gov/23005932
  3. [3]Artemy Kolchinsky, Iman Marvian, Can Gokler, Zi-Wen Liu, Peter Shor, Oles Shtanko, Kevin Thompson, David Wolpert, and Seth Lloyd. Maximizing free energy gain. Entropy, 27(1):91, 2025.DOI
  4. [4]Seth Lloyd. Thermodynamics+ natural selection= bayesian inference. arXiv preprint arXiv:2511.17641, 2025.arXiv
  5. [5]Dibyendu Mandal and Christopher Jarzynski. Work and information processing in a solvable model of maxwell’s demon. Proceedings of the National Academy of Sciences, 109(29):11641–11645, 2012.DOI
  6. [6]Alexander B Boyd, Dibyendu Mandal, and James P Crutchfield. Identifying functional thermodynamics in autonomous maxwellian ratchets. New Journal of Physics, 18(2):023049, 2016.DOI
  7. [7]Alexander B Boyd, Dibyendu Mandal, and James P Crutchfield. Leveraging environmental correlations: The thermodynamics of requisite variety. Journal of Statistical Physics, 167(6):1555–1585, 2017.DOI
  8. [8]Andrew JP Garner, Jayne Thompson, Vlatko Vedral, and Mile Gu. Thermodynamics of complexity and pattern manipulation. Physical Review E, 95(4):042140, 2017.DOI
  9. [9]Alexander B Boyd, Dibyendu Mandal, and James P Crutchfield. Thermodynamics of modularity: Structural costs beyond the landauer bound. Physical Review X, 8(3):031036, 2018.DOI
  10. [10]Andrew JP Garner. The fundamental thermodynamic bounds on finite models. Chaos: An Interdisciplinary Journal of Nonlinear Science, 31(6), 2021.DOI
  11. [11]Alexander B Boyd, James P Crutchfield, Mile Gu, and Felix C Binder. Thermodynamic overfitting and generalization: energetics of predictive intelligence. New Journal of Physics, 27(6):063901, 2025.DOI
  12. [12]Armen E Allahverdyan, Roger Balian, and Th M Nieuwenhuizen. Maximal work extraction from finite quantum systems. Europhysics Letters, 67(4):565, 2004.DOI
  13. [13]Johan Åberg. Truly work-like work extraction via a single-shot analysis. Nature communications, 4(1):1925, 2013.DOI
  14. [14]Fernando GSL Brandao, Michał Horodecki, Jonathan Oppenheim, Joseph M Renes, and Robert W Spekkens. Resource theory of quantum states out of thermal equilibrium. Physical review letters, 111(25):250404, 2013.pubmed.ncbi.nlm.nih.gov/24483734
  15. [15]Michał Horodecki and Jonathan Oppenheim. Fundamental limitations for quantum and nanoscale thermodynamics. Nature communications, 4(1):2059, 2013.DOI
  16. [16]Paul Skrzypczyk, Anthony J Short, and Sandu Popescu. Work extraction and thermodynamics for individual quantum systems. Nature communications, 5(1):4185, 2014.DOI
  17. [17]Dominik Šafránek, Dario Rosa, and Felix C Binder. Work extraction from unknown quantum sources. Physical Review Letters, 130(21):210401, 2023.DOI
  18. [18]Alexander I Lvovsky, Barry C Sanders, and Wolfgang Tittel. Optical quantum memory. Nature photonics, 3(12):706–714, 2009.arxiv.org/abs/1002.4659
  19. [19]Khabat Heshami, Duncan G England, Peter C Humphreys, Philip J Bustard, Victor M Acosta, Joshua Nunn, and Benjamin J Sussman. Quantum memories: emerging applications and recent advances. Journal of modern optics, 63(20):2005–2028, 2016.DOI
  20. [20]Ruo Cheng Huang, Paul M Riechers, Mile Gu, and Varun Narasimhachar. Engines for predictive work extraction from memoryful quantum stochastic processes. Quantum, 7:1203, 2023.DOI
  21. [21]Ronnie Kosloff. Quantum thermodynamics: A dynamical viewpoint. Entropy, 15(6):2100–2128, 2013.DOI
  22. [22]Johan Åberg. Catalytic coherence. Physical review letters, 113(15):150402, 2014.pubmed.ncbi.nlm.nih.gov/25375693
  23. [23]Gilad Gour, Markus P Müller, Varun Narasimhachar, Robert W Spekkens, and Nicole Yunger Halpern. The resource theory of informational nonequilibrium in thermodynamics. Physics Reports, 583:1–58, 2015.DOI
  24. [24]Matteo Lostaglio, David Jennings, and Terry Rudolph. Description of quantum coherence in thermodynamic processes requires constraints beyond free energy. Nature communications, 6(1):6383, 2015.DOI
  25. [25]Kamil Korzekwa, Matteo Lostaglio, Jonathan Oppenheim, and David Jennings. The extraction of work from quantum coherence. New Journal of Physics, 18(2):023045, 2016.DOI
  26. [26]Matteo Lostaglio, David Jennings, and Terry Rudolph. Thermodynamic resource theories, non-commutativity and maximum entropy principles. New Journal of Physics, 19(4):043008, 2017.DOI
  27. [27]Carlo Sparaciari, Jonathan Oppenheim, and Tobias Fritz. Resource theory for work and heat. Physical Review A, 96(5):052112, 2017.DOI
  28. [28]Paul M Riechers and Mile Gu. Initial-state dependence of thermodynamic dissipation for any quantum process. Physical Review E, 103(4):042145, 2021.DOI
  29. [29]Josep Lumbreras, Ruo Cheng Huang, Yanglin Hu, Mile Gu, and Marco Tomamichel. Quantum state-agnostic work extraction (almost) without dissipation. arXiv preprint arXiv:2505.09456, 2025.arXiv
  30. [30]Jaynes, E. T. (1946). Probability, frequency and reasonable expectation. American Journal of Physics, 14(1), 1–13.
  31. [31]Edwin T Jaynes. Probability theory: The logic of science. Cambridge university press, 2003.
  32. [32]James P Crutchfield, Christopher J Ellison, and Paul M Riechers. Exact complexity: The spectral structure of intrinsic computation. Physics Letters A, 380(9–10):998–1002, 2016.arxiv.org/abs/1309.3792
  33. [33]Karl Johan Åström. Optimal control of markov processes with incomplete state information i. Journal of mathematical analysis and applications, 10:174–205, 1965.DOI
  34. [34]Richard D Smallwood and Edward J Sondik. The optimal control of partially observable markov processes over a finite horizon. Operations research, 21(5):1071–1088, 1973.DOI
  35. [35]Eugene P Wigner. Die messung quantenmechanischer operatoren. Zeitschrift für Physik A Hadrons and nuclei, 133(1):101–108, 1952.DOI
  36. [36]Huzihiro Araki and Matsuo M Yanase. Measurement of quantum mechanical operators. Physical Review, 120(2):622, 1960.DOI
  37. [37]William S Lovejoy. Computationally feasible bounds for partially observed markov decision processes. Operations research, 39(1):162–175, 1991.DOI
  38. [38]Milos Hauskrecht. Value-function approximations for partially observable markov decision processes. Journal of artificial intelligence research, 13:33–94, 2000.DOI
  39. [39]Richard S Sutton, Andrew G Barto, et al. Reinforcement learning: An introduction, volume 1. MIT press Cambridge, 1998.DOI
  40. [40]Scherrer, B., & Lesner, B. (2012). On the use of non-stationary policies for stationary infinite-horizon markov decision processes. Advances in Neural Information Processing Systems, 25.
  41. [41]Wojciech Hubert Zurek. Quantum discord and maxwell’s demons. Physical Review A, 67(1):012320, 2003.DOI
  42. [42]Aharon Brodutch and Daniel R Terno. Quantum discord, local operations, and maxwell’s demons. Physical Review A—Atomic, Molecular, and Optical Physics, 81(6):062103, 2010.DOI
  43. [43]Kaito Watanabe and Ryuji Takagi. Black box work extraction and composite hypothesis testing. Physical Review Letters, 133(25):250401, 2024.DOI
  44. [44]Kaito Watanabe and Ryuji Takagi. Universal work extraction in quantum thermodynamics. Nature Communications, 17(1):1857, 2026.DOI
  45. [45]Jayne Thompson, Andrew JP Garner, John R Mahoney, James P Crutchfield, Vlatko Vedral, and Mile Gu. Causal asymmetry in a quantum world. Physical Review X, 8(3):031013, 2018.DOI
  46. [46]Spiros Kechrimparis, Mile Gu, and Hyukjoon Kwon. Causal asymmetry of classical and quantum autonomous agents. arXiv preprint arXiv:2309.13572, 2023.arXiv
  47. [48]Mischa P. Woods and Michał Horodecki. Autonomous quantum devices: When are they realizable without additional thermodynamic costs? Phys. Rev. X, 13:011016, Feb 2023.DOI
  48. [49]James P Crutchfield, Karl Young, et al. Inferring statistical complexity. Physical review letters, 63(2):105–108, 1989.pubmed.ncbi.nlm.nih.gov/10040781
  49. [50]Cosma R. Shalizi and James P. Crutchfield. Computational Mechanics: Pattern and Prediction, Structure and Simplicity. Journal of Statistical Physics, 104(3-4):817–879, 2001.DOI
  50. [51]Puterman, M. L. (2014). Markov decision processes: discrete stochastic dynamic programming.
  51. [52]Ellison, C. J., Mahoney, J. R., & Crutchfield, J. P. (2009). Prediction, retrodiction, and the amount of information stored in the present. Journal of Statistical Physics, 136(6), 1005–1034.
  52. [53]Nicholas F Travers and James P Crutchfield. Exact synchronization for finite-state sources. Journal of Statistical Physics, 145(5):1181–1201, 2011.DOI
  53. [54]Paul M Riechers and James P Crutchfield. Spectral simplicity of apparent complexity. ii. exact complexities and complexity spectra. Chaos: An Interdisciplinary Journal of Nonlinear Science, 28(3), 2018.DOI
  54. [55]Richard Bellman, Robert E Kalaba, et al. Dynamic programming and modern control theory, volume 81. Citeseer, 1965.
  55. [56]Nielsen, M. A., & Chuang, I. L. (2010). Quantum computation and quantum information.
  56. [57]Rudin, W. (2021). Principles of mathematical analysis.

Paper details

vlnc:W_cfrgawp2
Version
v1 · latest
Published
Fresh — added to the Hub within the last 30 days
Added
License
CC BY 4.0
Usage

views · 0

downloads · 0

A view counts once per browser per day, a download each time the PDF is requested.

  • Physics and Astronomy

Not recorded for this paper

View statement
Looking for related papers…
Contents