betaFeedback

WorksComputer Science

Bayesian Sequential Optimal Experimental Design for Single-Pixel Imaging using Reinforcement Learning

Abstract

Adaptive compressed sensing (CS) aims to select the most informative measurements sequentially, which is crucial for many modern imaging systems. This task can be formulated as a Bayesian sequential optimal experimental design (sOED) problem, where each new acquisition is chosen to maximally reduce uncertainty about the unknown image given prior information and subsequent data provided by acquisitions. However, in single-pixel imaging (SPI), the large design space consists of all possible binary illumination patterns. Directly optimizing the Bayesian sOED objective over this space using conventional methods is generally challenging. In this work, we reformulate sOED for SPI as a reinforcement learning (RL) problem. In RL, an agent learns a policy through repeated experimental trials to maximize a reward, here given as the expected information gain, the most complete measure of uncertainty reduction in Bayesian OED. This avoids the need to directly optimize complex information-theoretic objectives during a measurement to obtain the next design. Instead, the result is a learned policy function that directly maps the current posterior belief about the image to the next optimal binary measurement pattern, in a computationally efficient way. Using a simple yet representative SPI imaging scenario, this work evaluates the scalability of the learned policy under two different policy parameterizations. Furthermore, we investigate how performance deteriorates when only approximate posterior information is available or the measurements are impaired by noise, and propose several strategies to introduce additional design constraints motivated by biomedical applications. The numerical experiments demonstrate that the proposed Bayesian sOED-RL framework remains robust and can be extended to more complex, large-scale SPI scenarios.

Introduction

Single-pixel imaging (SPI) emerged as an alternative to conventional array-based imaging systems, with a simplified hardware and operation in wavelengths where detector arrays are expensive or unavailable [1] as their main advantages. Unlike traditional cameras, where an entire scene simultaneously using pixel arrays is captured, here an image is sequentially acquired as follows: A spatial light pattern illuminates the scene and a single photodetector records the total light intensity being reflected back. In practice, the patterns are structured via spatial light modulators or digital micromirror devices, the latter producing binary (on/off) illumination and making binary patterns a natural fit for SPI hardware [2, 3]. A clever choice of sensing patterns and reconstruction routines enable the recovery of images from far fewer measurements than the total number of pixels [4, 5]. Known as compressed sensing (CS), the mathematical framework display which properties should signals have so as to be exactly recovered from undersampled measurements, as long as the sensing patterns satisfy certain conditions.

Early approaches chose patterns for this CS framework in a generic manner, such as random binary projections or scrambled Hadamard matrices [6], which provide reasonable results over a broad range of images. When prior knowledge about the samples that are images is available, one can improve the results by adapting the acquisitions. In an optimal experimental design (OED) framework, a representation of the possible images enables the selection of maximally informative measurement patterns, allowing an efficient image recovery [7].

While conventional OED determines a fixed set of optimal patterns before any measurements are taken, sequential OED (sOED) proposes a way more efficient strategy [8]. Instead of deploying all patterns upfront, sOED adaptively selects each new pattern during the measurement process, based on what has already been learned from previous measurements. As early measurements mainly localize where interesting features lie, later ones focus on details around those specific regions. The ’a-posteriori’ distribution, a Bayesian update of the prior knowledge by each data acquisition, results in a natural state representation that captures all available information.

However, sOED for imaging faces a severe computational challenge. On each step, one must solve an optimization problem over all possible binary measurement patterns to find the one that maximally reduces the uncertainty about the image. For an image with MM pixels, the 2M2^{M} possible binary patterns make conventional per-step optimization computationally prohibitive, even at modest image sizes [9, 10], limiting the practical applicability of sOED to real-world imaging problems.

To overcome this barrier, we pose sOED as a reinforcement learning (RL) problem. In this approach, rather than solving an optimization problem on each step, one looks for a parametric policy mapping the current posterior distribution directly to measurement patterns. Using the expected information gain between prior and posterior image knowledge as the reward signal, policy training consists on selecting the most informative patterns given what has already been observed along experiments. At evaluation time, the policy acts as a pattern proposal function, without needing any further optimization.

In this work, two kinds of policy parameterizations are proposed and their scalability is studied. A categorical policy assigns a probability to each of the 2M2^{M} possible binary patterns, whereas a element-wise Bernoulli policy sets each pixel’s activation probability independently. Under a controlled SPI scenario, where the optimal policy is known and information gain can be computed exactly, their performance is compared after training against random and optimal design strategies. Moreover, the replacement of the posterior distribution as a state variable by a sample-based approximate and the addition of noise to the measurements are analyzed, as well as hard and soft energy constraints over the binary projections are proposed.

The paper is organized as follows. Section 2 presents the elements of single-pixel imaging and its treatment as an inverse problem, how Bayesian optimal experimental design can be applied to it and its close relation to sequential decision making processes. Section 3 bridges this theory using a reinforcement learning abstraction, and illustrates the technical implementation details for the single-pixel imaging case. Section 4 introduces setting considerations for the proposed learning scenario and numerical results obtained in the studies, with a discussion in Section 5 and conclusions in Section 6.

Background

Single-pixel imaging as a Bayesian inverse problem

A single measurement in SPI can be described as

y=⟨h,x⟩+ε,(1)y = \langle h, x \rangle+ \varepsilon, \tag*{(1)}

where x∈RMx \in\mathbb{R}^{M} is the image that one wants to recover in vector form, hN∈{0,1}Mh_{N} \in\{0,1\}^{M} is a discrete binary pattern and ε\varepsilon is a noise realization, usually modeled as a normally distributed uncorrelated random variable N(0,σ2)\mathcal{N}(0,\sigma^{2}) with mean zero and standard deviation σ\sigma. The SPI forward model, collecting NN observations yN∈RNy_{N} \in\mathbb{R}^{N}, is described in a matrix-vector form as

yN=HNx+εN,[y1⋮yN]=[h1T⋮hNT][x1⋮xM]+[ε1⋮εN],(2)\begin{aligned} y_{N} &= H_{N}x + \varepsilon_{N}, \\ \begin{bmatrix} y_{1} \\ \vdots\\ y_{N} \end{bmatrix} &= \begin{bmatrix} h_{1}^{T} \\ \vdots\\ h_{N}^{T} \end{bmatrix} \begin{bmatrix} x_{1} \\ \vdots\\ x_{M} \end{bmatrix} + \begin{bmatrix} \varepsilon_{1} \\ \vdots\\ \varepsilon_{N} \end{bmatrix}, \tag*{(2)} \end{aligned}

with HNH_{N} being the forward operator, which rows consist of discrete binary patterns hnh_{n}. Assuming that the noise from individual measurements is uncorrelated, the noise variable εN\varepsilon_{N} follows a multivariate normal distribution εN∼N(0N,σ2IN)\varepsilon_{N} \sim\mathcal{N}(0_{N},\sigma^{2}I_{N}). When N<MN < M, the inverse problem of recovering x\mathbf{x} from measurements yN\mathbf{y}_N is underdetermined, and usually ill-conditioned, which means that a perturbation in the image x\mathbf{x} does not necessarily lead to an identifiable change in the observed data yN\mathbf{y}_N [11].

To prevent this, assumptions based on ’a-priori’ knowledge of the image can be used to regularize the possible solutions of the problem. In combination with how consistent measurements are with respect to the forward model, an updated, ’a-posteriori’ knowledge of the image is obtained. This methodology can be formalized using the Bayesian framework, where x\mathbf{x} is treated as a random variable rather than merely a real-valued vector, describing the solution to Eq. (2) through the Bayes formula [12]

P(x∣yN;HN)=P(yN∣x;HN)P(x)P(yN;HN)(3)P(\mathbf{x}\mid\mathbf{y}_N;\mathbf{H}_N)=\frac{P(\mathbf{y}_N\mid\mathbf{x};\mathbf{H}_N)P(\mathbf{x})}{P(\mathbf{y}_N;\mathbf{H}_N)} \tag*{(3)}

Here, the prior distribution P(x)P(\mathbf{x}) encompasses the knowledge of the image before measurement takes place and can encode the properties one expects the image to have, e.g., total variation priors were proposed to model images with clearly defined edges between piecewise constant areas [13]. Note that the prior distribution is independent of the measurement operator HN\mathbf{H}_N. The likelihood distribution P(yN∣x;HN)P(\mathbf{y}_N\mid\mathbf{x};\mathbf{H}_N) quantifies the probability of measuring yN\mathbf{y}_N, which contains all measurements up to step NN, given the image x\mathbf{x} using the measurement operator HN\mathbf{H}_N. Here the noise and the unknown x\mathbf{x} are mutually independent variables

P(yN∣x;HN)∝exp⁡(−12σ2∥HNx−yN∥22)=∏n=1Nexp⁡(−(hnTx−yn)22σ2)=:∏n=1NP(yn∣x;hn)(4)\begin{aligned} P(\mathbf{y}_N\mid\mathbf{x};\mathbf{H}_N)&\propto\exp\left(-\frac{1}{2\sigma^2}\left\lVert\mathbf{H}_N\mathbf{x}-\mathbf{y}_N\right\rVert_2^2\right)\\ &=\prod_{n=1}^{N}\exp\left(-\frac{\left(\mathbf{h}_n^{\mathsf{T}}\mathbf{x}-y_n\right)^2}{2\sigma^2}\right)=:\prod_{n=1}^{N}P(y_n\mid\mathbf{x};\mathbf{h}_n) \tag*{(4)} \end{aligned}

The marginal likelihood P(yN;HN)P(\mathbf{y}_N;\mathbf{H}_N), also known as model evidence, is a normalizing factor for all possible images x\mathbf{x} given a particular HN\mathbf{H}_N

P(yN;HN)=EP(x)[P(yN∣x;HN)](5)P(\mathbf{y}_N;\mathbf{H}_N)=\mathbb{E}_{P(\mathbf{x})}\left[P(\mathbf{y}_N\mid\mathbf{x};\mathbf{H}_N)\right] \tag*{(5)}

As a result, the posterior distribution, P(x∣yN;HN)P(\mathbf{x}\mid\mathbf{y}_N;\mathbf{H}_N), combines information from the prior and likelihood distributions, assigning a probability score to each possible image x\mathbf{x} given observed data yN\mathbf{y}_N. Note that due to the sequential, independent SPI measurement process, the posterior distribution is updated during acquisition as

P(x∣yN;HN)∝P(yN∣x;HN)P(x)∝(4)P(yN∣x;hN)P(yN−1∣x;HN−1)P(x)∝P(yN∣x;hN)P(x∣yN−1;HN−1)(6)\begin{aligned} P(\mathbf{x}\mid\mathbf{y}_N;\mathbf{H}_N)&\propto P(\mathbf{y}_N\mid\mathbf{x};\mathbf{H}_N)P(\mathbf{x})\\ &\overset{(4)}{\propto}P(y_N\mid\mathbf{x};\mathbf{h}_N)P(\mathbf{y}_{N-1}\mid\mathbf{x};\mathbf{H}_{N-1})P(\mathbf{x})\\ &\propto P(y_N\mid\mathbf{x};\mathbf{h}_N)P(\mathbf{x}\mid\mathbf{y}_{N-1};\mathbf{H}_{N-1}) \tag*{(6)} \end{aligned}

Bayesian optimal experimental design

The aim in compressed SPI is to obtain a useful reconstruction of x\mathbf{x} using N<MN < M measurements. An important question is which of the 2NM2^{NM} possible experimental designs HN∈{0,1}N×M\mathbf{H}_N \in\{0,1\}^{N \times M} one should choose for a given prior P(x)P(\mathbf{x}). Bayesian OED formalizes this problem: First, it uses the Shannon entropy as an information-theoretic measure to quantify the uncertainty about the random image x\mathbf{x} before the measurement

Ent⁡(x):=−EP(x)[log⁡P(x)](7)\operatorname{Ent}(\mathbf{x}) := -\mathbb{E}_{P(\mathbf{x})}\left[\log P(\mathbf{x})\right] \tag*{(7)}

Then, it assesses how informative a particular collection of measurements yN\mathbf{y}_N acquired with a design HN\mathbf{H}_N is by introducing the information gain IG⁡x(HN,yN)\operatorname{IG}_{\mathbf{x}}(\mathbf{H}_N,\mathbf{y}_N) as a difference between prior and posterior entropy

IG⁡x(yN;HN):=Ent⁡(x)−Ent⁡(x∣yN;HN)(8)\operatorname{IG}_{\mathbf{x}}(\mathbf{y}_N;\mathbf{H}_N) := \operatorname{Ent}(\mathbf{x}) - \operatorname{Ent}(\mathbf{x}\mid\mathbf{y}_N;\mathbf{H}_N) \tag*{(8)}

Using the marginal likelihood in (5) and the information gain in (8), the expected information gain EIG⁡x(H)\operatorname{EIG}_{\mathbf{x}}(\mathbf{H}) provides an average uncertainty change with respect to all possible measurement realizations yN\mathbf{y}_N

EIG⁡x(HN):=EP(yN;HN)[IG⁡x(yN;HN)](9)\operatorname{EIG}_{\mathbf{x}}(\mathbf{H}_N) := \mathbb{E}_{P(\mathbf{y}_N;\mathbf{H}_N)}\left[\operatorname{IG}_{\mathbf{x}}(\mathbf{y}_N;\mathbf{H}_N)\right] \tag*{(9)}

BOED chooses the measurement patterns that maximize the expected information gain

HNPr:=arg⁡max⁡HN∈{0,1}N×M EIG⁡x(HN),(10)\mathbf{H}^{\mathrm{Pr}}_N := \underset{\mathbf{H}_N \in\{0,1\}^{N \times M}}{\arg\max}\ \operatorname{EIG}_{\mathbf{x}}(\mathbf{H}_N), \tag*{(10)}

where HNPr\mathbf{H}^{\mathrm{Pr}}_N refers to an ’a-priori’ design, as this choice only relies on information available before the measurement, i.e., the prior. Now consider the case where one can design pattern sequentially during the measurement, based on the information gathered so far, i.e., ’a-posteriori’ (sequential Optimal Experimental Design, sOED). In Bayesian sOED, consider the expected information gain of using pattern hn\mathbf{h}_n for the nthn^{\mathrm{th}} measurement step given that the previous yn−1\mathbf{y}_{n-1} measurements were acquired with the design Hn−1\mathbf{H}_{n-1}

EIG⁡x∣yn−1;Hn−1(hn):=EP(yn;hn)[IG⁡x∣yn−1;Hn−1(yn;hn)],(11)\operatorname{EIG}_{\mathbf{x}\mid\mathbf{y}_{n-1};\mathbf{H}_{n-1}}(\mathbf{h}_n) := \mathbb{E}_{P(\mathbf{y}_n;\mathbf{h}_n)}\left[\operatorname{IG}_{\mathbf{x}\mid\mathbf{y}_{n-1};\mathbf{H}_{n-1}}(\mathbf{y}_n;\mathbf{h}_n)\right], \tag*{(11)}

where the marginal likelihood can be expressed as an expectation over the new measurement yn\mathbf{y}_n likelihood using the the posterior until step n−1n-1

P(yn;hn):=EP(x∣yn−1;Hn−1)[P(yn∣x;hn)],P(x∣y0;H0):=P(x)(12)P(\mathbf{y}_n;\mathbf{h}_n) := \mathbb{E}_{P(\mathbf{x}\mid\mathbf{y}_{n-1};\mathbf{H}_{n-1})}\left[P(\mathbf{y}_n\mid\mathbf{x};\mathbf{h}_n)\right], \qquad P(\mathbf{x}\mid\mathbf{y}_0;\mathbf{H}_0) := P(\mathbf{x}) \tag*{(12)}

Note that the prior P(x)P(\mathbf{x}) in (8), (5) was merely exchanged for the posterior P(x∣yn−1;Hn−1)P(\mathbf{x}\mid\mathbf{y}_{n-1};\mathbf{H}_{n-1}) in (11), (12) and only consider the design of the following pattern hn\mathbf{h}_n. The greedy sOED HNGr(x)\mathbf{H}^{\mathrm{Gr}}_N(\mathbf{x}), which compared to (10) depends on the unknown realization x\mathbf{x}, maximizes (11) in every step

hnGr(x):=arg max⁡hn∈{0,1}M EIG⁡x∣yn−1;Hn−1(hn),HNGr=[h1Gr,…,hNGr]T(13)\mathbf{h}^{\mathrm{Gr}}_n(\mathbf{x}) := \underset{\mathbf{h}_n\in\{0,1\}^M}{\operatorname{arg\,max}}\ \operatorname{EIG}_{\mathbf{x}\mid\mathbf{y}_{n-1};\mathbf{H}_{n-1}}(\mathbf{h}_n),\qquad\mathbf{H}^{\mathrm{Gr}}_N = [\mathbf{h}^{\mathrm{Gr}}_1,\ldots,\mathbf{h}^{\mathrm{Gr}}_N]^{T} \tag*{(13)}

Greedy designs reduce the chain of sequential design decisions that all influence the informativeness of the final NN-step design HN\mathbf{H}_N into a sequence of isolated design decision steps. Although this is often computationally attractive, it is a suboptimal solution to the problem of maximizing the sum of expected information gains over all NN steps [14] [15]. As an alternative, the full sequential OED HNF(x)\mathbf{H}^{\mathrm{F}}_N(\mathbf{x}) is the globally optimal sequence of measurement patterns h1,…,hN\mathbf{h}_1,\ldots,\mathbf{h}_N that maximizes the cumulative expected information gain across all NN steps

HNF(x):=arg max⁡h1,…,hN∈{0,1}M ∑n=1NEIG⁡x∣yn−1;Hn−1(hn)(14)\mathbf{H}^{\mathrm{F}}_N(\mathbf{x}) := \underset{\mathbf{h}_1,\ldots,\mathbf{h}_N\in\{0,1\}^M}{\operatorname{arg\,max}}\ \sum_{n=1}^{N}\operatorname{EIG}_{\mathbf{x}\mid\mathbf{y}_{n-1};\mathbf{H}_{n-1}}(\mathbf{h}_n) \tag*{(14)}

Unlike greedy approaches that optimize each measurement independently, this formulation accounts for the long-term information-gathering strategy by jointly optimizing over the entire measurement sequence. The key insight is that early measurement choices influence the posterior distributions encountered in subsequent steps, creating a complex interdependence between decisions that greedy methods fail to capture.

The theoretical superiority of the full sOED strategy with respect to the greedy approach, introduced in [8], is formalized as

∑n=1NEIG⁡x∣yn−1;Hn−1F(hnF)≥∑n=1NEIG⁡x∣yn−1;Hn−1Gr(hnGr)(15)\sum_{n=1}^{N}\operatorname{EIG}_{\mathbf{x}\mid\mathbf{y}_{n-1};\mathbf{H}^{\mathrm{F}}_{n-1}}\left(\mathbf{h}^{\mathrm{F}}_n\right) \ge\sum_{n=1}^{N}\operatorname{EIG}_{\mathbf{x}\mid\mathbf{y}_{n-1};\mathbf{H}^{\mathrm{Gr}}_{n-1}}\left(\mathbf{h}^{\mathrm{Gr}}_n\right) \tag*{(15)}
∑n=1NEIG⁡x∣yn−1;Hn−1F(hnF)≥EIG⁡x(HNPr)(16)\sum_{n=1}^{N}\operatorname{EIG}_{\mathbf{x}\mid\mathbf{y}_{n-1};\mathbf{H}^{\mathrm{F}}_{n-1}}\left(\mathbf{h}^{\mathrm{F}}_n\right) \ge\operatorname{EIG}_{\mathbf{x}}\left(\mathbf{H}^{\mathrm{Pr}}_N\right) \tag*{(16)}

Inequality (15) states that the cumulative information gain achieved by the full sOED sequence HNF\mathbf{H}^{\mathrm{F}}_N is at least as large as that obtained by the greedy approach HNGr\mathbf{H}^{\mathrm{Gr}}_N, which myopically maximizes the EIG at each individual step without considering future measurements. This gap arises because the full sOED can strategically select measurements that may have lower immediate information gain but enable more informative measurements in subsequent steps—a form of temporal credit assignment that greedy methods cannot perform. Inequality (16) further shows that the full sOED outperforms the a-priori design HNPrH^{\mathrm{Pr}}_{N}, which selects all NN patterns before observing any data and thus cannot adapt to the information revealed by early measurements. Together, these inequalities establish the full sOED as the optimal approach within the Bayesian framework, motivating the development of tractable approximation methods such as the RL-based approach presented in this paper.

Computational considerations. While the full sOED (14) is theoretically superior to the a-priori OED design (10), it comes with similar computational challenges: Taking the interdependence of the terms of the sum in (14) into account means that one also needs to optimize over the exponentially large space {0,1}NM\{0,1\}^{NM} of binary patterns. Even for modest image sizes (e.g., M=64M = 64), the design space contains 264≈10192^{64} \approx10^{19} possible patterns at each step, making exhaustive search or traditional optimization methods computationally prohibitive.

Moreover, computing the expected information gain EIG⁡x∣yn−1;Hn−1(hn)\operatorname{EIG}_{x\mid y_{n-1};H_{n-1}}(h_n) in (11) requires integrating over all possible measurement outcomes through the marginal likelihood P(yn;hn)P(y_n;h_n) in (12), which itself involves an expectation over the posterior distribution from previous measurements. In practice, these integrations are often high-dimensional and lack closed-form expressions, relying on Monte Carlo-based approximations (see Section 3.4). This computation is repeated at each of the NN sequential steps, making the full sOED approach (14) challenging for real-time or resource-constrained imaging applications.

These computational challenges motivate the reformulation of the sOED problem as a sequential decision-making under uncertainty, and the introduction of RL-based approaches as a principled way to tackle the sOED problem (14). Instead of solving the complex optimization problems behind sOED, one learns parametric policies that directly map posterior distributions to near-optimal measurement patterns by simulating a large number of sOED instances offline, i.e. before any real measurement takes place. The complex integrals only have to be computed during this offline training phase, not during deployment.

Sequential decision-making processes

Solving the sOED problem (14) can be seen as an instance of a Markov decision process (MDP), a mathematical model to describe sequential and time-discrete decision processes under uncertainty. In MDPs, an agent interacts dynamically with an environment in order to accumulate a reward. The state S∈SS \in\mathcal{S} represents the current configuration of the agent and environment, and every possible decision the agent can make is represented by an action A∈AA \in\mathcal{A} that will lead to a new state S′S' with a certain probability. The agent’s decision making is represented by a policy π(S,A)\pi(S,A), which is a probability distribution in AA, encoding the likelihood that the agent takes the decision AA when in state SS. For every decision, the agent receives a reward R(S,A,S′)R(S, A, S'). The goal in MDP is to find policies that maximize the accumulated rewards for a given time window. Very often, the agent does not have full knowledge of the state SS, leading to a partially observable Markov decision process (POMDP) [16].

Several methodologies describe how to find a policy that maximizes the sum of rewards from the starting time n=1n = 1 until the end of the experiment. In complex scenarios, obtaining analytical solutions for this objective or using conventional optimization schemes is often infeasible. Instead, RL methods achieve the optimization goal by updating the agent’s policy based on the feedback provided by the environment at each time. The learning process accounts not only for exploiting stepwise successful actions, but also by exploring the landscape of possible environment states [16]. Two principal classes of RL algorithms exist: value-based methods and direct policy search methods. In value-based methods, the agent learns a scalar estimate of the expected cumulative rewards from a given state onward under a fixed policy, called the value function, from repeated environment interactions and derives a policy by acting greedily with respect to this function. In direct policy search, the policy is explicitly represented by a parametric function πθ\pi_\theta and θ\theta is updated directly via stochastic gradient ascent, i.e. policy gradient methods [17, 18].

In the sOED setting, however, the practical benefits of RL are only realized when the agent can be trained on a sufficiently large number of simulated trajectories, so that it can repeatedly experience the stochastic transitions S→S′S \to S' and the associated rewards R(S,A,S′)R(S, A, S') under many different policies. If the available simulation budget is too small, the agent cannot adequately explore the joint state-action space, and the resulting policy is prone to overfitting to a few sampled scenarios rather than approximating the truly optimal policy of the underlying MDP/POMDP, which substantially limits the applicability of RL-based sOED in practice.

Methods

SPI-sOED reinforcement learning framework

In this section, the sequential optimal experimental design problem for single-pixel imaging defined in Eq. (14) is formulated as a RL task and a methodology to solve it is developed. The key insight is that one must choose a complete sequence of patterns that jointly maximizes the cumulative information gain about the unknown image over the entire experiment, accounting for how each measurement shapes future decisions.

The Bayesian framework from Sec. 2.1 induces this sequential structure: At any measurement step nn, the current knowledge about the image x\mathbf{x} is summarized by the posterior distribution P(x∣yn−1;Hn−1)P(\mathbf{x}\mid\mathbf{y}_{n-1};\mathbf{H}_{n-1}), where yn−1\mathbf{y}_{n-1} contains all measurements up to step n−1n-1 and Hn−1\mathbf{H}_{n-1} contains all previously used patterns. This reduces to the prior P(x)P(\mathbf{x}) when n=0n=0 (see Equation (5)). When a new pattern hn\mathbf{h}_n is chosen and measurement yny_n is obtained, Bayes’ rule gives us an updated posterior P(x∣yn;Hn)P(\mathbf{x}\mid\mathbf{y}_n;\mathbf{H}_n), which represents the refined knowledge. This sequential updating process can be outlined using the following decision-theoretic components:

  • State: The posterior distribution P(x∣yn−1;Hn−1)P(\mathbf{x}\mid\mathbf{y}_{n-1};\mathbf{H}_{n-1}) after measurement n−1n-1 serves as the current state SnS_n. This state encodes all information from past measurements and patterns that is relevant for future decisions.

  • Action: The binary measurement pattern hn\mathbf{h}_n used at step nn becomes the action AnA_n. It leads to the scalar measurement yn=hnTx+ϵny_n=\mathbf{h}_n^{\mathsf{T}}\mathbf{x}+\epsilon_n.

  • State transition: The new posterior distribution P(x∣yn;Hn)P(\mathbf{x}\mid\mathbf{y}_n;\mathbf{H}_n) after measurement nn becomes the next state Sn+1S_{n+1}. This transition is stochastic because the measurement yny_n is corrupted by noise ϵn\epsilon_n, but the Bayesian update rule provides the deterministic mapping from (Sn,An,yn)(S_n,A_n,y_n) to Sn+1S_{n+1}.

  • Reward: The immediate reward for choosing pattern hn\mathbf{h}_n in state SnS_n is the expected information gain Rn:=EIG⁡x∣yn−1;Hn−1(hn)R_n := \operatorname{EIG}_{\mathbf{x}\mid\mathbf{y}_{n-1};\mathbf{H}_{n-1}}(\mathbf{h}_n), defined in Eq. (9). Note that this reward depends only on the current state SnS_n and action AnA_n, and not on the resulting new state Sn+1S_{n+1}. Computing EIG requires marginalizing over all possible outcomes yny_n, which is only tractable in small-scale settings as the one showcased on this work. For more complex settings, RnR_n can be approximated by the actual information gain IG⁡x∣yn−1;Hn−1(hn,yn)\operatorname{IG}_{\mathbf{x}\mid\mathbf{y}_{n-1};\mathbf{H}_{n-1}}(\mathbf{h}_n,y_n), which depends on the observed yny_n and thereby Sn+1S_{n+1}, cf. § 3.4.

  • Episode: A complete sequence of NN measurement steps forms an episode or trajectory τ:=(S0,A1,S1,R1,…,SN−1,AN,SN,RN)\tau:= (S_0,A_1,S_1,R_1,\ldots,S_{N-1},A_N,S_N,R_N), representing one complete imaging experiment from prior to final posterior.

  • Policy: The stochastic policy π(An∣Sn)\pi(A_n\mid S_n) maps the current state, the posterior distribution over x\mathbf{x}, to a probability distribution over binary measurement patterns.

With these definitions, the sequential optimal design problem of maximizing the cumulative expected information gain from Eq. (14) can be recast as finding an optimal policy π∗\pi^* that maximizes the total reward

π∗:=arg max⁡π Eτ∼P(τ∣π)[∑n=1NRn]=arg max⁡π Eτ∼P(τ∣π)[∑n=1NEIG⁡x∣yn−1;Hn−1(hn)]=:arg max⁡π J(π)(17)\begin{aligned} \pi^* &:= \underset{\pi}{\operatorname{arg\,max}}\ \mathbb{E}_{\tau\sim P(\tau\mid\pi)}\left[\sum_{n=1}^{N} R_n\right]\\ &= \underset{\pi}{\operatorname{arg\,max}}\ \mathbb{E}_{\tau\sim P(\tau\mid\pi)}\left[\sum_{n=1}^{N} \operatorname{EIG}_{\mathbf{x}\mid y_{n-1};\mathcal{H}_{n-1}}(\mathbf{h}_n)\right]\\ &=: \underset{\pi}{\operatorname{arg\,max}}\ J(\pi) \tag*{(17)} \end{aligned}

where J(π)J(\pi) is defined as the objective function for the policy training. The expectation in (17) over the distribution of trajectories τ∼P(τ∣π)\tau\sim P(\tau\mid\pi) integrates the stochasticity of the sample, policy and the observation noise. A trajectory can be generated as follows: a sample is drawn from the prior x∼P(x)\mathbf{x} \sim P(\mathbf{x}), set S0=P(x)S_0 = P(\mathbf{x}) and iterate for n=1,…,Nn = 1, \ldots, N

An=hn∼π(⋅∣Sn−1),yn∼P(yn∣x;hn),Sn=P(x∣yn;Hn−1)∝P(yn∣x;hn)P(x∣yn−1;Hn−1),Rn=EIG⁡x∣yn−1;Hn−1(hn)=EP(yn;hn)[IG⁡x∣yn−1;Hn−1(yn;hn)].(18)\begin{aligned} A_n &= \mathbf{h}_n \sim\pi(\cdot\mid S_{n-1}),\\ y_n &\sim P(y_n \mid\mathbf{x}; \mathbf{h}_n),\\ S_n &= P(\mathbf{x} \mid y_n; \mathcal{H}_{n-1}) \propto P(y_n \mid\mathbf{x}; \mathbf{h}_n)P(\mathbf{x} \mid y_{n-1}; \mathcal{H}_{n-1}),\\ R_n &= \operatorname{EIG}_{\mathbf{x}\mid y_{n-1};\mathcal{H}_{n-1}}(\mathbf{h}_n) = \mathbb{E}_{P(y_n;\mathbf{h}_n)}\left[\operatorname{IG}_{\mathbf{x}\mid y_{n-1};\mathcal{H}_{n-1}}(y_n;\mathbf{h}_n)\right]. \tag*{(18)} \end{aligned}

The remaining question is how to represent and optimize the policy π\pi in practice. The sOED state space consists of full posterior distributions over x\mathbf{x}, which are high-dimensional and continuous, making value function estimation intractable. This motivates the use of direct policy search methods, where π\pi is parameterized by a differentiable function πθ\pi_\theta and optimized end-to-end via policy gradient ascent on J(θ)J(\theta). In the following sections, the specific parameterizations of π\pi adopted for SPI-sOED and the resulting learning algorithm are detailed.

SPI design policy parameterizations

The fundamental question is how to represent this distribution over binary patterns. To construct a policy from the current posterior state, we use a feed-forward neural network that maps a state representation Sn−1S_{n-1} to a probability distribution over binary measurement patterns. We consider two policy parameterizations that share the same multilayer perceptron (MLP) backbone but differ in their output dimension and final activation function. The MLP follow the model

fθ:=wθL−1∘σL−1∘…∘σ1∘wθ1,(19)f_\theta:= w_{\theta_{L-1}} \circ\sigma_{L-1} \circ\ldots\circ\sigma_1 \circ w_{\theta_1}, \tag*{(19)}

where ww denotes an affine linear mapping with parameters θl\theta_l, σl(⋅)\sigma_l(\cdot) a non-linear activation function and ∘\circ the composition of functions such that (f∘g)(⋅)=f(g(⋅))(f \circ g)(\cdot) = f(g(\cdot)), with l=1,…,L−1l = 1, \ldots, L - 1 the layer index.

A categorical policy πθC\pi_\theta^C directly outputs a probability distribution over all 2M2^M possible patterns. With an output layer of 2M2^M units and a Softmax activation, the probability for a possible pattern hi\mathbf{h}_i given a particular past state Sn−1S_{n-1} is

πθC(hi∣Sn−1)=efθ(Sn−1)i∑jefθ(Sn−1)j,(20)\pi_\theta^C(\mathbf{h}_i \mid S_{n-1}) = \frac{e^{f_\theta(S_{n-1})_i}}{\sum_j e^{f_\theta(S_{n-1})_j}}, \tag*{(20)}

where j=1,…,2Mj = 1, \ldots, 2^M indexes all possible patterns. This approach allows the network to learn arbitrary correlations between pixel activations. However, it scales exponentially in MM, making it practical only for very small images.

Alternatively, each pixel activation can be modeled independently: with MM output units and a sigmoid activation σL\sigma_L, we obtain independent Bernoulli probabilities for the activation of each pixel hkh_k, k=1,…,Mk = 1, \ldots, M of a pattern h\mathbf{h}

πθB(hk=1∣Sn−1)=Bernoulli⁡(11+e−fθ(Sn−1)k).(21)\pi_\theta^B(h_k = 1 \mid S_{n-1}) = \operatorname{Bernoulli}\left(\frac{1}{1 + e^{-f_\theta(S_{n-1})_k}}\right). \tag*{(21)}

To sample a pattern, independent draws are sampled for each pixel from its corresponding Bernoulli distribution. This scales linearly in MM, making it practical for larger images. Although the network still can learn to modulate individual probabilities based on the full posterior state, it cannot model correlations between pixel activations directly. With these state representations in place, the architecture of the input layer is determined by how the state is encoded, since the model processes that representation in a compatible form as input.

State representation and approximation

The RL framework from § 3.1 requires at each step nn a state SnS_n merging the current knowledge about the image and a reward RnR_n quantifying information gained. In the SPI-sOED setting, both quantities are obtained via the Bayesian posterior

P(x∣yn−1;Hn−1),(22)P(\mathbf{x} \mid\mathbf{y}_{n-1}; \mathbf{H}_{n-1}), \tag*{(22)}

but their computation depends on whether this posterior is available in closed-form or only implicitly through sampling. As we introduced in § 2.2.1, using a stochastic variable as a state corresponds to a partially observable Markov decision process.

Explicit posterior (closed-form state). When the posterior admits a closed-form expression, the full distribution reads

Sn=P(x∣yn−1;Hn−1)(23)S_n = P(\mathbf{x} \mid\mathbf{y}_{n-1}; \mathbf{H}_{n-1}) \tag*{(23)}

as the state.

A typical example is a linear Gaussian model with prior x∼N(μ0,Σ0)\mathbf{x} \sim\mathcal{N}(\boldsymbol{\mu}_0,\boldsymbol{\Sigma}_0) and likelihood

yn∣x,hn∼N(hnTx,σ2),(24)y_n \mid\mathbf{x},\mathbf{h}_n \sim\mathcal{N}(\mathbf{h}_n^{\mathsf{T}}\mathbf{x},\sigma^2), \tag*{(24)}

for which the posterior is again Gaussian,

Σn=(Σn−1−1+σ−2hnhnT)−1,(25)\boldsymbol{\Sigma}_n = \left(\boldsymbol{\Sigma}_{n-1}^{-1}+\sigma^{-2}\mathbf{h}_n\mathbf{h}_n^{\mathsf{T}}\right)^{-1}, \tag*{(25)}
μn=Σn(Σn−1−1μn−1+σ−2ynhn),(26)\boldsymbol{\mu}_n = \boldsymbol{\Sigma}_n\left(\boldsymbol{\Sigma}_{n-1}^{-1}\boldsymbol{\mu}_{n-1}+\sigma^{-2}y_n\mathbf{h}_n\right), \tag*{(26)}

with (μn,Σn)(\boldsymbol{\mu}_n,\boldsymbol{\Sigma}_n) fully characterising SnS_n. In this regime, both the state and the expected information gain can be evaluated exactly, providing a high-fidelity learning signal.

Implicit posterior (sample-based state). For more complex priors or likelihoods, the posterior P(x∣yn−1,Hn−1)P(\mathbf{x}\mid y_{n-1},\mathbf{H}_{n-1}) is not available in closed-form and must instead be represented implicitly via a collection of posterior samples {x(i)}i=1Ns∼P(x∣yn−1,Hn−1)\{\mathbf{x}^{(i)}\}_{i=1}^{N_s} \sim P(\mathbf{x}\mid y_{n-1},\mathbf{H}_{n-1}). Among several possible ways to encode this implicit representation as a state, two natural choices are considered. The first is to pass a set of samples directly to the policy,

S^n={x(i)}i=1Ns,(27)\hat{S}_n = \left\{\mathbf{x}^{(i)}\right\}_{i=1}^{N_s}, \tag*{(27)}

which preserves the complete distributional information at the cost of a fixed-size input of dimension Ns×MN_s \times M. The second, computationally cheaper alternative is to summarize the set by its empirical mean and covariance analogues of the sufficient statistics (μn,Σn)(\boldsymbol{\mu}_n,\boldsymbol{\Sigma}_n) from the closed-form Gaussian case in Eqs. (26), (25)

μ^n:=1Ns∑i=1Nsx(i),(28)\hat{\boldsymbol{\mu}}_n := \frac{1}{N_s}\sum_{i=1}^{N_s}\mathbf{x}^{(i)}, \tag*{(28)}
Σ^n:=1Ns−1∑i=1Ns(x(i)−μ^n)(x(i)−μ^n)T.(29)\hat{\boldsymbol{\Sigma}}_n := \frac{1}{N_s-1}\sum_{i=1}^{N_s}\left(\mathbf{x}^{(i)}-\hat{\boldsymbol{\mu}}_n\right)\left(\mathbf{x}^{(i)}-\hat{\boldsymbol{\mu}}_n\right)^{\mathsf{T}}. \tag*{(29)}

which compresses the posterior into a low-dimensional pair of Gaussian moments that can be processed efficiently by the policy network. We study the numerical impact of the sample size NsN_s on policy learning in § 4 for the one-active-pixel prior.

Reward computation and approximation: Noiseless and noisy cases

The EIG reward Rn=EIG⁡x∣yn−1;Hn−1(hn)R_n = \operatorname{EIG}_{\mathbf{x}\mid y_{n-1};\mathbf{H}_{n-1}}(\mathbf{h}_n) is evaluated at each step of every training trajectory. Computing it exactly requires integrating over both the unknown image x\mathbf{x} and the future observation yny_n, which is tractable only under specific model assumptions. Two variants appear as the posterior P(x∣yn−1;Hn−1)P(\mathbf{x}\mid y_{n-1};\mathbf{H}_{n-1}) and the likelihood P(yn∣x;hn)P(y_n\mid\mathbf{x};\mathbf{h}_n) admit closed-form representations or not.

For a general prior P(x)P(\mathbf{x}) and likelihood P(yn∣x;hn)P(y_n\mid\mathbf{x};\mathbf{h}_n), the exact reward is

Rn=EIG⁡x∣yn−1;Hn−1(hn)=Ent⁡[P(x∣yn−1;Hn−1)]−Eyn[Ent⁡[P(x∣yn,yn−1;Hn)]],(30)R_n = \operatorname{EIG}_{\mathbf{x}\mid y_{n-1};\mathbf{H}_{n-1}}(\mathbf{h}_n) = \operatorname{Ent}\left[P(\mathbf{x}\mid y_{n-1};\mathbf{H}_{n-1})\right] - \mathbb{E}_{y_n}\left[\operatorname{Ent}\left[P(\mathbf{x}\mid y_n,y_{n-1};\mathbf{H}_n)\right]\right], \tag*{(30)}

where the expectation is over the marginal P(yn;hn)=∫XP(yn∣x;hn)P(x∣yn−1;Hn−1) dxP(y_n;\mathbf{h}_n) = \int_{\mathcal{X}} P(y_n\mid\mathbf{x};\mathbf{h}_n)P(\mathbf{x}\mid y_{n-1};\mathbf{H}_{n-1})\,\mathrm{d}x. In general, evaluating (30) exactly is intractable because it requires integrating over both x\mathbf{x} and yny_n.

Given NrN_r posterior samples {x(i)}i=1Nr∼P(x∣yn−1;Hn−1)\{\mathbf{x}^{(i)}\}_{i=1}^{N_r}\sim P(\mathbf{x}\mid y_{n-1};\mathbf{H}_{n-1}) one can approximate the marginal as

P^(yn;hn)=1Nr∑i=1NrP(yn∣x(i);hn).(31)\widehat{P}(y_n;\mathbf{h}_n) = \frac{1}{N_r}\sum_{i=1}^{N_r}P(y_n\mid\mathbf{x}^{(i)};\mathbf{h}_n). \tag*{(31)}

To estimate the EIG itself, a single held-out sample is drawn x(0)∼P(x∣yn−1;Hn−1)\mathbf{x}^{(0)}\sim P(\mathbf{x}\mid y_{n-1};\mathbf{H}_{n-1}), independent of the NrN_r denominator particles, and use the nested Monte Carlo (NMC) estimator [19]

R^n=∫YP(yn∣x(0);hn)[log⁡P(yn∣x(0);hn)−log⁡P^(yn;hn)]dyn.(32)\widehat{R}_n = \int_{\mathcal{Y}}P(y_n\mid\mathbf{x}^{(0)};\mathbf{h}_n)\left[\log P(y_n\mid\mathbf{x}^{(0)};\mathbf{h}_n)-\log\widehat{P}(y_n;\mathbf{h}_n)\right]\mathrm{d}y_n. \tag*{(32)}

The one-dimensional integral in (32) is over the observation space Y\mathcal{Y} and can be evaluated analytically or by quadrature depending on the form of the likelihood.

In in absence of noise, i.e. yn=⟨hn,x⟩y_n = \langle\mathbf{h}_n,\mathbf{x}\rangle, the likelihood collapses to a point mass P(yn∣x;hn)=δ(yn−⟨hn,x⟩)P(y_n\mid\mathbf{x};\mathbf{h}_n)=\delta(y_n-\langle\mathbf{h}_n,\mathbf{x}\rangle). In this case the integral over Y\mathcal{Y} in (32) vanishes and the estimator reduces to a purely discrete sum over particles. Concretely, the forward model is evaluated at the held-out sample to obtain yn(0)=⟨hn,x(0)⟩y_n^{(0)}=\langle\mathbf{h}_n,\mathbf{x}^{(0)}\rangle and at each denominator particle to form the empirical marginal likelihood. Because yny_n takes values in a finite set {⟨hn,x(i)⟩}i=0Nr\{\langle\mathbf{h}_n,\mathbf{x}^{(i)}\rangle\}_{i=0}^{N_r}, the EIG estimate becomes

R^nnoiseless=−log⁡(1Nr∑i=1Nr1[⟨hn,x(i)⟩=⟨hn,x(0)⟩]),(33)\widehat{R}^{\mathrm{noiseless}}_n = -\log\left(\frac{1}{N_r}\sum_{i=1}^{N_r}\mathbf{1}\left[\langle\mathbf{h}_n,\mathbf{x}^{(i)}\rangle=\langle\mathbf{h}_n,\mathbf{x}^{(0)}\rangle\right]\right), \tag*{(33)}

where 1[⋅]\mathbf{1}[\cdot] is the indicator function. This is simply the negative log of the fraction of particles that predict the same observation as the held-out sample.

When the observation is corrupted by additive uncorrelated Gaussian noise, yn=⟨hn,x⟩+ϵy_n=\langle\mathbf{h}_n,\mathbf{x}\rangle+\epsilon with ϵ∼N(0,σ2)\epsilon\sim\mathcal{N}(0,\sigma^2), the likelihood becomes P(yn∣x;hn)=N(⟨hn,x⟩,σ2)P(y_n\mid\mathbf{x};\mathbf{h}_n)=\mathcal{N}(\langle\mathbf{h}_n,\mathbf{x}\rangle,\sigma^2). One can approximate it via numerical quadrature on a uniform grid {yk}k=1K\{y_k\}_{k=1}^{K} spanning [min⁡i⟨hn,x(i)⟩−5σ,max⁡i⟨hn,x(i)⟩+5σ][\min_i\langle\mathbf{h}_n,\mathbf{x}^{(i)}\rangle-5\sigma,\max_i\langle\mathbf{h}_n,\mathbf{x}^{(i)}\rangle+5\sigma] with spacing Δy\Delta y. This quadrature scheme can be viewed as a deterministic variant of the nested Monte Carlo (NMC) estimator of [20], where the outer expectation over observations is resolved on a fixed grid rather than by sampling; this removes the variance of the outer loop at the cost of scaling linearly with the grid resolution KK, which remains tractable when the observation space is one-dimensional.

At each grid point yky_k, define the weighted likelihood as

qik=P(x(i)∣yn−1;Hn−1)⋅N(yk;⟨hn,x(i)⟩,σ2),(34)q_{ik}=P(\mathbf{x}^{(i)}\mid y_{n-1};\mathbf{H}_{n-1})\cdot\mathcal{N}(y_k;\langle\mathbf{h}_n,\mathbf{x}^{(i)}\rangle,\sigma^2), \tag*{(34)}

the marginal mk=∑i=1Nxqikm_k=\sum_{i=1}^{N_x}q_{ik}, and the posterior weights:

P(x(i)∣yk,yn−1;Hn−1,hn)=qikmk,(35)P(\mathbf{x}^{(i)}\mid y_k,y_{n-1};\mathbf{H}_{n-1},\mathbf{h}_n)=\frac{q_{ik}}{m_k}, \tag*{(35)}

where ⋅\cdot is the scalar product.

The expected posterior entropy is obtained by averaging the entropy of the posterior distribution over the observation space via quadrature:

E^yn[Ent⁡[P(x∣yn,yn−1;Hn)]]=∑k=1KmkEnt⁡[P(x∣yk,yn−1;Hn−1,hn)]Δy,(36)\widehat{\mathbb{E}}_{y_n}\left[\operatorname{Ent}\left[P(\mathbf{x}\mid y_n,y_{n-1};\mathbf{H}_n)\right]\right]=\sum_{k=1}^{K}m_k\operatorname{Ent}\left[P(\mathbf{x}\mid y_k,y_{n-1};\mathbf{H}_{n-1},\mathbf{h}_n)\right]\Delta y, \tag*{(36)}

where mkm_k acts as the quadrature weight, combining the marginal density with the grid spacing. The expected information gain is then

R^nGaussian=Ent⁡[P(x∣yn−1;Hn−1)]−E^yn[Ent⁡[P(x∣yn,yn−1;Hn)]].(37)\widehat{R}^{\mathrm{Gaussian}}_n=\operatorname{Ent}\left[P(\mathbf{x}\mid y_{n-1};\mathbf{H}_{n-1})\right]-\widehat{\mathbb{E}}_{y_n}\left[\operatorname{Ent}\left[P(\mathbf{x}\mid y_n,y_{n-1};\mathbf{H}_n)\right]\right]. \tag*{(37)}

Gaussian Prior case: D-Optimal Sequential Criterion

When the prior is Gaussian, P(x)=N(x;μ0,Σ0)P(\mathbf{x}) = \mathcal{N}(\mathbf{x}; \boldsymbol{\mu}_0, \boldsymbol{\Sigma}_0), the forward operator is linear and the noise is Gaussian, the posterior remains Gaussian at every step nn with mean μn−1\boldsymbol{\mu}_{n-1} and covariance Σn−1\boldsymbol{\Sigma}_{n-1} updated via the recursions in (25) and (26). In this setting, the model evidence for the nn-th observation is itself Gaussian, P(yn∣hn)=N(yn;hnTμn−1,sn2)P(y_n \mid\mathbf{h}_n) = \mathcal{N}(y_n; \mathbf{h}_n^{\mathsf{T}}\boldsymbol{\mu}_{n-1}, s_n^2), with predictive variance sn2=hnTΣn−1hn+σ2s_n^2 = \mathbf{h}_n^{\mathsf{T}}\boldsymbol{\Sigma}_{n-1}\mathbf{h}_n + \sigma^2. Since the posterior covariance Σn\boldsymbol{\Sigma}_n is independent of the observed value yny_n, averaging over the model evidence via Gaussian moment identities yields a closed-form reward

RnGauss(hn)=12log⁡det⁡(I+σ−2Σn−11/2hnhnTΣn−11/2)=12log⁡(1+hnTΣn−1hnσ2)(38)R_n^{\mathrm{Gauss}}(\mathbf{h}_n) = \frac{1}{2}\log\det\left(\mathbf{I}+\sigma^{-2}\boldsymbol{\Sigma}_{n-1}^{1/2}\mathbf{h}_n\mathbf{h}_n^{\mathsf{T}}\boldsymbol{\Sigma}_{n-1}^{1/2}\right) = \frac{1}{2}\log\left(1+\frac{\mathbf{h}_n^{\mathsf{T}}\boldsymbol{\Sigma}_{n-1}\mathbf{h}_n}{\sigma^2}\right) \tag*{(38)}

where the second equality follows from the matrix determinant lemma [21]. This is precisely the D-optimal design criterion evaluated sequentially [22, 23]; the optimal pattern hn∗\mathbf{h}_n^* at step nn maximises the predictive variance hnTΣn−1hn\mathbf{h}_n^{\mathsf{T}}\boldsymbol{\Sigma}_{n-1}\mathbf{h}_n of the current Gaussian belief, automatically adapting to all past measurements through the running covariance Σn−1\boldsymbol{\Sigma}_{n-1}. The total EIG over NN steps then decomposes additively as

EIG⁡(HN)=∑n=1NRnGauss(hn)=12log⁡det⁡(I+σ−2Σ01/2HNTHNΣ01/2)(39)\operatorname{EIG}(\mathbf{H}_N) = \sum_{n=1}^{N} R_n^{\mathrm{Gauss}}(\mathbf{h}_n) = \frac{1}{2}\log\det\left(\mathbf{I}+\sigma^{-2}\boldsymbol{\Sigma}_0^{1/2}\mathbf{H}_N^{\mathsf{T}}\mathbf{H}_N\boldsymbol{\Sigma}_0^{1/2}\right) \tag*{(39)}

recovering the standard Bayesian D-optimality objective [22]. Unlike the general estimator in (37), (39) requires no outer quadrature over Y\mathcal{Y} and no posterior samples, only the covariance update, making it exact and computationally negligible.

Pattern design including budget constraints

The framework developed so far optimises binary measurement patterns to maximise information gain, without placing any other explicit restrictions on them. In many real-world applications, however, sensing is not free: every observation consumes a physical resource, as time, energy, or radiation dose, whose accumulation can harm the very system being studied. In live-cell fluorescence microscopy, for instance, cumulative light exposure directly damages the cells through multiple photodamage mechanisms [24, 25]: excitation light generates reactive oxygen species that degrade cellular components, induces phototoxic stress responses altering gene expression, and accelerates photobleaching of fluorophores. The severity of these effects scales with total photon dose, which is simply the ℓ1,1\ell_{1,1} norm of HN\mathbf{H}_N. The dose-constrained sequential optimal experimental design for single-pixel imaging is defined as

πB∗:=arg⁡max⁡π J(π)s.t.∑i,j(HN)i,j≤B\pi_B^* := \underset{\pi}{\arg\max}\ J(\pi) \quad\text{s.t.} \quad\sum_{i,j}(\mathbf{H}_N)_{i,j} \le B

The budget BB encourages the policy πB∗\pi_B^* to focus sensing capacity on the most informative regions, naturally promoting sparse, adaptive measurement strategies.

One approach to enforce the hard constraint is to directly optimize over policies that fulfill it, e.g., by using the conditional Bernoulli policies πθ,BCBer(An∣Sn−1)\pi_{\theta,B}^{CBer}(A_n \mid S_{n-1}) [26], which generates each new pattern conditioned on the dose used so far. While using these policies guarantees strict dose compliance, it introduces sequential dependencies across pattern proposals, which can slow training and requires the policy to anticipate future sensing opportunities.

An alternative is to fold the cost into the objective as a soft constraint, penalizing dose expenditure directly within the policy gradient

πλ∗:=arg⁡max⁡π J(π)−λ∑i,j(HN)i,j(40)\pi_\lambda^* := \underset{\pi}{\arg\max}\ J(\pi) - \lambda\sum_{i,j}(\mathbf{H}_N)_{i,j} \tag*{(40)}

where λ>0\lambda> 0 balances information gain against dosage used. This Lagrangian relaxation allows more flexible pattern proposals and avoids the sequential conditioning required by the hard-constrained approach, but does not guarantee strict dose compliance: A too small λ\lambda may lead to BB routinely exceeded; too large λ\lambda and informative patterns may be suppressed.

Learning experimental designs using parametric policies

For policy gradient methods, improving the experimental design reduces to iteratively updating the policy parameters θ\theta introduced in § 3.2 to maximize the expected information gain objective J(πθ)=J(θ)J(\pi_\theta) = J(\theta) by gradient ascent

θn=θn−1+α∇θJ(θn)(41)\theta_n = \theta_{n-1} + \alpha\nabla_\theta J(\theta_n) \tag*{(41)}

where α\alpha is learning rate of the parameterization update.

As computing the true gradient is intractable, different methods rely on different stochastic approximation approaches. When a differentiable policy is available, one can make use of the gradient policy theorem [27], to reformulate the gradient of (17) as:

∇θJ(θ)=∇θEτ∼P(τ∣πθ)[∑n=1NRn(Sn−1,An,Sn)]=Eτ∼P(τ∣πθ)[∑n=1NGn∇θlog⁡πθ(An∣Sn−1)],(42)\begin{aligned} \nabla_\theta J(\theta) &= \nabla_\theta\mathbb{E}_{\tau\sim P(\tau\mid\pi_\theta)} \left[\sum_{n=1}^{N} R_n(S_{n-1}, A_n, S_n)\right] \\ &= \mathbb{E}_{\tau\sim P(\tau\mid\pi_\theta)} \left[\sum_{n=1}^{N} G_n \nabla_\theta\log\pi_\theta(A_n \mid S_{n-1})\right], \tag*{(42)} \end{aligned}

where Gn:=∑l=nNRlG_n := \sum_{l=n}^{N} R_l is the expected cumulative reward. At each step nn, the update is proportional to Gn=∑l=nNRlG_n = \sum_{l=n}^{N} R_l, the sum of all rewards collected from that step until the end of the experiment. This means the policy is geared towards choices that proved beneficial across the remainder of the episode. Actions that consistently produce rewards are reinforced, while those that led to diminishing returns are gradually discarded. By doing this, the expected cumulative return attributes long-term success back to the decisions that were crucial in the past.

The policy gradient update is the core step in the REINFORCE algorithm [16], which approximates the expected value in (42) by Monte Carlo estimation over NMCN_{MC} trajectory samples: For each trajectory sample, the expected cumulative reward is computed for each step from that point until n=Nn = N and using (42) the gradient in (41) is approximated as

∇θJ(θ)≈1NMC∑j=1NMC∑n=1NGn(j)∇θlog⁡πθ(An(j)∣Sn−1(j))(43)\nabla_{\theta}J(\theta) \approx\frac{1}{N_{MC}}\sum_{j=1}^{N_{MC}}\sum_{n=1}^{N}G_{n}^{(j)}\nabla_{\theta}\log\pi_{\theta}\left(A_{n}^{(j)} \mid S_{n-1}^{(j)}\right) \tag*{(43)}

The full REINFORCE algorithm formulated for the sOED problem is described in Algorithm 1.

Algorithm 1 REINFORCE for SPI Sequential Optimal Experimental Design

  1. Initialize policy parameters θ\theta randomly

  2. for episode i=1i = 1 to NepisodesN_{\mathrm{episodes}} do

  3. sample ground truth image x∼P(x)\mathbf{x} \sim P(\mathbf{x}) for this episode

  4. // Collect NMCN_{\mathrm{MC}} trajectories

  5. for trajectory j=1j = 1 to NMCN_{\mathrm{MC}} do

  6. τ(j)←∅\tau^{(j)} \leftarrow\emptyset

  7. S0(j)←P(x)S_0^{(j)} \leftarrow P(\mathbf{x}) (prior distribution evaluated at x\mathbf{x})

  8. for step n=1n = 1 to NN do

  9. hn(j)∼πθ(⋅∣Sn−1(j))\mathbf{h}_n^{(j)} \sim\pi_\theta(\cdot\mid S_{n-1}^{(j)})

  10. yn(j)∼P(y∣x;hn(j))y_n^{(j)} \sim P(y \mid\mathbf{x}; \mathbf{h}_n^{(j)})

  11. Rn(j)=EIG⁡x∣yn(j)(hn(j))R_n^{(j)} = \operatorname{EIG}_{\mathbf{x}\mid y_n^{(j)}}(\mathbf{h}_n^{(j)}) as described in Sec. 3.4

  12. Sn(j)←P(x∣yn(j);Hn(j))∝P(yn(j)∣x;hn(j))⋅Sn−1(j)S_n^{(j)} \leftarrow P(\mathbf{x} \mid y_n^{(j)}; \mathbf{H}_n^{(j)}) \propto P(y_n^{(j)} \mid\mathbf{x}; \mathbf{h}_n^{(j)}) \cdot S_{n-1}^{(j)} as described in Sec. 3.3

  13. τ(j)←τ(j)∪{(Sn−1(j),hn(j),Rn(j),Sn(j))}\tau^{(j)} \leftarrow\tau^{(j)} \cup\{(S_{n-1}^{(j)}, \mathbf{h}_n^{(j)}, R_n^{(j)}, S_n^{(j)})\} as described in Sec. 3.1

  14. end for 15.

  15. // Compute expected cumulative reward

  16. for step n=1n = 1 to NN do

  17. Gn(j)←∑l=nNRl(j)G_n^{(j)} \leftarrow\sum_{l=n}^{N} R_l^{(j)} (sum of rewards from step nn)

  18. end for

  19. end for 21.

  20. //Update policy using (43)

  21. ∇θJ(θ)←1NMC∑j=1NMC∑n=1NGn(j)⋅∇θlog⁡πθ(An(j)=hn(j)∣Sn−1(j))\nabla_\theta J(\theta) \leftarrow\frac{1}{N_{\mathrm{MC}}}\sum_{j=1}^{N_{\mathrm{MC}}}\sum_{n=1}^{N}G_n^{(j)} \cdot\nabla_\theta\log\pi_\theta(A_n^{(j)} = \mathbf{h}_n^{(j)} \mid S_{n-1}^{(j)})

  22. Update parameters θ←θ+α⋅∇θJ(θ)\theta\leftarrow\theta+ \alpha\cdot\nabla_\theta J(\theta) as in (41)

  23. end for

  24. return Trained policy πθ\pi_\theta

Numerical Studies

The proposed reinforcement learning approach for sOED is validated by learning optimal measurement strategies in a simple, yet representative, numerical SPI scenario. Concepts introduced in the previous section and their impact on learning performance, such as the policy parametrization (cf. Sec. 3.6), state approximation (cf. Sec 3.3), reward approximation (cf. Sec 3.4), dose constraints (cf. Sec. 3.5) performance are illustrated. The ability of the framework to find optimal strategies under increasing measurement noise is shown.

One-active-pixel SPI scenario

Consider a structured single-pixel imaging setup in which exactly one pixel of the unknown image is active with intensity 1, as described in the left side of Figure 1. This discrete sparsity-constraint means that the possible images are reduced to the discrete set of canonical basis vectors, and one can assume that the prior P(x)P(\mathbf{x}) accurately captures this a-priori information. In this setting, both the Bayesian posterior and the expected information gain are tractable to compute either in closed form, for the noiseless scenario, or via low-dimensional numerical quadrature in the noisy case. It provides a simple benchmark in which the reinforcement learning agent can use the exact posterior as the state and the sensing strategy can be compared against analytically optimal designs as well as against random sensing pattern. It also enables a systematic study of variability across independent policy initializations, since the discrete prior supports repeated runs with a modest computational cost.

Figure 1. The posterior is being updated using measurements of the ground truth x\mathbf{x} via the proposed patterns hn\mathbf{h}_n (in green). Bisection and raster scan are considered as two possible strategies to identify the ground truth from measurements via one-active-pixel prior knowledge.

Formally, the one-active-pixel set can be described by the images EM:={e1,…,eM}\mathcal{E}_M := \{\mathbf{e}_1,\ldots,\mathbf{e}_M\}, where em\mathbf{e}_m denotes the mm-th unit vector in RM\mathbb{R}^M. The prior P(x)P(\mathbf{x}) is a discrete probability measure on RM\mathbb{R}^M and can be written as

P(x)=∑m=1Mwm(0)δem(x),wm(0)=1/M(44)P(\mathbf{x}) = \sum_{m=1}^{M} w_m^{(0)} \delta_{\mathbf{e}_m}(\mathbf{x}), \qquad w_m^{(0)} = 1/M \tag*{(44)}

where δem(x)\delta_{\mathbf{e}_m}(\mathbf{x}) denotes a unit point mass concentrated at x=em\mathbf{x} = \mathbf{e}_m. Following (3), each measurement leads to posterior that is also discrete and has the same form as the prior, just with updated weights

P(x∣yn;Hn)=∑m=1Mwm(n)δem(x),wm(n)∝P(yn∣x=em;Hn),∑m=1Mwm(n)=1(45)P(\mathbf{x}\mid\mathbf{y}_n;\mathbf{H}_n)=\sum_{m=1}^{M}w_m^{(n)}\delta_{\mathbf{e}_m}(\mathbf{x}),\quad w_m^{(n)}\propto P(y_n\mid\mathbf{x}=\mathbf{e}_m;\mathbf{H}_n),\quad\sum_{m=1}^{M}w_m^{(n)}=1 \tag*{(45)}

Thus, the posterior and the RL state can be fully represented by the MM-dimensional vector Sn=w(n)=(w1(n),…,wM(n))S_n=\mathbf{w}^{(n)}=(w_1^{(n)},\ldots,w_M^{(n)}).

In the ideal noiseless one-active-pixel setting, a measurement of any x=em\mathbf{x}=\mathbf{e}_m that is supported by the prior with a binary measurement pattern h∈{0,1}M\mathbf{h}\in\{0,1\}^{M} yields either 0 or 1

y=⟨h,em⟩={1if (h)m=10otherwise(46)y=\langle\mathbf{h},\mathbf{e}_m\rangle=\begin{cases}1 & \text{if }(\mathbf{h})_m=1\\0 & \text{otherwise}\end{cases} \tag*{(46)}

If one measures y=1y=1, the active pixel then lies in the support of the illuminated mask, i.e., m∈supp⁡(h)m\in\operatorname{supp}(\mathbf{h}). Updating the prior with this likelihood information means removing all one-active-pixel images with active locations outside of the mask and renormalizing the weights to yield a uniform distribution over the remaining locations, as illustrated in the trajectories of Figure 1. Measuring y=0y=0 accordingly removes all locations that are in the support of the illuminated mask from the posterior. Intuitively, this scenario thereby corresponds to a simple ”finding a needle in a haystack” search problem: where, out of mm possible locations, a particular objects sits. Every measurement one can make with a pattern h\mathbf{h} corresponds to asking the question ”Is the object located in any of the illuminated positions?” and receiving a ”yes-or-no” answer.

For illustration, let us consider the case M=4M=4 case: The prior is uniform over {e1,…,e4}\{\mathbf{e}_1,\ldots,\mathbf{e}_4\}, leading to w(0)=(1/4,1/4,1/4,1/4)\mathbf{w}^{(0)}=(1/4,1/4,1/4,1/4). The simplest case is the conventional raster scan, where one pixel at a time is illuminated in a fixed order. The sequence of patterns would be h1=(1,0,0,0)\mathbf{h}_1=(1,0,0,0), h2=(0,1,0,0)\mathbf{h}_2=(0,1,0,0), h3=(0,0,1,0)\mathbf{h}_3=(0,0,1,0), h4=(0,0,0,1)\mathbf{h}_4=(0,0,0,1). Now suppose that the true image is x=e3\mathbf{x}=\mathbf{e}_3. The first measurement is then y1=0y_1=0, which discards e1\mathbf{e}_1 but leaves e2,e3,e4\mathbf{e}_2,\mathbf{e}_3,\mathbf{e}_4 as possible. After the likelihood update, the posterior weights become

w(1)=(0,13,13,13).(47)\mathbf{w}^{(1)}=\left(0,\frac{1}{3},\frac{1}{3},\frac{1}{3}\right). \tag*{(47)}

The second pattern yields again y2=0y_2=0, ruling out e2\mathbf{e}_2 and leaving only e3\mathbf{e}_3 and e4\mathbf{e}_4 feasible, so the posterior weights are

w(2)=(0,0,12,12).(48)\mathbf{w}^{(2)}=\left(0,0,\frac{1}{2},\frac{1}{2}\right). \tag*{(48)}

Only after the third measurement, with h3=(0,0,1,0)\mathbf{h}_3=(0,0,1,0) and outcome y3=1y_3=1, does the posterior collapses to a single delta distribution centered on the true image, such that weights are

w(3)=(0,0,1,0).(49)\mathbf{w}^{(3)} = (0, 0, 1, 0). \tag*{(49)}

In this worst-case trajectory, the posterior support shrinks from four to three, then to two, and only then to a single element, illustrating that, under a uniform prior, raster scanning requires on average (M+1)/2(M+1)/2 measurements to uniquely identify the active pixel, exhibiting a linear measurement complexity in the number of image elements MM.

In analogy to search problems, a bisection strategy would be more advantageous, where the patterns are chosen to half the current possible locations: In the same M=4M = 4 example with uniform prior w(0)\mathbf{w}^{(0)} and true image x=e3\mathbf{x} = \mathbf{e}_3, one can consider a first pattern h1=(1,1,0,0)\mathbf{h}_1 = (1, 1, 0, 0), which only illuminates pixels 1 and 2. The first outcome is then y1=0y_1 = 0 and after renormalizing the posterior becomes uniform over e3\mathbf{e}_3 and e4\mathbf{e}_4 so that the weights are

w(1)=(0,0,12,12),(50)\mathbf{w}^{(1)} = \left(0, 0, \frac{1}{2}, \frac{1}{2}\right), \tag*{(50)}

An adaptive second pattern is then h2=(0,0,1,0)\mathbf{h}_2 = (0, 0, 1, 0), which illuminates the third but not the fourth element. The second outcome is y2=1y_2 = 1, collapsing to a delta centered in e3\mathbf{e}_3 and the weights become

w(2)=(0,0,1,0),(51)\mathbf{w}^{(2)} = (0, 0, 1, 0), \tag*{(51)}

shrinking from four to two and then directly to a single element in just two measurements.

Another possibility is the usage of a pattern that, on top of halving the possible locations, also keeps illuminating a partial fraction of the already measured pixels. From the previous case, a pattern h2=(0,1,1,0)\mathbf{h}_2 = (0, 1, 1, 0) represents a fixed bisection strategy, and these strategies are different from the adaptive ones in the dose budget they require and their dependence on the measurements. Fixed bisection allows via redundancy to solve for all possible images in the one-active-pixel set without redesigning their activations given the measurements obtained over the experiment.

For both cases, bisection strategies half the support of the posterior distribution at each step and, more generally, identifies the active pixel in at most ⌈log⁡2M⌉\lceil\log_2 M \rceil measurements, reflecting its logarithmic complexity in this simplified SPI setting.

Optimal EIG strategies

For the scenario described, one can find that bisection strategies are optimal with respect to EIG. Using this result, learned policies can be compared against the strategies yielding the best achievable performance. Firstly, the EIG (9) is computed for images in the one-active-pixel set x∈E={e1,…,eM}\mathbf{x} \in\mathcal{E} = \{\mathbf{e}_1, \ldots, \mathbf{e}_M\} for the first measurement with h\mathbf{h}. The marginal likelihood

P(y;h)P(y;\mathbf{h}) is needed to compute the expectation over the prior and posterior entropy difference.

As illustrated in Figure 2, yy can only take the values 0 and 1 in the noiseless scenario: The marginal likelihood of y=1y=1 is the ratio between the number of illuminated sites in the measuring pattern h\mathbf{h}, ∣supp⁡(h)∣|\operatorname{supp}(\mathbf{h})| and the number of possible illuminated sites MM, so that P(y=1;h)=∣supp⁡(h)∣/MP(y=1;\mathbf{h})=|\operatorname{supp}(\mathbf{h})|/M and P(y=0;h)=(M−∣supp⁡(h)∣)/MP(y=0;\mathbf{h})=(M-|\operatorname{supp}(\mathbf{h})|)/M.

Figure 2. For a uniform one-active-pixel, strategies that illuminate either one or all but one site are suboptimal in an EIG sense. This can be seen from the unbalanced posterior at the different possible outcomes yy. For the bisection strategy, measurements 0 or 1 have equal probability and the posterior support halves, which produces the optimal one-step EIG.

Then, the entropy for the uniform one-active-pixel prior, as described in (44), reads

Ent⁡(x)=log⁡M(52)\operatorname{Ent}(\mathbf{x})=\log M \tag*{(52)}

Following the previous section, the posterior shrinks with respect to the uniform prior. One can compute the posterior entropy by averaging over the possible images of the set after the measurement, which leaves

Ent⁡(x∣y;h)={log⁡(M−∣supp⁡(h)∣)if y=0log⁡(∣supp⁡(h)∣)if y=1(53)\operatorname{Ent}(\mathbf{x}\mid y;\mathbf{h})= \begin{cases} \log(M-|\operatorname{supp}(\mathbf{h})|) & \text{if } y=0 \\ \log(|\operatorname{supp}(\mathbf{h})|) & \text{if } y=1 \end{cases} \tag*{(53)}

With this, one can write the one-step EIG as an information gain average weighted by the marginal likelihood P(y;h)P(y;\mathbf{h}). Decomposing the information gain as the difference between prior and posterior entropy, one obtains

EIGx(h)=∑y=0,1P(y;h)[Ent⁡(x)−Ent⁡(x∣y;h)]=∣supp⁡(h)∣Mlog⁡M∣supp⁡(h)∣+M−∣supp⁡(h)∣Mlog⁡MM−∣supp⁡(h)∣(54)\mathrm{EIG}_{x}(\mathbf{h})=\sum_{y=0,1}P(y;\mathbf{h})[\operatorname{Ent}(\mathbf{x})-\operatorname{Ent}(\mathbf{x}\mid y;\mathbf{h})] =\frac{|\operatorname{supp}(\mathbf{h})|}{M}\log\frac{M}{|\operatorname{supp}(\mathbf{h})|}+\frac{M-|\operatorname{supp}(\mathbf{h})|}{M}\log\frac{M}{M-|\operatorname{supp}(\mathbf{h})|} \tag*{(54)}

which means that EIGx(h)\mathrm{EIG}_{x}(\mathbf{h}) is a function of s=∣supp⁡(h)∣s=|\operatorname{supp}(\mathbf{h})|. If ss was a continuous variable on (0,M)(0,M)

ddsEIGx(s)=1Mlog⁡M−ss,d2ds2EIGx(s)=−1M(1s+1M−s)<0,(55)\frac{d}{ds}\mathrm{EIG}_{x}(s)=\frac{1}{M}\log\frac{M-s}{s},\qquad\frac{d^{2}}{ds^{2}}\mathrm{EIG}_{x}(s)=-\frac{1}{M}\left(\frac{1}{s}+\frac{1}{M-s}\right)<0, \tag*{(55)}

so the EIGx(s)\mathrm{EIG}_{x}(s) would be strictly concave w.r.t. ss and its unique stationary point

s∗=arg⁡max⁡s EIGx(s)=M2(56)s^{*}=\underset{s}{\arg\max}\ \mathrm{EIG}_{x}(s)=\frac{M}{2} \tag*{(56)}

is the global maximum. This means that for an even MM, the optimal pattern illuminates half the sites of the possible image set E\mathcal{E} in one step.

Because every posterior is again uniform, now over the remaining image set Ei⊆E\mathcal{E}_{i}\subseteq\mathcal{E} of size MiM_{i}, with M0=MM_{0}=M, the step-(i+1)(i+1) design problem is identical to the first with MM corresponding to MiM_{i} and supp⁡(hi+1)⊆Ei\operatorname{supp}(\mathbf{h}_{i+1})\subseteq\mathcal{E}_{i}. The same calculation gives si+1∗=Mi/2s^{*}_{i+1}=M_{i}/2, and at the maximum both outcomes leave Mi/2M_{i}/2 possible sites. Thereby, Mi+1=Mi/2M_{i+1}=M_{i}/2 and the support halves at every step, leading to Mi=M/2iM_{i}=M/2^{i}. Bisection is then EIG-optimal per step and identifies the active pixel in exactly ⌈log⁡2M⌉\lceil\log_{2}M\rceil measurements no matter where the active pixel is located.

Experimental settings

In all presented RL experiments, N=2N=2 is set as the number of measurements, so that the posterior is updated twice per experiment. Furthermore, NMC=1N_{\mathrm{MC}}=1 is always used in (43), i.e., the policy update is based on a single simulated trajectory. This corresponds to the setting one would use for large-scale imaging scenarios. Every RL experiment is repeated 100 times with different policy initializations to study the stability and convergence of learned designs and report the mean and 95% confidence intervals in the evaluations. Unless stated otherwise, the total expected information gain of a policy πθ\pi_{\theta} is normalized for visualization purposes, to allow for the comparison of policies that propose binary pattern strategies under different experimental setups

J(πθ)=J(πθ)−J‾randJ‾opt−J‾rand,(57)J(\pi_{\theta})=\frac{J(\pi_{\theta})-\overline{J}_{\mathrm{rand}}}{\overline{J}_{\mathrm{opt}}-\overline{J}_{\mathrm{rand}}}, \tag*{(57)}

Policy architecture comparison

Figure 3 shows two training comparisons: the scaling of image sizes M={4,8,12,16}M = \{4, 8, 12, 16\} and the usage of categorical or Bernoulli pattern proposal architectures, as described in § 3.2. Categorical-based policies, described in (20), reach optimal convergence in fewer episodes than the Bernoulli-based ones, formulated in (21). Scaling the image size shows a higher requirement of episodes for convergence for the optimal EIG in two experimental steps.

Figure 3. Total normalized EIG for categorical (lighter tone, square marker) and Bernoulli (darker tone, circle marker) policies trained over different image sizes M={4,8,12,16}M = \{4, 8, 12, 16\} and same experimental step size N=2N = 2. Optimal and random baselines are drawn as red and gray lines, respectively, where the maximum EIG value JoptJ_{\mathrm{opt}} is calculated for that experiment, obtained using a bisection strategy in the one-active-pixel prior case, and the average EIG J‾rand\overline{J}_{\mathrm{rand}} that a fully random policy reaches which illuminates each pixel with probability 0.5.

The normalized EIG at Figure 3 shows that Bernoulli policies are slower to train than categorical policies for all image sizes. However, Bernoulli policies can parameterize distributions over the action space with fewer trainable parameters, as the output of the network is the distribution over the pattern activations, and this output layer grows linearly with the image size MM. Actions sampled from the Bernoulli policies are rewarded indirectly, since there is not an output unit-to-pattern correspondence as there is in the categorical. Then, the reward carries more stochasticity than in the case of categorical policies.

In the categorical case, for an image size of M=16M = 16 the output layer has a size of ∼65k\sim65\mathrm{k} units given that the design space is {0,1}16×16\{0, 1\}^{16 \times16}. Larger images result challenging for a categorical parameterization, leaving Bernoulli kind of policies for these cases despite their requirement for longer training times. The stochastic reward assignment, instead of a direct pattern-to-distribution reward, and the design space size are the trade-off variables for the framework for scaling up to large imaging problems.

Effect of sample-based approximations

We study sample-based approximations for state and reward, as introduced in § 3.3 and § 3.4. This is important for translating this framework to more complex inference problems, where the posterior is accessible only through samples rather than in closed form. The focus of this section is to assess these effects separately, while in the Supplementary Data we present a brief study of both effects in combination with respect to the posterior without approximation.

State approximation

As described in § 4.1, the posterior and RL state at time nn is completely described by the discrete parameter w(n)\mathbf{w}^{(n)}, whose entries wi(n)w_i^{(n)} contains the probability that the active pixel is at location ii. Therefore, the empirical mean state approximation computed from NsN_s posterior samples, as defined in (28), converges to the true state w(n)\mathbf{w}^{(n)}. Here, we compare policies trained with this approximation against the exact posterior benchmark.

Figure 4 shows how approximating the state via sampling affects the learning process. Surprisingly, even the policy learned with approximating the state by a single sample reaches the optimal EIG. The inflection point around episode 6,000 marks the transition where modifying the number of samples produces a different learning performance. Strategies learned using fewer samples for the state approximation do need more episodes for reaching the optimal EIG. The CI95 plots show that the variability follows a similar behavior: Policies trained with a low number of samples need more episodes to reduce the variability, but eventually all of them to to zero.

Figure 4. Total expected information gain for policy training under sample-based state approximations, with Ns={1,2,4,8,16}N_s = \{1, 2, 4, 8, 16\}. First row figure displays mean total EIG, second row shows 95% confidence interval width. Curve for learning with no approximation, using exact state representation is illustrated in yellow for comparison of the sampling impact. Optimal policy reward is computed exactly and displayed in red for both plots.

Reward approximation

We train policies to recover the optimal one-active-pixel strategy using the sample-based EIG estimator introduced in § 3.4 as the reward signal. This setting allows us to assess whether policies trained with the estimator in (32) can recover the strategy that is optimal under the exact EIG objective in (30). Although evaluating the exact EIG requires the marginal likelihood, which is generally intractable in more complex settings, it is available in the scenario we designed and provides a benchmark for evaluation.

In Figure 5 we observe that using the sample-based reward not only enables learning the optimal EIG strategy, it seems to even accelerate training.

Figure 5. Total expected information gain using a sample-based reward approximation, with Nr={2,4,8,16,32}N_r = \{2, 4, 8, 16, 32\} samples and a exact state representation. Optimal rewards for each sample quantity are illustrated in red, with mean values in the first row plot and 95% confidence value widths over the second row. For comparison, EIG reward calculation using the posterior distribution representation without approximation (’No Approximation’) is displayed in yellow.

The only case displaying a slower training curve than the exact reward is the Nr=2N_r = 2 case. In contrast to Figure 4, after the inflection point in episode 6,000, using more than Nr=2N_r = 2 samples for the reward approximation show a similar trend in mean total EIG and confidence interval widths to the case where the exact reward (’No Approximation’) is used as a reward signal. Note that as in the sample-based state approximation (cf. Figure 4), the training curves converge to the exact reward curves for larger numbers of samples.

Learning under noisy measurements

In the noiseless case, the first measurement collapses the posterior to delta distributions over the regions of activity or inactivity in the case of measuring a 1 or 0. In the noisy case σ>0\sigma> 0, this is no longer true, as noisy measurements still leave posterior probability mass over the discarded areas after the first acquisition. We now examine the impact of this on learning design policies. As a reference, we again compare their performance to the EIG of the bisection strategy, normalized as in (57).

Among the two approaches introduced in § 3.4 to treat the Gaussian likelihood modeled in (1), our proposed one-active-pixel scenario requires the usage of a numerical integration method. For the experiments displayed in Figure 6 and Figure 7, the rewards using the Bayesian quadrature are computed as described in § 3.4, with K=1000K = 1000 elements over the grid determined by each sample.

Figure 6. Total normalized EIG for three noisy measurement levels σ∈{0.2,0.4,0.8}\sigma\in\{0.2, 0.4, 0.8\}. First column plot displays normalized EIG and second column plot 95% confidence interval widths, averaged across 100 categorical policies realizations. Curves are normalized using random and optimal baselines for each noise level, and represented in red and green, respectively.

Figure 7. Total normalized EIG for three noisy measurement levels under constant and a linear curriculum learning for noise scheduling. Columns represent noise measurement levels σ∈{0.2,0.4,0.8}\sigma\in\{0.2, 0.4, 0.8\} and rows display the normalized means and 95% confidence interval widths. Curves are normalized using random and optimal baselines for each noise level, and are drawn in gray and green lines, respectively.

We can see that the learning behavior depends on the noise level: For a lower noise level, the policies learn slower than for the higher noise level at first but then overtake them and reach the reference normalized EIG earlier (note however, that the normalization depends on σ\sigma). For higher noise levels, they reach a sub-optimal inflection point in less episodes, as we observe for σ=0.8\sigma= 0.8, but then it takes longer to converge to the reference EIG performance.

Learning an optimal EIG strategy directly at a fixed measurement noise level σ\sigma can be challenging, since noisy observations yield less informative posterior updates than in the noiseless setting. We investigate curriculum learning, in which training begins with noiseless measurements and the noise level is increased linearly to the target value σ\sigma over the course of training. The episode-dependent noise schedule is σtrain(i)=σiNepisodes\sigma_{\mathrm{train}}(i) = \sigma\frac{i}{N_{\mathrm{episodes}}}, for i=0,1,…,Nepisodesi = 0,1,\ldots,N_{\mathrm{episodes}}, where ii denotes the current episode. This curriculum is compared with the constant-noise training setting of Figure 6, in which σtrain(i)=σ\sigma_{\mathrm{train}}(i) = \sigma for every episode. To ensure a fair comparison, we evaluate both learned policies under the same target noise level σ\sigma and compute their total EIG exactly using (30).

Using σ∈{0.2,0.4,0.8}\sigma\in\{0.2, 0.4, 0.8\}, Figure 7 displays a faster learning of the linear schedule after the inflection point is reached around episode 15,000 for all

noise levels. After the first episodes, the 95% confidence intervals width shrink to zero for all σ\sigma, illustrating that using curriculum learning presents an performance advantage with respect to learning under constant noise. The linear curriculum learning policy is trained with cleaner measurement and evaluated in a noisier environment, which suggests that these strategies present a more stable training procedure for the proposed reinforcement learning framework.

Dose-constrained methods

In this study, we examine the two approaches to incorporate dose constraints introduced in § 3.5. In the first case, the policy is a conditional Bernoulli distribution as in [28], where the design policy proposes patterns using a dose budget BB along a experiment, as described in (40). Training is carried out for budgets B∈{1,…,8}B \in\{1,\ldots,8\}, the number of experiment steps is M=2M = 2 and the image size is N=4N = 4.

Figure 8 shows the performance of constrained Bernoulli policies normalized by the optimal EIG and the strategy obtained using unconstrained Bernoulli policies without training, i.e., projecting patterns at random without dose constraints, as described in § 4.3 (this results in an average dose of 4 for this setup). We observe that only the trained policy with budget B=1B = 1 cannot produce a policy that beats the unconstrained random policy, stabilizing at ∼−0.75\sim-0.75 of the normalized EIG. B=2B = 2 policies start below the unconstrained Bernoulli random baseline but after training, these policies stabilize around a normalized EIG of ∼0.25\sim0.25 with a large variability. This suggests that, even with a small budget, some of these policies produce dose-constrained strategies that are better than just drawing patterns from an unconstrained Bernoulli policy. Still, the budget is too small to reach optimal EIG strategies.

Figure 8. Total normalized EIG under dose budget constraints B∈{1,…,8}B \in\{1, \ldots, 8\} for conditional Bernoulli policies. First and second row display mean and 95% confidence interval width, respectively. Random baseline, in red, is computed using a unconstrained Bernoulli policy. Optimal baseline is drawn in green.

As we described in § 4.2, an adaptive policy, i.e., one that truly implements sOED, needs a budget of B=3B = 3 to reach optimal performance in this two-step experiment. We can see that the training curve indeed reaches an optimal EIG until episode 40,000 and then the performance drops in the last epochs. The policy presents the fastest learning of the optimal EIG strategy along with budget B=4B = 4, while this dose constraint is the first reaching almost zero variability around episode 25,000. Larger budget constraints B≥5B \ge5 follow a similar trend, with a slower, more stable performance curve and low variability after the end of training.

Figure 9 illustrates how to incorporate the constraint as a penalty term, where λ\lambda is the weight between the dose budget BB and the EIG to be optimized. We evaluate using λ∈{0.01,0.02,0.04,0.08,0.1,0.2,0.4}\lambda\in\{0.01, 0.02, 0.04, 0.08, 0.1, 0.2, 0.4\} and plot separately the EIG normalized between the optimal EIG strategy and the unconstrained random policy, the dose budget that each policy uses at the time of projecting patterns and the normalized 95% confidence interval.

Figure 9. From top to bottom, total normalized EIG, dose budgets BB and normalized 95% confidence interval widths. Dose constraints λ∈{0.01,0.02,0.04,0.08,0.1,0.2,0.4}\lambda\in\{0.01, 0.02, 0.04, 0.08, 0.1, 0.2, 0.4\} are used, with random and optimal baseline strategies in gray and red, respectively. For the dose budget plot, baselines represent a fixed (gray) or an adaptive (red) policy design choice.

When using the dose as a soft constraint, different EIG performances are observed by the end of the learning process. For small λ\lambda in the range of [0.01,0.04][0.01, 0.04], the policies converge to an optimal EIG policy that proposes designs with B=4B = 4. Intermediate λ∈[0.8,0.1]\lambda\in[0.8, 0.1] also display an optimal EIG behavior with a wider confidence interval and a mean budget B∼3.5B \sim3.5. In the range of large λ∈{0.2,0.4}\lambda\in\{0.2, 0.4\}, the dose constraint weights are so heavy that the resulting design no longer achieves optimal EIG. This constraint also proposes designs B∼2B \sim2 that are suboptimal for the one-active pixel scenario and the confidence intervals are the widest.

Discussion

In § 4 we illustrated the usage of the reinforcement learning framework to solve the Bayesian optimal experimental design of binary pattern for single-pixel imaging under a one-active pixel prior. For this methodology, we showed different aspects that affects the performance of the method and allow it to find optimal strategies incorporating constraints and measurement

noise.

When choosing a particular parameterization for the policy, the learning curves in § 4.4 display an inflection point that replicates over the experiments. We infer that, given the two steps we set in the proposed SPI experiment, namely projecting two binary patterns to the image of interest, learning the optimal design splits into two phases. As we examined intermediate pattern proposals along training, this behavior is likely caused by learning a bisection halving pattern in the first place and later on learning the remaining strategy. This phenomena can be related to the adaptive and fixed strategies introduced in § 4.2. At the first pattern, both strategies do not differ from each other, as they have a first bisection pattern with half of their pixels active. Moreover, the position of these active pixels is not crucial for the first pattern, but it is for the second one with respect to the first to achieve an optimal EIG step, since the input to the policy changed from being the prior in the first step to one of two possible posterior distributions in the second step. Then, the inflection point suggests a faster learning of the first pattern, where the policy training reaches a suboptimal EIG. Learning the second pattern takes more episodes for the policy due to the different possible posteriors as input for which a different optimal second pattern needs to be learned.

The sample-based state and reward studies in § 5 show the potential of RL methods in large-scale imaging scenarios, where a Bayesian approach is only feasible via sampling methods. Here, we explored the effect of using state and reward estimators to learn optimal policies in the one-active pixel scenario. Our results show that training with a surprisingly small number of posterior samples is similarly fast compared to using the exact state or reward. The downside of this approach is the additional cost to simulating trajectories, which becomes more expensive in terms of compute due to the sampling times to compute the state or the reward estimators.

Lastly, training of policies under dose constraints in § 5.4 demonstrate that better strategies than random Bernoulli patterns can be learned even for high dose constraints, i.e., low pattern activation budgets. These studies also directly demonstrate that our framework is able to learn truly adaptive policies, i.e., policies that solve the sOED problem (14), not just the a-priori OED problem (16): For both type of constraints, policies can only reach the optimal EIG with budget B=3B = 3 if they act adaptively to the first measurement, cf. § 4.2.

Conclusions

This work presents a way to formulate the design proposal of single-pixel imaging binary patterns as a Bayesian sequential optimal experimental design problem. By using a reinforcement learning framework, we proposed different ways to parameterize a design policy, and illustrated how they can be used depending on the scale of the design size and the image size. For random binary patterns, if the image size is small, a categorical policy over all possible options is preferred over a Bernoulli one due to their faster training procedure.

To investigate the scaling from proof-of-concept to large scale computational imaging scenarios, we carried out sample-based approximations using numerical simulations. Surprisingly, we found that approximating the state and/or the reward with with sample-based estimators had only minor effects on the learning dynamics, even for very low numbers of samples.

We validated the robustness of the framework by learning designs when the measurements contain different noise levels. To accelerate training in this scenario, we showed that a curriculum learning strategy can be beneficial that uses a gradual noise scaling over the training episodes, simulating a degree of difficulty from simple to complex.

Moreover, we studied two possible ways to impose constraints over the illumination dosage, enabling budget-constrained single-pixel experimental designs and proofing that the learned designs are truly adaptive.

In this work, we deliberately limited our numerical studies to a simple, small-scale scenario, for which we could deduce optimal policies in an intuitive way. This allowed us to evaluate the learned policies in a thorough way and compute many quantities that are usually not tractable. In future work, we will extend this framework to other image priors and large-scale SPI scenarios.

Data availability

The code supporting the findings of this study will be released publicly upon acceptance of the manuscript.

Ethics statement

This study did not involve research with human participants, human data, human tissue, or live animals. Therefore, ethical approval was not required.

Funding

The authors have no conflicts of interest to declare. This work was supported by the European Union (No. GA 101072354). Views and opinions expressed are, however, those of the author(s) only and do not necessarily reflect those of the European Union nor the granting authority can be held responsible for them.

Acknowledgements

M.A.O. thanks L. Veenstra for insightful discussions on Bayesian inverse problems.

Supplementary data

Section 4.5 introduced separate studies varying the number of samples over the state and reward computation. In Figure 10 we study the effect of reusing the same samples for approximating the state and the reward. In comparison to the previous approximation studies, we observe a similar behavior to the reward approximation for Ns,Nr≥4N_s, N_r \ge4, where the total EIG along training is similar with no different learning trend after the inflection point at episode 6,000. For Ns,Nr=2N_s, N_r = 2, the learning curve is slower and the number of episodes is not enough for reaching the optimal EIG. With respect to the reward approximation alone, using Ns,Nr=2N_s, N_r = 2 seems to have a stronger impact in learning the optimal strategy, suggesting that there is a strong interplay between state and reward approximation when samples are scarce.

Figure 10. Impact of reward approximation on policy learning. Training curves showing normalized cumulative EIGEIG for categorical policy learning under a Monte Carlo reward approximation with Ns,Nr∈{2,4,8,16,32}N_s, N_r \in\{2, 4, 8, 16, 32\} posterior samples. The same samples are used to compute both state and reward approximations. Curves are averaged across 100 independent policy training realizations.

References

  1. [1]M. F. Duarte, M. A. Davenport, D. Takhar, J. N. Laska, T. Sun, K. F. Kelly, and R. G. Baraniuk, “Single-pixel imaging via compressive sampling,” IEEE Signal Processing Magazine, vol. 25, no. 2, pp. 83–91, 2008.DOI
  2. [2]Z. Zhang, X. Wang, G. Zheng, and J. Zhong, “Hadamard single-pixel imaging versus fourier single-pixel imaging,” Optics Express, vol. 25, no. 16, pp. 19619–19639, 2017.DOI
  3. [3]R. Stojek, A. Pastuszczak, P. Wróbel, and R. Kotyński, “Single pixel imaging at high pixel resolutions,” Optics express, vol. 30, no. 13, pp. 22730–22745, 2022.DOI
  4. [4]D. L. Donoho, “Compressed sensing,” IEEE Transactions on Information Theory, vol. 52, no. 4, pp. 1289–1306, 2006.DOI
  5. [5]E. J. Candès, J. Romberg, and T. Tao, “Robust uncertainty principles: Exact signal reconstruction from highly incomplete frequency information,” IEEE Transactions on Information Theory, vol. 52, no. 2, pp. 489–509, 2006.arxiv.org/abs/math/0409186
  6. [6]L. Gan, T. T. Do, and T. D. Tran, “Fast compressive imaging using scrambled block hadamard ensemble,” in 2008 16th European Signal Processing Conference. IEEE, 2008, pp. 1–5.DOI
  7. [7]T. Rainforth, A. Foster, D. R. Ivanova, and F. Bickford Smith, “Modern bayesian experimental design,” Statistical Science, vol. 39, no. 1, pp. 100–114, 2024.arxiv.org/abs/2302.14545
  8. [8]W. Shen and X. Huan, “Bayesian sequential optimal experimental design for nonlinear models using policy gradient reinforcement learning,” Computer Methods in Applied Mechanics and Engineering, vol. 416, p. 116304, 2023.DOI
  9. [9]W. J. Welch, “Branch-and-bound search for experimental designs based on D optimality and other criteria,” Technometrics, vol. 24, no. 1, pp. 41–48, 1982.
  10. [10]A. Foster, D. R. Ivanova, I. Malik, and T. Rainforth, “Deep adaptive design: Amortizing sequential Bayesian experimental design,” in Proceedings of the 38th International Conference on Machine Learning. PMLR, 2021, pp. 3384–3395.arxiv.org/abs/2103.02438
  11. [11]O. Scherzer, M. Grasmair, H. Grossauer, M. Haltmeier, and F. Lenzen, Variational Methods in Imaging, ser. Applied Mathematical Sciences. New York, NY: Springer, 2009, vol. 167.DOI
  12. [12]A. M. Stuart, “Inverse problems: A bayesian perspective,” Acta Numerica, vol. 19, pp. 451–559, 2010.DOI
  13. [13]M. Lassas and S. Siltanen, “Can one use total variation prior for edge-preserving bayesian inversion?” Inverse problems, vol. 20, no. 5, p. 1537, 2004.DOI
  14. [14]G. L. Nemhauser, L. A. Wolsey, and M. L. Fisher, “An analysis of approximations for maximizing submodular set functions,” Mathematical Programming, vol. 14, no. 1, pp. 265–294, 1978.
  15. [15]A. Alexanderian and S. Maio, “Submodularity of the expected information gain in infinite-dimensional linear inverse problems,” arXiv preprint arXiv:2602.09285, 2026.arxiv.org/abs/2602.09285
  16. [16]R. S. Sutton and A. G. Barto, Reinforcement learning: An introduction. MIT press, 2018.
  17. [17]R. J. Williams, “Simple statistical gradient-following algorithms for connectionist reinforcement learning,” Machine learning, vol. 8, pp. 229–256, 1992.
  18. [18]R. S. Sutton, D. A. McAllester, S. P. Singh, and Y. Mansour, “Policy gradient methods for reinforcement learning with function approximation,” in Advances in Neural Information Processing Systems, vol. 12, 2000.
  19. [19]B. T. Vincent and T. Rainforth, “The darc toolbox: automated, flexible, and efficient delayed and risky choice experiments using bayesian adaptive design,” PsyArXiv preprint, vol. 5, p. 9, 2017.
  20. [20]X. Huan and Y. M. Marzouk, “Simulation-based optimal Bayesian experimental design for nonlinear systems,” Journal of Computational Physics, vol. 232, no. 1, pp. 288–317, 2013.DOI
  21. [21]R. A. Horn and C. R. Johnson, Matrix Analysis, 2nd ed. Cambridge: Cambridge University Press, 2013.
  22. [22]K. Chaloner and I. Verdinelli, “Bayesian experimental design: A review,” Statistical Science, vol. 10, no. 3, pp. 273–304, 1995.
  23. [23]V. V. Fedorov, Theory of Optimal Experiments. New York: Academic Press, 1972.
  24. [24]P. P. Laissue, R. A. Alghamdi, P. Tomançak, E. G. Reynaud, and H. Shroff, “Assessing phototoxicity in live fluorescence imaging,” Nature Methods, vol. 14, no. 7, pp. 657–661, 2017.DOI
  25. [25]J. Icha, M. Weber, J. C. Waters, and C. Norden, “Phototoxicity in live fluorescence microscopy, and how to avoid it,” BioEssays, vol. 39, no. 8, p. 1700003, 2017.DOI
  26. [26]A. Attia, “Probabilistic approach to black-box binary optimization with budget constraints: Application to sensor placement,” arXiv preprint arXiv:2406.05830, 2024.arxiv.org/abs/2406.05830
  27. [27]R. S. Sutton and A. G. Barto, “Temporal-difference learning,” Reinforcement learning: an introduction, pp. 133–160, 1998.
  28. [28]A. Attia and E. Constantinescu, “Optimal experimental design for inverse problems in the presence of observation correlations,” SIAM Journal on Scientific Computing, vol. 44, no. 4, pp. A2808–A2842, 2022.arxiv.org/abs/2007.14476

Paper details

vlnc:W_vxrtknrg
Version
v1 · latest
Published
Fresh — added to the Hub within the last 30 days
Added
License
CC BY 4.0
Usage

views · 0

downloads · 0

A view counts once per browser per day, a download each time the PDF is requested.

  • Computer Science

Not recorded for this paper

View statement
Looking for related papers…
Contents