betaFeedback

WorksPhysics and Astronomy

Leveraging Multiplexed Metasurfaces for Multi-Task Learning with All-Optical Diffractive Processors

Abstract

Diffractive Neural Networks (DNNs) leverage the power of light to enhance computational performance in machine learning, offering a pathway to high-speed, low-energy, and large-scale neural information processing. However, most existing DNN architectures are optimized for single tasks and thus lack the flexibility required for the simultaneous execution of multiple tasks within a unified artificial intelligence platform. In this work, we utilize the polarization and wavelength degrees of freedom of light to achieve optical multi-task identification using the MNIST, FMNIST, and KMNIST datasets. Employing bilayer cascaded metasurfaces, we construct dual-channel DNNs capable of simultaneously classifying two tasks, using polarization and wavelength multiplexing schemes through a meta-atom library. Numerical evaluations demonstrate performance accuracies comparable to those of individually trained single-channel, single-task DNNs. Extending this approach to three-task parallel recognition reveals an expected performance decline yet maintains satisfactory classification accuracies of greater than 80% for all tasks. We further introduce a novel end-to-end joint optimization framework to redesign the three-task classifier, demonstrating substantial improvements over the meta-atom library design and offering the potential for future multi-channel DNN designs. Our study could pave the way for the development of ultrathin, high-speed, and high-throughput optical neural computing systems.

Introduction

Optical computing, especially through Optical Neural Networks (ONNs), has long been recognized for its potential to enhance computational speed and energy efficiency. The first optical implementation of neural networks in 1987 used optical components to emulate neuron configurations, sparking decades of research into optical neuromorphic technologies [1]. Recently, advancements in deep learning and photonic technology have revitalized interest in this field, enabling the development of scalable, ultra-fast, and energy-efficient ONNs [2, 3, 4, 5, 6, 7, 8, 9]. Diffractive Neural Networks (DNNs) [10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22] are a type of ONN consisting of multiple spatially engineered transmissive diffractive layers. Utilizing light-matter interactions, these diffractive surfaces perform element-wise multiplication, with each ‘pixel’ acting as a ‘neuron’, interconnected through the physics of optical diffraction. The complex-valued transmission coefficient of each neuron serves as a trainable network parameter, systematically adjusted via an error back-propagation algorithm executed on a digital computer to perform a specific machine learning task.

On the other hand, metasurfaces – engineered two-dimensional arrays of subwavelength nanostructures – allow one to precisely manipulate optical properties such as phase, amplitude, and polarization, all through the adjustment of the size and shape of meta-atoms [23, 24, 25, 26, 27]. Over the past decade, this capability has revolutionized applications in several fields, including imaging and holography [28, 29, 30, 31], sensing [32, 33], information processing [34, 35, 36], and quantum photonics [37, 38]. Their integration into DNNs has facilitated the development of advanced ultra-thin diffractive processors, which show promise for large-scale on-chip integration in future computing systems. Additionally, the capability of metasurfaces for multi-dimensional light modulation makes them ideal for constructing multi-channel, multi-functional computing devices. Several studies have demonstrated how multiplexing of various optical properties – such as wavelength, polarization, and angle of incidence [39, 40, 41, 42] can be harnessed to develop compact, parallel optical computing systems capable of performing mathematical operations such as differentiation and integration within a single element.

Although most DNNs focus on performing a single machine learning task, the ability to handle multiple tasks within a single DNN is crucial for advancing toward more generalized artificial intelligence devices that are both high-speed and energy-efficient, for applications such as autonomous driving and machine vision. Recently, significant steps have been taken toward implementing versatile, multi-functional DNNs. For instance, one study introduced a reconfigurable metasurface-based pluggable DNN capable of switching between two tasks by altering its pluggable components [43]. Another approach experimentally demonstrated an on-chip two-task optical classifier using birefringent nanostructures and a polarization multiplexing scheme, despite limited performance with a single-layer architecture [44]. Additionally, a numerical investigation explored multi-wavelength parallel image recognition of more than two tasks, utilizing a joint optimization approach to adjust the height map of diffractive optical elements [45]. However, achieving high parallel classification accuracy in this case requires at least five diffractive layers and a large number of modulation elements per layer, which makes the device bulky. In short, while these developments mark important progress toward versatile, multifunctional diffractive processors, the full potential of multiplexed metasurfaces has yet to be fully explored in terms of utilizing physical parametric degrees of freedom to implement compact, highly parallel multi-task DNNs.

In this work, we rigorously investigate the potential of both polarization-multiplexed and wavelength-multiplexed metasurfaces in realizing DNNs capable of simultaneously classifying multiple inputs. Using our meta-atom library, we initially design a dual-channel Polarization-Multiplexed DNN (PM-DNN) and a dual-channel Wavelength-Multiplexed-DNN (WM-DNN), to simultaneously classify the MNIST and Fashion-MNIST (FMNIST) databases with high classification accuracies. Extending this approach, we introduce a tri-channel WM-DNN to perform three tasks, MNIST, FMNIST, and Kuzushiji-MNIST (KMNIST), in parallel. Numerical results demonstrate satisfactory outcomes, despite a moderate decline in accuracy for the two more challenging tasks of FMNIST and KMNIST, primarily due to increased task competition. To further enhance the system performance, we develop a novel end-to-end design methodology to redesign the tri-channel WM-DNN. This framework utilizes surrogate models to map the complex transmission responses of the meta-atoms to their structural parameters, which are then used in a joint training framework to optimize the network parameters for all three tasks simultaneously. This approach not only improves classification accuracy but also has the potential to train multi-channel DNNs capable of handling a large number of tasks in parallel, thereby enabling massively parallel, multifunctional neural architectures.

Design

Figure 1(a) schematically shows the concept of metasurface-assisted Multiplexed DNNs (M-DNNs) for parallel optical classification. The system includes a multi-channel optical field with different targets encoded in specific light channels as the input layer, multiplexed metasurfaces as hidden layers, and a segmented detection plane for multi-channel detection functioning as the output layer. The two orthogonal polarization states x and y in the upper panel of Figure 1(a) and the three wavelengths λ1\lambda_1, λ2\lambda_2, and λ3\lambda_3 in the lower panel serve as independent channels (i.e., without any cross talk) for information processing. Adjusting the structural parameters of each meta-atom allows for spatially varying, channel-dependent transmission responses, enabling independent processing of the multi-input light. Each meta-atom in a designated polarization or wavelength state acts as an ‘optical neuron’, which is interconnected with the neurons in the subsequent layers through the physics of optical diffraction. According to the Rayleigh–Sommerfeld integral [46], the complex field of the (l+1)(l+1)th layer of the M-DNN can be expressed as:

Concept of metasurface-based M-DNN, doublet metasurface schematic, detection areas, and simulated transmission amplitude and phase

Figure 1. (a) Concept of metasurface-based M-DNN for performing multiple tasks. Top: Dual-channel PM-DNN. Bottom: Tri-channel WM-DNN. After the multi-input light, comprising various datasets and encoded in specific polarization (top panel) or wavelength (bottom panel) channels, passes through the doublet metasurface, it is focused onto the corresponding detection areas for each task’s class, enabling parallel recognition. (b) Schematic of the M-DNN featuring a doublet metasurface. The axial distance between the layers is set at 500 μm\mu\mathrm{m}. The metasurface consists of TiO2_2 rectangular nanopillars on a glass substrate. The height of each meta-atom hh is fixed at 600 nm, while the widths, wxw_x and wyw_y, vary from 60 nm to 350 nm; the periodicity of the unit cells, denoted as aa, is 400 nm. Each meta-atom exhibits a polarization- and wavelength-dependent transmission response. A supercell, consisting of a 3×33 \times3 array of identical meta-atoms, serves as our optical neuron. Their channel-dependent transmission responses are optimized through the training process to perform multiple classification tasks. (c) Designated detection areas corresponding to each task category for the dual-channel PM-DNN (top), dual-channel WM-DNN (middle), and tri-channel WM-DNN (bottom). (d-f) Simulated values of the transmission amplitude (∣Tx∣,∣Ty∣|T_x|, |T_y|) and phase (∠Tx,∠Ty\angle T_x, \angle T_y) for different geometries, under x- and y-polarized light. The incident wavelengths for (d), (e), and (f) are 450 nm, 550 nm, and 650 nm, respectively.

Ep,λl+1(x,y)=∬−∞+∞Ep,λl(x′,y′)Tp,λl(x′,y′)zr′2(12πr′−1jλ)exp⁡(j2πr′λ)dx′dy′(1)E_{p,\lambda}^{l+1}(x,y)=\iint_{-\infty}^{+\infty}E_{p,\lambda}^{l}(x',y')T_{p,\lambda}^{l}(x',y')\frac{z}{r'^{2}}\left(\frac{1}{2\pi r'}-\frac{1}{j\lambda}\right)\exp\left(j\frac{2\pi r'}{\lambda}\right)\mathrm{d}x'\mathrm{d}y' \tag*{(1)}

Where Ep,λl(x′,y′)E_{p,\lambda}^{l}(x',y') is the optical field irradiated to the llth layer, for the specific channel with polarization pp and wavelength λ\lambda. For the l=1l=1, Ep,λl(x′,y′)E_{p,\lambda}^{l}(x',y') represents the projected light of the target image. The term z=zl+1−zlz=z^{l+1}-z^l denotes the axial distance between successive layers, r′=(x−x′)2+(y−y′)2+z2r'=\sqrt{(x-x')^2+(y-y')^2+z^2} and j=−1j=\sqrt{-1}. Tp,λl(x′,y′)T_{p,\lambda}^{l}(x',y') is the spatially varying complex transmittance of the metasurface of the llth layer corresponding to the channel with polarization pp and wavelength λ\lambda, which is a function of the structural parameters of the meta-atoms. The multi-channel light is modulated by the metasurfaces, and the multi-dimensional transmission responses of the meta-atoms—i.e., their modulation coefficients—can be optimized to maximize the detected light intensity within the correct detection sub-region for each task, corresponding to its respective channel, thereby enabling optical multitask learning. It is important to note that while channel crosstalk does not occur as light diffracts through the air or substrate and is modulated by the metasurfaces, it does manifest at the detection plane due to the superposition of optical intensities. Employing polarization-selective and wavelength-selective filters in each sub-area can effectively mitigate crosstalk during intensity measurements across various channels, thereby enhancing the performance of the M-DNN. In this work, we assume the presence of such filters on the detectors.

To illustrate the capabilities of our M-DNN, we present several examples of multitask optical systems. As shown in Figure 1(b), these multiplexed metasurfaces consist of rectangular TiO2_2 nanofins on a glass substrate with a fixed height hh and two independently tunable widths wxw_x and wyw_y. When exposed to linearly polarized light, the asymmetric meta-units modulate the phase and amplitude of the incoming light in a polarization- and wavelength-dependent manner. By adjusting wxw_x and wyw_y, the desired transmission responses for each channel are achieved, facilitating parallel multitasking across various polarization and wavelength incidences. The phase and amplitude of the complex transmission responses of the meta-atoms are modeled using COMSOL Multiphysics software, which utilizes the Finite Element Method (FEM). The nanofins are set with a fixed height of 600 nm and a period of 400 nm. Figure 1(d-Figure 1f) display the computed electromagnetic response of each unit cell under x- and y-polarizations for specific wavelengths of 450 nm, 550 nm, and 650 nm, with the nanofins’ widths ranging from 60 nm to 350 nm in 5 nm increments.

All designs presented in this paper feature two hidden layers, i.e., multiplexed metasurfaces, each containing 210 × 210 optical neurons. Instead of utilizing a single unit cell, we intentionally adopted a supercell configuration consisting of a 3×33 \times3 array of identical meta-atoms, serving as our optical neuron, as depicted in Figure 1(b). This enlargement of the neuron size to 1.2 μm×1.2 μm1.2\ \mu\mathrm{m} \times1.2\ \mu\mathrm{m} facilitates the feasible experimental realization of the designs using commercially available spatial light modulators and CCD cameras, whose smallest pixel sizes are in the range of a few microns. Therefore, utilizing a supercell structure instead of a single cell allows us to maintain the desired pixel size for efficient light encoding and detection, while also benefiting from the light modulation capabilities of meta-atoms at shorter periodicities, specifically 400 nm. Another critical parameter in the design of a DNN is the distance between successive layers. Since 500 μ\mum is the most commonly available thickness for SiO2\mathrm{SiO}_2 wafers, we have fixed the axial distance between the layers at 500 μ\mum to support the ease of implementation in the future.

Designing M-DNNs using a meta-atom library

In this section, we verify the application of our metasurface-based M-DNNs in parallel multi-task classification utilizing our library of unit cell configurations—sourced from our simulations. We first design a dual-channel PM-DNN and a dual-channel WM-DNN to simultaneously classify two distinct datasets: the MNIST database (task I) and the Fashion-MNIST (FMNIST) database (task II), containing handwritten digits and fashion items, respectively. In the PM-DNN, MNIST and FMNIST data are encoded in x- and y-polarized light, respectively, both using a wavelength of 550 nm. In contrast, for the WM-DNN, the handwritten digits from task I are encoded at a wavelength of 450 nm, and the fashion products from task II at 550 nm, both under x-polarization. Figure 1(c) illustrates the detection planes for the dual-channel PM-DNN and WM-DNN, where specific sub-areas are designated for each category, with the upper and lower regions corresponding to tasks I and II, respectively.

Training is performed by individually training four single-task DNNs, each specifically designed for a distinct polarization and wavelength corresponding to the channels of our PM-DNN and WM-DNN. Each DNN is tasked with performing a specific classification task. The transmission amplitudes of the neurons are set to unity, and the transmission phases φ\varphi are the trainable parameters (phase-only DNNs). These phases are optimized via an error back-propagation algorithm to focus the output light intensity on the designated sub-area corresponding to each task's category. The top panels of Figure 2(a) and 3(a) display the phase distributions of the hidden layers obtained from the training sessions for the dual-channel PM-DNN and WM-DNN, respectively, designed to classify MNIST and FMNIST datasets. Subsequently, for each location on the metasurfaces, we search within our existing unit cell library to select the optimal meta-atom structure that most accurately replicates the desired local phase shifts for both tasks (More details on the selection process can be found in the Materials and Methods section). The lower panels of Figure 2(a) and 3(a) show the mismatch between the optimized phase shifts and those realized by the PM-DNN and WM-DNN, respectively, highlighting that the metasurfaces successfully reproduce the learned phase profiles with high fidelity. Numerical evaluations of the metasurface-based PM-DNN and WM-DNN using unseen data confirm the capabilities of both dual-channel classifiers, as they achieve classification results comparable to those of their individually trained single-task counterparts. The classification accuracies obtained are 97.72% for MNIST and 88.01% for FMNIST by the PM-DNN, and 97.19% for MNIST and 86.35% for FMNIST by the WM-DNN, respectively, as presented in Table 1.

Final phase distributions, detector intensity patterns, normalized energy distributions, and confusion matrices for PM-DNN classification

Figure 2. Figure 2(a) (Top) Final phase distributions of the two hidden layers of the dual-channel PM-DNN under x- and y-polarized incident light with the incident wavelength set to 550 nm. (Bottom) The absolute phase difference between the desired phase and the phase realized by the meta-atoms at each pixel. (b) Exemplary results of simultaneously classifying a handwritten digit and a fashion item encoded in the x- and y-polarized light, respectively. Output intensity patterns and normalized energy distributions across the category sub-areas show the success of the PM-DNN in parallel two-task categorization. (c) Confusion matrices for MNIST and FMNIST processed by the PM-DNN, demonstrating its performance across individual classes, with average classification accuracies of 97.72% and 88.01% for Task I and Task II, respectively.

DNN ModelDesign MethodTask I (MNIST)Task II (FMNIST)Task III (KMNIST)
Single-taskMulti-taskSingle-taskMulti-taskSingle-taskMulti-task
Dual-Channel PM-DNNMeta-atom library97.75%97.72%88.04%88.01%--
Dual-Channel WM-DNNMeta-atom library97.66%97.19%87.80%86.35%--
Tri-Channel WM-DNNMeta-atom library97.49%96.73%88.03%80.9%88.88%81.13%
Tri-Channel WM-DNNEnd-to-end method97.49%96.48%88.03%85.68%88.88%85.35%

Table 1. Accuracy of the M-DNNs

Figure 2(b) and 3(b) demonstrate the performance of these M-DNNs in parallel recognition of a handwritten digit and a fashion item, where the digit ‘8’ and the item ‘dress’ are processed by the PM-DNN, while ‘0’ and ‘trouser’ are processed by the WM-DNN. For both two-task classifiers, the detector intensity patterns, and normalized output energy distributions clearly indicate that the correct sub-regions corresponding to Task I and Task II receive the maximum signal. The confusion matrices, which statistically summarize the correct and incorrect identification results for all samples, are displayed in Figure 2(c) and 3(c) for Task I and Task II performed by the PM-DNN and WM-DNN, respectively. Numerical evaluations on the test datasets confirm that the dual-channel PM-DNN and WM-DNN successfully achieve parallel categorization of MNIST and FMNIST targets with high accuracy, reaching rates above 97% and 86%, respectively. While polarization multiplexing is limited to two orthogonal channels, wavelength multiplexing can support a large number of channels, thus allowing for higher-capacity computation. To assess the ability of metasurface-based M-DNNs to handle a greater number of tasks simultaneously, we construct a tri-channel WM-DNN to perform three-task classification: MNIST (Task I), FMNIST (Task II), and KMNIST, which contains 10 classes of kanji Japanese characters (Task III). The datasets for these tasks are encoded at wavelengths of 450 nm, 550 nm, and 650 nm, respectively, under x-polarized light. The bottom panel of Figure 1(c) shows the detection plane used

Final phase profiles, classification results, and confusion matrices for the dual-channel WM-DNN

Figure 3a. (Top) Final phase profiles of the two hidden layers of the dual-channel WM-DNN under x-polarized incident light at 450 nm (left) and 550 nm (right). (Bottom) The absolute phase difference between the desired phase and the phase realized by the metasurfaces at each pixel. (b) Exemplary results of simultaneously classifying a handwritten digit and a fashion item, encoded at wavelengths of 450 nm and 550 nm, respectively. Output intensity patterns and normalized energy distributions across the category sub-regions illustrate the capability of the WM-DNN in parallel two-task classification. (c) Confusion matrices for MNIST and FMNIST processed by the WM-DNN, presenting the classifier’s performance across individual classes, with average classification accuracies of 97.19% and 86.35% for Task I and Task II, respectively.

for the tri-channel WM-DNN, with specific sub-areas dedicated to specific tasks. Three individual single-task DNNs are trained at 450 nm, 550 nm, and 650 nm, tasked with recognizing MNIST, FMNIST, and KMNIST, respectively. The upper panel of Figure 4(a) displays the phase maps corresponding to the hidden layers of the three single-task DNNs for Tasks I to III, obtained from training. The tri-channel metasurface-based WM-DNN is realized by locally assigning meta-atoms from our library, each meticulously selected to meet the three-dimensional phase requirement at each pixel. The errors are more substantial than the dual-channel cases, particularly for the FMNIST and KMNIST tasks. This is expected as increasing the number of tasks exacerbates task competition, making it challenging for the meta-atoms to meet all the phase modulation requirements. The classification accuracies yielded from assessing the tri-channel WM-DNN on the MNIST, FMNIST, and KMNIST test datasets are 96.73%, 80.9%, and 81.13%, respectively.

Phase profiles, exemplary classification results, and confusion matrices for the tri-channel WM-DNN

Figure 4. (a) (Top) Final phase profiles of the two hidden layers of the tri-channel WM-DNN with x-polarized incident light at 450 nm (left), 550 nm (middle), and 650 nm (right). (b) Exemplary results of simultaneously classifying a handwritten digit, a fashion item, and a Kanji character, encoded at wavelengths of 450 nm, 550 nm, and 650 nm, respectively. (Bottom) The absolute phase error between the optimized phase and the phase realized by the metasurfaces at each pixel. Output intensity patterns and normalized energy distributions across the category sub-areas demonstrate the effectiveness of the WM-DNN in performing parallel recognition of three tasks. (c) Confusion matrices for MNIST, FMNIST, and KMNIST processed by the WM-DNN, summarizing the classifier’s performance across individual classes, with average classification accuracies of 96.73%, 80.9%, and 81.13%, for TaskI, Task II, and Task III, respectively.

As shown in Table 1, the identification accuracies for FMNIST and KMNIST are notably lower than those in their corresponding single-task DNNs, which aligns with the phase map results discussed earlier. Moreover, these two tasks are more challenging than MNIST and, therefore, more sensitive to errors.

Figure 4(b) depicts the performance of the tri-channel WM-DNN in parallel recognition, processing ‘2’ from MNIST, ankle boot’ from FMNIST, and ‘お’ from KMNIST. The detector intensity patterns and the normalized energy distributions of the sub-areas for different tasks indicate that the multi-wavelength incident light is successfully directed onto the correct sub-regions corresponding to each task. Figure 4(c) presents the overall identification results across all samples from the three test datasets through confusion matrices, revealing that the average performance across all classes is commendable.

It is worth discussing the extent of neuronal connectivity in our trained DNNs. Connectivity between layers is a crucial factor influencing the computational implementation complexity and the inference performance of the DNN. At a given wavelength, the interconnectivity between the neurons is dictated by the neuron size within each layer, which determines the diffraction angle, along with the axial distance between the layers. As it is analyzed in [22], for a DNN to be considered fully connected, the radius of the diffraction spot of each neuron RR should be larger than the side length of the diffractive layer. This radius is defined as: R=z×tan⁡φmax⁡R = z \times\tan\varphi_{\max}, where φmax⁡=sin⁡−1(λ/2d)\varphi_{\max} = \sin^{-1}(\lambda/2d), with φmax⁡\varphi_{\max} representing the half-maximum-diffraction angle, dd the neuron size and zz the axial distance between successive layers.

Accordingly, the diffraction radii for the neurons in our designed DNNs at wavelengths of 450 nm, 550 nm, and 650 nm are 140 μ\mum, 175 μ\mum, and 212 μ\mum, respectively. These values are smaller than the side length of the layers, which is 252 μ\mum, thus making our DNNs partially connected. Nevertheless, although our DNNs are not fully connected neural networks, the large coverage of the diffraction spots still provides sufficient connectivity for effective information processing. This is evidenced by the training results of our single-task DNNs (as presented in Table 1), which achieved accuracies nearly comparable to those of fully connected networks reported in the literature [10, 45]. Moreover, it is important to note that, in the design of a multiwavelength DNN, despite the fixed layer spacing and neuron size across different channels, channels with higher wavelengths exhibit greater connectivity due to larger diffraction angles, enhancing computational capability. As such, assigning more demanding tasks to channels with higher wavelengths is beneficial; for instance, we assigned MNIST, the least complex task, to our channel with the lowest wavelength.

Designing M-DNNs using an end-to-end approach

Lastly, we investigate an end-to-end design strategy to address issues encountered in the previous examples, with the imperfect phase map implementation using the meta-atom library approach. Unlike the previous approach, where the DNN for each task was optimized separately, we now incorporate a joint optimization framework to minimize classification errors across all three tasks simultaneously. By constructing surrogate models that relate the structural parameters of the meta-atoms to their channel-dependent complex transmission responses and integrating these models as proxy functions during training, we can directly optimize the structural parameters of the meta-atoms. Therefore, in contrast to the approach in the previous section, the trainable parameters of the diffractive networks here are the structural parameters of the meta-atoms, rather than their transmission responses. This approach eliminates the need for a further search step in the design process, thus addressing the issues caused by the imperfect phase map realization. In addition, it ensures that the training process accounts for the physical constraints of the unit cells, producing feasible and physically realizable complex modulation responses at each channel, thereby enhancing the performance of M-DNNs implemented by the metasurfaces.

As a final demonstration, we implement a tri-channel WM-DNN to simultaneously perform MNIST, FMNIST, and KMNIST tasks using similar wavelength channels and sub-area designations on the detection plane, as in the previous design. For the surrogate models that map the widths of the meta-atoms to their optical modulation coefficients at each wavelength, we utilize deep Artificial Neural Networks (ANNs) as differentiable proxy functions. The use of deep learning in nanophotonics is a well-established practice and has proven highly effective, particularly in inverse design, as evidenced by references [47, 48, 49, 50, 51, 52, 53].

The architecture used for the ANN models, depicted in Figure 5(a), consists of four hidden layers, each with 512 neurons and a ReLU activation function, along with two input neurons representing wxw_x and wyw_y, and three output neurons representing transmittance, and the cosine and sine of the phase shift. We design three models, f1(wx,wy)f_1(w_x,w_y), f2(wx,wy)f_2(w_x,w_y), and f3(wx,wy)f_3(w_x,w_y), using this architecture to approximate the transmission responses of the nanofins under the x-polarized light at wavelengths of 450 nm, 550 nm, and 650 nm, respectively. For each wavelength, 2784 cell instances—sourced from our previous COMSOL simulations—are used for training, while a separate set of 696 unseen cell instances is reserved for testing. The assessment of the unseen dataset shows that, while there are variations in performance across different models, they all exhibit low prediction errors, as presented in Table 2. As a representative example, Figure 5(b) shows the progress of training and test losses for the model f1(wx,wy)f_1(w_x,w_y) trained at 450 nm over 2500 epochs, reaching a mean squared error (MSE) of 0.047 on the test samples. Figure 5(c-Figure 5e) provide a visual comparison between the transmission responses of the meta-atoms derived from the COMSOL simulations and those predicted by the ANN models for all three wavelengths, verifying that the ANN models can serve as reliable approximation functions.

Architecture of the ANN models, training and test losses, COMSOL and ANN outputs, and amplitude and phase modulation profiles

Figure 5. (a) The architecture of the ANN models, which are used to map the phase and transmittance values of the meta-atoms to their structural parameters. (b) Training and test losses for the ANN model trained at 450 nm, over 2500 epochs, illustrating the network's convergence to final training and test losses of 0.0056 and 0.0471, respectively. (c-e) Comparison of COMSOL and ANN outputs for different under x-polarized light at 450 nm, 550 nm, and 650 nm. Data points from COMSOL simulations used for training are evaluated at 5 nm intervals, while the ANN outputs are at a resolution of 1 nm. (f-h) Final amplitude and phase modulation profiles for the two hidden layers of the tri-channel WM-DNN obtained through joint training, utilizing the pre-trained models from parts (c-e).

ANN ModelTraining Loss (MSE)Test Loss (MSE)Number of Epochs
f1(wx,wy)f_1(w_x,w_y)0.00560.04712500
f2(wx,wy)f_2(w_x,w_y)0.00190.02071500
f3(wx,wy)f_3(w_x,w_y)0.00370.00741500

Table 2. Training and Test Losses for ANN Models Used in the Inverse Design Process

Once trained, the ANN models have their network weights fixed and are then employed to design our tri-channel WM-DNN. In each iteration of training, the structural parameters of the meta-atoms are updated based on the back-propagation algorithm, with the models predicting their associated complex modulation coefficients, thereby facilitating the simultaneous training of multiple tasks. The complex transmittance maps of the hidden layers obtained from the joint training session are displayed in Figure 5(f, Figure 5g, and Figure 5h), respectively.

It’s worth noting that compared to previous cases where the networks were trained with a single objective, the learning process in this scenario faces a more complex optimization landscape, primarily due to the multi-objective joint training. Additionally, the modulation coefficient of the unit cell has a more intricate relationship with the trainable parameters wxw_x and wyw_y, represented as ∣fi∣exp⁡(i∠fi(wx,wy))\lvert f_i\rvert\exp\left(i\angle f_i(w_x,w_y)\right). This complexity is particularly pronounced at 450 nm, where resonances and fluctuations are most significant. In contrast, the previous phase-only scenario involved a simpler relationship, where the learnable parameter φ\varphi (i.e., the transmission phase) had a straightforward exponential relationship with the modulation coefficient, exp⁡(iφ)\exp(i\varphi), and the amplitude was set to unity. As a result, more time and computational resources are required for the joint network to converge. To facilitate better convergence of the joint training, we utilize a weighted sum loss function:

Total loss=w1∗lossMNIST+w2∗lossFMNIST+w3∗lossKMNIST(2)\mathit{Total\ loss}=w_1 * \mathit{loss}_{MNIST}+w_2 * \mathit{loss}_{FMNIST}+w_3 * \mathit{loss}_{KMNIST} \tag*{(2)}

We conduct multiple training sessions with various weight combinations, and the best results are achieved with coefficients w1w_1, w2w_2 and w3w_3 set at 0.13, 0.2, and 0.67, respectively. Utilizing a genetic algorithm to optimize these weights can further enhance the model’s performance, though it would involve higher computational costs.

Blind testing of the tri-channel WM-DNN trained with our end-to-end method demonstrates accuracies of 96.48% for MNIST, 85.68% for FMNIST, and 85.35% for KMNIST. Compared to the previous tri-channel WM-DNN designed using the meta-atom library approach, these results reproduce the high accuracy for MNIST and show improvements of 4.7% and 4.2% for FMNIST and KMNIST, respectively, highlighting the effectiveness of this method. Figure 6(a) illustrates the performance of the end-to-end designed tri-channel WM-DNN in simultaneously classifying ‘1’, ‘trouser’, and ‘ց’ from the MNIST, FMNIST, and KMNIST datasets, respectively. The system successfully predicts the correct category for each task. The confusion matrices, shown in Figure 6(b), display the network's performance across all individual classes for all test samples in the three tasks. A comparison with the confusion matrices in Figure 4(c) reveals an increase in the total number of correct identifications, confirming the improvement in average accuracies. For instance, in the KMNIST dataset, the number of correct identifications for class 4—the worst-performing class—increased from 561 to 842, demonstrating a clear improvement in the recognition of KMNIST images.

Simultaneous three-task classification results and confusion matrices

Figure 6. (a) Exemplary results of simultaneously classifying a handwritten digit (left), a fashion item (middle), and a Kanji character (right), encoded at wavelengths of 450 nm, 550 nm, and 650 nm, respectively, and processed by the jointly trained tri-channel WM-DNN. Output intensity patterns and normalized energy distributions across the category sub-areas confirm the capability of the WM-DNN for simultaneous three-task classification. (b) Confusion matrices for MNIST, FMNIST, and KMNIST processed by the WM-DNN, summarizing the classifier’s performance across individual classes, with average classification accuracies of 96.48%, 85.68%, and 85.35%, for Task I, Task II, and Task III, respectively, showing improvement compared to the previous WM-DNN design with the meta-atom library approach.

Conclusion

In this work, we have demonstrated the potential of wavelength and polarization multiplexing schemes to facilitate all-optical multi-task learning using DNNs with bilayer cascaded metasurfaces. We design dual-channel PM-DNN and WM-DNN, utilizing polarization and wavelength multiplexing, respectively, using our meta-atom library. Numerical results for both systems in parallel processing of two tasks show high accuracies comparable to their individually trained single-task counterparts, thereby validating the effectiveness of our dual-channel M-DNN design methodology. Additionally, we explore the implementation of a tri-channel WM-DNN to perform three classification tasks in parallel, using two different design strategies: the meta-atom library approach and the end-to-end joint training framework. While the meta-atom library approach achieves high accuracy for the MNIST task, it exhibits a moderate decline in performance for the FMNIST and KMNIST tasks. This decline can be attributed to the increased number of tasks, which makes finding meta-atoms that satisfy all the local modulation coefficient requirements more challenging.

On the other hand, the end-to-end joint training framework uses approximate models to map the structural parameters of the unit cells to their complex transmission responses for each channel, ensuring that the obtained modulation coefficients are realizable by the meta-atoms, thereby enhancing the multi-tasking performance of the network. However, despite the improvements, the performance of the M-DNN still falls short when compared to the individually trained DNNs for the respective single tasks. We anticipate that performance could be further improved by incorporating more complex meta-atom geometries, increasing the number of layers, and applying hyperparameter tuning, possibly using a genetic algorithm, to the joint training. Finally, it is worth noting that the significance of the end-to-end training approach becomes increasingly apparent as the number of parallel tasks grows, offering considerable advantages for designing highly parallel optical machine learning systems capable of handling numerous tasks simultaneously. Such systems could be further utilized in high-throughput computational imaging, real-time data processing, and autonomous systems.

Materials and Methods

Training of the Neural Networks

All neural networks, including DNNs and ANNs, were implemented using Python version 3.10 and TensorFlow framework version 2.16.1 on Google Colab Pro, equipped with a V100 NVIDIA GPU and 32 GB of RAM. We used the MNIST, FMNIST, and KMNIST datasets, with each image—originally a 28×2828 \times28 grayscale image—zero-padded to 70×7070 \times70 and then resized to 210×210210 \times210. These images were encoded in the amplitude of light for the corresponding channels. Each dataset contains 60,000 training samples and 10,000 testing samples across 10 classes. The Adam optimizer was used to train all networks, with an exponential decay learning rate schedule starting at 0.001 and a decay rate of 0.9 applied every 10,000 steps, using a batch size of 32. A cross-entropy loss function was employed to maximize light intensity in the target region for the DNN designs.

Meta-atom Search and Selection Process

As described in the paper, the proposed M-DNNs outlined in § 3 were implemented through a comprehensive search within our unit cell library. Specifically, to select the optimal meta-atom at each location, we first excluded meta-atoms with transmission amplitudes below 0.5. We then applied a weighted sum of the errors between the target phases and the phases obtained from the meta-atoms across all tasks as our search criterion, selecting the meta-atoms that minimized this total error. Utilizing the weighted sum error as the search metric enables us to prioritize the more challenging and sensitive tasks, thereby enhancing the overall multitasking performance. We conducted several iterations of searches with varying weight combinations to find the optimal set of weights that achieves accuracies greater than 80% for all tasks, rather than maximizing the accuracy for one task at the expense of weaker performance in the others. The optimal error weights for the metasurface implementation of the dual-channel PM-DNN were identified as 1 and 1.2 for Task I and Task II, respectively, while for the dual-channel WM-DNN, weights of 1.2 and 2 for Task I and Task II yielded the desired accuracies across both tasks. Lastly, for the tri-channel WM-DNN, the optimal weights were determined to be 0.8, 1.28, and 1.23 for Task I, Task II, and Task III, respectively.

Acknowledgment

This work is supported by the National Science Foundation Award number ECCS-2240448.

References

  1. [1]1-Abu-Mostafa, Yaser S., and Demetri Psaltis. "Optical neural computers." Scientific American 256.3 (1987): 88-95.
  2. [2]2-Hamerly, Ryan, et al. "Large-scale optical neural networks based on photoelectric multiplication." Physical Review X 9.2 (2019): 021032.
  3. [3]3-Farhat, Nabil H., et al. "Optical implementation of the Hopfield model." Applied optics 24.10 (1985): 1469-1475.pubmed.ncbi.nlm.nih.gov/18223740
  4. [4]4-Tait, Alexander N., et al. "Silicon photonic modulator neuron." Physical Review Applied 11.6 (2019): 064043.DOI
  5. [5]5-Tait, Alexander N., et al. "Neuromorphic photonic networks using silicon photonic weight banks." Scientific reports 7.1 (2017): 7430.DOI
  6. [6]6-Feldmann, Johannes, et al. "All-optical spiking neurosynaptic networks with self-learning capabilities." Nature 569.7755 (2019): 208-214.DOI
  7. [7]7-Pierangeli, D., et al. "Deep optical neural network by living tumour brain cells." arXiv preprint arXiv:1812.09311 (2018).arxiv.org/abs/1812.09311
  8. [8]8-Cheng, Tian-You, et al. "Optical neural networks based on optical fiber-communication system." Neurocomputing 364 (2019): 239-244.DOI
  9. [9]9-Ashtiani, Farshid, Alexander J. Geers, and Firooz Aflatouni. "An on-chip photonic deep neural network for image classification." Nature 606.7914 (2022): 501-506.DOI
  10. [10]10-Lin, Xing, et al. "All-optical machine learning using diffractive deep neural networks." Science 361.6406 (2018): 1004-1008.DOI
  11. [11]11-Zhao, Qingsong, et al. "Orbital angular momentum detection based on diffractive deep neural network." Optics Communications 443 (2019): 245-249.DOI
  12. [12]12-Zhou, Tiank uang, et al. "Large-scale neuromorphic optoelectronic computing with a reconfigurable diffractive processing unit." Nature Photonics 15.5 (2021): 367-373.DOI
  13. [13]13-Yan, Tao, et al. "Fourier-space diffractive deep neural network." Physical review letters 123.2 (2019): 023901.DOI
  14. [14]14-Rahman, Md Sadman Sakib, et al. "Ensemble learning of diffractive optical networks." Light: Science & Applications 10.1 (2021): 14.DOI
  15. [15]15-Chang, Julie, et al. "Hybrid optical-electronic convolutional neural networks with optimized diffractive optics for image classification." Scientific reports 8.1 (2018): 1-10.DOI
  16. [16]16-Shi, Jiashuo, et al. "Broad-spectrum diffractive network via ensemble learning." Optics Letters 47.3 (2022): 605-608.DOI
  17. [17]17-Veli, Muhammed, et al. "Terahertz pulse shaping using diffractive surfaces." Nature Communications 12.1 (2021): 37.
  18. [18]18-Huang, Zebin, et al. "All-optical signal processing of vortex beams with diffractive deep neural networks." Physical Review Applied 15.1 (2021): 014037.DOI
  19. [19]19-Dou, Hongkun, et al. "Residual D 2 NN: training diffractive deep neural networks via learnable light shortcuts." Optics Letters 45.10 (2020): 2688-2691.
  20. [20]20-Li, Yiming, et al. "Multiscale diffractive U-Net: a robust all-optical deep learning framework modeled with sampling and skip connections." Optics Express 30.20 (2022): 36700-36710.DOI
  21. [21]21-Lee, Seokho, Cherry Park, and Junsuk Rho. "Mapping information and light: Trends of AI-enabled metaphotonics." Current Opinion in Solid State and Materials Science 29 (2024): 101144.DOI
  22. [22]22-Chen, Hang, et al. "Diffractive deep neural networks at visible wavelengths." Engineering 7.10 (2021): 1483-1491.DOI
  23. [23]23-Yu, Nanfang, et al. "Light propagation with phase discontinuities: generalized laws of reflection and refraction." science 334.6054 (2011): 333-337.DOI
  24. [24]24-Yu, Nanfang, and Federico Capasso. "Flat optics with designer metasurfaces." Nature materials 13.2 (2014): 139-150.DOI
  25. [25]25-Grady, Nathaniel K., et al. "Terahertz metamaterials for linear polarization conversion and anomalous refraction." Science 340.6138 (2013): 1304-1307.DOI
  26. [26]26-Chang, Shengyuan, Xuexue Guo, and Xingjie Ni. "Optical metasurfaces: progress and applications." Annual Review of Materials Research 48.1 (2018): 279-302.DOI
  27. [27]27-Overvig, Adam C., et al. "Dielectric metasurfaces for complete and independent control of the optical amplitude and phase." Light: Science & Applications 8.1 (2019): 92.DOI
  28. [28]28-Khorasaninejad, Mohammadreza, et al. "Metalenses at visible wavelengths: Diffraction-limited focusing and subwavelength resolution imaging." Science 352.6290 (2016): 1190-1194.DOI
  29. [29]29-Lin, Ren Jie, et al. "Achromatic metalens array for full-colour light-field imaging." Nature nanotechnology 14.3 (2019): 227-231.DOI
  30. [30]30-So, Sunae, et al. "Multicolor and 3D holography generated by inverse-designed single-cell metasurfaces." Advanced Materials 35.17 (2023): 2208520.DOI
  31. [31]31-Kim, Joohoon, et al. "Dynamic Hyperspectral Holography Enabled by Inverse-Designed Metasurfaces with Oblique Helicoidal Cholesterics." Advanced Materials (2024): 2311785.DOI
  32. [32]32-Liu, Lei, et al. "Terahertz polarization sensing based on metasurface microsensor display anti-proliferation of tumor cells with aspirin." Biomedical optics express 11.5 (2020): 2416-2430.DOI
  33. [33]33-Gao, Jun, et al. "Superabsorbing Metasurfaces with Hybrid Ag–Au Nanostructures for Surface-Enhanced Raman Spectroscopy Sensing of Drugs and Chemicals." Small Methods 2.7 (2018): 1800045.DOI
  34. [34]34-Dai, Jun Yan, et al. "Wireless communication based on information metasurfaces." IEEE Transactions on Microwave Theory and Techniques 69.3 (2021): 1493-1510.DOI
  35. [35]35-Nie, Shuai, and Ian F. Akyildiz. "Metasurfaces for multiplexed communication." Nature Electronics 4.3 (2021): 177-178.DOI
  36. [36]36- Badloe, Trevon, Seokho Lee, and Junsuk Rho. "Computation at the speed of light: metamaterials for all-optical calculations and neural networks." Advanced Photonics 4.6 (2022): 064002-064002.DOI
  37. [37]37- Solntsev, Alexander S., Girish S. Agarwal, and Yuri S. Kivshar. "Metasurfaces for quantum photonics." Nature Photonics 15.5 (2021): 327-336.DOI
  38. [38]38-Li, Quanwei, et al. "A non-unitary metasurface enables continuous control of quantum photon–photon interactions from bosonic to fermionic." Nature Photonics 15.4 (2021): 267-271.DOI
  39. [39]39-Sol, J., Smith, D. R. & del Hougne, P. Meta-programmable analog differentiator. Nat. Commun. 13, 1713 (2022)DOI
  40. [40]40-Camacho, M., Edwards, B. & Engheta, N. A single inverse-designed photonic structure that performs parallel computing. Nat. Commun. 12, 1466 (2021) )
  41. [41]41-Momeni, A. et al. Generalized optical signal processing based on multioperator metasurfaces synthesized by susceptibility tensors. Phys. Rev. Appl. 11, 064042 (2019) )DOI
  42. [42]42-Fang, Yisheng, and Zhichao Ruan. "Optical spatial differentiator for a synthetic three-dimensional optical field." Optics letters 43.23 (2018): 5893-5896.DOI
  43. [43]43-He, Cong, et al. "Pluggable multitask diffractive neural networks based on cascaded metasurfaces." Opto-Electron Adv 7 (2024): 230005.DOI
  44. [44]44-Luo, Xuhao, et al. "Metasurface-enabled on-chip multiplexed diffractive neural networks in the visible." Light: Science & Applications 11.1 (2022): 158.DOI
  45. [45]45-Duan, Zhengyang, Hang Chen, and Xing Lin. "Optical multi-task learning using multi-wavelength diffractive deep neural networks." Nanophotonics 12.5 (2023): 893-903.DOI
  46. [46]46-Goodman, Joseph W. Introduction to Fourier optics. Roberts and Company publishers, 2005.
  47. [47]47-Cheng, Jierong, et al. "Inverse design of generic metasurfaces for multifunctional wavefront shaping based on deep neural networks." Optics & Laser Technology 159 (2023): 100938.DOI
  48. [48]48-Jiang, Li, et al. "Neural network enabled metasurface design for phase manipulation." Optics Express 29.2 (2021): 2521-2528.DOI
  49. [49]49-Ghorbani, Fardin, et al. "Deep neural network-based automatic metasurface design with a wide frequency range." Scientific Reports 11.1 (2021): 7102.arxiv.org/abs/2101.10866
  50. [50]50-Liu, Changhao, et al. "Deep-learning-empowered inverse design for freeform reconfigurable metasurfaces." arXiv preprint arXiv:2211.08296 (2022).arxiv.org/abs/2211.08296
  51. [51]51-Nadell, Christian C., et al. "Deep learning for accelerated all-dielectric metasurface design." Optics express 27.20 (2019): 27523-27535.DOI
  52. [52]52-Jing, Guoqing, et al. "Neural network-based surrogate model for inverse design of metasurfaces." Photonics Research 10.6 (2022): 1462-1471.DOI
  53. [53]53-Hazineh, Dean, et al. "Polarization multi-image synthesis with birefringent metasurfaces." 2023 IEEE International Conference on Computational Photography (ICCP). IEEE, 2023.

Paper details

vlnc:W_5q6g7fvg
Version
v1 · latest
Published
Fresh — added to the Hub within the last 30 days
Added
License
CC BY 4.0
Usage

views · 0

downloads · 0

A view counts once per browser per day, a download each time the PDF is requested.

  • Physics and Astronomy

Not recorded for this paper

View statement
Looking for related papers…
Contents