betaFeedback

WorksPhysics and Astronomy

High performance artificial visual system with plasmon-enhanced 2D material neural network

Abstract

Artificial visual systems (AVS) have gained tremendous momentum because of its huge potential in areas such as autonomous vehicles and robotics as part of artificial intelligence (AI) in recent years. However, current machine visual systems composed of complex circuits based on complementary metal oxide semiconductor (CMOS) platform usually contains photosensor array, format conversion, memory and processing module. The large amount of redundant data shuttling between each unit, resulting in large latency and high power consumption, which greatly limits the performance of the AVS. Here, we demonstrate an AVS based on a new design concept, which consists of hardware devices connected in an artificial neural network (ANN) that can simultaneously sense, pre-process and recognize optical images without latency. The Ag nanograting and the two-dimensional (2D) heterostructure integrated plasmonic phototransistor array (PPTA) constitute the hardware ANN, and its synaptic weight is determined by the adjustable regularized photoresponsivity matrix. The eye-inspired pre-processing function of the device under photoelectric synergy ensures the considerable improvement of the efficiency and accuracy of subsequent image recognition. The comprehensive performance of the proof-of-concept device demonstrates great potential for machine vision applications in terms of large dynamic range (180 dB), high speed (500 ns) and ultralow energy consumption per spike (2.4e(-17) J).

demonstrates great potential for machine vision applications in terms of large dynamic range (180 dB), high speed (500 ns) and ultralow energy consumption per spike (2.4×10−172.4 \times10^{-17} J).

Introduction

The human visual system is mainly composed of the eyes and the visual cortex of the brain[1, 2]. The retina of the eye is normally used to capture external optical information and perform first-stage image pre-processing[3, 4, 5]. The regulated visual signals are transmitted to the neural network of the visual center for final processing and recognition[6, 7]. Accordingly, a variety of bio-inspired AVS for emulating certain functions of the human eye and neural network image processing have emerged that are used to perform typical image processing functionalities, which include image-contrast enhancement[1, 2, 8, 9], noise suppression[10, 11], visual adaptation[5, 12], detection and recognition[13, 14, 15, 16, 17, 18], and auto-encoding[19]. However, for current AVS, a hardware solution with both the pre-processing function of the human retina and the image recognition capability of the visual cortex has not been reported, especially in time-critical applications[18, 19]. There is a high demand to develop multifunctional electronic devices to meet the challenges of next generation machine vision. Additionally, developing low-power and high-efficiency AVS has become a major research focus, where the most critical issue to be addressed is the efficient conversion of optical images into electrical digital signals.

Plasmonic energy conversion has been considered as a promising alternative to drive a wide range of physical and chemical processes[20]. This emerging method is based on the generation of hot electrons with energy distribution deviating substantially from equilibrium Fermi-Dirac distribution in plasmonic nanostructures after light absorption through non-radiative electromagnetic decay of surface plasmons[21, 22, 23, 24, 25]. while the 2D semiconductor itself has excellent optoelectronic properties[28, 29, 30] such as ultrafast response[31, 32], external tunability[19, 33] and large photothermoelectric effect[34] plasmonics can further enable strong light-matter interactions in 2D materials[26, 27]. 2D materials technology has by now achieved a sufficiently high level of maturity for integration with conventional complex electronic systems[35, 36, 38]. Herein, we present a PPTA constructed of nanogratings and 2D heterostructures, which constitutes an ANN that integrates simultaneous sensing, pre-processing and image recognition functions. The plasmonic phototransistor (PPT) takes advantage of the strong coupling of photonic and electronic resonances in an elaborately designed device, in which hot electrons are injected efficiently into the floating gate and produce a large photoelectric effect, to simulate the response of the human retina to optical color information. Moreover, the electrical dynamic modulation of the gate electrode can effectively enlarge the dynamic range of the device for image pre-processing functions (image contrast enhancement). Further real-time image recognition is realized by training the network through varying the drain-source voltage to set the photoresponsivity value of each pixel individually. As a result, the AVS integrated with image pre-processing and ANN can effectively improve the image quality, and increase the efficiency and the accuracy of image recognition.

Results

fig:1a illustrates the schematic structure of a 2D PPT, which consists of a 2D MoS2_2/Ag nanograting integrated structure on the left and a 2D MoS2_2/h-BN/WSe2_2 heterostructure on the right. The left part of the device mimics the sensing and pre-processing functions of the human retina for color information (Extended Data Figure 1) using light-excited waveguide-plasmon polaritons (WPPs)[37] and electrical modulation of the gate electrode, respectively (see Figure 2 for more details of the mechanism). The photocurrent signal processed in the first stage can be passed to the floating gate on the right side of the device to induce the channel current, which is similar to that visual information can be transmitted through the optical nerve to each neuron in the visual center via synaptic interconnection (Extended Data Figure 1, Extended Data Figure 1). The photoresponsivity (synaptic weight) of the device is modulated by changing the drain-source voltage to emulate the regulation of neurotransmitter release between biological synapses (Extended Data Figure 1). To avoid unnecessary direct photocurrents in the channel, the right side is covered by the Al2_2O3_3/Au layer. Interconnecting each 2D PPT (subpixel) in the form of an ANN constitutes an AVS with image sensing, pre-processing and recognition functions (Fig. 1b). It contains NN pixels, which form the imaging array, and each pixel is divided into MM subpixels. The circuit connections of MM subpixels and NN pixels are presented in Figs. 1c,d, respectively. Each subpixel delivers a photocurrent of Imn=RmnPnI_{mn}=R_{mn}P_n under illumination, where RmnR_{mn} is the regularized photoresponsivity of the subpixel and PnP_n denotes the optical power at the nth pixel. n=1,2,…,Nn=1, 2, \ldots, N and m=1,2,…,Mm=1, 2, \ldots, M denote the pixel and subpixel indices, respectively.

Schematic illustrations of the mechanisms of 2D PPT

Figure 2. Schematic illustrations of the mechanisms of 2D PPT. a, A diagrammatic model proposed to describe the whole physical process of the strong coupling between the LSPR mode and the waveguide mode and its relaxation. b, Simplified band diagram illustrating the hot electron injection process taking place at the Ag-MoS2\mathrm{MoS}_{2} interface. In addition to receiving hot electrons emitted from the Ag nanograting, MoS2\mathrm{MoS}_{2} itself can also generate a small amount of electrons after receiving light. c, The simulated transmittance spectrum of the Ag nanograting-ITO waveguide integrated structure dependent on the grating period, showing the classic Rabi splitting. d, The calculated electric field distribution at the 320 nm grating period corresponding to each branch at red (R), green (G) and blue (B) wavelengths in the strong coupling regime. Scale bar, 180 nm. e-h, Charge-flow illustrations and schematic band diagrams at different operation modes: light on (e), light off (f), light on and apply −VG-V_{\mathrm{G}} (g), light on and apply +VG+V_{\mathrm{G}} (h). The blue balls denote the holes, the magenta balls denote the electrons, and the magenta arrows indicate the flow direction of the electrons. ETE_{\mathrm{T}} represents the thermoelectric potential, EAE_{\mathrm{A}} represents the accumulated

Schematics of the human visual system

Extended Data Figure 1. Schematics of the human visual system. a, Illustration of an neuromorphic visual system. A biological visual system consisting of the retina (sensing and pre-processing), optic nerve (transducing) and the visual cortex (image recognition). The inset shows a magnified view of the retina organized in a hierarchical way. Photoreceptors (cone and rod cells) are used to sense external color optical information, and the information can be pre-processed through ON and OFF bipolar cells[8]. b, Schematic diagram of a neural network formed by synaptic connections between neurons. c, Diagram of neurotransmitter transmission between synapses.

Schematic of the device architecture, circuit implementation, neural network, microscopy image, and elemental mapping

Figure 1.

Image not extracted

Figure 1 (PDF p. 5). AVS-inspired 2D ANN PPTA. a, Schematic of a 2D PPT. b, Disassembled diagram of the 2D ANN PPTA. The current induced by subpixels of the same color in WSe2_2 channel layer is connected in parallel by wires of the same color to generate an output current IMI_M. c, d, Circuit diagram of the nnth pixel (c) and MϵNϵM^{\epsilon}N^{\epsilon} subpixels (d) in the array, where MϵM^{\epsilon} is a subset of MM, representing a certain number of subpixels among MM subpixels with the same MM indice, and NϵN^{\epsilon} is a subset of NN, representing a certain number of pixels among NN pixels. e, Illustration of an AVS based on the 2D PPT for image pre-processing and an ANN for image recognition. f, Scanning electron microscopy (SEM) image of the PPTA. Scale bar, 20 μm. GND, ground electrode. g, High-resolution scanning transmission electron microscope image captured from the black box in (f) and energy dispersive X-ray spectroscopy mapping. Scales bar are 8 (left) and 40 nm (right), respectively.

The schematic of a classifier is provided in Fig. 1e. The array is operated as a single-layer perceptron using pre-processed visual information as the input layer. Here, we chose the softmax function

ϕm(I)=eImξ/∑k=1MeIkξ\phi_m(I)=e^{I_m\xi}/\sum_{k=1}^{M}e^{I_k\xi}

as the nonlinear activation function to generate the neuron output off-chip, where ξ=1011 A−1\xi=10^{11}\,\mathrm{A}^{-1} is a scaling factor. In one type of ANN representing a supervised learning algorithm, in order to facilitate the classification of images P into different categories y, we chose a binary code encoding, where each of the three letters corresponds to an output code. Following the elaborated design concept of the 2D PPTA, we fabricated the actual device as shown in Fig. 1f. The sample fabrication process is provided in Extended Data Figure 2 (for details, see Methods). This device consists of 27 subpixels (N×M=27N \times M=27), of which every 9 subpixels were arranged to form a 3×33 \times3 imaging array (N=9N=9) with a subpixel size of about 17×5 μm217 \times5\ \mu\mathrm{m}^{2}. A schematic of the entire circuit connections of the array is presented in Extended Data Figure 3. Summing all photocurrents generated by 9 PPTs with the same subpixel index mm according to Kirchhoff’s law, the output ImI_m is expressed as

Schematic fabrication process of the device

Extended Data Figure 2. Schematic fabrication process of the device. a, Sputtering a layer of indium tin oxide (ITO) on SiO2\mathrm{SiO}_{2} substrate by magnetron sputtering. b, Growth of a thin film of Al2O3\mathrm{Al}_{2}\mathrm{O}_{3} using atomic layer deposition technology. c, An MoS2\mathrm{MoS}_{2} flake is first transferred to a transparent PDMS film by mechanical exfoliation method. d, e, After the PDMS is lifted off, the MoS2\mathrm{MoS}_{2} is left on the substrate just like stamping a pattern with a stamp. f, Fabrication of Ag nanograting on MoS2\mathrm{MoS}_{2} flake using EBL through overlay process. g, RIE of the MoS2\mathrm{MoS}_{2} flake using a mask made of EBL. h, i, j, Using the same method, an h-BN flake is transferred on the MoS2\mathrm{MoS}_{2} flake. k, l, m, With the same method, a WSe2\mathrm{WSe}_{2} flake is transferred on the h-BN/MoS2\mathrm{MoS}_{2} flake. n, RIE of h-BN/WSe2\mathrm{WSe}_{2} flakes using an EBL mask exposes the Ag nanograting to air. o, EBL is used

Circuit of the ANN PPTA

Extended Data Figure 3. Circuit of the ANN PPTA.

Im=∑n=1NImn=∑n=1NRmnPn(1)I_m=\sum_{n=1}^{N}I_{mn}=\sum_{n=1}^{N}R_{mn}P_n \tag*{(1)}

Figure 1g shows the high-resolution scanning transmission electron microscope and energy dispersive X-ray spectroscopy element mapping characterizations of a single subpixel in the black box in Fig. 1f, indicating a clean heterostructure interface.

potential, and EGE_G represents the gate potential. The black arrows represent the direction of each potential.

The black dotted arrow represents the direction of the electron transition.

In order to understand the mechanism of 2D PPT, we present a scenario for elaboration below. As shown in Figure 2a, following light absorption and localized surface plasmon resonance (LSPR) excitation in the Ag nanograting, the electromagnetic resonance can be damped radiatively by re-emission of photons, or non-radiatively through transferring the energy to hot electrons via Landau damping[20, 21]. In the subsequent hot electron injection[24] (Figure 2b), hot electrons with momentum within the escape cone[23] can be rapidly emitted into MoS2_2 through ohmic contacts during the relaxation time[22, 27]. At the same time, 2D MoS2_2 itself also produces a fraction of energetic hot electrons after absorbing light energy, although the effect of this fraction is minimal (see Extended Data Fig. 4a-c). Figure 2c shows the simulated normalized transmittance mapping of the grating period from 250 to 450 nm in the visible region (for details, see Methods), where Rabi splitting can be clearly observed as a distinguishing characteristic of the strong coupling. It is worth mentioning that the upper, middle and lower three hybrid branches are caused by the coupling of the symmetric and antisymmetric modes in the waveguide with the LSPR mode, respectively, and the bottom two branches are caused by the presence of the mode in the quartz substrate, which is independent of the strong coupling modes. We choose the three eigenenergies corresponding to the red (632 nm), green (535 nm) and blue light (469 nm) when the grating period is 320 nm as the eigenvalues of the three-coupled oscillator model to analyze the strong coupling of this structure. The obtained Rabi splitting (Ω≈680\Omega\approx680 meV) is satisfied with the strong coupling criterion between these three oscillators, that is, Ω>W∑i=Pl, Sym, AsymPiγi\Omega> \mathbf{W}\sum_{i=\mathrm{Pl},\ \mathrm{Sym},\ \mathrm{Asym}}\mathbf{P}^{i}\gamma_i, where W=(WUpper,WMiddle,WLower)\mathbf{W} = (W_{Upper}, W_{Middle}, W_{Lower}) are the weight of each hybrid branch, and Pi=(PUpperi,PMiddlei,PLoweri)\mathbf{P}^{i} = (P^{i}_{Upper}, P^{i}_{Middle}, P^{i}_{Lower}) represents the proportion of uncoupled states in each branch, and γi\gamma_i represents the linewidth of each uncoupled mode. (for details, see Methods). The electric field distribution corresponding to the eigenenergy of different branches at the period 320nm is provided in Figure 2d. It can be clearly found that the coupling between LSPR mode and waveguide mode leads to energy exchange. The above mechanism suggests that the 2D PPT can respond to optical color information, and it is also the first time that the splitting of three absorption peaks in the visible range has been achieved compared to previous studies[27, 37]. Thus, by exploiting the hybrid LSPR and waveguide modes, we realize highly efficient photoelectric conversion, while the limitation on the narrow responding wavelength of LSPR could be surmounted by adjusting the dimension of the Ag nanograting structure.

On the other hand, the hot electrons that can not be emitted from the decay of plasmons can generate enormous heat on the picosecond scale, which leads to a balance between thermoelectric potential ETE_{\mathrm{T}} (Extended Data Fig. 4d-f) and the accumulated electropotential EAE_{\mathrm{A}} as shown in Figure 2e,Figure 2f[34]. With such mechanism, the device can respond to different luminance (gray scale of image). When the light is turned on and the negative side gate voltage −VG-V_{\mathrm{G}} is applied, the electrons will be more easily transferred from the left side of MoS2_2 to the right side, as there is an additional gate potential EGE_{\mathrm{G}} (Figure 2g). Accordingly, the larger channel current will be induced by the floating gate. Conversely, by applying a positive gate voltage +VG+V_{\mathrm{G}} while the light is turned on, the electrons will be dragged to the left side because of the additional gate potential EGE_{\mathrm{G}} (Figure 2h). The holes left on the right side of the floating gate lead to electron doping to the channel, which gives low conductance since WSe2_2 is a p type semiconductor. The mechanism of the device described in Figure 2g,Figure 2h can be used to eliminate the redundant information. Finally, the regulation of the photoresponsivity of a single device can be realized by changing the drain-source voltage, which can be used to train the weights in the ANN formed by interconnected devices.

Having described the design concept of AVS, we next present its feasibility from an experimental perspective. The optical experimental setup is shown in Extended Data Fig. 5a,b and the electrical experimental setup is shown in Extended Data Figure 6 (for details, see Methods). Here we choose the red light of λ=635\lambda= 635 nm, and its power (0-10 μ\muW) is divided into 11 orders. Figure 3a presents the multi-state photocurrents corresponding to different levels of optical power. These photocurrents are graphically visualised as 11 grey levels in the normalized 0-1 interval. By measuring the photocurrent corresponding to three

Functional implementation of 2D PPT

Figure 3. Functional implementation of 2D PPT. a, Multi-state photocurrents corresponding to different levels of optical power (grey levels), where the laser wavelength is 635 nm and the drain-source voltage is 0.1 V. b, Photocurrent of different colors of light (R: 635nm, G: 532nm, B: 473nm) under the same measurement conditions, where the power is 10 μ\muW and the drain-source voltage is 0.1 V. c, Experimentally measured normalized transmittance spectra of the WPPs structure on the left side of the device. d, IPHI_{\mathrm{PH}}-VDSV_{\mathrm{DS}} curves at different optical powers without any applied gate voltage. e, Voltage tunability of the regularized photoresponsivity. The inset shows IPHI_{\mathrm{PH}} versus PP for different VDSV_{\mathrm{DS}} values. f, The voltage (VDSV_{\mathrm{DS}}) tunable photocurrent corresponding to each gray scale. g-i, The transfer characteristic curves of the devices with red (g), green (h) and blue (i) light measured under different PP values at VDS=1V_{\mathrm{DS}}=1 V, respectively.

Experimental setup of an AVS classifier, time-resolved measurements, and image-processing flow chart

Extended Data Figure 6. Experimental setup. a, Experimental setup of an AVS used as a classifier. CW, continuous wave. Optical setup is shown in Extended Data Fig. 5. b, Experimental setup for time-resolved measurements. c, Flow chart of AVS used for image processing. The blue shaded boxes indicate the ANN PPTA in action.

wavelengths of light at the same power P=10 μWP = 10\,\mu\mathrm{W}, we can distinguish red (635 nm), green (532 nm) and blue colors (473 nm) when VDS=0.1V_{\mathrm{DS}} = 0.1 V (Figure 3b). This is caused by the different absorption rates of the device for the corresponding three wavelengths of light in the strong coupling mechanism, as shown in Figure 3c. Next, we performed photocurrent-voltage (IPHI_{\mathrm{PH}}-VDSV_{\mathrm{DS}}) characteristic measurements under different optical powers (Figure 3d). It shows a linear dependence of the photocurrent on the voltage over a wide voltage range, which indicates that the device is dominated by ohmic contacts. Then, we extracted photocurrent as a function of optical power under different VDSV_{\mathrm{DS}} values (inset in Figure 3e). An almost symmetrical and adjustable (trainable) linear photoresponsivity between -15 and +15 pA/μ\muW can be obtained by varying the VDSV_{\mathrm{DS}} (Figure 3e). Considering the subsequent ANN training, we plotted the voltage tunable photocurrents corresponding to each grey level, as shown in Figure 3f. Similar measurements of the optoelectronic characterization of green and blue light and the uniformity of each device are presented in Extended Data Figure 7-Extended Data Figure 7. We also performed the photoresponsivity measurement when VG=−1V_{\mathrm{G}} = -1 V (Extended Data Fig. 8a-i), and the increase of photoresponsivity in the order of magnitude can be applied to image detection and recognition under weak light. Figure 3g-Figure 3i show the transfer characteristic curves of the PPTs obtained under the illumination with wavelength of 635, 532 and 473 nm and different incident optical power. The dynamic range (DR) is defined by the equation: DR =20×log⁡[Imax⁡/Imin⁡]= 20 \times\log\left[I_{\max}/I_{\min}\right] (dB), where Imax⁡I_{\max} and Imin⁡I_{\min} are the photocurrent values corresponding to the maximum and minimum gate voltages, respectively. The calculated effective DR is up to 180 dB, which equals almost the highest value reported up to date[5, 11]. Therefore, the characteristic allows us to realize image pre-processing such as contrast enhancement and noise reduction by locally modulating the gate voltage of each pixel.

Measurement of optoelectronic characteristics of 2D PPT

Extended Data Figure 7. Extended Data Fig. 7|Measurement of optoelectronic characteristics of 2D PPT. a, IPHI_{\mathrm{PH}}-VDSV_{\mathrm{DS}} curves at different green light powers without any applied gate voltage. b, Voltage tunability of the regularized photoresponsivity extracted from (a). The inset shows IPHI_{\mathrm{PH}} versus PP for different VDSV_{\mathrm{DS}} values. c, The voltage (VDSV_{\mathrm{DS}}) tunable photocurrent (green light) corresponding to each gray scale. d, IPHI_{\mathrm{PH}}-VDSV_{\mathrm{DS}} curves at different blue light powers without any applied gate voltage. e, Voltage tunability of the regularized photoresponsivity extracted from (d). The inset shows IPHI_{\mathrm{PH}} versus PP for different VDSV_{\mathrm{DS}} values. f, The voltage (VDSV_{\mathrm{DS}}) tunable photocurrent (blue light) corresponding to each gray scale. g-h. The multi-state photocurrents corresponding to different levels of optical power (gray levels), where the laser wavelengths are 532 nm (g) and 473 nm (h), respectively, and the drain-source voltage is 0.1 V. i, The transfer characteristic curves of 27 devices measured under dark conditions at VDS=1V_{\mathrm{DS}} = 1 V.

To test the integrated sensing, pre-processing and image recognition functions of the AVS chip, we used it as a classifier to recognize the letters ‘z’, ‘j’ and ‘u’. For training and testing of the chip, a point-by-point scan is used to project the optical image using the setup shown in Fig. 4a (for details, see Methods). In this example of supervised learning algorithm, cross-entropy is used as the loss/cost function, the weight values were updated by backpropagation of the gradient of the loss function[19]. A detailed flow chart of the whole AVS including the training algorithm is presented in Extended Data Figure 6. Figure 4b illustrates the input image with different Gaussian noise (σ=0.2,0.4\sigma= 0.2, 0.4) added and the pre-processed image (σ=0.4\sigma= 0.4), which is extracted from the drain-source current IDI_{\mathrm{D}}. After applying gate voltage VGV_{\mathrm{G}} to the certain pixel (the white pixels in Fig. 4b), the body feature of the letters in the pre-processed image has been enhanced obviously. The complete dataset used for training after pre-processing is given in Extended Data Figure 9. In Fig. 4c, the accuracy of recognition with and without pre-processing of the images is plotted. For the pre-processed image, it is faster to reach recognition accuracy of 100%. The initial and final responsivities/weights of the classifier are shown in Fig. 4d, and the measured currents and corresponding codes of the target port for each letter are depicted in Fig. 4e. Each code corresponds to a letter, and the corresponding letter is reconstructed through post-processing, as shown in Fig. 4f. To evaluate the overall performance (processing speed and energy consumption) of this network, we also performed time-resolved measurements. The experimental setup is shown in Extended Data Figure 6. The trigger/measurement pulse is provided in Extended Data Figure 10 (see Method for details). The response of a single spike in a single device measured with the assistance of gate voltage is approximately 500 ns (Extended Data Figure 10) and the leakage current is shown in Extended Data Figure 10. The dissipated energy per spike of the device with such sensitive photoresponse is approximately 2.4×10−172.4 \times10^{-17} J, according to P=I×V×t16P = I \times V \times t^{16}. Such a system may hence provide great potential for the development of ultrafast and ultralow power machine vision.

Training datasets with and without preprocessing

Extended Data Figure 9. Training datasets. a, b, Dataset with noise level σ=0.4\sigma= 0.4 for classifier training with (a) and without (b) preprocessing.

Time-resolved measurements: electrical pulse, drain-source voltage signal, and leakage current

Extended Data Figure 10. Time-resolved measurements. a, Electrical pulse for synchronously triggering laser and drain-source current measurement. b, The drain-source voltage signal when the laser is turned on or off after preamplifier conversion. c, Leakage current when side gate voltage is applied to assist measurement.

In conclusion, we have presented an AVS composed of PPTA, which integrates multifunctions of sensing, preprocessing and image recognition simultaneously. By performing image pre-processing using this PPT, the image quality is effectively improved, and the efficiency and accuracy of subsequent image recognition is increased. This device exhibits great potential in terms of large dynamic range, ultrafast and ultralow power consumption for machine vision applications.

AVS operation as a classifier

Figure 4. AVS operation as a classifier. a, Schematic illustration of the optical setup for network training/operation. The resulting image is projected onto the photodiode array in a point-by-point scanning manner. b, Examples of images with (σ=0.4\sigma= 0.4) and without (σ=0.2,0.4\sigma= 0.2, 0.4) pre-processing of the device. c, Comparison of image recognition rate before and after pre-processing of the device. d, Responsivity distributions before (initial) and after (final) training. e, The measured three currents corresponding to ‘z’, ‘j’ and ‘u’ target ports, which are converted by the nonlinearity into binary activation codes. In each experiment, the letters ‘z’, ‘j’ and ‘u’ were projected onto the chip separately. f, The reconstructed letters after post-processing.

Methods

Device fabrication

The fabrication of the chip follows the procedure described in Extended Data Figure 2. A quartz wafer was used as the original substrate, which was cleaned with acetone, isopropyl alcohol and deionized water, respectively. The cleaned quartz wafer was deposited with a layer of ITO film (~200 nm) using magnetron sputtering(equip? ). Subsequently, an Al2O3\mathrm{Al_2O_3} layer was grew on top of the ITO film by atomic layer deposition (~40nm, Kurt J. Lesker ALD150LX). 2D crystals including MoS2\mathrm{MoS_2}, h-BN and WSe2\mathrm{WSe_2} flakes were derived from bulk source materials by a mechanical peel-transfer method. For the transfer of MoS2\mathrm{MoS_2} flake, it was first mechanically exfoliated on a transparent polydimethylsiloxane film and then transferred to the substrate with the help of an optical microscope. To eliminate unnecessary stresses, the transferred 2D MoS2\mathrm{MoS_2} was annealed in an argon atmosphere. Standard e-beam lithography (EBL, Raith Voyager) and magnetron sputtering were then employed to define the Ti/Ag nanogratings on the produced structures by a lift-off approach. Next, we defined the mask with EBL and carried out reactive ion etching (RIE) with Ar/SF6\mathrm{Ar/SF_6} plasma to separate the previously transferred MoS2\mathrm{MoS_2} sheet into 27 pixelsAfterwards, the mask was removed with acetone. 2D h-BN and WSe2\mathrm{WSe_2} flakes were also transferred to the structure using the same method described above . In order to maximize the absorption of nanogratings, Ar/SF6\mathrm{Ar/SF_6} plasma was again used to perform RIE towards 2D heterostructure on the mask defined by EBL. The top metal layer (gate electrode and drain-source electrode) was added by another EBL process and Cr/Au (3 nm/15 nm) evaporation. Finally, Al2O3\mathrm{Al_2O_3} (20 nm) and Cr/Au (20 nm/50 nm) layers were deposited on the produced heterostructures by lift-off methods using standard EBL process and magnetron sputtering/thermal evaporation of materials.

Experimental setup

Schematics of the experimental setup are shown in Extended Data Fig. 5 and Extended Data Figure 6, Extended Data Figure 6.

Light from a semiconductor laser (635/532/473 nm wavelength) was collimated by a lens before passing through a linear polarizer. The polarization direction of the linear polarizer was mounted perpendicular to the long axis of the Ag nanograting, and the linearly polarized light was projected on the structure in a normal incidence manner. The gray level of each pixel in the optical image was achieved by adjusting the laser power, and then the optical image was projected onto the sample using a microscope objective with a long working distance. A source meter (Keithley, 2400) was used to supply gate voltage to the PPT, and a source meter (Keithley, 2450) was used to supply drain-source voltage to the PPT while measuring the output current. The sample was connected to the source meter via a home-made measurement box and BNC connection cable. For time-resolved measurements, a femtosecond pulsed laser source (BFL-1030-20B, BWT) was used, which was triggered using a lock-in amplifier (Stanford Research Systems, SR830) to emit a single pulse at a wavelength of 515 nm. The 500ns cycle drain-source pulse voltage was provided by an arbitrary waveform generator (Keithley, 3390), and the output current was amplified by a preamplifier (Stanford Research Systems, SR570) and converted into a voltage signal, which was finally recorded by an oscilloscope (Siglent). In addition, all measurements were carried out at room temperature in an air environment.

Simulation and strong coupling model

The transmittance spectra and electromagnetic field distributions of the structures with strong coupling were simulated using finite-difference time-domain (FDTD) method. The plane wave light source was projected onto the structure with normal incidence in the direction of polarization perpendicular to the long axis of Ag nanogratings. In order to highlight the strong coupling effect, we neglected the effect of 2D materials in our experimental and theoretical simulations. Here, small volumes Ag nanorods with a height of 20 nm were selected to form the grating in order to achieve large photoelectric conversion efficiency by reducing the proportion of radiation damping and increasing the ballistic transport probability[22] and hot electron relaxation time[27].All calculated data were collected while satisfying the steady state energy criteria.

A coupled oscillator model was introduced to analyze the strong coupling behavior of the hybrid architecture under specific parameters. The plasmon of Ag nanogratings, symmetrized photonic mode, and antisymmetrized mode can be assumed as three oscillators. Therefore, the Hamiltonian of this three-coupled system can be written as:

(EPl−iγPl/2gwgsgwEAsym−iγAsym/20gs0ESym−iγSym/2)(2)\begin{pmatrix} E_{Pl}-i\gamma_{Pl}/2 & g_w & g_s \\ g_w & E_{Asym}-i\gamma_{Asym}/2 & 0 \\ g_s & 0 & E_{Sym}-i\gamma_{Sym}/2 \tag*{(2)} \end{pmatrix}

Where γPl\gamma_{Pl}, γAsym\gamma_{Asym}, and γSym\gamma_{Sym} are the linewidths of plasmon, antisymmetrized and symmetrized modes, EPlE_{Pl}, EAsymE_{Asym}, and ESymE_{Sym} are the resonance energies of plasmon, antisymmetrized and symmetrized modes, while gwg_w and gsg_s represent plasmon-antisymmetrized mode and plasmon-symmetrized mode interaction constants. In the three-oscillator model, the eigenstates of Hamiltonian correspond to the three hybrid branches. The wave function of each branch from the admixture contribution of plasmon, symmetric mode and antisymmetric mode can be expressed as ∣ψj⟩=αPlj∣Pl⟩+αSymj∣Sym⟩+αAsymj∣Asym⟩|\psi_j\rangle=\alpha_{Pl}^{j}|Pl\rangle+\alpha_{Sym}^{j}|Sym\rangle+\alpha_{Asym}^{j}|Asym\rangle, where αij\alpha_i^j (i=Pl,Sym,Asym;j=Upper,Middle,Loweri=Pl, Sym, Asym; j=Upper, Middle, Lower) denotes Hopfield coefficients. The modular square of the Hopfield coefficient represents the proportion of uncoupled states Pi=(PUpperi,PMiddlei,PLoweri)\mathbf{P}^{i}=(P_{Upper}^{i},P_{Middle}^{i},P_{Lower}^{i}) in hybrid state. Also, the weight of each hybrid branch W=(WUpper,WMiddle,WLower)\mathbf{W}=(W_{Upper},W_{Middle},W_{Lower}) in this strong coupling regime can be calculated as Wj=γj/∑jγjW_j=\gamma_j/\sum_j\gamma_j.

Data Availability

The data that support the findings of this study are available from the corresponding authors upon reasonable request.

Acknowledgements

This work was supported by the National Key R&D Program of China (2019YFA0308602), the National Science Foundation of China (general program 12174336 & major program 91950205) and the Natural Science Foundation of Zhejiang Province (LR20A040002). We thank the Micro and Nano Fabrication Centre at Zhejiang University for facility support and W. Wang at the State Key Laboratory of Modern Optical Instrumentation for suggestions on nanofabrication. We appreciate the equipment support provided by the Center of Electron Microscopy of Zhejiang University for the preparation of samples to be characterized, as well as the assistance provided by H. H. Huang from the Center for Micro/Nano Fabrication of Westlake University for sample characterization. We also acknowledge useful comments from Prof. D. Xiang of Frontier Institute of Chip and System, Fudan University.

Author contributions

L.L. and T.Z. conceived and designed the project. T.Z. designed and built the experimental setup, programmed the machine-learning algorithm, fabricated the ANN PPTA, carried out the material and device characterization. T.Z. and L.L. analyzed data and wrote the manuscript. All authors commented on the manuscript.

Competing interests

The authors declare no competing interests.

to define Au/Cr electrodes on the obtained heterostructures. p, q, Finally, the Al2_2O3_3 and Cr/Au layers are sputtered sequentially on the EBL-defined pattern. r, SEM view of the sample during the (g) processing step. Scale bar, 20 μ\mum. s, Enlarged SEM view of the sample in (r). Scale bar, 5 μ\mum. t, Full SEM view of the PPTA. Scale bar, 200 μ\mum.

Schematic, SEM image, photocurrent plot, and simulated temperature distributions for 2D PPT with and without Ag nanogratings

Figure 4 (PDF p. 24). a, Schematic of a single 2D PPT without Ag nanogratings. b, False-coloured SEM image of a fabricated device without Ag nanograting. The red dot indicates the location of the laser on MoS2_2. Scale bar, 20 μ\mum. c, When the laser is turned on and off, only subtle changes of photocurrent can be detected in the WSe2_2 channel, indicating that very few electrons in the MoS2_2 excited by the device without Ag nanograting can transfer to the MoS2_2 floating gate below the WSe2_2 channel through the photothermoelectric effect. d–e, Simulated temperature distribution of the left part of the phototransistor without (d) and with Ag nanograting (e). The colored images indicate the temperature distribution and the green arrows indicate the heat flux. The heat source (heat power density) is written as Q=12ωIm⁡(εr)∣E∣2Q = \frac{1}{2}\omega\operatorname{Im}(\varepsilon_r)|\mathbf{E}|^2, where ω\omega is angular frequency of the light, εr\varepsilon_r is the relative permittivity of silver, and E\mathbf{E} is electric field calculated in Figure 2d. In the absence of the right part of the device, the heat generated by plasmon decay is mainly transmitted downward, resulting in an increase in the temperature of MoS2_2. Scale bar, 100 nm. f, The simulated temperature distribution of the MoS2_2 floating gate layer of the device extending to the right. The heat generated by the plasmon decay is transmitted to the lower right, but the heat is still mainly concentrated in the Ag nanograting region, resulting in a thermoelectric potential between the left and right sides of the MoS2_2 floating gate layer. Scale bar, 100 nm.

Optical setup

Figure 5. Extended Data Fig. 5 Optical setup. a, Schematic illustration of the optical setup. The polarizer enables that the linearly polarized light (TM polarization) generated by the incident laser is perpendicular to the Ag nanograting. b, Photograph of the optical setup. OAM: Optical Adjustable Mount. c, Front view photo of the optical setup.

Extended Data Fig. 8 Implementation of PPT photoresponsivity

Figure 8. a, IPHI_{\mathrm{PH}}-VDSV_{\mathrm{DS}} curves at different red light powers with applied gate voltage. b, The relationship between IPHI_{\mathrm{PH}} and PP under different VDSV_{\mathrm{DS}} values extracted from (a). c, The voltage (VDSV_{\mathrm{DS}}) tunable red light regularized photoresponsivity corresponding to each gray scale. d, IPHI_{\mathrm{PH}}-VDSV_{\mathrm{DS}} curves at different green light powers with applied gate voltage. e, The relationship between IPHI_{\mathrm{PH}} and PP under different VDSV_{\mathrm{DS}} values extracted from (d). f, The voltage (VDSV_{\mathrm{DS}}) tunable green light regularized photoresponsivity corresponding to each gray scale. g, IPHI_{\mathrm{PH}}-VDSV_{\mathrm{DS}} curves at different blue light powers with applied gate voltage. h, The relationship between IPHI_{\mathrm{PH}} and PP under different VDSV_{\mathrm{DS}} values extracted from (g). i, The voltage (VDSV_{\mathrm{DS}}) tunable blue light regularized photoresponsivity corresponding to each gray scale.

References

  1. [1]Zhou, F. et al. Optoelectronic resistive random access memory for neuromorphic vision sensors. Nat. Nanotechnol. 14, 776-782 (2019).DOI
  2. [2]Choi, C. et al. Curved neuromorphic image sensor array using a MoS₂-organic heterostructure inspired by the human visual recognition system. Nat. Commun. 11, 5934 (2020).DOI
  3. [3]Kolb, H. How the retina works: much of the construction of an image takes place in the retina itself through the use of specialized neural circuits. Am. Sci. 91, 28-35 (2003).
  4. [4]Wang, H. et al. A ferroelectric/electrochemical modulated organic synapse for ultraflexible, artificial visual-perception system. Adv. Mater. 30, 1803961 (2018).DOI
  5. [5]Liao, F. et al. Bioinspired in-sensor visual adaptation for accurate perception. Nat. Electron. 5, 84-91 (2022).DOI
  6. [6]Gollisch, T. & Meister, M. Eye smarter than scientists believed: neural computations in circuits of the retina. Neuron 65, 150-164 (2010).DOI
  7. [7]Prezioso, M. et al. Training and operation of an integrated neuromorphic network based on metal-oxide memristors. Nature 521, 61-64 (2015).DOI
  8. [8]Wang, C.-Y. et al. Gate-tunable van der Waals heterostructure for reconfigurable neural network vision sensor. Sci. Adv. 6, eaba6173 (2020).DOI
  9. [9]Zhu, Q. et al. A flexible ultrasensitive optoelectronic sensor array for neuromorphic vision systems. Nat. Commun. 12, 1798 (2021).DOI
  10. [10]Choi, C. et al. Human eye-inspired soft optoelectronic device using high-density MoS₂-graphene curved image sensor array. Nat. Commun. 8, 1664 (2017).DOI
  11. [11]Dodda, A. et al. Active pixel sensor matrix based on monolayer MoS₂ phototransistor array. Nat. Mater. 21, 1379-1387 (2022).DOI
  12. [12]Meng, J. et al. Integrated In-Sensor Computing Optoelectronic Device for Environment-Adaptable Artificial Retina Perception Application. Nano Lett. 22, 81-89 (2022).DOI
  13. [13]Cottini, N., Gottardi, M., Massari, N., Passerone, R. & Smilansky, Z. A 33 μW 64×64 pixel vision sensor embedding robust dynamic background subtraction for event detection and scene interpretation. IEEE J. Solid-State Circuits 48, 850-863 (2013).
  14. [14]Du, C. et al. Reservoir computing using dynamic memristors for temporal information processing. Nat. Commun. 8, 2204 (2017).
  15. [15]Yao, P. et al. Face classification using electronic synapses. Nat. Commun. 8, 15199 (2017).DOI
  16. [16]Seo, S. et al. Artificial optic-neural synapse for colored and color-mixed pattern recognition. Nat. Commun. 9, 5106 (2018).DOI
  17. [17]Zhang, Z. et al. All-in-one two-dimensional retinomorphic hardware device for motion detection and recognition. Nat. Nanotechnol. 17, 27-32 (2022).DOI
  18. [18]Cui, B. et al. Ferroelectric photosensor network: an advanced hardware solution to real-time machine vision. Nat. Commun. 13, 1707 (2022).DOI
  19. [19]Mennel, L. et al. Ultrafast machine vision with 2D material neural network image sensors. Nature 579, 62-66 (2020).DOI
  20. [20]Clavero, C. Plasmon-induced hot-electron generation at nanoparticle/metal-oxide interfaces for photovoltaic and photocatalytic devices. Nature Photon. 8, 95-103 (2014).DOI
  21. [21]Sönnichsen, C. et al. Drastic reduction of plasmon damping in gold nanorods. Phys. Rev. Lett. 88, 077402 (2002).
  22. [22]Hartland, G. V. Optical studies of dynamics in noble metal nanostructures. Chem. Rev. 111, 3858-3887 (2011).DOI
  23. [23]Leenheer, A. J., Narang, P., Lewis, N. S. & Atwater, H. A. Solar energy conversion via hot electron internal photoemission in metallic nanostructures: efficiency estimates. J. Appl. Phys. 115, 134301 (2014).
  24. [24]Brongersma, M. L., Halas, N. J. & Nordlander, P. Plasmon-induced hot carrier science and technology. Nat. Nanotechnol. 10, 25-34 (2015).DOI
  25. [25]Reddy, H. et al. Determining plasmonic hot-carrier energy distributions via single-molecule transport measurements. Science 369, 423-426 (2020).
  26. [26]Mueller, T. & Malic, E. Exciton physics and device application of two-dimensional transition metal dichalcogenide semiconductors. npj 2D Mater. Appl. 2, 29 (2018).
  27. [27]Shan, H. Y. et al. Direct observation of ultrafast plasmonic hot electron transfer in the strong coupling regime. Light Sci. Appl. 8, 9 (2019).
  28. [28]Fang, Z. et al. Plasmon-induced doping of graphene. ACS Nano 6, 10222-10228 (2012).
  29. [29]Lopez-Sanchez, O., Lembke, D., Kayci, M., Radenovic, A. & Kis, A. Ultrasensitive photodetectors based on monolayer MoS₂. Nat. Nanotechnol. 8, 497-501 (2013).
  30. [30]Long, M., Wang, P., Fang, H. & Hu, W. Progress, challenges, and opportunities for 2D material based photodetectors. Adv. Funct. Mater. 29, 1803807 (2019).
  31. [31]Hong, X. et al. Ultrafast charge transfer in atomically thin MoS₂/WS₂ heterostructures. Nat. Nanotech. 9, 682-686 (2014).
  32. [32]Liu, C. et al. A semi-floating gate memory based on van der Waals heterostructures for quasi-non-volatile applications. Nat. Nanotechnol. 13, 404-410 (2018).
  33. [33]Pospischil, A., Furchi, M. M. & Mueller, T. Solar-energy conversion and light emission in an atomic monolayer p-n diode. Nat. Nanotechnol. 9, 257-261 (2014).
  34. [34]Buscema, M. et al. Large and tunable photothermoelectric effect in single-layer MoS₂. Nano Lett. 13, 358-363 (2013).
  35. [35]Goossens, S. et al. Broadband image sensor array based on graphene-CMOS integration. Nat. Photon. 11, 366-371 (2017).
  36. [36]Akinwande, D. et al. Graphene and two-dimensional materials for silicon technology. Nature 573, 507-518 (2019).
  37. [37]Christ, A., Tikhodeev, S. G., Gippius, N. A., Kuhl, J. & Giessen, H. Waveguide-Plasmon Polaritons: Strong Coupling of Photonic and Electronic Resonances in a Metallic Photonic Crystal Slab. Phys. Rev. Lett. 91, 183901 (2003).
  38. [38]Chen, S. et al. Wafer-scale integration of two-dimensional materials in high-density memristive crossbar arrays for artificial neural networks. Nat. Electron. 3, 638–645 (2020).

Paper details

vlnc:W_hm5jbusm
Version
v1 · latest
Published
Fresh — added to the Hub within the last 30 days
Added
License
CC BY 4.0
Usage

views · 1

downloads · 0

A view counts once per browser per day, a download each time the PDF is requested.

  • Physics and Astronomy

Not recorded for this paper

View statement
Looking for related papers…
Contents