Computation at the speed of light: metamaterials for all-optical calculations and neural networks

Trevon Badloe; Seokho Lee; Junsuk Rho

doi:10.1117/1.AP.4.6.064002

21 December 2022 Computation at the speed of light: metamaterials for all-optical calculations and neural networks

Trevon Badloe, Seokho Lee, Junsuk Rho

Author Affiliations +

Advanced Photonics, Vol. 4, Issue 6, 064002 (December 2022). https://doi.org/10.1117/1.AP.4.6.064002

Abstract

The explosion in the amount of information that is being processed is prompting the need for new computing systems beyond existing electronic computers. Photonic computing is emerging as an attractive alternative due to performing calculations at the speed of light, the change for massive parallelism, and also extremely low energy consumption. We review the physical implementation of basic optical calculations, such as differentiation and integration, using metamaterials, and introduce the realization of all-optical artificial neural networks. We start with concise introductions of the mathematical principles behind such optical computation methods and present the advantages, current problems that need to be overcome, and the potential future directions in the field. We expect that our review will be useful for both novice and experienced researchers in the field of all-optical computing platforms using metamaterials.

1. Introduction

Computers and digital processors that are used daily by a large number of people in modern society are now an integral part of life. They are generally made up of numerous transistors, resistors, capacitors, and sensors that communicate through the flow of electrons to perform calculations and store information using bits that are represented by either a “0” or “1.” As the amount of online digital content being created and consumed increases continuously, the demands for data transfer, storage, and larger bandwidths are simultaneously rising. However, the physical size of transistors is reaching the angstrom scale, meaning that further reduction in size will be limited by the size of atoms themselves. Various new technologies have been proposed as new ways to perform calculations with higher speed and efficiency. Notably, quantum computing and photonic computing are two of the major advancements that are being actively researched.¹ Both have their individual merits and obstacles that must be overcome for application in real-life situations. Quantum computing promises increased parallelism and therefore raw speed, but at the moment, must run at extremely low temperatures that require a lot of power and specialized equipment.² Photonic computing, on the other hand, aims to replace electronic systems with their photonic equivalent to simultaneously increase the speed and efficiency of data processing³ and decrease the amount of data corruption that can occur in electronics due to the lack of interference of photons compared to electrons but is somewhat hindered by the requirements of wavelength scale building blocks, which are tens to hundreds of times larger than the latest electronics architectures that use a 3-nm die.⁴ As both of these technologies are in their infancy, there is still a long way for them to mature before being commercially viable, by which time, they could potentially be complementary, rather than competing, technologies.

Research into photonic devices for all-optical computations has been fruitful over the past few decades.⁵^–¹² All-optical analog computing opens the door for large real-time parallel computations with minimal energy requirements, which in the energy-conscious reality that we live in with such massive reliance on big data calculations, practical realizations of low-energy real-time image, and data processing could be fundamental to unlocking the next generation of all-optical, or optoelectronic devices. Rather than using the standard “0” and “1” bits used for digital calculations and postprocessing of data, electromagnetic (EM) fields can be exploited to perform calculations and operations optically at the speed of light. Such analog processing can exploit all of the optical properties of light simultaneously, including the phase, amplitude, and polarization to manipulate the complex amplitude of EM waves. This could facilitate both high-throughput and low-power devices, all inside a miniaturized footprint, compared to that of traditional bulky optics made up of lenses and other components. Silva et al.¹³ proposed the concept of using metamaterials for spatial differentiation, integration, or convolution, with simulations based on metasurfaces combined with graded-index waveguides, as well as multilayered slabs designed to achieve specific Green’s functions. Since then, there have been numerous experimental demonstrations of optical calculations using metamaterials in various wavelength regimes. Along with the development of high-fidelity and large-scale nanofabrication techniques that have been produced over the past decade, the opportunity to create fully on-chip photonic devices with extremely miniaturized footprints appears to have come to light. Fully optical computing and image processing could have impacts on numerous fields, including on-chip solid-state light detection and ranging (LiDAR),¹⁴^,¹⁵ bioimaging,¹⁶ and preprocessing of big data.¹⁷

Here we will review the recent advancements of all-optical photonic calculations using metamaterials, including simple single-layered structures, multilayer structures, structured flat optical elements, known as metasurfaces, and other photonic devices, such as photonic crystals. We note here that although photonic crystals are not exactly classified as metamaterials, they are still photonic devices that demonstrate interesting characteristics at the wavelength scale, which become nanophotonic devices for visible light. We will start by introducing the basic mathematical functions that underpin the speed-of-light calculations. We will then introduce the latest research on metamaterials for linear differentiation, integration, and Laplacian calculations and discuss their potential applications in terms of all-optical computing. We will conclude by discussing the current progress in all-optical machine learning, from the mathematical basis of how it works, to the latest research, and current difficulties. We will finally provide our thoughts on the outlook and future of all-optical computing toward fully integrated photonic brains. We hope that this review will be useful for both experts and novices in the field of all-optical computations using metamaterials and to serve as a valuable reference to guide new research.

2. All-Optical Calculations at the Speed of Light

2.1.

Mathematical Operations

We start by describing how light can be used to perform mathematical operations. An interesting mathematical parallel exists between convolution in Fourier space between the input function and the systems impulse response, and the interaction of a single-wavelength EM wave interacting with an optical element. That is, for a system with an impulse response given by $g (x, y)$ , any input function given by $f (x, y)$ produces an output of $w (x, y)$ that can be defined by

Eq. (1)

w (x, y) = \int g (x - x^{'}, y - y^{'}) f (x^{'}, y^{'}) d x^{'} d y^{'},

which can be written in terms of Fourier transforms (FTs) as

Eq. (2)

w (x, y) = IFT {G (k_{x}, k_{y}) FT {f (x, y)}},

where IFT denotes an inverse Fourier transform, and

G (k_{x}, k_{y}) = FT {g (x, y)}

, where

k_{x}

and

k_{y}

are the spatial frequency variables and

x

and

y

are real-space locations. This is analogous to an input EM wave given by

E_{in} (x, y)

that interacts with a metamaterial imparting a spatially varying function

m (x, y)

that performs some form of optical operation in Fourier space. The resultant EM field,

E_{out} (x, y)

can be defined as

Eq. (3)

E_{out} (x, y) = IFT {m (x, y) FT {E_{in} (x, y)}} .

It is therefore clear to see that the phase and amplitude that are encoded by the metamaterial represent the term $G (k_{x}, k_{y})$ in Eq. (2). It should be noted that, although it does not cause any additional difficulties, based on the well-known identity $FT {FT {A (x, y)}} \propto A (- x, - y)$ , the output EM field will be flipped, i.e., it will represent $E_{out} (- x, - y)$ . This FT is generally performed using standard refractive lenses but has also been successfully encoded directly into a metamaterial itself to further miniaturize the system. This switching between real and Fourier space can also be avoided by designing metamaterials that directly process the required transfer function in the Fourier space. This is generally achieved by designing a metamaterial with an angular-dependent response and has been termed as the Green’s function approach.

Differentiation is one of the fundamental mathematical operations that appears in numerous places in physics, from Newton’s famous second law of motion, to Maxwell’s equations that underpin classical electromagnetism. When applied to an image, a process of edge detection is performed, which could have potential impact in numerous fields that benefit from instantaneous edge detection without the need for computationally expensive and time-costly postprocessing, such as in object and scene detection or biomicroscopy. Machine-learning algorithms also rely on such numerical calculations, such as differentiation and backpropagation, further highlighting the potential for all-optical computing at the speed of light. First, without loss of generality, we consider one-dimensional (1D) operations and use the notation of $m$ to represent the optical coefficient of the metamaterial, which could be interchanged freely with the reflection or transmission coefficient, $r$ and $t$ , respectively, depending on the metamaterial design and direction of operation.

2.1.1.

First-order differentiation

In order to perform first-order spatial differentiation in 1D, for an arbitrary function $f (x)$ , the operation $\partial f (x) / \partial_{x}$ is equal to $i k_{x}$ in Fourier space. Therefore, the metamaterial $[m (x)]$ should physically impart the transfer function:

Eq. (4)

m (k_{x}) = \frac{\max (| m |) i k_{x}}{L},

where

2 L

is the size of the metamaterial bounding the spatial locations

- L \leq x \leq L

. This function is depicted in Fig. 1(a). The maximum reflection or transmission that can be achieved imparts a limitation that is related to the fact that the operations performed by the metamaterials are in fact scaled when compared to the exact mathematical definition. The phase of the transfer function required for first-order differentiation is odd with respect to

k_{x}

, which necessarily requires some form of mirror symmetry to be broken with respect to the

x

axis as well as the

z

axis.¹⁸ This also holds true for other odd integer orders of differentiation. In terms of optical image processing, the first-order differential will produce an edge-enhanced image. Since the first derivative is dependent only on the neighboring intensities, it can be quite sensitive to fluctuations and noise in the image.

Fig. 1

Required optical transfer functions for all-optical calculations for (a) first- and (b) second-order differentiation and (c) integration. The black lines represent the required transmission or reflection coefficients, whereas the red dashed lines denote the required phase. Note the $π$ phase shift required at $k_{x} / k_{0}$ for the first-order differentiation, due to its being an odd function. The absolute value of the phase is arbitrary. The limits of the $x$ axis are determined by the angular response of the designed metamaterial, which denotes the working NA of the system. Note the arbitrary truncated region around 0 for the integration operation that is necessary to avoid unphysical gain requirements, as suggested in Ref. 13.

2.1.2.

Second-order differentiation

Considering an arbitrary function $f (x)$ , second-order spatial differentiation in 1D, i.e., $\frac{\partial^{2} f (x)}{\partial x^{2}}$ , can be represented in Fourier space as $- k_{x}^{2}$ . This means that designing a metamaterial that imparts a parabolic transfer function and a near-constant phase profile to the incoming EM wave allows access to completely optical second-order differentiation, i.e., the following function should be achieved:

Eq. (5)

m (k_{x}) = \frac{\max (| m |) k_{x}^{2}}{L},

where the coefficient is the same as Eq. (4). This function is depicted visually in Fig. 1(b). Since

k_{x} = k \sin θ

, the transmission or reflection should be zero at normal incidence, while increasing quadratically under increasing angle of incidence. Second-order differentiation is generally less prone to noise in images but can, however, struggle when there are large local fluctuations in intensity.

2.1.3.

Integration

To perform 1D integration on an arbitrary function $f (x)$ , the operation defined by ${(i k_{x})}^{- 1}$ must be achieved. It should be noted that the singularity at $k_{x} = 0$ should be taken care of appropriately when designing the metamaterial.¹³ This has been achieved through the use of a constant value around the singularity. This function is depicted in Fig. 1(c). In all-optical image processing, the integral of an image has implications in real-time face detection, where high-resolution images can be processed quickly to uncover and highlight important features.¹⁹

Although we introduced 1D operation for simplicity, two-dimensional (2D) operations can be easily achieved by simply including the second dimension. The Laplacian is a specific example of a well-known 2D operation that can be implemented by manipulating EM waves.

2.1.4.

Laplacian

The Laplacian ( $\nabla^{2}$ ) is the 2D version of the second-order differentiation operation discussed above. Therefore, for a function $f (x, y)$ , the 2D second-order differentiation is given as $\frac{\partial^{2} f (x)}{\partial x^{2}} + \frac{\partial^{2} f (y)}{\partial y^{2}}$ . Subsequently, given an arbitrary EM field $E (x, y)$ , $\nabla^{2} E (x, y) = - k_{∥}^{2} E (x, y)$ , where $k_{∥}^{2} = k_{x}^{2} + k_{y}^{2}$ . This results in an additional requirement that the metamaterial should be symmetrical in both the $x$ and $y$ directions. Such even operations are fairly simple to achieve physically; however, for odd functions extra steps must be undertaken to create a physical asymmetry. Various solutions have been proposed and developed, which we will review in terms of all-optical mathematical operations with different designs of metamaterials and photonic devices in the following section.

2.2.

Metamaterials for All-Optical Calculations

2.2.1.

Static metamaterials

Single and multilayers

The simplest way of manipulating EM waves is by passing them through the interface between two media. Since layering materials does not require any form of patterning or complex procedures other than deposition, they provide an obvious and straightforward method of manipulating EM waves through the correct design of the wavelength, incident angle, and thicknesses. Fabry–Perot (FP) resonators have found wide use in metamaterials for various applications from selective absorbers²⁰^,²¹ and radiative cooling,²² to structural color, with multilayer structures such as distributed Bragg reflectors being extensively researched. They have also been utilized for all-optical computing through various mechanisms, which will be presented here.

In one of the first examples of optical computing using metamaterials, Zhu et al.²³ utilized surface plasmon polaritons (SPPs) to demonstrate first-order differentiation using a simple single layer of silver on a glass prism [Figs. 2(a)–2(c)]. The total reflected amplitude was determined by the interference between the reflection from the glass–metal interface and the SPP radiation under the illumination of 532 nm incident light. The thickness of the film was only around 50 nm, providing a tiny footprint for the device that has the potential to be produced at scale. The minimum resolution of edge detection was determined to be around $7.2 μ m$ , which could be sufficient for image-processing applications. However, the device operates in quite a narrow spatial region with limited efficiency due to the requirement of exciting the SPP, which in turn, also restricts the design to work in reflection. The requirement of the use of a bulky prism, also somewhat offsets the benefit of the reduction of size of the super-thin film. By adding a few extra layers, rather than the extremely simple single layer reported previously, Wesemann et al.²⁴ experimentally showed the use of a near-perfect absorbing mirror for all-optical image processing of phase and amplitude objects with a Salisbury screen with a two-layer design [Figs. 2(d)–2(f)]. The metal–insulator–metal layers made up of gold (Au) and silicon dioxide ( ${SiO}_{2}$ ) provide an FP response, so by making the dielectric insulator layer thickness approximately a quarter wavelength, perfect absorption is released at the working wavelength. Since this absorption is dependent on the incident angle of light ( $θ$ ), the reflectance is therefore also related, since the transverse component $k_{x} = k \sin θ$ , where $k$ is the wave vector of light in free space. The successful demonstration of phase imaging of a biological sample using the designed device highlights the potential for such simple systems, without any requirements for complex and time-consuming structuring processes, to achieve all-optical computation.

Fig. 2

All-optical calculations using planar films. (a) Experimentally measured (green dots) and numerically fitted (solid lines) spatial transfer functions for three different samples. $θ_{0}$ corresponds to the incident angle required for the phase matching to support the SPP. All optical first-order differentiation of (b) (i) amplitude and (c) (i) phase images. (ii) The reflected intensity after differentiation. Scale bars, $100 μ m$ . (d) Calculated reflectance for p-polarization (solid line) and s-polarization (dashed line) at the absorption wavelength of 631 nm. (e) Demonstration of edge enhancement in reflected amplitude images. (i) Bright-field microscope image of the amplitude object and experimentally reflected image from the thin film mirror at (ii) 598 nm, (iii) 640 nm, and (iv) 694 nm. Scale bar, $100 μ m$ . (f) Experimental demonstration of phase imaging of a biological sample. Images reflected from the mirror at (i) 598 nm (ii), 638 nm, and (iii) 694 nm. Scale bar, $250 μ m$ . First-order differentiation using the SHE of light. (g) Experimental (dots), theoretical (dashed lines), and the ideal (solid lines) spatial spectral transfer functions for first-order differentiation. (h) Experimental demonstration of the material and angle independence of SPE of light for image differentiation; (i) air to glass interface with the Chinese character for “light” and (ii) air to Au and glass interface with the English word “LIGHT.” Images (a)–(c) reproduced with permission from Ref. 23. Copyright 2017 Springer Nature Group. Images (d)–(f) reproduced with permission from Ref. 24. Copyright 2019 AIP Publishing LLC. Images (h)–(j) reproduced with permission from Ref. 25. Copyright 2019 American Physical Society.

Noting the fact that most multilayered structures can only operate for even functions, such as second-order differentiation, due to their inherent symmetry, Youssefi et al.²⁶ provided a solution to achieve odd functions, specifically first-order differentiation. This was achieved by simply rotating the device, or equivalently, using light at an oblique incidence. Since the reflection coefficient of TM-polarized light becomes zero at the Brewster angle, odd functions can be subsequently released; however, they are limited to 1D operation. However, Xu et al.²⁷ later provided a solution for 2D differentiation using the properties of the Brewster angle. 2D edge detection was also demonstrated by Zhu et al.²⁸. They identified that the complex transfer function should carry a topological charge (TC) in order to exhibit broadband isotropic 2D differentiation. By selecting the appropriate input and output polarization and using the incident angles that correspond to total internal reflection or the Brewster angle, 2D differentiation was successfully proven for both green and red light with wavelengths of 532 and 633 nm.

In addition to exploiting the Brewster angle, Zhu et al.²⁵ showed that since the reflected light from a planar interface exhibits a different transverse shift based on its polarization state, the spin Hall effect (SHE) of light can be utilized for spatial differentiation of EM waves [Figs. 2(g)–2(j)]. They showed that by using specific orthogonal polarization states under the paraxial approximation, the reflected and refracted light from any planar interface, regardless of the material or incident angle, can be employed, since the differentiation operation is due to the intrinsic nature of the SHE.²⁹ Since SHE is a nonresonant effect, the spatial differentiation frequency bandwidth is infinitely large, which enables ultrafast all-optical calculations at all wavelengths. Accordingly, 1D differentiation was experimentally demonstrated at a wavelength of 532 nm, both for various angles and an air–glass and an air–Au–glass interface, proving the generality of using SHE for differentiation. One drawback of SHE is that it disappears at normal incidence, which means that the signal is decreased at extremely small incident angles. However, recently, SHE has been proven for arbitrarily polarized and unpolarized light,³⁰ as well as high-efficiency operation,³¹ which could open the path for further developments of SHE for all-optical calculations. Another interesting application of spatial differentiation using SHE was demonstrated as a method to calibrate spatial light modulators (SLMs) by converting phase information into amplitude information that can be imaged directly.³² In turn, this concept was extended to develop a phase-mining method by measuring a shadow-cast effect in the measured images.³³ 2D differentiation has also been proposed using SHE near the Brewster angle due to a double-peak profile.³⁴ A common drawback of the work presented so far is that they operate in reflection mode, whereas for practical applications, transmission mode would be preferred.

This was successfully proposed and demonstrated by Xue and Miller,³⁵ who also noted that the conventional Laplacian-based transmission response may be not the ideal case when considering the scattering physics of real-world designs. Therefore, through the optimization of alternating layers of silicon (Si) and ${SiO}_{2}$ , 2D second-order differentiation was demonstrated with a high numerical aperture (NA), along with the benefits of straightforward fabrication of simple multilayers. Transmission up to angles of 80 deg was designed, which corresponds to an NA of 0.98. Since any transmission function can be optimized by changing the target, various operations were designed. In particular, for the functionality of edge detection, the authors suggest that since a multilayer design with a transmission $(θ) \approx θ^{3}$ dependency is more closely designed than the standard quadratic form, suggesting that a more data-driven approach could be taken to optimize the output fields rather than designing the optical response of the metamaterial directly. Other multilayer dielectric slabs have also been proven for integration calculations.³⁶

Plasmonic structures

Pors et al.³⁷ demonstrated plasmonic metasurfaces for analog computing by exciting gap-surface plasmons (GSPs) through the design of an Au meta-atom on top of a ${SiO}_{2}$ spacer, above an Au mirror [Figs. 3(a)–3(c)]. The GSPs propagate within the ${SiO}_{2}$ spacer, demonstrating FP-like resonances that originate from the multiple reflections due to the meta-atoms. Both first-order differentiation and integration were demonstrated numerically and experimentally. The metasurfaces operate in reflection mode at the wavelength of 800 nm, providing all-optical mathematical operations at visible wavelengths. The fundamental design of using the GSP allows for the full phase and amplitude control of incident light; however, it must be used in reflection, which somewhat limits its potential uses for integration in optical systems that generally require transmission-type components. Hwang et al.³⁸ subsequently numerically presented a transmission mode plasmonic metasurface based on localized surface plasmon resonances driven by evanescent coupling of five metallic meta-atoms [Fig. 3(d)]. Simulations of second-order differentiation of phase were demonstrated; however, gaps of around 10 nm could limit the feasibility of fabrication of such devices. Various double split ring resonator (SRR) devices have been developed to perform second-order differentiation in the microwave regime.⁴⁰ Recently, a double SRR fabricated on either side of a dielectric substrate was demonstrated by Wang et al.³⁹ to perform a number of all-optical operations [Figs. 3(e)–3(f)]. Namely, first-order differentiation and cross-correlation are experimentally proven using the Huygens’ metasurface at microwave frequencies. A Huygens’ metasurface is so-called because each meta-atom being considered as the source of a secondary wave is emitted from the specific spatial location. This means that each spatial location has a defined response due to the designed meta-atom, without considering any long-distance effects from neighboring meta-atoms. Twenty-five distinct meta-atoms were designed in order to encode the required complex amplitude into the metasurface, which acts as the transfer function to modulate the Fourier spectrum directly. This was achieved by superimposing a phase factor related to the input and output focal length of the system, which removes the requirement of the 4f system, reducing it down to a 2f one. The single-layer metasurface is extremely small, down to a thickness of around $λ / 6$ , which means that the entire system is effectively miniaturized.

Fig. 3

Plasmonic-based metamaterials for all-optical calculations. (a) (i) Reflection coefficient and (ii) meta-atom dimensions for $50 μ m$ wide plasmonic differentiator and integrator metasurfaces. ( $d = 2.5 μ m$ ). (b) Bright-field images and measured (solid lines) average normalized reflectivity along the $x$ coordinate, along with the ideal (dashed lines) reflectivity of the (i) differentiator and (ii) integrator plasmonic metasurfaces. (c) (i) Input object with varied phase and amplitude and (ii) reflected image after integration by the plasmonic metasurface, along with the measured (solid lines) and ideal (dashed lines) normalized intensity along the $x$ direction. (d) Scattered intensities of the plasmonic circuits for first- and second-order differentiations as functions of incidence angle $θ$ for a plane wave polarized along the meta-atoms. The inset indicates the oblique incidence for the second-order differentiation circuit. (e) Simulated transmission amplitude and phase of the 25 designed meta-atoms at 10 GHz. (f) Schematics, numerical simulations, and experimental results of the normalized electric field intensity distribution at the focal length of 100 mm and working frequency of 10 GHz for (i) 1D-edge detection along the $x$ axis, (ii) 2D-edge detection, and (iii) 2D-edge detection of a regular hexagon. Images (a)–(c) reproduced with permission from Ref. 37. Copyright 2015 American Chemical Society. Image (d) reproduced with permission Ref. 38. Copyright 2018 Optica Publishing Group. Images (e) and (f) reproduced with permission from Ref. 39. Copyright 2022 Springer Nature Group.

Rather than using SRRs as the meta-atoms for Huygens’ metasurfaces, their long-distance interactions can also be exploited in what is known as a nonlocal metasurface. As one of the first nonlocal metasurfaces for all-optical computations, Kwon et al.¹⁸ implemented the concept of nonlocal metasurfaces to demonstrate first- and second-order differentiation and integration at normal incidence. A sinusoidal modulation of the permittivity of the dielectric element in the SRR meta-atoms is imparted along the $x$ axis to provide a larger variation in the nonlocal response by engineering the magnetic resonance. This gives rise to momentum matching to leaky guided modes, known as quasi-guided mode resonances (qGMRs), which propagate along the surface. By aligning the operating frequency with the location where the transmission of the meta-atom response due to the leaky qGMR is zero, a strong variation in the response with regard to the incident angle, and therefore the incoming wave vectors, is produced. The maximum spatial resolution of the designed metasurface is determined by the angle of where the transmission becomes unity, which is dependent on the dispersion of the leaky GMR. To provide the asymmetry for first-order differentiation, an array of metallic wires was placed under the metasurface parallel to the $x$ axis. Furthermore, 2D second-order differentiation was demonstrated by simply rotating two 1D operation metasurfaces by 90 deg. Although the ideal Laplacian operator defined by $\frac{d^{2}}{d x^{2}} + \frac{d^{2}}{d y^{2}}$ requires azimuthal symmetry, using unpolarized light to illuminate the image provides successful 2D edge detection using the simplified design.

Despite these successful demonstrations of all-optical calculations using plasmonic metamaterials, designing and fabricating nanoscale meta-atoms that can control the complex amplitude of light at visible wavelengths is still a challenge. Although plasmonics has shown the potential of all-optical computations, they are generally limited at visible wavelengths due to ohmic losses and rely on exciting strong plasmonic resonances within small gaps. Therefore, all-dielectric metamaterials have been proposed as an alternative direction with numerous successful advances in the past decade.

All-dielectric structures

Nanophotonic devices based on dielectrics have been the subject of copious research in recent years. In one early demonstration of all-optical computational dielectric metasurfaces by Zhou et al.,⁴¹ a 1D phase gradient was patterned into a glass substrate using a femtosecond pulse laser to create a grating structure. Under illumination of linearly polarized light with the metasurface placed in the Fourier plane of a lens, 1D edge detection was successfully proven. The requirement to place the metasurface in the Fourier plane is due to its acting as a filter in the Fourier regime, rather than directly processing the information, as the layered structures in the previous section demonstrated. The interplay between the slight difference in the transmitted LCP and RCP light provides the final image with a horizontal variation in images that, when an analyzer is used, produces edge detection with high efficiency of over around 90%. A similar idea was used to prove the ability of a two-layer photonic crystal slab structure to perform first- and second-order differentiation using incoherent light.⁴² By perturbing the location of the layers, mirror symmetry can be broken in order to achieve first-order differentiation. This concept has been further demonstrated for 1D, 2D, and even 3D edge detection using an array of metalenses.⁴³ Through postprocessing, the variation of the spatial location of each individual metalens allows for the calculation of 3D edge detection. However, as we hope to move toward fully optical calculations, methods that require no postprocessing would be more desirable.

To that note, Cordaro et al.⁴⁴ demonstrated all-dielectric metasurfaces that perform 1D first or second differentiation by engineering a 1D grating of Si nanobeams. The spatial dispersion of the metasurfaces is controlled using the nonlocal response due to the qGMR, which allows for direct processing of the angular information without the need for a 4f system. A Fano-type spectrum is generated by combining the FP resonance from the thickness and filling factor of the nanostructures with the qGMR, which occurs when the in-plane $k$ -vectors of the incoming light match that of the nanoscale grating. This design demonstrated an operational NA of 0.35 and experimentally measured transmission of over 0.8, allowing for processing of high-resolution images at the speed of light. This was successfully experimentally proven for both simple images of text, and a more complex painting. Various other examples and designs of gratings that exploit the qGMR in a 1D grating structure have been proven for 1D differentiation.⁴⁵ For practical applications, however, 2D calculations are in demand.

To tackle this issue, Zhou et al.⁴⁶ demonstrated flat optics for image differentiation using Si meta-atoms embedded in PMMA on a glass substrate to develop a photonic crystal that can perform 2D second-order differentiation [Figs. 4(a)–4(c)]. An NA up to 0.315 is experimentally demonstrated to produce a resolution of around a micrometer. Moreover, the realistic application of such metasurfaces was proven by integrating them directly into an optical microscope and camera sensor, highlighting the potential for such miniaturization of optical components. The photonic crystal slab supports leaky qGMR modes that can lead to near-unity backreflection or transmission when Kerker’s conditions are met. Bound states in the continuum (BIC) were uncovered, and the working wavelength was chosen to be 1120 nm to avoid it. At oblique incidence, there is no s-polarized output due to reciprocity; however, under p-polarized incidence, the required quadratic response for the transfer function is satisfied. Furthermore, after redesigning the metasurface to work at 740 and 1450 nm, metasurfaces were inserted into a commercial optical microscope and above an image sensor to directly perform 2D edge detection on various biosamples. Additionally, the possibility for further miniaturization was demonstrated using a metalens for the focusing element and spatially layering the differentiation metasurface and metalens on top of each other, highlighting the potential that all-dielectric optical computing holds for integration into existing systems. In another example, Guo et al.⁴⁹ implemented a dielectric photonic crystal slab above a dielectric layer with an air gap to create an unusual band structure that is isotropic for the two orthogonal polarizations in transmission mode. Another obvious solution to further miniaturize 2D all-optical calculations is to use metasurfaces with 2D arrays of meta-atoms. By combining the whole 4f system with flat lenses such as gradient index lenses, or metalenses before and after the computing metasurface, the system can be easily miniaturized to perform all-optical calculations.⁵⁰^–⁵³ Alternatively, the opposite can also be implemented to achieve similar responses, i.e., air holes in a dielectric layer. Kwon et al.⁵⁴ employed a triangular lattice of holes in a Si membrane to 2D second-order differentiation with an NA of 0.1392. Since the metasurface is designed to be polarization-independent, it can also be used to perform edge detection using unpolarized light. In order to achieve first-order differentiation, a ${SiO}_{2}$ layer was added to break the symmetry in the $z$ direction, while in-plane symmetry was broken by adding three holes around the main hole. In another example, Long et al.⁵⁵ designed a photonic crystal slab made of daisy-shaped holes in a hexagonal lattice shape for polarization-independent functionality. At two different frequencies, the device acts as a free-space compressor or a second-order differentiator.

Fig. 4

All-dielectric metamaterials for all-optical calculations. (a) Calculated optical transfer function for p-polarization at 1120 nm (red and green lines) and quadratic fitting (blue dashed line). (b) (i) Optical image of the 3D macroscopic imaging target of a plastic flower and its bright-field and differentiated imaging results. (ii) The same with a second target. (c) Imaging results for (i) bright-field and (ii) differentiated onion cells. Scale bars: $50 μ m$ . (iii) Optical image of the compound metalens and differentiator system. (d) Simulated angular dispersion of the metasurface at different incident angles at the wavelength of 1400 nm, corresponding to the magnetic resonance mode. (e) Edge detection using the magnetic dipole mode at 1400 nm for horizontal and vertical stripes. (f) (i) Calculated (lines) and experimental (dots) amplitude of the meta-atoms and (ii) phase determined by the rotation angle of the meta-atoms. (g) (i) Amplitude object and (ii) differentiated images. Scale bars: $50 μ m$ . (h) (i) The transmitted and (iii) differentiated images of a phase object. Scale bars: $50 μ m$ . Images (a)–(c) reproduced with permission from Ref. 46. Copyright 2020 Springer Nature Group. Images (d) and (e) reproduced with permission from Ref. 47. Copyright 2021 American Chemical Society. Images (f)–(h) reproduced with permission from Ref. 48. Copyright 2022 Springer Nature Group.

Other resonant effects have also been effectively exploited for optical computation. Komar et al.⁴⁷ explored the effect of the electric and magnetic Mie-type resonances to demonstrate that the magnetic resonance provides a better mechanism for second-order differentiation [Figs. 4(d)–4(e)]. The hexagonal lattice of Si meta-atoms provided a magnetic resonance at the working wavelengths of 1400 and 1570 nm for the magnetic and electric dipole resonances, respectively, and demonstrated polarization-insensitive 2D edge detection. The intensity of the edge detection is fairly low, which is expected as the high-power harmonics are cut during the edge detection process. BICs are nonradiating resonant modes that exist in open systems that cannot outcouple to radiating channels that propagate out of the system.⁵⁶^,⁵⁷ Optical BICs have been a hot topic recently, with deep research into symmetry-protected BICs, accidental BICs, and quasi-BICs,⁵⁸^–⁶⁰ the latter of which exist close to the BIC resonances and can therefore be accessed for experimental demonstration. Quasi-BICs also rely on long-distance interactions between meta-atoms that result in nonlocal responses. In order to exploit the quasi-BIC resonance, Pan et al.⁶¹ designed a modified square meta-atom with a square notch on each face made of Si in a square lattice. The symmetry of the lattice and meta-atom ensures the isotropic transfer function that is required for 2D second-order differentiation. By calculating the dispersion band structure of the metasurface, the existence of an infinite quality factor ( $q$ -factor) resonance was proven. Therefore, normally incident plane waves on the metasurface cannot access the BIC; however, the finite $q$ -factors of the quasi-BIC can be accessed at different incident angles for p-polarized light. The designed metasurface demonstrated the desired isotropic parabolic shape up to angles of around 8 deg, which indicates an NA of about 0.14 at the working wavelength of 740 nm. Exploiting the quasi-BIC mode for optical calculations leads to extremely high-quality results, as the $q$ -factor of the resonance can be theoretically infinite. However, this comes with additional difficulties in terms of fabrication, especially at visible wavelengths. The robustness of fabricated devices that demonstrate BICs has been actively studied, proving the potential that such high- $q$ resonances could bring.⁶²

Furthermore, Goh et al.⁶³ proved that any arbitrary shaped scatterer can be exploited by designing a meta-atom with an optimized nonlocal response using inverse design. The problem is defined as a partial differential equation, and the inexact Newton conjugate gradient method is used to find an optimum solution to the number of harmonics for the problem. The scatterer is designed using Si with a working frequency of 135 THz. Since the response of the designed scatterer does not rely on sharp resonances, it shows an increased robustness to external factors, such as added noise due to differences in the working frequency, relative permittivity of the scatterer, and pollution of the scattered fields with random noise up to 20% of the mean value. Scatterers that provide solutions to both a Fredholm equation of type II and a second-order differential equation were demonstrated. The design flexibility offered by the inverse design process is extremely beneficial, as the scatterer can be designed through numerous different EM solvers; however, such freeform objects could be challenging for fabrication, especially if the working wavelength is moved to the visible regime and nanofabrication is required. Various other methods of inverse design for all-optical calculating metasurfaces have also been demonstrated, including the ability to perform parallel computing.⁶⁴^–⁶⁶

Finally, in a recent demonstration of the power of all-optical calculations using metasurfaces, Fu et al.⁴⁸ introduced metaimagers that were able to perform 1D and 2D differentiations, alongside experimental results that proved the capabilities of analog processing through examples of optical and biological samples showing real-time edge detection and denoising [Figs. 4(f)–4(g)]. The complex amplitude of the incident EM waves was controlled through the design of the Si meta-atoms at the working wavelength of 633 nm. The geometry of the meta-atom is modulated to control the amplitude of the transmitted light, whereas the in-plane rotation exploits a geometric phase to control the phase response. The device consists of a metalens element combined with an arbitrary secondary metasurface that imparts the desired transfer function onto the transmitted light. Both doublet and singlet metasurfaces are presented. Although cascaded metasurfaces have been proven to provide interesting optical responses,⁶⁷^–⁷¹ extra orders of complexity are introduced into the fabrication and design in terms of alignment and consideration of the optical path length of the light that has been focused by the metalens. However, the distance between the metalens and computational metasurface provides a useful degree of freedom that helps to increase the field of view and magnification of the resulting field. By combining the required phase into a single metasurface, the complete system is miniaturized at the expense of a reduced field of view and magnification.

2.2.2.

Tunable and multiplexing metamaterials

One of the key advantages that metamaterials, especially metasurfaces, hold is for multiplexing numerous responses into a single device. Various methods have been proven to achieve this, such as simply interleaving metasurfaces with different functionalities,⁷²^,⁷³ designing anisotropic meta-atoms that provide different optical responses depending on the polarization of the incident light,⁷⁴^,⁷⁵ as well as wavelength⁷⁶ and incident angle dependency.⁷⁷ Recently, there has been a push toward developing metamaterials and metasurfaces that include tunable properties after they have been fabricated.⁷⁸ This has opened up access to devices that have multifunctionality within a tiny footprint. Numerous methods of modulating the optical properties of metadevices using external stimuli have been demonstrated, such as through the use of liquid crystals,⁷⁹^–⁸² phase-change materials,⁸³^,⁸⁴ and physical strain.⁸⁵^,⁸⁶ By arduously designing the metasurface to have a specific response under different stimuli, it has been shown that a single device can perform multiple functions. This section will review the progress of such multiplexed and tunable systems for applications of all-optical computation.

Since multiple functionalities can be encoded into a single metasurface, multiplexing different responses for different polarizations of light has been demonstrated extensively. One benefit of metasurfaces is that phase information, which is generally disregarded in conventional optics, can be accessed directly and manipulated for direct phase contrast imaging. Huo et al.⁸⁷ used this idea to produce a metasurface that can be introduced to the Fourier plane of a 4f system that acts either as a constant phase filter, or a spiral phase filter, depending on the helicity of the incident CP light [Figs. 5(a)–5(b)]. The spiral phase imparted on the Fourier plane acts to introduce an optical vortex or orbital angular momentum (OAM) with a donut-shaped intensity and spiral phase to the incoming light, which when used to image an object will result in edge detection. Sixteen titanium dioxide ( ${TiO}_{2}$ ) meta-atoms are designed and arranged to make the metasurface for a resolution of around $3 μ m$ that can work across the whole visible regime. This functionality is enabled due to the use of achromatic lenses in the 4f system and the broadband response on the metasurface itself. This concept was expanded by exploiting the multiplexing possibilities that metasurfaces offer by Kim et al.⁸⁹ By integrating the phase required for a metalens with a spiral phase of TC 1, the single device was able to perform isotropic 2D edge detection within an extremely small form factor. Since the metalens phase is also encoded into the metasurface, the requirement for the bulky 4f system was relieved and confined into a single ultrathin device. Hydrogenated amorphous Si (a-Si:H) was used as the material for the meta-atoms, which were placed to encode the required combination of metalens and spiral phases using Pancharatnam–Berry (PB), also known as geometric phase. Since a PB phase is related only to the orientation angle of the meta-atom, it inherently allows for broadband functionality that depends only on the dispersion of the material. By designing the NA of the metalens to be fairly high, determined to be 0.8, an impressive resolution down to $0.78 μ m$ was experimentally demonstrated through the successful demonstration of the edge detection of red blood cells, highlighting the potential of such systems for biomedical applications.

Fig. 5

Multiplexed and tunable metamaterials for all-optical calculations. (a) Calculated and experimental demonstrations of the two spin-dependent masking functions under 530 nm incidence. (i) The intensity distribution and (ii) phase profile of a Gaussian beam for LCP incidence, and (iii) donut-shaped intensity distribution and (iv) spiral phase profile for a 1ℏ OAM beam for RCP incidence. (v) and (vii) show measured nonparaxial interference patterns with a plane wavefront and (vi) and (viii) show the output states corresponding to LCP and RCP incidence. Insets, handedness of the incident light. (b) Images of undyed onion epidermal cells with a 20× objective lens. (i)–(iv) Bright-field images captured with LCP incident light at the wavelengths of 480, 530, 580, and 630 nm. (v)–(viii) Spiral phase contrast images captured under RCP incidence at the same wavelengths. Scale bar, $100 μ m$ . (c) Schematic of the strain tunable metasurface under (i) no strain and (ii) 36% strain, along with the corresponding calculated transfer functions at the working wavelength of 1230 nm. (d) Experimental demonstration of the switchable image properties for (i) a sample of onion cells and (ii) an amplitude object of the number “2.” Images (a) and (b) reproduced with permission from Ref. 87. Copyright 2020 American Chemical Society. Images (c) and (d) reproduced with permission from Ref. 88. Copyright 2021 American Chemical Society.

In an interesting example of a dynamically tunable metasurface that can be switched between bright field and differentiation, Zhang et al.⁸⁸ designed and experimentally demonstrated a stretchable metasurface [Figs. 5(c)–5(d)]. By embedding a Si metasurface into a layer of stretchable polydimethylsiloxane (PDMS), under different amounts of strain, the two different functionalities can be switched. The periodicity of the array of meta-atoms is controlled through stretching the PDMS to produce a highly transparent film or a Laplacian operator. The physical modulation of the meta-atom locations changes the coupling between neighbors, allowing for the two different responses. A bandwidth of 60 nm with an NA of up to 0.25 was proven at the working wavelength of 1230 nm. The device showed robustness to up to 50 cycles with no reduction in functionality, highlighting the potential for use in real-life situations. Furthermore, since the impulse response of the metasurface is designed directly, there is no need for any bulky imaging systems, as no lens is required to perform an FT.

Another widely used method of achieving tunable optical responses with metasurfaces is to employ phase-change materials, such as germanium antimony telluride (GST).⁹⁰^–⁹² Yang et al.⁹³ used a GST variant, ${Ge}_{2} {Sb}_{2} {Se}_{4} {Te}_{1}$ (GSST) to dynamically switch between bright-field imaging and edge detection through the reversible, nonvolatile phase transition from crystalline to amorphous GST. The designed metasurface was placed in the focal plane of a 4f system and showed a polarization-insensitive functionality at the working wavelength of 1550 nm. This is where GSST shows the biggest modulation in optical properties over the phase change, while maintaining a reasonable amount of optical loss. For those reasons, using GST at visible wavelengths is generally quite difficult, as there is a large amount of loss and the variance in optical properties is fairly weak. Nevertheless, in the near-infrared regime, bright-field imaging and edge detection were successfully demonstrated numerically. Recently, a completely reconfigurable device that could possibly act as the basis for solutions to any problem using the gradient descent method has been proposed, highlighting the potential that metamaterials have in all-optical computing.⁹⁴

The progress of metamaterials for all-optical computational operations has been profound, and such advances point directly at the development of completely optical neural networks (ONNs) that could be able to perform complex computations at the speed of light. With the current speed of development of high-resolution cameras in consumer products, such as smartphones, along with the large file sizes and an increase in the amount of data that is being processed daily, such low-power fully optical solutions could lead the way for the next generation of computing.

3. All-Optical Machine Learning

Machine learning has been employed extensively in the field of nanophotonics and metamaterials for forward modeling and inverse design over the past few years.⁹⁵^–¹⁰⁰ More recently, metamaterials have been employed directly as the calculating elements, allowing for processing of information at the speed of light. Metasurfaces in particular have been proven to perform all-optical matrix calculations through the ability to manipulate the properties of light at defined spatial locations through the correct design of subwavelength-sized meta-atoms.¹⁰¹^–¹⁰³ This functionality is extremely important in the field of machine learning, where models are trained through a combination of matrix multiplications and backpropagation.

The processing of optical information has been a challenge for optical circuits that are needed to produce all-optical computers.¹⁰⁴^–¹⁰⁶ Mathematically, artificial neural networks (ANNs) consist of sequences of matrix products and nonlinear functions. They must be implemented physically in the shape of optical elements for the realization of all-optical ANNs. By employing metasurfaces to perform the matrix calculations, there is no need for additional hardware, such as optical cables, as each calculation can be mapped exactly to a specific meta-atom that diffracts light to another specific location. Therefore, there has been a lot of development of all-optical matrix operations using metasurfaces that have been implemented as fully optical ANNs that perform calculations at the speed of light with an extremely low-energy consumption.¹⁰⁵^–¹¹⁰ Conventional ANNs have already shown great promise for regression and classification tasks, as well as at controlling the response of metamaterials in real time.¹¹¹ All-optical versions could be the next step toward all-optical computations for complex tasks that require machine learning.¹¹² In this section, we review the recent progress in the use of metamaterials, especially metasurfaces, for applications of all-optical machine learning. We will start with a brief overview of the principles of how the metasurface-based ONNs work based on the diffraction and propagation of EM waves. For a more detailed breakdown of ANNs and machine learning in general with relation to photonic applications, we refer the readers to the following interesting review papers.¹¹³^–¹¹⁶

ONNs are a physical manifestation of the multiple complex matrix computations that are required for traditional ANNs based on classical computing. However, rather than using digital bits of “0” and “1,” the diffraction of the complex amplitude of light can be exploited using flat optical elements with spatially varying phase and amplitude modulation. Metasurfaces are an obvious and noteworthy choice due to their similarity and almost one-to-one correspondence to ANNs, where each physical meta-atom in a metasurface relates to one element, or neuron, of the ANN. A schematic illustration is shown in Fig. 6(a). First, we will compare the constituent elements of ANNs and ONNs and explain the principle of ONNs in detail numerically.

Fig. 6

Machine learning using all-optical neural networks. (a) Comparison of the correspondence between ANNs and ONNs. The ANN is made up of neurons and hidden layers, which relate to the physical meta-atoms and cascaded metasurfaces in ONNs. (b) The coordinate system required to design ONNs using diffraction.

For traditional computational ANNs, each neuron in each layer plays four main roles: (1) collecting data transferred from the previous layer, (2) multiplying the data by the set weight, (3) processing the data with a nonlinear activation function, and (4) sending the processed data to the next layer. Without any nonlinear activation functions, the resulting ANN would only be able to produce a linear function, regardless of the number of hidden layers; therefore, it is equivalent to a single-layer perceptron.¹¹⁷ However, real-life data are generally nonlinear, which means that nonlinearities in the ANN are of utmost importance to solve complex problems. In an ONN, each neuron is replaced with a physical meta-atom that plays a similar part, namely, (1) collecting the light transmitted from the previous layer, (2) modulating the phase, amplitude, and polarization of the incoming light, and (3) propagating the light to the next layer, which is generally modeled and calculated as a diffraction problem. The biggest difference between ANNs and current ONNs is the use of nonlinear activation functions. It should be noted that due to the distinct lack of nonlinearity in the system, the subsequent ONNs are strictly linear in nature. To ensure brevity, here we refer to these linear ONNs as ONNs throughout. Cascaded metasurfaces play the role of the physical counterpart of the hidden layers in ANNs. 2D plane waves containing specific images have been generally used as the input data for ONNs as shown in Fig. 6(b). The desired output target that is set as the output layer in an ANN is generally designed to be a specific intensity of the light at a designated position. The diffraction of light from each individual meta-atom can be described by the Huygens principle; thereby, the light passing through a meta-atom at each spatial location on a metasurface layer is considered to be a secondary source that irradiates from that location. The multiplied weight of an ONN is realized by manipulating the complex amplitude of the light at that position, that is to say, the amplitude and phase of the incident light are modulated. The mathematical basis for ONNs is explained in the next section.

3.1.

Mathematical Basis for ONNs

We examine the mathematical basis for ONNs here to help understand the fundamentals of how diffractive ONNs work. The coordinate system for the following formulas is shown in Fig. 7(b). Using the Rayleigh–Sommerfeld formulation of diffraction by a planar screen, we can calculate the light transmitted from the $i$ ’th meta-atom of the $l$ ’th metasurface to the $j$ ’th meta-atom of the ( $l + 1$ )’th metasurface.¹²⁰ In other words, the light collected by the $j$ ’th meta-atom of the ( $l + 1$ )’th metasurface is the integration of all the secondary sources generated by every meta-atom of the $l$ ’th metasurface. The location of each secondary source ${\vec{r}}_{i}^{l}$ can be expressed as ${\vec{r}}_{i}^{l} = (x_{i}^{l}, y_{i}^{l}, z^{l})$ . Then the $j$ ’th position of the ( $l + 1$ )’th layer, ${\vec{r}}_{j}^{l + 1}$ , where the light is gathered, can be expressed as ${\vec{r}}_{j}^{l + 1} = (x_{j}^{l + 1}, y_{j}^{l + 1}, z^{l + 1})$ . As shown in Fig. 7(b), it is assumed that light propagates in the $z$ direction, so the values of $z$ are all equal for a given layer. The contribution of the $i$ ’th meta-atom on the $l$ ’th layer to the $j$ ’th meta-atom on the ( $l + 1$ )’th layer is, therefore, determined by the following equation:

Eq. (6)

H_{z}^{Huy} ({\vec{r}}_{j}^{l + 1} - {\vec{r}}_{i}^{l}) = G ({\vec{r}}_{i}^{l}) \cdot h_{z}^{Huy} ({\vec{r}}_{j}^{l + 1} - {\vec{r}}_{i}^{l}),

where

G ({\vec{r}}_{i}^{l})

is the state of light immediately after exiting the

i

’th meta-atom of the

l

’th layer.

G ({\vec{r}}_{i}^{l})

is the product of the electric field

u ({\vec{r}}_{i}^{l})

that is incident on the meta-atom, and the transmission coefficient

t ({\vec{r}}_{i}^{l})

that is determined by the design of the meta-atom [Eq. (7)]. The meta-atom can be designed to manipulate the complex amplitude of the incident light, i.e., both the real part, related to the intensity

t

, and the imaginary part, related to phase delay

φ

:

Eq. (7)

G ({\vec{r}}_{i}^{l}) = u ({\vec{r}}_{i}^{l}) \cdot t ({\vec{r}}_{i}^{l}),

Eq. (8)

t ({\vec{r}}_{i}^{l}) = t \cdot e^{i \cdot φ} .

h_{z}^{Huy} ({\vec{r}}_{j}^{l + 1} - {\vec{r}}_{i}^{l})

is the Huygens wavelet of such a secondary wave source of the

i

’th meta-atom of the

l

’th layer as shown in the following equation:

Eq. (9)

h_{z}^{Huy} ({\vec{r}}_{j}^{l + 1} - {\vec{r}}_{i}^{l}) = \frac{- 1}{2 π} (i k - \frac{1}{R}) \frac{i k R z^{l + 1}}{R} \frac{- z^{l}}{R},

where

R

is

\sqrt{{(x_{j}^{l + 1} - x_{i}^{l})}^{2} + {(y_{j}^{l + 1} - y_{i}^{l})}^{2} + {(z^{l + 1} - z^{l})}^{2}}

, and

k

is the wave vector of light in free space. Finally, the total electric field at

{\vec{r}}_{j}^{l + 1}

,

u ({\vec{r}}_{j}^{l + 1})

, is calculated as the sum of all contributions of the sources at

{\vec{r}}_{i}^{l}

as expressed as Eq. (6), which is given by

Eq. (10)

u ({\vec{r}}_{j}^{l + 1}) = \iint_{- \infty}^{\infty} H_{z}^{Huy} ({\vec{r}}_{j}^{l + 1} - {\vec{r}}_{i}^{l}) d x d y,

where

x

and

y

are integrated in the plane of the

l

’th metasurface.

Fig. 7

Utilization of metasurfaces for ONNs. (a) Schematic of the input mask layer, showing the logic operation of “1 + 0.” (b) Amplitude and phase response of the dielectric meta-atoms. (c) Experimental setup of the two cascaded metasurfaces. (d) Distribution of the measured normalized intensity of the EM field. Most of the energy is focused to the region denoted for 0 or 1. (e), (f) ONNs for image classification and imager. (e) Schematic of the (i) MNIST data set classification and (ii) imager. (f) (i) An example of an input image expressed in phase and (ii) the output energy distribution. (iii) Confusion matrices and (iv) energy distribution percentages for experimental and numerical results of the fashion data set. (g) On-chip ONN working at visible wavelengths for image classification tasks using (i) MNIST and (ii) fashion data sets for $x$ and $y$ polarized light, respectively. (h) The confusion matrices for the (i) MNIST and (ii) fashion data sets for 80 randomly selected images. Images (a)–(d) reproduced with permission from Ref. 108. Copyright 2020 Springer Nature Group. Images (e) and (f) reproduced with permission from Ref. 118. Copyright 2018 The American Association for Advancement of Science. Images (g) and (h) reproduced with permission from Ref. 119. Copyright 2022 Springer Nature Group.

To calculate the required phase and amplitude modulations at each spatial location of the metasurface that needs to be encoded with meta-atoms, Eq. (8) can be used as the forward function of the ONN. It should be noted that any alternative propagation formula can be used in place of Eq. (8), as long as it is physically correct. The “weight” of each neuron is optimized through an iterative backpropagation algorithm, in the same way as ANNs are trained. The output is generally calculated as the mean squared error between the calculated and desired amplitude of the output electric field, i.e., ${(| u ({\vec{r}}_{j}^{l + 1}) |)}^{2}$ . The “weights” of each neuron are then encoded to the metasurface through the optical properties of the designed meta-atoms based on their calculated complex transmission or reflection coefficients.

3.2.

Metasurface-Based ONNs

Since the propagation of light is a linear operation, a fundamental challenge occurs for ONNs with regard to performing complex functions, one of the key abilities that ANNs have been exploited for. Furthermore, unlike electric signals that can be boosted using simple amplifiers, light signals do not have an analogous operation to help increase the transmission. Therefore, the intensity of the light unavoidably becomes weaker as it passes through each metasurface. This makes it difficult to construct deep ONNs with many layers and therefore limits more complex network structures. In addition, since static metasurfaces are fixed after fabrication, backpropagation through the ONN cannot be performed optically, so the weights of each neuron must be calculated computationally before implementing them physically with meta-atoms. ONNs can be interpreted as all-optical computations that convert the optical properties of a given input EM wave into an output EM wave that provides some information about the input, such as classification. All properties of light can be exploited in this way, including but not limited to the amplitude, phase, and polarization. In this section, we will review the latest work in the field and discuss the key limitations of ONNs that must be overcome.

In one interesting experimental demonstration of an ONN using metasurfaces, Qian et al.¹⁰⁸ demonstrated two cascaded metasurfaces capable of performing basic logic operations at microwave frequencies (17 GHz). Seven logical operations (“NOT,” “OR,” “AND,” “NOR,” “XNOR,” “NAND,” and “XOR”) were shown to be feasible through numerical calculations and furthermore, three of these operations (NOT, OR, and AND) were demonstrated experimentally. Each logical operation was spatially encoded in the input mask layer, as shown in Fig. 7(a). The amplitude of the output EM wave was designated to be focused into one of two spatial locations depending on the result of the operation. The ONN was trained to find the required transmission coefficients needed for the metasurfaces; however, rather than using the complex amplitude of the EM field, phase-only metasurfaces were implemented. Therefore, meta-atoms with the phase to cover the whole $2 π$ range and high transmission were designed by varying the height of a square dielectric meta-atom. The high transmission and almost linear phase of the designed meta-atoms are shown in Fig. 7(b). Using this two hidden-layer ONN [Fig. 7(c)], the results were experimentally verified [Fig. 7(d)], with most of the energy in the EM fields being focused into one of the two regions. Rather than using a simple mask that shapes the input EM field, active modulators such as SLM or digital micromirror devices can be used to change the input EM fields at high frequency, which could then be processed at the speed of light using the well-trained ONN. Various other examples of optical logic gates have been demonstrated, such as on-chip 1D metamaterials¹²¹ and 2D images.¹²²^,¹²³ These kinds of logic operations could have profound implications for applications, such as real-time object recognition.

In an example of image classification using ONNs, Lin et al.¹¹⁸ proposed ONNs made up of five cascaded metasurfaces to classify the well-known handwritten number data set from MNIST and the fashion item data set from ImageNet, as well as an imager, which could be considered to be a physical implementation of an autoencoder [Fig. 7(e)]. The ONN was designed to operate in the terahertz regime. To demonstrate the robustness of image classification, the MNIST data were used as the input EM wave as amplitude objects, whereas the fashion items were defined as phase objects; both were upsampled and binarized before being physically encoded through 3D printed masks. The required phase at each spatial location in the hidden layers was physically encoded by modulating the thickness of each 3D-printed meta-atom, allowing for complete coverage of the $2 π$ space. The intensity of the output light was spatially encoded to 10 different areas to determine the classification of the images. The imager produced a lens type functionality that displayed a clear image of the input after propagating through the five-layer ONN. The classification accuracy of the MNIST and fashion data sets was 93.39% and 91.75%, respectively, for a numerically implemented ANN [Fig. 7(f)], while the experimental accuracy of the ONN reached 86.60% and 81.13%. Using full complex amplitude modulation rather than phase-only metasurfaces as the hidden layers, the accuracy of the ONN for the fashion item data set was improved to 86.33%, highlighting the promise of fully exploiting the vectorial nature of light for optical calculations. Unlike logic operations, where all of the output energy should be confined to the true or false regions, similar to ANNs, the intensity at each location can be interpreted as a probability for the classification images. However, postprocessing of the output is required, as there is no physical implementation of a softmax layer.

Recently, research about metasurface-based ONNs working at visible wavelengths has been proposed and experimentally demonstrated. By exploiting the multiplexing nature of metasurfaces for use in ONNs, the number of functions that a single ONN can achieve has been increased.¹¹⁹ By employing polarization-dependent dual-channel metasurfaces in the ONN, the exciting possibility of image classification of both the MNIST and fashion data sets in a single device was proven. By designing asymmetric meta-atoms that have different optical responses to $x$ - and $y$ -polarized light, an on-chip ONN integrated with a complementary metal-oxide semiconductor sensor with an operating wavelength of 532 nm was experimentally demonstrated for eight different classes [Fig. 7(g)]. This showed a match of 93.75% and 95% between the simulations and experiments for the MNIST and fashion data sets, respectively, with only a few errors [Fig. 7(h)]. Subsequently, a metasurface-based ONN operating at the wavelength of 633 nm has also been proven.¹²⁴ An accuracy of 98.05% in experiment was demonstrated for image classification using 10 classes of the MNIST data set. By integrating the metasurface with an SLM, the input EM field can be electrically modulated. However, for both of these systems, only a single hidden layer was fabricated, highlighting both the difficulties for precise alignment of the cascaded components, as well as the power of analog calculations using EM waves for image classification problems.

The ability of ONNs to process light and perform calculations at the speed of light has been proven extensively for a diverse number of applications so far; however, these have been mostly focused on image classification. Examples of OAM processing have been numerically demonstrated,¹²⁵^,¹²⁶ as well as using metamaterials in the acoustic regime.¹²⁷ Up to now, experimental demonstrations of ONNs have been generally limited to longer wavelengths due to difficulties in the fabrication of nanoscale metasurfaces for use at visible wavelengths, and especially due to the added complications that arise from aligning metasurfaces spatially at such small-length scales. Nevertheless, there has been significant progress in cascading metasurfaces that operate at visible wavelengths with successful demonstrations in metaholography and metalenses,¹²⁸^,¹²⁹ which prove the feasibility of ONNs using visible light. The next step toward ONNs that can be trained and modulated after fabrication is to include a form of tunability to the metasurfaces. In the next section, we will look at the current state of tunable metasurface-based ONNs.

3.3.

Programmable Metasurface-Based ONNs

The metasurface-based ONNs presented so far were all passive systems that were preoptimized computationally before being implemented physically. After fabrication, the designed function is fixed and cannot be modulated due to the lack of active modulation at the meta-atom scale. Active and multiplexed metamaterials have been a hot topic recently as discussed in the previous section. For the active modulation of ONNs, so-called coding, digital or programmable metamaterials are required.¹³⁰ This allows for the complex transmission or reflection coefficient of each meta-atom to be modulated at will after fabrication. In this section, we will discuss the recent progress in tunable metasurface-based all-optical ONNs.

By employing a 2-bit programmable metasurface, Li et al.¹³¹ demonstrated an in situ digital imager using field-programmable gate arrays (FPGAs) to program the metasurface in real time, modulating the desired radiation patterns based on machine-learning guided imaging at 3.2 GHz. After training, the imager was able to monitor and recognize human movement, and even infer the body position when blocked from direct sight [Fig. 8(a)]. With just 100 measurements, reasonable results were acquired when training using machine-learning methods through the data embedding principal component analysis technique. A similar idea was proposed but with the addition of an ANN to further process the data from the programmable metasurface imager.¹³³ It was shown that the system was robust to noise from Wi-Fi signals that could interfere with the signals at $\sim 2.4 GHz$ , and that it could successfully monitor the movements of humans, on both the macroscale, in terms of movements and gestures, and the microscale, in terms of subtle physiological states. More recently, a high-performance imager using a programmable metasurface trained using the multilayer perceptron method has been reported.¹³⁴ Under noisy conditions, the results show an 18% improvement on the random scattering method, along with an improvement in the reconstruction of the input image. Such programmable metasurface-based imagers have set the foundations for tunable ONNs; however, each only employed a single tunable metasurface to achieve its functionality.

Fig. 8

Programmable metasurfaces for ONNs. (a) Optical images and reconstructions from the single-layer programmable metasurface. Experimental demonstrations of (b) image classification between the letter “I” and square brackets, (c) simultaneous transmission of four orthogonal codes, and (d) on-site reinforcement learning using the programmable five-layer ONN. Image (a) reproduced with permission from Ref. 131. Copyright 2022 Springer Nature Group. Images (b)–(d) reproduced with permission from Ref. 132. Copyright 2022 Springer Nature Group.

In a notable example of a tunable ONN, Liu et al.¹³² proposed a five-layer tunable ONN using FPGAs to modulate the complex transmission coefficient of each meta-atom by manipulating the voltage of each amplifier chip. The working frequency of the ONN of 5.4 GHz lies in the microwave regime, and each metasurface layer was made up of just $8 \times 8$ meta-atoms, which is generally fewer than the number of neurons that are used in ANNs. Nevertheless, the ONN demonstrated an experimental accuracy of 100% for a simple classification problem with images of the letter $I$ and square brackets [Fig. 8(b)] as well as digits from 1 to 4. Furthermore, the same ONN with different voltages provided by the FPGAs was used to demonstrate a code division multiple access task using the first metasurface as an encoder and the following four as a decoder [Fig. 8(c)]. Due to the independence of wave propagation, each code can be successfully decoded, even when they are overlapping. Space-time telecommunications were then demonstrated using this encoder and decoder system with an error rate of just 0.52% at a transmission rate of $1000 bit s^{- 1}$ . Finally, the fully programmable ONN was dynamically trained to focus the EM energy into arbitrary locations using reinforcement learning [Fig. 8(d)]. More than 90% of the energy in the output EM field was successfully concentrated in the target area after the active training. Typically, the complex amplitude of the transmission coefficient has been calculated numerically in advance using a standard ANN; however, the on-site training of the ONN using reinforcement learning demonstrates the key ability of being able to adjust and program the optical response in relation to interactions with an unknown environment. This important result indicates the potential for programmable ONNs to be trained actively on any new optical problem without the need for a training data set, while also alleviating the requirement of accurate simulations and exact physical implementation as the ONN directly updates its optical properties based on the exact experimental environment.

Although the field of metasurface-based ONNs has been developing at a remarkable rate, there are still a few major problems that must be solved in order to fully utilize all optically trainable and reconfigurable ONNs that can solve complex problems. First, only a few experimental demonstrations of the full modulation of the complex amplitude of the transmission or reflection coefficient have been proven, a component that has been proven to improve the accuracy of ONNs.¹³⁵ A form of feedback between the output of the ONN and the optical properties of the individual meta-atoms must be developed to allow for all optical backpropagation and training without the need for a computer. One potential solution has been suggested theoretically.¹³⁶ This could also lead to the requirement of the development of physical implementations of other layers that are often used in ANNs, as currently, the metasurfaces are limited to acting as standard linear layers. For example, the development of physical layers for functions, such as softmax, max pooling, and normalization layers, could greatly improve the performance of ONNs, especially if they are to be trained optically. An optical version of a dropout layer has already been demonstrated¹³⁷ and also implementations of recurrent ONNs.¹³⁸^–¹⁴⁰ For metasurface-based ONNs to achieve the ability to solve complex nonlinear functions, a problem that ANNs are particularly astute at, a method of implementing an optically activated nonlinearity should be developed, as we highlight again that the current state-of-the-art work presented here consists of only linear systems.¹⁴¹ It has been shown that nonlinear ONNs outperform their linear counterparts,¹¹⁸^,¹⁴² and various methods of physically implementing nonlinearity in ONNs without metasurfaces have been demonstrated, such as using thermal atoms,¹⁴³ Kerr-type nonlinear materials,¹⁴⁴ and induced transparency through sandwiching quantum dots between metallic nanoparticles.¹⁴⁵ Materials, such as phase-change materials,¹⁴⁶ semiconductors,¹⁴⁷ and liquids,¹⁴⁸ are potential options to produce nonlinear responses, and the correct choice would depend on the wavelength regime under consideration to provide a response at a reasonable power and efficiency.¹⁴⁹^–¹⁵³ Finally, with respect to miniaturization and operation at visible wavelengths, new methods of manipulating individual meta-atoms at the nanoscale must be developed experimentally. This has been performed optically through the use of phase-change materials; however, this requires high-power lasers and long quenching times. Additionally, their optical properties in the visible regime limit their widespread application due to large extinction coefficients that would limit the performance of ONNs.¹⁵⁴^–¹⁵⁷ Nevertheless, the rapid development of ONNs has proven their capability to perform all-optical calculations at the speed of light and has shown great potential for classification problems in particular, which have direct applications in fields such as computer vision for semantic segmentation and LiDAR.¹⁵⁸^,¹⁵⁹

4. Conclusion and Outlook

The general consensus is that the likelihood of photonics to completely replace electronics is fairly low, due to the extremely mature electronic systems and fabrication processes that are already in place. However, as discussed in this review, the progress of photonics toward all-optical computation has developed at an impressive rate. Before there is any chance of photonics to replace electronics, it is highly likely that the two will be developed hand in hand as complementary technologies rather than competing ones. The integration of photonics with Si nanoelectronics has already begun,¹⁶⁰^–¹⁶² with notable examples of photonic systems on a chip.¹⁶³^,¹⁶⁴ The biggest drawback of such systems is that a lot of energy is wasted converting photons to electrical signals and back.

All-optical computing systems are naturally larger than the electronic chips, which could be a limitation that needs to be addressed in the future. However, their ability to use multiple wavelengths of light highlights their potential for parallel processing of big data with little energy cost. Another interesting possibility that is opened up with using completely photonic systems is the chance to directly integrate optical data storage.¹⁶⁵^,¹⁶⁶ In the short term, integrated photonic systems using metasurfaces have already shown great promise for specific applications such as all-solid-state LiDAR, which could be influential in autonomous driving,¹⁶⁷^,¹⁶⁸ as well as in biophotonics for noninvasive diagnostics,¹⁶⁹^–¹⁷² and advanced sensors with extremely small physical footprints. The use of EM waves for all-optical computation has additional benefits due to the vectorial nature of light, and the capacity of such diffractive surfaces has been investigated.¹⁷³ This means that advanced functionalities can be designed into the optical components themselves, as well as unlocking new options that cannot be accessed using traditional computations. One example in particular that exploits the vectorial nature of EM waves is phase imaging, where the contrast of phase objects can be significantly increased, promoting new avenues of bioimaging and sensing.¹⁷⁴^–¹⁷⁶

The engineering of materials to suppress optical losses could be influential in designing meta-atoms with unity transmission,¹⁷⁷^–¹⁷⁹ allowing for highly efficient metasurfaces, and therefore, easing the limitations of how many metasurfaces can be cascaded in the hidden layers of ONNs. In addition, large-scale fabrication techniques, such as nanoimprint lithography,¹⁸⁰^–¹⁸⁷ holographic lithography,¹⁸⁸ and self-assembly,¹⁸⁹^–¹⁹¹ could allow for the mass production of metasurfaces for ONNs. Together with advancements in nonlinear responses,¹⁹² deep ONNs with nonlinear activation functions could be a reality. Furthermore, with coding metasurfaces that allow for the manipulation of the properties of single meta-atoms or unit cells in real time, ONNs could be trained completely optically, proving that the future has the possibility to be driven by optics. Work on such devices has recently been undertaken in the microwave regime¹⁹³^,¹⁹⁴ and programming metasurfaces through the power of touch,¹⁹⁵ as well as examples of spatiotemporal functionality.¹⁹⁶^–²⁰¹ Limitations related to controlling and aligning metasurfaces down to the meta-atom level at shorter wavelengths hinder the progress of fully optical ONNs working at visible wavelengths. However, the standard working wavelengths of LiDAR lie in the near-infrared region, possibly easing such intricate fabrication and control constraints, as well as opening the door for the inclusion of tunable materials such as ITO and GST.

Current methods for all-optical computing are based on either direct processing of EM waves, through the design of metamaterials with inherent incident angle dependencies, or through the manipulation of the information in the Fourier plane of a 4f system. Although these techniques allow for numerical calculations at the speed of light, their optical properties are generally fixed, which allows them to only solve a single function after fabrication. The design of actively tunable metamaterials that can be modulated at will to solve arbitrary functions would be a giant leap toward optical computing and even quantum computations using single-photon sources. To be able to solve both even and odd functions with a single metasurface, a method of introducing tunable symmetry breaking in the $x$ and $z$ directions must be realized. In terms of ONNs, the obvious limitation is that they are currently distinctively linear solvers, so new ways to introduce optical nonlinearities at reasonable energies should be uncovered. Furthermore, there must be substantial advancements in 3D nanofabrication methods to produce ONNs that operate at visible wavelengths with multiple hidden layers.

In summary, all-optical computing at the speed of light using metamaterials has developed extensively over the past decade, with numerous demonstrations that could have potential impact in real-life applications. One exciting advancement is that of all-optical machine learning using diffractive neural networks; however, there is still a lot of research yet to be done before their implementation outside of research labs. Although there is a long way to go, especially regarding all-optical training, including backpropagation and active feedback, along with operation at visible wavelengths and optical nonlinearities, the fundamental work toward a completely photonic brain that can learn, process, and store masses of data is well under way.

Acknowledgments

This work was financially supported by the POSCO-POSTECH-RIST Convergence Research Center Program funded by POSCO and the National Research Foundation (NRF) (Grant Nos. NRF-2022M3C1A3081312, NRF-2022M3H4A1A02074314, NRF-2022M3H4A1A02085335, CAMM-2019M3A6B3030637, and NRF-2019R1A5A8080290) funded by the Ministry of Science and ICT, Republic of Korea. The authors declare that they have no competing interests. J. R. conceived the idea and initiated the project. T. B. and S. L. designed the research and mainly wrote this paper. All authors confirmed the final version of the paper. J. R. guided the entire research.

References

1.

H. J. Caulfield and S. Dolev, “Why future supercomputing requires optics,” Nat. Photonics, 4 (5), 261 –263 https://doi.org/10.1038/nphoton.2010.94 (2010). Google Scholar

2.

E. Pelucchi et al., “The potential and global outlook of integrated photonics for quantum technologies,” Nat. Rev. Phys., 4 (3), 194 –208 https://doi.org/10.1038/s42254-021-00398-z (2022). Google Scholar

3.

Z. Ying et al., “Electronic-photonic arithmetic logic unit for high-speed computing,” Nat. Commun., 11 (1), 2154 https://doi.org/10.1038/s41467-020-16057-3 NCAOBW 2041-1723 (2020). Google Scholar

4.

“Samsung begins chip production using 3 nm process technology with GAA architecture,” https://news.samsung.com/global/samsung-begins-chip-production-using-3nm-process-technology-with-gaa-architecture Google Scholar

5.

L. Li et al., “Intelligent metasurfaces: control, communication and computing,” eLight, 2 (1), 7 https://doi.org/10.1186/s43593-022-00013-3 (2022). Google Scholar

6.

F. Zangeneh-Nejad et al., “Analogue computing with metamaterials,” Nat. Rev. Mater., 6 (3), 207 –225 https://doi.org/10.1038/s41578-020-00243-2 (2021). Google Scholar

7.

S. He, R. Wang and H. Luo, “Computing metasurfaces for all-optical image processing: a brief review,” Nanophotonics, 11 (6), 1083 –1108 https://doi.org/10.1515/nanoph-2021-0823 (2022). Google Scholar

8.

L. Wesemann, T. J. Davis and A. Roberts, “Meta-optical and thin film devices for all-optical information processing,” Appl. Phys. Rev., 8 (3), 031309 https://doi.org/10.1063/5.0048758 (2021). Google Scholar

9.

S. Abdollahramezani, O. Hemmatyar and A. Adibi, “Meta-optics for spatial optical analog computing,” Nanophotonics, 9 (13), 4075 –4095 https://doi.org/10.1515/nanoph-2020-0285 (2020). Google Scholar

10.

L. L. Doskolovich et al., “Spatial differentiation of Bloch surface wave beams using an on-chip phase-shifted Bragg grating,” J. Opt., 18 (11), 115006 https://doi.org/10.1088/2040-8978/18/11/115006 (2016). Google Scholar

11.

D. A. Bykov et al., “Optical computation of the Laplace operator using phase-shifted Bragg grating,” Opt. Express, 22 (21), 25084 –25092 https://doi.org/10.1364/OE.22.025084 OPEXFF 1094-4087 (2014). Google Scholar

12.

Z. Ruan, “Spatial mode control of surface plasmon polariton excitation with gain medium: from spatial differentiator to integrator,” Opt. Lett., 40 (4), 601 –604 https://doi.org/10.1364/OL.40.000601 OPLEDP 0146-9592 (2015). Google Scholar

13.

A. Silva et al., “Performing mathematical operations with metamaterials,” Science, 343 (6167), 160 –163 https://doi.org/10.1126/science.1242818 SCIEAS 0036-8075 (2014). Google Scholar

14.

I. Kim et al., “Nanophotonics for light detection and ranging technology,” Nat. Nanotechnol., 16 (5), 508 –524 https://doi.org/10.1038/s41565-021-00895-3 NNAABX 1748-3387 (2021). Google Scholar

15.

J. Park et al., “All-solid-state spatial light modulator with independent phase and amplitude control for three-dimensional LiDAR applications,” Nat. Nanotechnol., 16 (1), 69 –76 https://doi.org/10.1038/s41565-020-00787-y NNAABX 1748-3387 (2021). Google Scholar

16.

T. Cao et al., “Tuneable thermal emission using chalcogenide metasurface,” Adv. Opt. Mater., 6 (16), 1800169 https://doi.org/10.1002/adom.201800169 2195-1071 (2018). Google Scholar

17.

B. Muminov and L. T. Vuong, “Fourier optical preprocessing in lieu of deep learning,” Optica, 7 (9), 1079 –1088 https://doi.org/10.1364/OPTICA.397707 (2020). Google Scholar

18.

H. Kwon et al., “Nonlocal metasurfaces for optical signal processing,” Phys. Rev. Lett., 121 (17), 173004 https://doi.org/10.1103/PhysRevLett.121.173004 PRLTAO 0031-9007 (2018). Google Scholar

19.

P. Viola and M. J. Jones, “Robust real-time face detection,” Int. J. Comput. Vision, 57 (2), 137 –154 https://doi.org/10.1023/B:VISI.0000013087.49260.fb IJCVEQ 0920-5691 (2004). Google Scholar

20.

S. J. Kim et al., “Reconfigurable, vivid reflective colors based on solution-processed Fabry–Perot absorber using thermochromic vanadium dioxide,” Appl. Surf. Sci., 565 150610 https://doi.org/10.1016/j.apsusc.2021.150610 ASUSEE 0169-4332 (2021). Google Scholar

21.

B. Ko et al., “Multilevel absorbers via the integration of undoped and tungsten-doped multilayered vanadium dioxide thin films,” ACS Appl. Mater. Interfaces, 14 (1), 1404 –1412 https://doi.org/10.1021/acsami.1c19223 AAMICK 1944-8244 (2022). Google Scholar

22.

A. Lalbakhsh, M. U. Afzal and K. P. Esselle, “Multiobjective particle swarm optimization to design a time-delay equalizer metasurface for an electromagnetic band-gap resonator antenna,” IEEE Antennas Wireless Propag. Lett., 16 912 –915 https://doi.org/10.1109/LAWP.2016.2614498 IAWPA7 1536-1225 (2017). Google Scholar

23.

T. Zhu et al., “Plasmonic computing of spatial differentiation,” Nat. Commun., 8 (1), 15391 https://doi.org/10.1038/ncomms15391 NCAOBW 2041-1723 (2017). Google Scholar

24.

L. Wesemann et al., “Selective near-perfect absorbing mirror as a spatial frequency filter for optical image processing,” APL Photonics, 4 (10), 100801 https://doi.org/10.1063/1.5113650 (2019). Google Scholar

25.

T. Zhu et al., “Generalized spatial differentiation from the spin Hall effect of light and its application in image processing of edge detection,” Phys. Rev. Appl., 11 (3), 034043 https://doi.org/10.1103/PhysRevApplied.11.034043 PRAHB2 2331-7019 (2019). Google Scholar

26.

A. Youssefi et al., “Analog computing by Brewster effect,” Opt. Lett., 41 (15), 3467 –3470 https://doi.org/10.1364/OL.41.003467 OPLEDP 0146-9592 (2016). Google Scholar

27.

D. Xu et al., “Optical analog computing of two-dimensional spatial differentiation based on the Brewster effect,” Opt. Lett., 45 (24), 6867 –6870 https://doi.org/10.1364/OL.413104 OPLEDP 0146-9592 (2020). Google Scholar

28.

T. Zhu et al., “Topological optical differentiator,” Nat. Commun., 12 (1), 680 https://doi.org/10.1038/s41467-021-20972-4 NCAOBW 2041-1723 (2021). Google Scholar

29.

M. Kim et al., “Generalized analytic formula for spin Hall effect of light: shift enhancement and interface independence,” Nanophotonics, 11 (11), 2803 –2809 https://doi.org/10.1515/nanoph-2021-0794 (2022). Google Scholar

30.

M. Kim, D. Lee and J. Rho, “Spin Hall effect under arbitrarily polarized or unpolarized light,” Laser Photonics Rev., 15 (7), 2100138 https://doi.org/10.1002/lpor.202100138 (2021). Google Scholar

31.

M. Kim et al., “Reaching the highest efficiency of spin Hall effect of light in the near-infrared using all-dielectric metasurfaces,” Nat. Commun., 13 (1), 2036 https://doi.org/10.1038/s41467-022-29771-x NCAOBW 2041-1723 (2022). Google Scholar

32.

J. Huang, T. Zhu and Z. Ruan, “Two-shot calibration method for phase-only spatial light modulators with generalized spatial differentiator,” Phys. Rev. Appl., 14 (5), 054040 https://doi.org/10.1103/PhysRevApplied.14.054040 PRAHB2 2331-7019 (2022). Google Scholar

33.

T. Zhu, J. Huang and Z. Ruan, “Optical phase mining by adjustable spatial differentiator,” Adv. Photonics, 2 (1), 016001 https://doi.org/10.1117/1.AP.2.1.016001 AOPAC7 1943-8206 (2020). Google Scholar

34.

C. Mi et al., “Tunable optical spatial differentiation in the photonic spin Hall effect,” Opt. Express, 28 (20), 30222 –30232 https://doi.org/10.1364/OE.406202 OPEXFF 1094-4087 (2020). Google Scholar

35.

W. Xue and O. D. Miller, “High-NA optical edge detection via optimized multilayer films,” J. Opt., 23 (12), 125004 https://doi.org/10.1088/2040-8986/ac370c (2021). Google Scholar

36.

F. Zangeneh-Nejad and A. Khavasi, “Spatial integration by a dielectric slab and its planar graphene-based counterpart,” Opt. Lett., 42 (10), 1954 –1957 https://doi.org/10.1364/OL.42.001954 OPLEDP 0146-9592 (2017). Google Scholar

37.

A. Pors, M. G. Nielsen and S. I. Bozhevolnyi, “Analog computing using reflective plasmonic metasurfaces,” Nano Lett., 15 (1), 791 –797 https://doi.org/10.1021/nl5047297 NALEFD 1530-6984 (2015). Google Scholar

38.

Y. Hwang et al., “Plasmonic circuit for second-order spatial differentiation at the subwavelength scale,” Opt. Express, 26 (6), 7368 https://doi.org/10.1364/OE.26.007368 OPEXFF 1094-4087 (2018). Google Scholar

39.

Z. Wang et al., “Single-layer spatial analog meta-processor for imaging processing,” Nat. Commun., 13 (1), 2188 https://doi.org/10.1038/s41467-022-29732-4 NCAOBW 2041-1723 (2022). Google Scholar

40.

L. Bao et al., “Mathematical operations of transmissive near fields controlled by metasurface with phase and amplitude modulations,” Ann. Phys., 532 (6), 2000069 https://doi.org/10.1002/andp.202000069 (2020). Google Scholar

41.

J. Zhou et al., “Optical edge detection based on high-efficiency dielectric metasurface,” Proc. Natl. Acad. Sci. U. S. A., 116 (23), 11137 –11140 https://doi.org/10.1073/pnas.1820636116 (2019). Google Scholar

42.

H. Wang et al., “Compact incoherent image differentiation with nanophotonic structures,” ACS Photonics, 7 (2), 338 –343 https://doi.org/10.1021/acsphotonics.9b01465 (2020). Google Scholar

43.

M. K. Chen et al., “Edge detection with meta-lens: from one dimension to three dimensions,” Nanophotonics, 10 (14), 3709 –3715 https://doi.org/10.1515/nanoph-2021-0239 (2021). Google Scholar

44.

A. Cordaro et al., “High-index dielectric metasurfaces performing mathematical operations,” Nano Lett., 19 (12), 8418 –8423 https://doi.org/10.1021/acs.nanolett.9b02477 NALEFD 1530-6984 (2019). Google Scholar

45.

D. A. Bykov et al., “First-order optical spatial differentiator based on a guided-mode resonant grating,” Opt. Express, 26 (8), 10997 https://doi.org/10.1364/OE.26.010997 OPEXFF 1094-4087 (2018). Google Scholar

46.

Y. Zhou et al., “Flat optics for image differentiation,” Nat. Photonics, 14 (5), 316 –323 https://doi.org/10.1038/s41566-020-0591-3 NPAHBY 1749-4885 (2020). Google Scholar

47.

A. Komar et al., “Edge detection with Mie-resonant dielectric metasurfaces,” ACS Photonics, 8 (3), 864 –871 https://doi.org/10.1021/acsphotonics.0c01874 (2021). Google Scholar

48.

W. Fu et al., “Ultracompact meta-imagers for arbitrary all-optical convolution,” Light Sci. Appl., 11 (1), 62 https://doi.org/10.1038/s41377-022-00752-5 (2022). Google Scholar

49.

C. Guo et al., “Photonic crystal slab Laplace operator for image differentiation,” Optica, 5 (3), 251 –256 https://doi.org/10.1364/OPTICA.5.000251 (2018). Google Scholar

50.

S. Abdollahramezani et al., “Dielectric metasurfaces solve differential and integro-differential equations,” Opt. Lett., 42 (7), 1197 –1200 https://doi.org/10.1364/OL.42.001197 OPLEDP 0146-9592 (2017). Google Scholar

51.

M. Farmahini-Farahani, J. Cheng and H. Mosallaei, “Metasurfaces nanoantennas for light processing,” J. Opt. Soc. Am. B, 30 (9), 2365 –2370 https://doi.org/10.1364/JOSAB.30.002365 JOBPDE 0740-3224 (2013). Google Scholar

52.

A. Chizari et al., “Analog optical computing based on a dielectric meta-reflect array,” Opt. Lett., 41 (15), 3451 –3454 https://doi.org/10.1364/OL.41.003451 OPLEDP 0146-9592 (2016). Google Scholar

53.

L. Wan et al., “Optical analog computing of spatial differentiation and edge detection with dielectric metasurfaces,” Opt. Lett., 45 (7), 2070 –2073 https://doi.org/10.1364/OL.386986 OPLEDP 0146-9592 (2020). Google Scholar

54.

H. Kwon et al., “Dual-polarization analog 2D image processing with nonlocal metasurfaces,” ACS Photonics, 7 (7), 1799 –1805 https://doi.org/10.1021/acsphotonics.0c00473 (2020). Google Scholar

55.

O. Y. Long et al., “Polarization-independent isotropic nonlocal metasurfaces with wavelength-controlled functionality,” Phys. Rev. Appl., 17 (2), 024029 https://doi.org/10.1103/PhysRevApplied.17.024029 PRAHB2 2331-7019 (2022). Google Scholar

56.

A. C. Overvig et al., “Selection rules for quasibound states in the continuum,” Phys. Rev. B, 102 (3), 035434 https://doi.org/10.1103/PhysRevB.102.035434 (2020). Google Scholar

57.

A. Overvig, N. Yu and A. Alù, “Chiral quasi-bound states in the continuum,” Phys. Rev. Lett., 126 (7), 073001 https://doi.org/10.1103/PhysRevLett.126.073001 PRLTAO 0031-9007 (2021). Google Scholar

58.

S. Joseph et al., “Bound states in the continuum in resonant nanostructures: an overview of engineered materials for tailored applications,” Nanophotonics, 10 (17), 4175 –4207 https://doi.org/10.1515/nanoph-2021-0387 (2021). Google Scholar

59.

X. Zhang et al., “Chiral emission from resonant metasurfaces,” Science, 377 (6611), 1215 –1218 https://doi.org/10.1126/science.abq7870 SCIEAS 0036-8075 (2022). Google Scholar

60.

A. Kodigala et al., “Lasing action from photonic bound states in continuum,” Nature, 541 (7636), 196 –199 https://doi.org/10.1038/nature20799 (2017). Google Scholar

61.

D. Pan et al., “Laplace metasurfaces for optical analog computing based on quasi-bound states in the continuum,” Photonics Res., 9 (9), 1758 –1766 https://doi.org/10.1364/PRJ.426827 (2021). Google Scholar

62.

J. Kühne et al., “Fabrication robustness in BIC metasurfaces,” Nanophotonics, 10 (17), 4305 –4312 https://doi.org/10.1515/nanoph-2021-0391 (2021). Google Scholar

63.

H. Goh and A. Alù, “Nonlocal scatterer for compact wave-based analog computing,” Phys. Rev. Lett., 128 (7), 073201 https://doi.org/10.1103/PhysRevLett.128.073201 PRLTAO 0031-9007 (2022). Google Scholar

64.

M. Camacho, B. Edwards and N. Engheta, “A single inverse-designed photonic structure that performs parallel computing,” Nat. Commun., 12 (1), 1466 https://doi.org/10.1038/s41467-021-21664-9 NCAOBW 2041-1723 (2021). Google Scholar

65.

D. Xu et al., “Inverse design of Pancharatnam–Berry phase metasurfaces for all-optical image edge detection,” Appl. Phys. Lett., 120 (24), 241101 https://doi.org/10.1063/5.0090606 APPLAB 0003-6951 (2022). Google Scholar

66.

N. M. Estakhri, B. Edwards and N. Engheta, “Inverse-designed metastructures that solve equations,” Science, 363 (6433), 1333 –1338 https://doi.org/10.1126/science.aaw2498 SCIEAS 0036-8075 (2019). Google Scholar

67.

P. Georgi et al., “Optical secret sharing with cascaded metasurface holography,” Sci. Adv., 7 (16), eabf9718 https://doi.org/10.1126/sciadv.abf9718 STAMCV 1468-6996 (2021). Google Scholar

68.

J. Jang, T. Badloe and J. Rho, “Unlocking the future of optical security with metasurfaces,” Light Sci. Appl., 10 144 https://doi.org/10.1038/s41377-021-00589-4 (2021). Google Scholar

69.

O. Avayu et al., “Composite functional metasurfaces for multispectral achromatic optics,” Nat. Commun., 8 (1), 14992 https://doi.org/10.1038/ncomms14992 NCAOBW 2041-1723 (2017). Google Scholar

70.

Y. Zhou et al., “Multilayer noninteracting dielectric metasurfaces for multiwavelength metaoptics,” Nano Lett., 18 (12), 7529 –7537 https://doi.org/10.1021/acs.nanolett.8b03017 NALEFD 1530-6984 (2018). Google Scholar

71.

B. O. Raeker et al., “All-dielectric meta-optics for high-efficiency independent amplitude and phase manipulation,” Adv. Photonics Res., 3 (3), 2100285 https://doi.org/10.1002/adpr.202100285 (2022). Google Scholar

72.

E. Maguid et al., “Multifunctional interleaved geometric-phase dielectric metasurfaces,” Light Sci. Appl., 6 (8), e17027 https://doi.org/10.1038/lsa.2017.27 (2017). Google Scholar

73.

K. Du et al., “Optical metasurfaces towards multifunctionality and tunability,” Nanophotonics, 11 (9), 1761 –1781 https://doi.org/10.1515/nanoph-2021-0684 (2022). Google Scholar

74.

J. P. Balthasar Mueller et al., “Metasurface polarization optics: independent phase control of arbitrary orthogonal states of polarization,” Phys. Rev. Lett., 118 (11), 113901 https://doi.org/10.1103/PhysRevLett.118.113901 PRLTAO 0031-9007 (2017). Google Scholar

75.

A. H. Dorrah et al., “Metasurface optics for on-demand polarization transformations along the optical path,” Nat. Photonics, 15 (4), 287 –296 https://doi.org/10.1038/s41566-020-00750-2 NPAHBY 1749-4885 (2021). Google Scholar

76.

Y. Zhou et al., “Multifunctional metaoptics based on bilayer metasurfaces,” Light Sci. Appl., 8 80 https://doi.org/10.1038/s41377-019-0193-3 (2019). Google Scholar

77.

S. M. Kamali et al., “Angle-multiplexed metasurfaces: encoding independent wavefronts in a single metasurface under different illumination angles,” Phys. Rev. X, 7 (4), 041056 https://doi.org/10.1103/PhysRevX.7.041056 PRXHAE 2160-3308 (2017). Google Scholar

78.

T. Badloe et al., “Tunable metasurfaces: the path to fully active nanophotonics,” Adv. Photonics Res., 2 (9), 2000205 https://doi.org/10.1002/adpr.202000205 (2021). Google Scholar

79.

T. Badloe et al., “Liquid crystal-powered Mie resonators for electrically tunable photorealistic color gradients and dark blacks,” Light Sci. Appl., 11 (1), 118 https://doi.org/10.1038/s41377-022-00806-8 (2022). Google Scholar

80.

T. Badloe et al., “Electrically tunable bifocal metalens with diffraction-limited focusing and imaging at visible wavelengths,” Adv. Sci., 8 (21), 2102646 https://doi.org/10.1002/advs.202102646 (2021). Google Scholar

81.

I. Kim et al., “Pixelated bifunctional metasurface-driven dynamic vectorial holographic color prints for photonic security platform,” Nat. Commun., 12 (1), 3614 https://doi.org/10.1038/s41467-021-23814-5 NCAOBW 2041-1723 (2021). Google Scholar

82.

I. Kim et al., “Holographic metasurface gas sensors for instantaneous visual alarms,” Sci. Adv., 7 (15), eabe9943 https://doi.org/10.1126/sciadv.abe9943 STAMCV 1468-6996 (2021). Google Scholar

83.

J. H. Ko et al., “Flexible, large-area covert polarization display based on ultrathin lossy nanocolumns on a metal film,” Adv. Funct. Mater., 30 (11), 1908592 https://doi.org/10.1002/adfm.201908592 AFMDC6 1616-301X (2020). Google Scholar

84.

T. Badloe, I. Kim and J. Rho, “Moth-eye shaped on-demand broadband and switchable perfect absorbers based on vanadium dioxide,” Sci. Rep., 10 (1), 4522 https://doi.org/10.1038/s41598-020-59729-2 SRCEC3 2045-2322 (2020). Google Scholar

85.

C. Zhang et al., “Stretchable all-dielectric metasurfaces with polarization-insensitive and full-spectrum response,” ACS Nano, 14 (2), 1418 –1426 https://doi.org/10.1021/acsnano.9b08228 ANCAC3 1936-0851 (2020). Google Scholar

86.

A. She et al., “Adaptive metalenses with simultaneous electrical control of focal length, astigmatism, and shift,” Sci. Adv., 4 (2), eaap9957 https://doi.org/10.1126/sciadv.aap9957 STAMCV 1468-6996 (2018). Google Scholar

87.

P. Huo et al., “Photonic spin-multiplexing metasurface for switchable spiral phase contrast imaging,” Nano Lett., 20 (4), 2791 –2798 https://doi.org/10.1021/acs.nanolett.0c00471 NALEFD 1530-6984 (2020). Google Scholar

88.

X. Zhang et al., “Reconfigurable metasurface for image processing,” Nano Lett., 21 (20), 8715 –8722 https://doi.org/10.1021/acs.nanolett.1c02838 NALEFD 1530-6984 (2021). Google Scholar

89.

Y. Kim et al., “Spiral metalens for phase contrast imaging,” Adv. Funct. Mater., 32 (5), 2106050 https://doi.org/10.1002/adfm.202106050 AFMDC6 1616-301X (2022). Google Scholar

90.

Y. Zhang et al., “Electrically reconfigurable non-volatile metasurface using low-loss optical phase-change material,” Nat. Nanotechnol., 16 (6), 661 –666 https://doi.org/10.1038/s41565-021-00881-9 NNAABX 1748-3387 (2021). Google Scholar

91.

C. Choi et al., “Hybrid state engineering of phase-change metasurface for all-optical cryptography,” Adv. Funct. Mater., 31 (4), 2007210 https://doi.org/10.1002/adfm.202007210 AFMDC6 1616-301X (2021). Google Scholar

92.

Y. Qu et al., “Thermal camouflage based on the phase-changing material GST,” Light Sci. Appl., 7 (1), 26 https://doi.org/10.1038/s41377-018-0038-5 (2018). Google Scholar

93.

H. Yang et al., “Switchable imaging between edge-enhanced and bright-field based on a phase-change metasurface,” Opt. Lett., 46 (15), 3741 –3744 https://doi.org/10.1364/OL.428870 OPLEDP 0146-9592 (2021). Google Scholar

94.

D. C. Tzarouchis et al., “Mathematical operations and equation solving with reconfigurable metadevices,” Light Sci. Appl., 11 (1), 263 https://doi.org/10.1038/s41377-022-00950-1 (2022). Google Scholar

95.

T. Badloe, I. Kim and J. Rho, “Biomimetic ultra-broadband perfect absorbers optimised with reinforcement learning,” Phys. Chem. Chem. Phys., 22 (4), 2337 –2342 https://doi.org/10.1039/C9CP05621A PPCPFQ 1463-9076 (2020). Google Scholar

96.

S. So and J. Rho, “Designing nanophotonic structures using conditional deep convolutional generative adversarial networks,” Nanophotonics, 8 (7), 1255 –1261 https://doi.org/10.1515/nanoph-2019-0117 (2019). Google Scholar

97.

S. So et al., “Inverse design of ultra-narrowband selective thermal emitters designed by artificial neural networks,” Opt. Mater. Express, 11 (7), 1863 –1873 https://doi.org/10.1364/OME.430306 (2021). Google Scholar

98.

Z. Liu et al., “Compounding meta-atoms into metamolecules with hybrid artificial intelligence techniques,” Adv. Mater., 32 (6), 1904790 https://doi.org/10.1002/adma.201904790 ADVMEW 0935-9648 (2020). Google Scholar

99.

T. Phan et al., “High-efficiency, large-area, topology-optimized metasurfaces,” Light Sci. Appl., 8 (1), 48 https://doi.org/10.1038/s41377-019-0159-5 (2019). Google Scholar

100.

H. Ren et al., “Three-dimensional vectorial holography based on machine learning inverse design,” Sci. Adv., 6 (16), eaaz4261 https://doi.org/10.1126/sciadv.aaz4261 STAMCV 1468-6996 (2020). Google Scholar

101.

H. Zhou et al., “Photonic matrix multiplication lights up photonic accelerator and beyond,” Light Sci. Appl., 11 (1), 30 https://doi.org/10.1038/s41377-022-00717-8 (2022). Google Scholar

102.

W. Zhu et al., “Design and experimental verification for optical module of optical vector-matrix multiplier,” Appl. Opt., 52 (18), 4412 –4418 https://doi.org/10.1364/AO.52.004412 APOPAI 0003-6935 (2013). Google Scholar

103.

J. Cheng, H. Zhou and J. Dong, “Photonic matrix computing: from fundamentals to applications,” Nanomaterials, 11 (7), 1683 https://doi.org/10.3390/nano11071683 (2021). Google Scholar

104.

W. Bogaerts and L. Chrostowski, “Silicon photonics circuit design: methods, tools and challenges,” Laser Photonics Rev., 12 (4), 1700237 https://doi.org/10.1002/lpor.201700237 (2018). Google Scholar

105.

Y. Shen et al., “Deep learning with coherent nanophotonic circuits,” Nat. Photonics, 11 (7), 441 –446 https://doi.org/10.1038/nphoton.2017.93 NPAHBY 1749-4885 (2017). Google Scholar

106.

Z. Wang et al., “Metasurface on integrated photonic platform: from mode converters to machine learning,” Nanophotonics, 11 (16), 3531 –3546 https://doi.org/10.1515/nanoph-2022-0294 (2022). Google Scholar

107.

M. K. Chen et al., “Artificial intelligence in meta-optics,” Chem. Rev., 122 15356 –15413 https://doi.org/10.1021/acs.chemrev.2c00012 CHREAY 0009-2665 (2022). Google Scholar

108.

C. Qian et al., “Performing optical logic operations by a diffractive neural network,” Light Sci. Appl., 9 (1), 59 https://doi.org/10.1038/s41377-020-0303-2 (2020). Google Scholar

109.

J. Liu et al., “Research progress in optical neural networks: theory, applications and developments,” PhotoniX, 2 (1), 5 https://doi.org/10.1186/s43074-021-00026-0 (2021). Google Scholar

110.

B. J. Shastri et al., “Photonics for artificial intelligence and neuromorphic computing,” Nat. Photonics, 15 (2), 102 –114 https://doi.org/10.1038/s41566-020-00754-y NPAHBY 1749-4885 (2021). Google Scholar

111.

C. Qian et al., “Deep-learning-enabled self-adaptive microwave cloak without human intervention,” Nat. Photonics, 14 (6), 383 –390 https://doi.org/10.1038/s41566-020-0604-2 NPAHBY 1749-4885 (2020). Google Scholar

112.

G. Wetzstein et al., “Inference in artificial intelligence with deep optics and photonics,” Nature, 588 (7836), 39 –47 https://doi.org/10.1038/s41586-020-2973-6 (2020). Google Scholar

113.

S. So et al., “Deep learning enabled inverse design in nanophotonics,” Nanophotonics, 9 (5), 1041 –1057 https://doi.org/10.1515/nanoph-2019-0474 (2020). Google Scholar

114.

J. Jiang, M. Chen and J. A. Fan, “Deep neural networks for the evaluation and design of photonic devices,” Nat. Rev. Mater., 6 (8), 679 –700 https://doi.org/10.1038/s41578-020-00260-1 (2021). Google Scholar

115.

W. Ma et al., “Deep learning for the design of photonic structures,” Nat. Photonics, 15 (2), 77 –90 https://doi.org/10.1038/s41566-020-0685-y NPAHBY 1749-4885 (2021). Google Scholar

116.

N. Wang et al., “Intelligent designs in nanophotonics: from optimization towards inverse creation,” PhotoniX, 2 (1), 22 https://doi.org/10.1186/s43074-021-00044-y (2021). Google Scholar

117.

C. M. Bishop, Pattern Recognition and Machine Learning, 4th ed.Springer, New York (2006). Google Scholar

118.

X. Lin et al., “All-optical machine learning using diffractive deep neural networks,” Science, 361 (6406), 1004 –1008 https://doi.org/10.1126/science.aat8084 SCIEAS 0036-8075 (2018). Google Scholar

119.

X. Luo et al., “Metasurface-enabled on-chip multiplexed diffractive neural networks in the visible,” Light Sci. Appl., 11 (1), 158 https://doi.org/10.1038/s41377-022-00844-2 (2022). Google Scholar

120.

J. W. Goodman, Introduction to Fourier Optics, 3rd ed.Roberts and Company Publishers, Englewood, Colorado (2004). Google Scholar

121.

S. Zarei and A. Khavasi, “Realization of optical logic gates using on-chip diffractive optical neural networks,” Sci. Rep., 12 (1), 15747 https://doi.org/10.1038/s41598-022-19973-0 SRCEC3 2045-2322 (2022). Google Scholar

122.

M. Papaioannou, E. Plum and N. I. Zheludev, “All-optical pattern recognition and image processing on a metamaterial beam splitter,” ACS Photonics, 4 (2), 217 –222 https://doi.org/10.1021/acsphotonics.6b00921 (2017). Google Scholar

123.

M. Papaioannou et al., “Two-dimensional control of light with light on metasurfaces,” Light Sci. Appl., 5 (4), e16070 https://doi.org/10.1038/lsa.2016.70 (2016). Google Scholar

124.

G. Qu et al., “All-dielectric metasurface empowered optical-electronic hybrid neural networks,” Laser Photonics Rev., 16 2100732 https://doi.org/10.1002/lpor.202100732 (2022). Google Scholar

125.

Z. Huang et al., “All-optical signal processing of vortex beams with diffractive deep neural networks,” Phys. Rev. Appl., 15 (1), 014037 https://doi.org/10.1103/PhysRevApplied.15.014037 PRAHB2 2331-7019 (2021). Google Scholar

126.

J. Zhang et al., “Polarized deep diffractive neural network for sorting, generation, multiplexing, and de-multiplexing of orbital angular momentum modes,” Opt. Express, 30 (15), 26728 –26741 https://doi.org/10.1364/OE.463137 OPEXFF 1094-4087 (2022). Google Scholar

127.

J. Weng et al., “Meta-neural-network for real-time and passive deep-learning-based object recognition,” Nat. Commun., 11 (1), 6309 https://doi.org/10.1038/s41467-020-19693-x NCAOBW 2041-1723 (2020). Google Scholar

128.

Q. Wei et al., “Rotational multiplexing method based on cascaded metasurface holography,” Adv. Opt. Mater., 10 (8), 2102166 https://doi.org/10.1002/adom.202102166 2195-1071 (2022). Google Scholar

129.

W. Feng et al., “RGB achromatic metalens doublet for digital imaging,” Nano Lett., 22 (10), 3969 –3975 https://doi.org/10.1021/acs.nanolett.2c00486 NALEFD 1530-6984 (2022). Google Scholar

130.

T. J. Cui et al., “Coding metamaterials, digital metamaterials and programmable metamaterials,” Light Sci. Appl., 3 (10), e218 https://doi.org/10.1038/lsa.2014.99 (2014). Google Scholar

131.

L. Li et al., “Machine-learning reprogrammable metasurface imager,” Nat. Commun., 10 (1), 1082 https://doi.org/10.1038/s41467-019-09103-2 NCAOBW 2041-1723 (2019). Google Scholar

132.

C. Liu et al., “A programmable diffractive deep neural network based on a digital-coding metasurface array,” Nat. Electron., 5 (2), 113 –122 https://doi.org/10.1038/s41928-022-00719-9 NEREBX 0305-2257 (2022). Google Scholar

133.

L. Li et al., “Intelligent metasurface imager and recognizer,” Light Sci. Appl., 8 (1), 97 https://doi.org/10.1038/s41377-019-0209-z (2019). Google Scholar

134.

Z. Gu et al., “High-resolution programmable metasurface imager based on multilayer perceptron network,” Adv. Opt. Mater., 10 (18), 2200619 https://doi.org/10.1002/adom.202200619 2195-1071 (2022). Google Scholar

135.

H. Zhang et al., “An optical neural chip for implementing complex-valued neural network,” Nat. Commun., 12 (1), 457 https://doi.org/10.1038/s41467-020-20719-7 NCAOBW 2041-1723 (2021). Google Scholar

136.

T. W. Hughes et al., “Training of photonic neural networks through in situ backpropagation and gradient measurement,” Optica, 5 (7), 864 https://doi.org/10.1364/OPTICA.5.000864 (2018). Google Scholar

137.

Y. L. Xiao et al., “Optical random phase dropout in a diffractive deep neural network,” Opt. Lett., 46 (20), 5260 https://doi.org/10.1364/OL.428761 OPLEDP 0146-9592 (2021). Google Scholar

138.

H. Dou et al., “Residual D 2 NN: training diffractive deep neural networks via learnable light shortcuts,” Opt. Lett., 45 (10), 2688 https://doi.org/10.1364/OL.389696 OPLEDP 0146-9592 (2020). Google Scholar

139.

J. Bueno et al., “Reinforcement learning in a large-scale photonic recurrent neural network,” Optica, 5 (6), 756 https://doi.org/10.1364/OPTICA.5.000756 (2018). Google Scholar

140.

T. W. Hughes et al., “Wave physics as an analog recurrent neural network,” Sci. Adv., 5 (12), eaay6946 https://doi.org/10.1126/sciadv.aay6946 STAMCV 1468-6996 (2019). Google Scholar

141.

H. Wei et al., “All-optical machine learning using diffractive deep neural networks,” Science, 361 (6406), 1004 –1008 (2018). Google Scholar

142.

T. Wang et al., “Image sensing with multilayer, nonlinear optical neural networks,” (2022). Google Scholar

143.

A. Ryou et al., “Free-space optical neural network based on thermal atomic nonlinearity,” Photonics Res., 9 (4), B128 –B134 https://doi.org/10.1364/PRJ.415964 (2021). Google Scholar

144.

S. R. Skinner, J. E. Steck and E. C. Behrman, “Optical neural network using Kerr-type nonlinear materials,” 12 –15 (1994). https://doi.org/10.1109/ICMNN.1994.593141 Google Scholar

145.

M. Miscuglio et al., “All-optical nonlinear activation function for photonic neural networks,” Opt. Mater. Express, 8 (12), 3851 https://doi.org/10.1364/OME.8.003851 (2018). Google Scholar

146.

M. Nauman et al., “Tunable unidirectional nonlinear emission from transition-metal-dichalcogenide metasurfaces,” Nat. Commun., 12 (1), 5597 https://doi.org/10.1038/s41467-021-25717-x NCAOBW 2041-1723 (2021). Google Scholar

147.

L. Carletti et al., “Reconfigurable nonlinear response of dielectric and semiconductor metasurfaces,” Nanophotonics, 10 (17), 4209 –4221 https://doi.org/10.1515/nanoph-2021-0367 (2021). Google Scholar

148.

S. Rubin and Y. Fainman, “Nonlinear, tunable, and active optical metasurface with liquid film,” Adv. Photonics, 4 (6), 066003 https://doi.org/10.1117/1.AP.1.6.066003 AOPAC7 1943-8206 (2019). Google Scholar

149.

A. Krasnok, M. Tymchenko and A. Alù, “Nonlinear metasurfaces: a paradigm shift in nonlinear optics,” Mater. Today, 21 (1), 8 –21 https://doi.org/10.1016/j.mattod.2017.06.007 MATOBY 1369-7021 (2018). Google Scholar

150.

A. E. Minovich et al., “Functional and nonlinear optical metasurfaces,” Laser Photonics Rev., 9 (2), 195 –213 https://doi.org/10.1002/lpor.201400402 (2015). Google Scholar

151.

G. Li, S. Zhang and T. Zentgraf, “Nonlinear photonic metasurfaces,” Nat. Rev. Mater., 2 (5), 17010 https://doi.org/10.1038/natrevmats.2017.10 (2017). Google Scholar

152.

S. Chen et al., “Controlling the phase of optical nonlinearity with plasmonic metasurfaces,” Nanophotonics, 7 (6), 1013 –1024 https://doi.org/10.1515/nanoph-2018-0011 (2018). Google Scholar

153.

Y. Zuo et al., “All-optical neural network with nonlinear activation functions,” Optica, 6 (9), 1132 –1137 https://doi.org/10.1364/OPTICA.6.001132 (2019). Google Scholar

154.

M. Wuttig, H. Bhaskaran and T. Taubner, “Phase-change materials for non-volatile photonic applications,” Nat. Photonics, 11 (8), 465 –476 https://doi.org/10.1038/nphoton.2017.126 NPAHBY 1749-4885 (2017). Google Scholar

155.

B. Ko, T. Badloe and J. Rho, “Vanadium dioxide for dynamically tunable photonics,” ChemNanoMat., 7 (7), 713 –727 https://doi.org/10.1002/cnma.202100060 (2021). Google Scholar

156.

R. E. Simpson, J. K. W. Yang and J. Hu, “Are phase change materials ideal for programmable photonics?: opinion,” Opt. Mater. Express, 12 (6), 2368 –2373 https://doi.org/10.1364/OME.456895 (2022). Google Scholar

157.

T. Y. Teo et al., “Programmable chalcogenide-based all-optical deep neural networks,” Nanophotonics, 11 (17), 4073 –4088 https://doi.org/10.1515/nanoph-2022-0099 (2022). Google Scholar

158.

“LiDAR drives forwards,” Nat. Photonics, 12 (8), 441 https://doi.org/10.1038/s41566-018-0235-z NPAHBY 1749-4885 (2018). Google Scholar

159.

Y. Jiang, S. Karpf and B. Jalali, “Time-stretch LiDAR as a spectrally scanned time-of-flight ranging camera,” Nat. Photonics, 14 (1), 14 –18 https://doi.org/10.1038/s41566-019-0548-6 NPAHBY 1749-4885 (2020). Google Scholar

160.

I. A. D. Williamson et al., “Tunable nonlinear activation functions for optical neural networks,” in Conf. Lasers and Electro-Opt., SM1E.2 (2020). Google Scholar

161.

D. Mengu et al., “Analysis of diffractive optical neural networks and their integration with electronic neural networks,” IEEE J. Sel. Top. Quantum Electron., 26 (1), 1 –14 https://doi.org/10.1109/JSTQE.2019.2921376 IJSQEN 1077-260X (2020). Google Scholar

162.

T. Zhou et al., “Large-scale neuromorphic optoelectronic computing with a reconfigurable diffractive processing unit,” Nat. Photonics, 15 (5), 367 –373 https://doi.org/10.1038/s41566-021-00796-w NPAHBY 1749-4885 (2021). Google Scholar

163.

A. H. Atabaki, “Integrating photonics with silicon nanoelectronics for the next generation of systems on a chip,” Nature, 556 (7701), 349 –354 https://doi.org/10.1038/s41586-018-0028-z (2018). Google Scholar

164.

W. Liu et al., “A fully reconfigurable photonic integrated signal processor,” Nat. Photonics, 10 (3), 190 –195 https://doi.org/10.1038/nphoton.2015.281 NPAHBY 1749-4885 (2016). Google Scholar

165.

C. Jung et al., “Metasurface-driven optically variable devices,” Chem. Rev., 121 (21), 13013 –13050 https://doi.org/10.1021/acs.chemrev.1c00294 CHREAY 0009-2665 (2021). Google Scholar

166.

P.R. Wiecha et al., “Pushing the limits of optical information storage using deep learning,” Nat. Nanotechnol., 14 (3), 237 –244 https://doi.org/10.1038/s41565-018-0346-1 NNAABX 1748-3387 (2019). Google Scholar

167.

Z. Li et al., “Full-space cloud of random points with a scrambling metasurface,” Light Sci. Appl., 7 (1), 63 https://doi.org/10.1038/s41377-018-0064-3 (2018). Google Scholar

168.

S. Q. Li et al., “Phase-only transmissive spatial light modulator based on tunable dielectric metasurface,” Science, 364 (6445), 1087 –1090 https://doi.org/10.1126/science.aaw6747 SCIEAS 0036-8075 (2019). Google Scholar

169.

J. Liu et al., “Intrinsic optical spatial differentiation enabled quantum dark-field microscopy,” Phys. Rev. Lett., 128 (19), 193601 https://doi.org/10.1103/PhysRevLett.128.193601 PRLTAO 0031-9007 (2022). Google Scholar

170.

R. Wang et al., “Computing metasurfaces enabled chiral edge image sensing,” iScience, 25 (7), 104532 https://doi.org/10.1016/j.isci.2022.104532 (2022). Google Scholar

171.

L. Wesemann et al., “Nanophotonics enhanced coverslip for phase imaging in biology,” Light Sci. Appl., 10 (1), 98 https://doi.org/10.1038/s41377-021-00540-7 (2021). Google Scholar

172.

K. Toda et al., “Molecular contrast on phase-contrast microscope,” Sci. Rep., 9 (1), 9957 https://doi.org/10.1038/s41598-019-46383-6 SRCEC3 2045-2322 (2019). Google Scholar

173.

O. Kulce et al., “All-optical information-processing capacity of diffractive surfaces,” Light Sci. Appl., 10 (1), 25 https://doi.org/10.1038/s41377-020-00439-9 (2021). Google Scholar

174.

S. Zhang et al., “Metasurfaces for biomedical applications: imaging and sensing from a nanophotonics perspective,” Nanophotonics, 10 (1), 259 –293 https://doi.org/10.1515/nanoph-2020-0373 (2020). Google Scholar

175.

D. Mengu and A. Ozcan, “All-optical phase recovery: diffractive computing for quantitative phase imaging,” Adv. Opt. Mater., 10 (15), 2200281 https://doi.org/10.1002/adom.202200281 2195-1071 (2022). Google Scholar

176.

H. Kwon et al., “Single-shot quantitative phase gradient microscopy using a system of multifunctional metasurfaces,” Nat. Photonics, 14 (2), 109 –114 https://doi.org/10.1038/s41566-019-0536-x NPAHBY 1749-4885 (2020). Google Scholar

177.

Y. Yang et al., “Revealing structural disorder in hydrogenated amorphous silicon for a low-loss photonic platform at visible frequencies,” Adv. Mater., 33 (9), 2005893 https://doi.org/10.1002/adma.202005893 ADVMEW 0935-9648 (2021). Google Scholar

178.

J. Kim et al., “Photonic encryption platform via dual-band vectorial metaholograms in the ultraviolet and visible,” ACS Nano, 16 (3), 3546 –3553 https://doi.org/10.1021/acsnano.1c10100 ANCAC3 1936-0851 (2022). Google Scholar

179.

M. Wang et al., “Suppressing material loss in the visible and near-infrared range for functional nanophotonics using bandgap engineering,” Nat. Commun., 11 (1), 5055 https://doi.org/10.1038/s41467-020-18793-y NCAOBW 2041-1723 (2020). Google Scholar

180.

K. Kim et al., “Facile nanocasting of dielectric metasurfaces with sub-100 nm resolution,” ACS Appl. Mater. Interfaces, 11 (29), 26109 –26115 https://doi.org/10.1021/acsami.9b07774 AAMICK 1944-8244 (2019). Google Scholar

181.

G. Yoon et al., “Single-step manufacturing of hierarchical dielectric metalens in the visible,” Nat. Commun., 11 (1), 2268 https://doi.org/10.1038/s41467-020-16136-5 NCAOBW 2041-1723 (2020). Google Scholar

182.

D. K. Oh et al., “Nanoimprint lithography for high-throughput fabrication of metasurfaces,” Front. Optoelectron., 14 (2), 229 –251 https://doi.org/10.1007/s12200-021-1121-8 (2021). Google Scholar

183.

G. Yoon et al., “Printable nanocomposite metalens for high-contrast near-infrared imaging,” ACS Nano, 15 (1), 698 –706 https://doi.org/10.1021/acsnano.0c06968 ANCAC3 1936-0851 (2021). Google Scholar

184.

J. Kim et al., “Metasurface holography reaching the highest efficiency limit in the visible via one-step nanoparticle-embedded-resin printing,” Laser Photonics Rev., 16 (8), 2200098 https://doi.org/10.1002/lpor.202200098 (2022). Google Scholar

185.

B. Ko et al., “Tunable metasurfaces via the humidity responsive swelling of single-step imprinted polyvinyl alcohol nanostructures,” Nat. Commun., 13 (1), 6256 https://doi.org/10.1038/s41467-022-32987-6 NCAOBW 2041-1723 (2022). Google Scholar

186.

W. Zhang et al., “2.5D, 3D and 4D printing in nanophotonics—a progress report,” Mater Today Proc., 70 304 –309 https://doi.org/10.1016/j.matpr.2022.09.242 (2022). Google Scholar

187.

J. Kim et al., “Scalably manufactured high-index atomic layer-polymer hybrid metasurfaces for high-efficiency virtual reality metaoptics in the visible,” (2022). Google Scholar

188.

C. Chen et al., “Large-area, low-cost near-infrared meta-surface reflector based on a pixelated two-dimensional silicon disk array,” Opt. Express, 28 (25), 38355 –38365 https://doi.org/10.1364/OE.412521 OPEXFF 1094-4087 (2020). Google Scholar

189.

H. Zheng et al., “Large-scale metasurfaces based on grayscale nanosphere lithography,” ACS Photonics, 8 (6), 1824 –1831 https://doi.org/10.1021/acsphotonics.1c00424 (2021). Google Scholar

190.

S. Kim et al., “Self-assembled pagoda-like nanostructure-induced vertically stacked split-ring resonators for polarization-sensitive dichroic responses,” Nano Converg., 9 (1), 40 https://doi.org/10.1186/s40580-022-00331-9 (2022). Google Scholar

191.

G. A. Ozin et al., “Nanofabrication by self-assembly,” Mater. Today, 12 (5), 12 –23 https://doi.org/10.1016/S1369-7021(09)70156-7 MATOBY 1369-7021 (2009). Google Scholar

192.

J. Zhou et al., “Nonlinear computational edge detection metalens,” Adv. Funct. Mater., 32 (34), 2204734 https://doi.org/10.1002/adfm.202204734 AFMDC6 1616-301X (2022). Google Scholar

193.

J. Sol, D. R. Smith and P. del Hougne, “Meta-programmable analog differentiator,” Nat. Commun., 13 (1), 1713 https://doi.org/10.1038/s41467-022-29354-w NCAOBW 2041-1723 (2022). Google Scholar

194.

H. Rajabalipanah et al., “Analog signal processing through space-time digital metasurfaces,” Nanophotonics, 10 (6), 1753 –1764 https://doi.org/10.1515/nanoph-2021-0006 (2021). Google Scholar

195.

L. Chen et al., “Touch-programmable metasurface for various electromagnetic manipulations and encryptions,” Small, 18 (45), 2203871 https://doi.org/10.1002/smll.202203871 SMALBC 1613-6810 (2022). Google Scholar

196.

A. Momeni, K. Rouhi and R. Fleury, “Switchable and simultaneous spatiotemporal analog computing with computational graphene-based multilayers,” Carbon, 186 599 –611 https://doi.org/10.1016/j.carbon.2021.10.001 (2022). Google Scholar

197.

J. Huang et al., “Spatiotemporal differentiators generating optical vortices with transverse orbital angular momentum and detecting sharp change of pulse envelope,” Laser Photonics Rev., 16 (5), 2100357 https://doi.org/10.1002/lpor.202100357 (2022). Google Scholar

198.

K. Chen et al., “All-optical 1st- and 2nd-order differential equation solvers with large tuning ranges using Fabry–Pérot semiconductor optical amplifiers,” Opt. Express, 23 (3), 3784 –3794 https://doi.org/10.1364/OE.23.003784 OPEXFF 1094-4087 (2015). Google Scholar

199.

L. Zhang et al., “Space-time-coding digital metasurfaces,” Nat. Commun., 9 (1), 4334 https://doi.org/10.1038/s41467-018-06802-0 NCAOBW 2041-1723 (2018). Google Scholar

200.

J. C. Ke et al., “Frequency-modulated continuous waves controlled by space-time-coding metasurface with nonlinearly periodic phases,” Light Sci. Appl., 11 (1), 273 https://doi.org/10.1038/s41377-022-00973-8 (2022). Google Scholar

201.

H. Li et al., “Performing calculus with epsilon-near-zero metamaterials,” Sci. Adv., 8 (30), eabq6198 https://doi.org/10.1126/sciadv.abq6198 STAMCV 1468-6996 (2022). Google Scholar

Biography

Trevon Badloe received his MPhys (hons) degree from the University of Sheffield, United Kingdom, in 2012, with a year of study abroad at the National University of Singapore in 2010. After three years of teaching English and classical mechanics as an assistant professor at Yeungjin University, Republic of Korea, he started working toward his PhD in mechanical engineering at POSTECH, Republic of Korea, in 2017. His interests include tunable metamaterials and metasurfaces, and machine learning for the design and optimization of nanophotonic applications.

Seokho Lee received his BS degree in mechanical engineering in 2020 from Pohang University of Science and Technology, Pohang, Republic of Korea, where he is currently working toward an integrated MS/PhD. His interests include metasurfaces, machine learning for the design and sensing of biocontents, and optimization of nanophotonic applications.

Junsuk Rho received his BS and MS degrees in mechanical engineering from Seoul National University and University of Illinois, Urbana–Champaign, in 2007 and 2008, respectively. After receiving his PhD in 2013 in mechanical engineering and nanoscale science and engineering from University of California Berkeley, he worked as a postdoctoral fellow in the Materials Sciences Division at Lawrence Berkeley National Laboratory and as a Ugo Fano fellow in the Nanoscience and Technology Division at Argonne National Laboratory. He is a Mu-Eun-Jae endowed chair professor with a joint appointment in the Department of Mechanical Engineering and the Department of Chemical Engineering at POSTECH.

CC BY: © The Authors. Published by SPIE and CLP under a Creative Commons Attribution 4.0 International License. Distribution or reproduction of this work in whole or in part requires full attribution of the original publication, including its DOI.

Citation Download Citation

Trevon Badloe, Seokho Lee, and Junsuk Rho "Computation at the speed of light: metamaterials for all-optical calculations and neural networks," Advanced Photonics 4(6), 064002 (21 December 2022). https://doi.org/10.1117/1.AP.4.6.064002

Received: 18 October 2022; Accepted: 29 November 2022; Published: 21 December 2022

Access the abstract

JOURNAL ARTICLE
21 PAGES

DOWNLOAD PAPER SAVE TO MY LIBRARY

GET CITATION

CITATIONS

Cited by 24 scholarly publications.

Explore citations on Lens.org

KEYWORDS

Artificial neural networks

Photonic metamaterials

Optical metamaterials

Design and modelling

Edge detection

Reflection

Computing systems

1.

Introduction

2.

All-Optical Calculations at the Speed of Light

2.1.

Mathematical Operations

Eq. (1)

Eq. (2)

Eq. (3)

2.1.1.

First-order differentiation

Eq. (4)

Fig. 1

2.1.2.

Second-order differentiation

Eq. (5)

2.1.3.

Integration

2.1.4.

Laplacian

2.2.

Metamaterials for All-Optical Calculations

2.2.1.

Static metamaterials

Single and multilayers

Fig. 2

Plasmonic structures

Fig. 3

All-dielectric structures

Fig. 4

2.2.2.

Tunable and multiplexing metamaterials

Fig. 5

3.

All-Optical Machine Learning

Fig. 6

3.1.

Mathematical Basis for ONNs

Eq. (6)

Eq. (7)

Eq. (8)

Eq. (9)

Eq. (10)

Fig. 7

3.2.

Metasurface-Based ONNs

3.3.

Programmable Metasurface-Based ONNs

Fig. 8

4.

Conclusion and Outlook

Acknowledgments

References

Biography

Show All Keywords

Keywords/Phrases

Search In:

Publication Years