Abstract
Innovative approaches and tools play an important role in shaping design, characterization and optimization for the field of photonics. As a subset of machine learning that learns multilevel abstraction of data using hierarchically structured layers, deep learning offers an efficient means to design photonic structures, spawning data-driven approaches complementary to conventional physics- and rule-based methods. Here, we review recent progress in deep-learning-based photonic design by providing the historical background, algorithm fundamentals and key applications, with the emphasis on various model architectures for specific photonic tasks. We also comment on the challenges and perspectives of this emerging research direction.
This is a preview of subscription content, access via your institution
Access options
Access Nature and 54 other Nature Portfolio journals
Get Nature+, our best-value online-access subscription
$29.99 / 30 days
cancel any time
Subscribe to this journal
Receive 12 print issues and online access
$209.00 per year
only $17.42 per issue
Buy this article
- Purchase on Springer Link
- Instant access to full article PDF
Prices may be subject to local taxes which are calculated during checkout
Similar content being viewed by others
References
Joannopoulos, J. D., Johnson, S. G., Winn, J. N. & Meade, R. D. Photonic Crystals: Molding the Flow of Light (Princeton Univ. Press, 2011).
Smith, D., Pendry, J. & Wiltshire, M. Metamaterials and negative refractive index. Science 305, 788–792 (2004).
Liu, Y. & Zhang, X. Metamaterials: a new frontier of science and technology. Chem. Soc. Rev. 40, 2494–2507 (2011).
Cai, W. & Shalaev, V. M. Optical Metamaterials: Fundamentals and Applications (Springer, 2010).
Maier, S. A. Plasmonics: Fundamentals and Applications (Springer, 2007).
Bohren, C. F. & Huffman, D. R. Absorption and Scattering of Light by Small Particles (Wiley, 2008).
Pendry, J. B., Holden, A., Robbins, D. & Stewart, W. Magnetism from conductors and enhanced nonlinear phenomena. IEEE Trans. Microw. Theory Tech. 47, 2075–2084 (1999).
John, S. Strong localization of photons in certain disordered dielectric superlattices. Phys. Rev. Lett. 58, 2486–2489 (1987).
Molesky, S. et al. Inverse design in nanophotonics. Nat. Photon. 12, 659–670 (2018).
Li, W., Meng, F., Chen, Y., Li, Y. & Huang, X. Topology optimization of photonic and phononic crystals and metamaterials: a review. Adv. Theory Simul. 2, 1900017 (2019).
Campbell, S. D. et al. Review of numerical optimization techniques for meta-device design. Opt. Mater. Express 9, 1842–1863 (2019).
LeCun, Y., Bengio, Y. & Hinton, G. Deep learning. Nature 521, 436–444 (2015).
Krizhevsky, A., Sutskever, I. & Hinton, G. E. ImageNet classification with deep convolutional neural networks. In Proc. 25th Int. Conf. Neural Information Processing Systems 1097–1105 (NIPS, 2012).
Cho, K. et al. Learning phrase representations using RNN encoder–decoder for statistical machine translation. In Proc. 2014 Conf. Empirical Methods in Natural Language Processing (EMNLP) 1724–1734 (2014).
Hinton, G. et al. Deep neural networks for acoustic modeling in speech recognition: the shared views of four research groups. IEEE Signal Proc. Mag. 29, 82–97 (2012).
Socher, R., Chen, D., Manning, C. D. & Ng, A. Reasoning with neural tensor networks for knowledge base completion. In NIPS’13: Proc. 26th Int. Conf. Neural Information Processing Systems 926–934 (NIPS, 2013).
Silver, D. et al. Mastering the game of Go with deep neural networks and tree search. Nature 529, 484–489 (2016).
Sanchez-Lengeling, B. & Aspuru-Guzik, A. Inverse molecular design using machine learning: generative models for matter engineering. Science 361, 360–365 (2018).
Goh, G. B., Hodas, N. O. & Vishnu, A. Deep learning for computational chemistry. J. Comput. Chem. 38, 1291–1307 (2017).
Zahavy, T. et al. Deep learning reconstruction of ultrashort pulses. Optica 5, 666–673 (2018).
Baldi, P., Sadowski, P. & Whiteson, D. Searching for exotic particles in high-energy physics with deep learning. Nat. Commun. 5, 4308 (2014).
Carrasquilla, J. & Melko, R. G. Machine learning phases of matter. Nat. Phys. 13, 431–434 (2017).
White, A., Khial, P., Salehi, F., Hassibi, B. & Hajimiri, A. A silicon photonics computational lensless active-flat-optics imaging system. Sci. Rep. 10, 1869 (2020).
Rivenson, Y. et al. Deep learning microscopy. Optica 4, 1437–1443 (2017).
Goodfellow, I., Bengio, Y. & Courville, A. Deep Learning (MIT Press, 2016).
McClelland, J. L., McNaughton, B. L. & O’Reilly, R. C. Why there are complementary learning systems in the hippocampus and neocortex: insights from the successes and failures of connectionist models of learning and memory. Psychol. Rev. 102, 419–457 (1995).
Rumelhart, D. E., Hinton, G. E. & Williams, R. J. Learning representations by back-propagating errors. Nature 323, 533–536 (1986).
Hinton, G. E., Osindero, S. & Teh, Y.-W. A fast learning algorithm for deep belief nets. Neural Comput. 18, 1527–1554 (2006).
Russakovsky, O. et al. ImageNet Large Scale Visual Recognition Challenge. Int. J. Comput. Vis. 115, 211–252 (2015).
Zhang, Q.-J., Gupta, K. C. & Devabhaktuni, V. K. Artificial neural networks for RF and microwave design-from theory to practice. IEEE Trans. Microw. Theory Tech. 51, 1339–1350 (2003).
Patnaik, A., Mishra, R., Patra, G. & Dash, S. An artificial neural network model for effective dielectric constant of microstrip line. IEEE Trans. Antennas Propag. 45, 1697 (1997).
Watson, P. M. & Gupta, K. C. EM-ANN models for microstrip vias and interconnects in dataset circuits. IEEE Trans. Microw. Theory Tech. 44, 2495–2503 (1996).
Kabir, H., Wang, Y., Yu, M. & Zhang, Q.-J. Neural network inverse modeling and applications to microwave filter design. IEEE Trans. Microw. Theory Tech. 56, 867–879 (2008).
Zaabab, A. H., Zhang, Q.-J. & Nakhla, M. A neural network modeling approach to circuit optimization and statistical design. IEEE Trans. Microw. Theory Tech. 43, 1349–1358 (1995).
Southall, H. L., Simmers, J. A. & O’Donnell, T. H. Direction finding in phased arrays with a neural network beamformer. IEEE Trans. Antennas Propag. 43, 1369–1374 (1995).
Nair, V. & Hinton, G. E. Rectified linear units improve restricted Boltzmann machines. In Proc. 27th Int. Conf. Machine Learning (ICML-10) 807–814 (2010).
Srivastava, N., Hinton, G., Krizhevsky, A., Sutskever, I. & Salakhutdinov, R. Dropout: a simple way to prevent neural networks from overfitting. J. Machine Learning Res. 15, 1929–1958 (2014).
Ioffe, S. & Szegedy, C. Batch normalization: accelerating deep network training by reducing internal covariate shift. In Proc. 32nd Int. Conf. Machine Learning 448–456 (PMLR, 2015).
Hochreiter, S. & Schmidhuber, J. Long short-term memory. Neural Comput. 9, 1735–1780 (1997).
Goodfellow, I. et al. Generative adversarial nets. In NIPS’14: Proc. 27th Int. Conf. Neural Information Processing Systems 2672–2680 (NIPS, 2014).
Kingma, D. P. & Welling, M. Auto-encoding variational Bayes. In Proc. 2nd Int. Conf. Learning Representations (ICLR, 2014); preprint at https://arxiv.org/abs/1312.6114.
Haykin, S. S. Neural Networks and Learning Machines (Prentice Hall, 2009).
Malkiel, I. et al. Plasmonic nanostructure design and characterization via deep learning. Light Sci. Appl. 7, 60 (2018).
Ma, W., Cheng, F. & Liu, Y. Deep-learning-enabled on-demand design of chiral metamaterials. ACS Nano 6, 6326–6334 (2018).
Liu, D., Tan, Y., Khoram, E. & Yu, Z. Training deep neural networks for the inverse design of nanophotonic structures. ACS Photonics 5, 1365–1369 (2018).
Peurifoy, J. et al. Nanophotonic particle simulation and inverse design using artificial neural networks. Sci. Adv. 4, eaar4206 (2018).
Pilozzi, L., Farrelly, F. A., Marcucci, G. & Conti, C. Machine learning inverse problem for topological photonics. Commun. Phys. 1, 57 (2018).
Tahersima, M. H. et al. Deep neural network inverse design of integrated photonic power splitters. Sci. Rep. 9, 1368 (2019).
Hemmatyar, O., Abdollahramezani, S., Kiarashinejad, Y., Zandehshahvar, M. & Adibi, A. Full color generation with Fano-type resonant HfO2 nanopillars designed by a deep-learning approach. Nanoscale 11, 21266–21274 (2019).
Sajedian, I., Badloe, T. & Rho, J. Optimisation of colour generation from dielectric nanostructures using reinforcement learning. Opt. Express 27, 5874–5883 (2019).
Chen, Y., Zhu, J., Xie, Y., Feng, N. & Liu, Q. H. Smart inverse design of graphene-based photonic metamaterials by an adaptive artificial neural network. Nanoscale 11, 9749–9755 (2019).
Nadell, C. C., Huang, B., Malof, J. M. & Padilla, W. J. Deep learning for accelerated all-dielectric metasurface design. Opt. Express 27, 27523–27535 (2019).
Qiu, T. et al. Deep learning: a rapid and efficient route to automatic metasurface design. Adv. Sci. 6, 1900128 (2019).
Zhang, T. et al. Efficient spectrum prediction and inverse design for plasmonic waveguide systems based on artificial neural networks. Photonics Res. 7, 368–380 (2019).
Asano, T. & Noda, S. Optimization of photonic crystal nanocavities based on deep learning. Opt. Express 26, 32704–32717 (2018).
Alagappan, G. & Png, C. E. Modal classification in optical waveguides using deep learning. J. Mod. Opt. 66, 557–561 (2019).
Alagappan, G. & Png, C. E. Deep learning models for effective refractive indices in silicon nitride waveguides. J. Opt. 21, 035801 (2019).
Kiarashinejad, Y., Abdollahramezani, S., Zandehshahvar, M., Hemmatyar, O. & Adibi, A. Deep learning reveals underlying physics of light-matter interactions in nanophotonic devices. Adv. Theory Simul. 2, 1900088 (2019).
Comin, A. & Hartschuh, A. Efficient optimization of SHG hotspot switching in plasmonic nanoantennas using phase-shaped laser pulses controlled by neural networks. Opt. Express 26, 33678–33686 (2018).
Li, L. et al. DeepNIS: deep neural network for nonlinear electromagnetic inverse scattering. IEEE Trans. Antennas Propag. 67, 1819–1825 (2018).
Turpin, A., Vishniakou, I. & Seelig, J. D. Light scattering control in transmission and reflection with neural networks. Opt. Express 26, 30911–30929 (2018).
Zhang, Q. et al. Machine‐learning designs of anisotropic digital coding metasurfaces. Adv. Theory Simul. 2, 1800132 (2019).
Li, L. et al. Machine-learning reprogrammable metasurface imager. Nat. Commun. 10, 1082 (2019).
Maksov, A. et al. Deep learning analysis of defect and phase evolution during electron beam-induced transformations in WS2. npj Comput. Mater. 5, 12 (2019).
Xie, T. & Grossman, J. C. Crystal graph convolutional neural networks for an accurate and interpretable prediction of material properties. Phys. Rev. Lett. 120, 145301 (2018).
He, K., Zhang, X., Ren, S. & Sun, J. Deep residual learning for image recognition. In Proc. 2016 IEEE Conf. Computer Vision and Pattern Recognition https://doi.org/10.1109/CVPR.2016.90 (IEEE, 2016).
Szegedy, C. et al. Going deeper with convolutions. In Proc. IEEE Conf. Computer Vision and Pattern Recognition https://doi.org/10.1109/CVPR.2015.7298594 (IEEE, 2015).
Sajedian, I., Kim, J. & Rho, J. Finding the optical properties of plasmonic structures by image processing using a combination of convolutional neural networks and recurrent neural networks. Microsyst. Nanoeng. 5, 27 (2019).
Zhou, Q., Yang, C., Liang, A., Zheng, X. & Chen, Z. Low computationally complex recurrent neural network for high speed optical fiber transmission. Opt. Commun. 441, 121–126 (2019).
Liu, Z., Raju, L., Zhu, D. & Cai, W. A hybrid strategy for the discovery and design of photonic nanostructures. IEEE Trans. Emerg. Sel. Top. Circuits Systems 10, 126–135 (2020).
Ma, W., Cheng, F., Xu, Y., Wen, Q. & Liu, Y. Probabilistic representation and inverse design of metamaterials based on a deep generative model with semi-supervised learning strategy. Adv. Mater. 31, 201901111 (2019).
So, S. & Rho, J. Designing nanophotonic structures using conditional deep convolutional generative adversarial networks. Nanophotonics 8, 1255–1261 (2019).
Liu, Z., Zhu, D., Rodrigues, S. P., Lee, K.-T. & Cai, W. Generative model for the inverse design of metasurfaces. Nano Lett. 18, 6570–6576 (2018).
Liu, Z. et al. Compounding meta-atoms into meta-molecules with hybrid artificial intelligence techniques. Adv. Mater. 32, 1904790 (2019).
Wiecha, P. R. & Muskens, O. L. Deep learning meets nanophotonics: a generalized accurate predictor for near fields and far fields of arbitrary 3D nanostructures. Nano Lett. 20, 329–338 (2019).
Jiang, J. & Fan, J. A. Global optimization of dielectric metasurfaces using a physics-driven neural network. Nano Lett. 19, 5366–5372 (2019).
Bogdanov, S. I., Boltasseva, A. & Shalaev, V. M. Overcoming quantum decoherence with plasmonics. Science 364, 532–533 (2019).
Linic, S., Christopher, P. & Ingram, D. B. Plasmonic-metal nanostructures for efficient conversion of solar to chemical energy. Nat. Mater. 10, 911–921 (2011).
Ilic, O. & Atwater, H. A. Self-stabilizing photonic levitation and propulsion of nanostructured macroscopic objects. Nat. Photon. 13, 289–295 (2019).
Li, J. et al. Addressable metasurfaces for dynamic holography and optical information encryption. Sci. Adv. 4, eaar6768 (2018).
Jiang, J. et al. Free-form diffractive metagrating design based on generative adversarial networks. ACS Nano 13, 8872–8878 (2019).
Kudyshev, Z. A., Kildishev, A. V., Shalaev, V. M. & Boltasseva, A. Machine-learning-assisted metasurface design for high-efficiency thermal emitter optimization. Appl. Phys. Rev. 7, 021407 (2020).
Melati, D. et al. Mapping the global design space of nanophotonic components using machine learning pattern recognition. Nat. Commun. 10, 4775 (2019).
Watanabe, T., Ayata, M., Koch, U., Fedoryshyn, Y. & Leuthold, J. Perpendicular grating coupler based on a blazed antiback-reflection structure. J. Light. Technol. 35, 4663–4669 (2017).
Bar-Sinai, Y., Hoyer, S., Hickey, J. & Brenner, M. P. Learning data-driven discretizations for partial differential equations. Proc. Natl Acad. Sci. USA 116, 15344–15349 (2019).
Rudy, S. H., Brunton, S. L., Proctor, J. L. & Kutz, J. N. Data-driven discovery of partial differential equations. Sci. Adv. 3, e1602614 (2017).
Han, J., Jentzen, A. & Weinan, E. Solving high-dimensional partial differential equations using deep learning. Proc. Natl Acad. Sci. USA 115, 8505–8510 (2018).
Trivedi, R., Su, L., Lu, J., Schubert, M. F. & Vuckovic, J. Data-driven acceleration of photonic simulations. Sci. Rep. 9, 19728 (2019).
Qu, Y., Jing, L., Shen, Y., Qiu, M. & Soljacic, M. Migrating knowledge between physical scenarios based on artificial neural networks. ACS Photonics 6, 1168–1174 (2019).
Liu, C.-X., Yu, G.-L. & Zhao, G.-Y. Neural networks for inverse design of phononic crystals. AIP Adv. 9, 085223 (2019).
Ma, W. & Liu, Y. M. A data-efficient self-supervised deep learning model for design and characterization of nanophotonic structures. Sci. China Phys. Mech. Astron. 63, 284212 (2020).
Sirignano, J. & Spiliopoulos, K. DGM: a deep learning algorithm for solving partial differential equations. J. Comput. Phys. 375, 1339–1364 (2018).
Raissi, M. & Karniadakis, G. E. Hidden physics models: machine learning of nonlinear partial differential equations. J. Comput. Phys. 357, 125–141 (2018).
Raissi, M., Perdikaris, P. & Karniadakis, G. E. Physics-informed neural networks: a deep learning framework for solving forward and inverse problems involving nonlinear partial differential equations. J. Comput. Phys. 378, 686–707 (2019).
Hansen, N., Müller, S. D. & Koumoutsakos, P. Reducing the time complexity of the derandomized evolution strategy with covariance matrix adaptation (CMA-ES). Evol. Comput. 11, 1–18 (2003).
Hegde, R. S. Photonics inverse design: pairing deep neural networks with evolutionary algorithms. IEEE J. Sel. Top. Quant. Electron. 26, https://doi.org/10.1109/JSTQE.2019.2933796 (2019).
Guler, U., Boltasseva, A. & Shalaev, V. M. Refractory plasmonics. Science 344, 263–264 (2014).
Marino, G. et al. Spontaneous photon-pair generation from a dielectric nanoantenna. Optica 6, 1416–1422 (2019).
Zhang, Q., Yu, H., Barbiero, M., Wang, B. & Gu, M. Artificial neural networks enabled by nanophotonics. Light Sci. Appl. 8, 42 (2019).
Khoram, E. et al. Nanophotonic media for artificial neural inference. Photonics Res. 7, 823–827 (2019).
Shen, Y. et al. Deep learning with coherent nanophotonic circuits. Nat. Photon. 11, 441–446 (2017).
Feldmann, J., Youngblood, N., Wright, C., Bhaskaran, H. & Pernice, W. All-optical spiking neurosynaptic networks with self-learning capabilities. Nature 569, 208–214 (2019).
Hughes, T. W., Minkov, M., Shi, Y. & Fan, S. Training of photonic neural networks through in situ backpropagation and gradient measurement. Optica 5, 864–871 (2018).
Lin, X. et al. All-optical machine learning using diffractive deep neural networks. Science 361, 1004–1008 (2018).
Bao, Q. et al. Monolayer graphene as a saturable absorber in a mode-locked laser. Nano Res. 4, 297–307 (2011).
Tait, A. N. et al. Silicon photonic modulator neuron. Phys. Rev. Appl. 11, 064043 (2019).
Williamson, I. A. et al. Reprogrammable electro-optic nonlinear activation functions for optical neural networks. IEEE J. Sel. Top. Quant. Electron. 26, https://doi.org/10.1109/JSTQE.2019.2930455 (2019).
Shastri, B. J. et al. Spike processing with a graphene excitable laser. Sci. Rep. 6, 19126 (2016).
Tait, A. N. et al. Neuromorphic photonic networks using silicon photonic weight banks. Sci. Rep. 7, 7430 (2017).
Estakhri, N. M., Edwards, B. & Engheta, N. Inverse-designed metastructures that solve equations. Science 363, 1333–1338 (2019).
Hughes, T. W., Williamson, I. A., Minkov, M. & Fan, S. Wave physics as an analog recurrent neural network. Sci. Adv. 5, eaay6946 (2019).
Acknowledgements
Y.L. acknowledges financial support from the US National Science Foundation (NSF) (ECCS-1916839) and the Office of Naval Research (N00014-16-1-2409). W.C. acknowledges support from the Office of Naval Research (N00014-17-1-2555) and the NSF (DMR-2004749). The Purdue team acknowledges financial support from DARPA/DSO (HR00111720032, Z.A.K.), the US National Science Foundation (ECCS-2029553, A.B.) and the Air Force Office of Scientific Research (AFOSR) (FA9550-20-1-0124, A.B.).
Author information
Authors and Affiliations
Corresponding authors
Ethics declarations
Competing interests
The authors declare no competing interests.
Additional information
Publisher’s note Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Rights and permissions
About this article
Cite this article
Ma, W., Liu, Z., Kudyshev, Z.A. et al. Deep learning for the design of photonic structures. Nat. Photonics 15, 77–90 (2021). https://doi.org/10.1038/s41566-020-0685-y
Received:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1038/s41566-020-0685-y
This article is cited by
-
Non-orthogonal optical multiplexing empowered by deep learning
Nature Communications (2024)
-
A deep learning method for empirical spectral prediction and inverse design of all-optical nonlinear plasmonic ring resonator switches
Scientific Reports (2024)
-
Stochastic and multi-objective design of photonic devices with machine learning
Scientific Reports (2024)
-
Controlling water waves with artificial structures
Nature Reviews Physics (2024)
-
Highly efficient vortex generation at the nanoscale
Nature Nanotechnology (2024)