Super-resolution structured illumination microscopy (SR-SIM) provides an up to twofold enhanced spatial resolution of fluorescently labeled samples. The reconstruction of high-quality SR-SIM images critically depends on patterned illumination with high modulation contrast. Noisy raw image data (e.g., as a result of low excitation power or low exposure time), result in reconstruction artifacts. Here, we demonstrate deep-learning based SR-SIM image denoising that results in high-quality reconstructed images. A residual encoding–decoding convolutional neural network (RED-Net) was used to successfully denoise computationally reconstructed noisy SR-SIM images. We also demonstrate the end-to-end deep-learning based denoising and reconstruction of raw SIM images into high-resolution SR-SIM images. Both image reconstruction methods prove to be very robust against image reconstruction artifacts and generalize very well across various noise levels. The combination of computational image reconstruction and subsequent denoising via RED-Net shows very robust performance during inference after training even if the microscope settings change.

Deep learning in the context of nano-photonics is mostly discussed in terms of its potential for inverse design of photonic devices or nano-structures. Many of the recent works on machine-learning inverse design are highly specific, and the drawbacks of the respective approaches are often not immediately clear. In this review we want therefore to provide a critical review on the capabilities of deep learning for inverse design and the progress which has been made so far. We classify the different deep-learning-based inverse design approaches at a higher level as well as by the context of their respective applications and critically discuss their strengths and weaknesses. While a significant part of the community’s attention lies on nano-photonic inverse design, deep learning has evolved as a tool for a large variety of applications. The second part of the review will focus therefore on machine learning research in nano-photonics “beyond inverse design.” This spans from physics-informed neural networks for tremendous acceleration of photonics simulations, over sparse data reconstruction, imaging and “knowledge discovery” to experimental applications.

All-optical binary convolution with a photonic spiking vertical-cavity surface-emitting laser (VCSEL) neuron is proposed and demonstrated experimentally for the first time, to the best of our knowledge. Optical inputs, extracted from digital images and temporally encoded using rectangular pulses, are injected in the VCSEL neuron, which delivers the convolution result in the number of fast (<100ps long) spikes fired. Experimental and numerical results show that binary convolution is achieved successfully with a single spiking VCSEL neuron and that all-optical binary convolution can be used to calculate image gradient magnitudes to detect edge features and separate vertical and horizontal components in source images. We also show that this all-optical spiking binary convolution system is robust to noise and can operate with high-resolution images. Additionally, the proposed system offers important advantages such as ultrafast speed, high-energy efficiency, and simple hardware implementation, highlighting the potentials of spiking photonic VCSEL neurons for high-speed neuromorphic image processing systems and future photonic spiking convolutional neural networks.

Imaging through scattering media is one of the hotspots in the optical field, and impressive results have been demonstrated via deep learning (DL). However, most of the DL approaches are solely data-driven methods and lack the related physics prior, which results in a limited generalization capability. In this paper, through the effective combination of the speckle-correlation theory and the DL method, we demonstrate a physics-informed learning method in scalable imaging through an unknown thin scattering media, which can achieve high reconstruction fidelity for the sparse objects by training with only one diffuser. The method can solve the inverse problem with more general applicability, which promotes that the objects with different complexity and sparsity can be reconstructed accurately through unknown scattering media, even if the diffusers have different statistical properties. This approach can also extend the field of view (FOV) of traditional speckle-correlation methods. This method gives impetus to the development of scattering imaging in practical scenes and provides an enlightening reference for using DL methods to solve optical problems.

Imaging through nonstatic scattering media is one of the major challenges in optics, and encountered in imaging through dense fog, turbid water, and many other situations. Here, we propose a method to achieve single-shot incoherent imaging through highly nonstatic and optically thick turbid media by using an end-to-end deep neural network. In this study, we use fat emulsion suspensions in a glass tank as a turbid medium and an additional incoherent light to introduce strong interference noise. We calibrate that the optical thickness of the tank of turbid media is as high as 16, and the signal-to-interference ratio is as low as -17dB. Experimental results show that the proposed learning-based approach can reconstruct the object image with high fidelity in this severe environment.

Being invisible at will has been a long-standing dream for centuries, epitomized by numerous legends; humans have never stopped their exploration steps to realize this dream. Recent years have witnessed a breakthrough in this search due to the advent of transformation optics, metamaterials, and metasurfaces. However, the previous metasurface cloaks typically work in a reflection manner that relies on a high-reflection background, thus limiting the applications. Here, we propose an easy yet viable approach to realize the transmitted metasurface cloak, just composed of two planar metasurfaces to hide an object inside, such as a cat. To tackle the hard-to-converge issue caused by the nonuniqueness phenomenon, we deploy a tandem neural network (T-NN) to efficiently streamline the inverse design. Once pretrained, the T-NN can work for a customer-desired electromagnetic response in one single forward computation, saving a great amount of time. Our work opens a new avenue to realize a transparent invisibility cloak, and the tandem-NN can also inspire the inverse design of other metamaterials and photonics.

Structural color based on Fabry–Perot (F-P) cavity enables a wide color gamut with high resolution at submicroscopic scale by varying its geometrical parameters. The ability to design such parameters that can accurately display the desired color is therefore crucial to the manufacturing of F-P cavities for practical applications. This work reports the first inverse design of F-P cavity structure using deep learning through a bidirectional artificial neural network. It enables the production of a significantly wider coverage of color space that is over 215% of sRGB with extremely high accuracy, represented by an average ΔE2000 value below 1.2. The superior performance of this structural color-based neural network is directly ascribed to the definition of loss function in the uniform CIE 1976-Lab color space. Over 100,000 times improvement in the design efficiency has been demonstrated by comparing the neural network to the metaheuristic optimization technique using an evolutionary algorithm when designing the famous painting of “Haystacks, end of Summer” by Claude Monet. Our results demonstrate that, with the correct selection of loss function, deep learning can be very powerful to achieve extremely accurate design of nanostructured color filters with very high efficiency.