Deep learning assisted variational Hilbert quantitative phase imaging
Quantitative phase imaging (QPI), as a powerful label-free imaging technique, enables dynamic 2D and 3D non-destructive imaging of completely transparent structures1-3. It uses the refractive index as an endogenous contrast agent to generate subcellular-specific quantitative maps of analyzed live bio-structure4, 5. QPI solutions based on digital holographic microscopy (DHM) encode a complex wavefront information into intensity modulations by the interference of a scattered sample wave and a reference wave6-9. And it can robustly perform the quantitative analysis of wave-matter interactions by decoding phase delay from a hologram. DHM has emerged as a valuable means in the biomedical fields, such as measurements for stain-free biological cells3, 10 , optical metrology of nanostructures11-14, and drug release monitoring invitro15.
Regarding the phase demodulation strategy employed, there are two main configurations for holographic wavefront acquisition in DHM, i.e., in-line and off-axis digital holography (DH). In-line DH records complete wavefront information by the interference of the object light and the reference light on the same optical axis, which can realize full detector-bandwidth phase reconstruction. However, due to the superimposed twin image, the phase retrieval results of samples are severely impacted by imaging artifacts. It always needs to be processed via iterative phase retrieval16, 17 or noniterative phase-shifting methods18-20, which dramatically sacrifices the temporal resolution. Therefore, it is difficult for the in-line DH, which is vulnerable to external disturbance and vibration, to be applied to dynamic measurement. Alternatively, off-axis DH implements twin-image separation by introducing a slight angle between the object beam and reference beam and recovers the complex wavefront of the sample from the single-frame off-axis hologram. Whereas, for achieving the separation of autocorrelation and cross-correlation terms in the spatial frequency domain (SFD), the off-axis DH needs to provide a sufficiently high carrier frequency at the expense of the space-bandwidth product (SBP) of the imaging system21. The slightly off-axis DH regime, as a single-frame high-SBP DH imaging solutions, is therefore proposed22-24. It optimizes SBP through full spectral separation of conjugated object lobes while leaving the autocorrelation term partially overlapped with information-carrying cross-correlation terms. Under this configuration, the inevitable spectrum overlapping causes phase artifacts, which greatly degrades the imaging quality and impairs the practicality of the slightly off-axis DH configuration.
High-accuracy artifacts-free phase recovery from the low-carrier frequency holograms is the key to slightly off-axis DH application. This process is presently implemented by suppressing autocorrelation term iteratively25, utilizing dual-frame decoding scheme26, 27, employing second wavelength assistance28 and performing the 1D limited processing29, 30. With inspiration from the theory of “cepstrum” and homomorphic filtering31, a slightly off-axis DH demodulation scheme based on the Kramers-Kronig (KK) relations is proposed, which utilizes the half-space bandwidth of the sensor to achieve high-SBP imaging32, 33. Although it is able to increase the SBP of full complex field recovery significantly, it inevitably requires intensity restrictions on the object and reference beams and the separation of the cross-correlation terms of the interferogram in the extended SFD. Noteworthily, an exquisite low-carrier frequency fringe demodulation approach has been presented recently, namely variational Hilbert quantitative phase imaging (VHQPI)34. The VHQPI, as an end-to-end pure numerical add-on module, deploys the merger of tailored variational image decomposition35 and enhanced Hilbert spiral transform36 to achieve quantitative phase recovery. It adaptively alleviates the overlapped-spectrum problem and robustly demodulates high-quality phase information, performing excellent practicality in biological applications.
Although VHQPI has demonstrated excellent low-carrier frequency fringe demodulation capability, the algorithm-inherent limitations (e.g., parameter robustness and iterative stability) still cause non-sufficient image frequency component extraction, resulting in imaging artifacts in the phase reconstruction results. Deep learning (DL), as a subfield of machine learning, has currently gained extensive attention in the field of optical metrology and demonstrated great potential in solving optical metrology tasks37-46. When sufficient training data is collected in an environment that reproduces real experimental conditions, the trained model may have advantages over physics-model-based approaches on some issues (e.g., computing speed, parameter adaptivity, algorithm complexity)37. Specifically, in terms of a series of ill-posed inverse phase retrieval problems, the traditional physical model tends to exhibit higher physics complexity and time consumption. Driven by a large dataset, the deep neural network (DNN) can directly and efficiently reconstruct the phase and amplitude images of the objects from the captured holograms47-49. Nevertheless, in DL-based phase recovery tasks, it is pretty tricky and laborious to capture massive datasets and generate the corresponding ground truth, especially when applied to bio-samples. Deep image prior (DIP) applies an untrained network to the solution of several inverse problems without a massive training dataset and ground truth, which can fit a randomly initialized DNN to a single corrupted image50. Inspired by the DIP, an untrained network model named “PhysenNet” is proposed, which incorporates a complete physical model into the conventional DNN to achieve phase retrieval from a single intensity image51.
Inspired by the successful application of the interplay between DNN and the physical model, in this work, we propose a DL-assisted variational Hilbert quantitative phase imaging approach (DL-VHQPI). Unlike the massive-data-driven DL training model, DL-VHQPI, which utilizes DNN to compensate and optimize the possible solutions of the physics-driven model, can achieve high-precision artifacts-free phase recovery using only a small fraction datasets. Specifically, VHQPI, as the underlying physical model, can complete the preliminary extraction of the background components of the fringes to provide a physical prior for the deep learning model. The DNN compensates for the image frequencies that cannot be extracted by the physical model using the idea of residual compensation. Due to the physical model reducing the information entropy of the dataset, the DL-VHQPI performs higher reconstruction accuracy utilizing less than one-tenth of the dataset of the conventional end-to-end model (without the physical model). The simulation experiments quantitatively demonstrate that the proposed method can achieve high-accuracy artifacts-free quantitative phase imaging from single-frame low-carrier frequency holograms. And the results of live-cell experiments demonstrate the practicality of the method in biological research.
2 Principle of VHQPI
The VHQPI, as the physical model of the DL-VHQPI, adaptively and effectively completes the low-carrier frequency fringe demodulation employing the unsupervised variational image decomposition (uVID) and enhanced Hilbert spiral transform (HST). This section will focus on describing the process details and physical limitations of this method. In the DH wavefront recording, the interferogram containing the required object information is constructed upon the coherent superimposition of the object and reference beams. The intensity distribution of the recorded hologram can be expressed as:
It consists of a sum of three fundamental intensity components: background (
Flow chart of slightly off-axis interferometric fringe demodulation based on VHQPI.
To recover the phase information of the object, the uVID-filtered noise-free zero-mean-valued interferogram is then analyzed using the HST algorithm36, as shown in Step 2 of
3 Deep learning assisted VHQPI model
VHQPI has been proven to have excellent robustness and practicality in low-carrier frequency fringe demodulation issue though34. However, the algorithm-inherent iterative instability and parameter robustness restrict the image frequency component extraction capability, which will cause the non-perfect background term removal. DL methods driven by massive datasets provide a new route to address this problem by virtue of their high-powerful image feature extraction characteristics. Whereas, when encountering insufficient training data, which is very common, the DL method based on massive datasets may have a poor effect. A feasible scheme is to train the DNN on a stronger-constrained available standardized dataset57. Here, we employ Shannon entropy theory of the images in the dataset for that purpose: the lower the entropy of the datasets is, the more constrained prior information is, giving it a better same-domain generalization ability58, 59. Therefore, in the proposed DL-assisted VHQPI model, the uVID is utilized to extract the image background term as the physical prior of the network to reduce the dataset's entropy. The first convolutional neural network (CNN1) is used to “learn” the residual terms and assists the physical model to complete the preliminary estimation of the background components of the fringes. Furthermore, to further improve the imaging accuracy, the original hologram and the preliminary estimation background are re-fed into the model (CNN2) for advanced component extraction. Dual-channel input is used because the preliminarily estimated background terms have been very close to the ground truth after the first residual compensation by CNN1. Hence, the preliminary estimated background can be used to provide the network with feature guidance and helps CNN2 achieve the advanced component extraction.
As depicted in
Deep learning-assisted VHQPI. ( a) Total network structure, combining uVID and HST with CNN respectively for phase reconstruction. ( b) CNN1 takes a hologram as input and consists of three convolutional layers and a group of residual blocks to achieve compensation of background residuals by learning. ( c) The CNN2 network structure is the same as CNN1, except that CNN2 combines the original hologram and the result of the first process into a two-channel input for advanced background compensation.
Moreover, both CNN1 and CNN2 networks are composed of a convolutional layer (Conv), a group of residual blocks (containing four residual blocks), and two convolutional layers. Each residual block comprises two sets of Convs stacked one above the other. The network architecture uses Batch Normalization60 and ReLU activation61 to accelerate the model convergence. It establishes a shortcut between input and output, which can solve the problem of accuracy decline as the network deepens, thereby easing the training process. The output of the Convs is a 3-D tensor of shape
4 Experiments and results
In this section, we demonstrate the performance of the proposed DL-VHQPI method over the conventional physics-driven low-carrier frequency fringe demodulation techniques and pure DL approach without a physical model (DL-noPhy) through numerical simulation and live-cell experiment. A rich set of paired training data is the prerequisite for network generalization during DL training. It is challenging to acquire a reliable ground truth in the real-world DH system due to environment-induced instability and system-inherent speckle noise. Consequently, we simulated low-carrier frequency holograms and the corresponding ground truth for training and quantitative analysis. We separately constructed the complex amplitude distributions of the object and reference light waves, and then the holograms can be constructed by solving the square of the modulus of the sum of the two. The sum of the squares of the modulus values of the two was calculated to obtain the background (ground truth) needed for training. The more specific process can be found in Supplementary information Section 1.
In the live-cell experiment, we used the Digital holographic smart computational light microscope (DH-SCLM) developed by SCILab, and turned it to a slightly off-axis state for hologram acquisition1. In the DH-SCLM, the object wave transmitting the objective lens (UPLanSAPO ×20/0.45NA, Olympus, Japan) interferes with the reference light and is recorded by the camera (The Imaging Source DMK 23U274, 1600×1200, 4.4 μm). The central wavelength of the illumination is 532 nm. The used sample is Henrietta Lacks (HeLa) human cervical cancer cells cultured in DMEM medium with 10% fetal bovine serum under standard cell culture conditions (37.2 °C in 5% CO2 in a humidified incubator). To acquire the ground truth from the configuration, each intensity map of the object and reference light paths needs to be captured separately under the highly stable condition of the holographic system (Refer to Section 2 of the Supplementary Information for detailed processing). The complete training process was implemented using the TensorFlow framework (Google) and was computed on a GTX Titan graphics card (NVIDIA). A fixed learning rate of 0.0001 for the experiment is adopted for the Adam optimizer62.
The experiment results under the numerical simulation. ( a) The FT method phase recovery result. ( b) The phase recovery result of VHQPI. ( c) The phase result reconstructed by DL-VHQPI. ( d) The ground truth. ( e– g) The difference between the phase results of the three methods (i.e. FT, VHQPI, DL-VHQPI) and the ground truth. ( h) Quantitative error analysis of three methods. ( i) The cross-section of the phase results of FT, VHQPI, DL-VHQPI, and ground truth, and ( j1– j4) are the DIC views of the partially enlarged views of their corresponding phase maps respectively.
In addition, we also designed a comparison experiment with DL-noPhy (The specific network is provided in the Section 4 of Supplementary information) to demonstrate the high-efficiency and high-accuracy characteristics exhibited by the proposed method.
Table 1. The quantitative comparison results of DL-VHQPI and DL-noPhy.
4.4 Live-cell experiment on HeLa cells
We performed holographic biological experiments on HeLa cells under a ×20/0.45NA lens to demonstrate the application of the method in biological research. The denoised interferogram presented in
Results of holographic experiments on HeLa cells. ( a) Low-carrier-frequency high-contrast hologram collected by slightly off-axis interferometry system. ( b) Corresponding spatial frequency spectrum. ( c) The result of phase recovery by slightly off-axis holography using FT method under ×20 lens. ( d) The result of phase recovery using DL-VHQPI. ( e1– e4) and ( f1– f4) correspond to the local amplification results of “Area1” and “Area2” for the two samples under different phase recovery methods. Where (e2, e4, f2, f4) are the corresponding DIC views, respectively. ( g) and ( h) The DIC views after partial magnification of the phase map in the corresponding red box. ( i) The numerical distribution of the cross-section and detail-preservation feature of the DL-VHQPI.
Indeed, reducing the size of the FT filter window may also be a good way to alleviate artifacts, but this will not fundamentally address the problem of the overlapped spectrum and will cause phase imaging blur. The reason is that reducing the filtering window is at the expense of the system’s SBP and the high-frequency information of the object cannot be enclosed in the limited filtering window. In the Section 3 of Supplementary information, we experimentally present the imaging effects under different FT filtering windows for living cells. To verify the generalization of DL-VHQPI, we supplemented a new group of experimental results for living cells in Supplementary Section 5, in which we added a comparison and discussion with the VHQPI method and the traditional FT method. The results demonstrate that DL-VHQPI still performs the best artifact-suppression ability and generalizability under a new group of biological applications.
5 Conclusions and discussions
In summary, we proposed a high-accuracy artifacts-free single-frame low-carrier frequency fringe demodulation approach for the slightly off-axis DH system, i.e., a model using the DNN-assisted physical process. When the cross-correlation and autocorrelation are inevitably aliased in the SFD, the phase reconstruction based on the conventional FT method cannot eliminate the effect of phase artifacts caused by zero-order term6. Although reducing the size of the FT filter window may alleviate the problem of imaging artifacts, the high-frequency information loss of the object caused by the limited filtering window will cause imaging blur. The method based on Kramers-Kronig relation is proposed on the basis of the concept of “cepstrum” and homomorphic filtering31, however, this method must depend on the limited condition of the object-reference ratio and need the separation of the high-order terms in the extended SFD32, 33. Furthermore, the VHQPI implements the background component removal of single-frame hologram utilizing the principle of image frequency components extraction, while it inevitably suffers from the non-sufficient background term removal caused by the physical method34. In contrast, DL-VHQPI, a novel DL-assisted physical model method, can better suppress phase artifacts while improving imaging accuracy. The simulation result quantitatively demonstrates that the phase recovery accuracy obtained by DL-VHQPI is greatly superior to that by FT and VHQPI. Moreover, the live-cell experiment results demonstrate that our method is applicable in biological research.
In addition, it is noteworthy that in the classical end-to-end DNN model (without a physical model), massive data pairs are required to train the network model for a higher reconstruction precision. However, it may be prohibitively laborious and time-consuming for the real-world DH system to collect datasets and generate the corresponding ground truth. Conversely, the proposed DL-VHQPI can perform better same-domain generalization ability and image data-feature extraction capability without a large of datasets. Compared to the classical end-to-end DNN model (i.e., DL-noPhy), DL-VHQPI can achieve a higher reconstruction accuracy utilizing only a small fraction of the datasets due to the physical model reducing the information entropy of DL training objects. Meanwhile, fewer datasets mean shorter training time and higher training efficiency.
The significance of our work lies in the multiple possibilities of applying the proposed DL-assisted physical model idea to the QPI. This idea can be applied to many scenarios in which deep learning methods are applied to the QPI, e.g., addressing a series of ill-posed inverse phase retrieval problems and holography-based high-throughput optical diffraction tomography (ODT) problems63-65. Specifically, the artifacts-free low-carrier-frequency fringe demodulation capability of the proposed method has application possibilities for ODT imaging of wide-bandwidth objects. In addition, it has also implications for high-throughput studies of high-robust common-path off-axis interferometer systems66, 67. We envision that the idea presented in this research can be applicable to a diverse range of future computational imaging techniques, not just limited to what we discussed here.
We are grateful for financial supports from the National Natural Science Foundation of China (61905115, 62105151, 62175109, U21B2033, 62227818), Leading Technology of Jiangsu Basic Research Plan (BK20192003), Youth Foundation of Jiangsu Province (BK20190445, BK20210338), Biomedical Competition Foundation of Jiangsu Province (BE2022847), Key National Industrial Technology Cooperation Foundation of Jiangsu Province (BZ2022039), Fundamental Research Funds for the Central Universities (30920032101), and Open Research Fund of Jiangsu Key Laboratory of Spectral Imaging & Intelligent Sense (JSGP202105, JSGP202201), National Science Center, Poland (2020/37/B/ST7/03629). The authors thank F. Sun for her contribution to this paper in terms of language expression and grammatical correction.
The authors declare no competing financial interests.
 BettenworthD, LenzP, KrausewitzP, BrücknerM, KetelhutSet alQuantitative stain-free and continuous multimodal monitoring of wound healing in vitro with digital holographic microscopy PLoS One20149e10731710.1371/journal.pone.0107317
 CoppolaG, FerraroP, IodiceM, De NicolaS, FinizioAet alA digital holographic microscope for complete characterization of microelectromechanical systemsMeas Sci Technol20041552953910.1088/0957-0233/15/3/005
 PoonTCDigitalHolographyandThree-DimensionalDisplay: PrinciplesandApplications (Springer, New York, 2006).
 León-RodríguezM, RayasJA, CorderoRR, Martínez-GarcíaA, Martínez-GonzalezAet alDual-plane slightly off-axis digital holography based on a single cube beam splitterAppl Opt2018572727273510.1364/AO.57.002727
 LarkinKG, BoneDJ, OldfieldMANatural demodulation of two-dimensional fringe patterns. I. General background of the spiral phase quadrature transformJ Opt Soc Am A2001181862187010.1364/JOSAA.18.001862
 CywińskaM, BrzeskiF, KrajnikW, PatorskiK, ZuoCet alDeepDensity: convolutional neural network based estimation of local fringe pattern densityOpt Lasers Eng202114510667510.1016/j.optlaseng.2021.106675
 LiYX, QianJM, FengSJ, ChenQ, ZuoCDeep-learning-enabled dual-frequency composite fringe projection profilometry for single-shot absolute 3D shape measurementOpto-Electron Adv2022521002110.29026/oea.2022.210021
 ChenHL, HuangLZ, LiuTR, OzcanAFourier Imager Network (FIN): a deep neural network for hologram reconstruction with superior external generalizationLight Sci Appl20221125410.1038/s41377-022-00949-8
 LempitskyV, VedaldiA, UlyanovD.Deep image prior. In Proceedings of 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition 9446–9454 (IEEE, 2018);http://doi.org/10.1109/CVPR.2018.00984.
 CoverTMElementsofInformationTheory. John Wiley & Sons, 1999).
 IoffeS, SzegedyC.Batch normalization: accelerating deep network training by reducing internal covariate shift. In Proceedings of the 32nd International Conference on Machine Learning (JMLR.org, 2015).
 NairV, HintonGE.Rectified linear units improve restricted Boltzmann machines. In Proceedings of the 27th International Conference on Machine Learning 807–814 (Omnipress, 2010).
 ChoiW, Fang-YenC, OhS, LueN, DasariRRet alTomographic phase microscopy: quantitative 3D-mapping of refractive index in live cellsImaging Microsc2008104850
Zhuoshi Li, Jiasong Sun, Yao Fan, Yanbo Jin, Qian Shen, Maciej Trusiak, Maria Cywińska, Peng Gao, Qian Chen, Chao Zuo. Deep learning assisted variational Hilbert quantitative phase imaging[J]. Opto-Electronic Science, 2023, 2(4): 220023.