The digital lensless in-line holographic microscope (LIHM) was originated from the holography scheme of Gabor^[1] and has attracted many research efforts in recent years^{[2–5" target="_self" style="display: inline;">–5]} because it provides a compact and low-cost solution for microscopic imaging with promising applications in biomedical or other areas^{[6–9" target="_self" style="display: inline;">–9]}. The setup of an LIHM is very simple and only consists of the illumination light source, the sample, and an imaging sensor located behind the sample. Not surprisingly, imaging resolution is one of the main considerations in LIHM, which is generally limited by the pixel size of imaging sensors because of the short sample-to-sensor distance^[10,11]. The pixel super-resolution (PSR) technique was one of the popular methods to overcome this issue by reconstructing the enhanced-resolution image from multiple low-resolution images with different sample displacements^{[12–17" target="_self" style="display: inline;">–17]}. The displacements can be achieved by changing the position of the illumination source or by moving the sample either mechanically or non-mechanically. In previously reported methods, the objects within the sample are usually supposed to move together in the same speed and direction. Thus, the relative object displacements are invariant, which is one of the fundamental assumptions of the popular PSR reconstruction method^[18]. Besides PSR, the method of obtaining enhanced-resolution images from multiple low-resolution images was also used in Fourier ptychographic microscopy^[19], where the processing was in the Fourier domain instead.

Previously, we demonstrated that the PSR technique can be used in LIHM for multilayer samples, where the enhanced-resolution sample image was reconstructed from the high-resolution in-line hologram obtained from multiple low-resolution holograms recorded by the imaging sensor while moving the sample randomly by hand^[20]. The random sample movements by hand have greatly simplified the actuation requirement for the PSR technique, and our PSR-LIHM system has been successfully used to image objects located in different layers when all the objects are moved in the same direction and speed. However, in certain multilayer samples, especially samples in water solutions where the in-line holograms of objects in different layers are overlapped and move in different directions or speeds, applying the PSR directly to the multiple low-resolution in-line holograms will result in a distorted hologram instead of an enhanced-resolution hologram because of the overlapping of holograms of different objects.

In this Letter, we propose to solve this issue by using the hologram segmentation method before applying the PSR technique. In recent years, many hologram segmentation techniques have been reported, including, for example, the entropy minimization method^[21], the blocked partitioned entropy minimization method^[22], the autofocusing method^[23], the virtual diffraction plane method^[24], the level set method^[25], and the bivariate jointly distributed region snake method^[26]. These methods relied on image segmentation at certain holographic reconstruction planes, which could be determined by entropy^[21–22], autofocusing^[23], or randomly^[24]. These methods are generally suitable for hologram segmentations in situations where an off-axis holography scheme is used or twin-image disturbance and hologram overlapping are insignificant. Here, we choose to use the hologram segmentation method proposed by Orzo et al.^[27,28] which can effectively separate the in-line holograms of objects at different layers, even when the twin-image disturbance and hologram overlapping exist. Thus, we can obtain the low-resolution in-line holograms of the target object in one layer only and then apply the PSR technique to get the enhanced-resolution holograms, which can be used to reconstruct the enhanced-resolution image. Another additional advantage of applying the hologram segmentation technique is that the twin-image effects can also be suppressed, which results in reduction of background noises. Our proposed PSR-LIHM with hologram segmentation provides a better way to image specific target objects in multilayer samples, especially for samples in water solutions where the moving directions or speeds of different objects in the solution might be different.

The schematic and the photograph of our LIHM system are shown in Fig. 1. In the system, the multilayer sample was illuminated by a fiber-coupled laser source with a wavelength of 473 nm (MBL-III-473, Changchun New Industries Optoelectronics Technology Co., China), and the in-line hologram was recorded by a complementary metal–oxide–semiconductor (CMOS) imaging sensor with pixel size of 2.2 μm (DMK72AUC02, the Imaging Sources Europe GmbH) located behind the sample. In our experiment, the distance between the light source and the sample was around 5–8 cm, and the distance between the sample and sensor was around 1–5 mm. During the experiment, a sequence of low-resolution in-line holograms was acquired while the sample was randomly moved by hand or by self-actuation if the sample was alive.

Fig. 1. (a) Schematic of the LIHM system setup; (b) photograph of the system.

下载图片 查看所有图片

The process of reconstructing the enhanced-resolution image of the target object is illustrated in the flowchart shown in Fig. 2, where we first segmented the holograms of the target object in each acquired in-line hologram, and then reconstructed a sequence of low-resolution images of the target object by the conventional scalar diffraction formula with angular spectrum propagation^[29]; finally, the PSR technique was applied to obtain the enhanced-resolution image of the target object.

Fig. 2. Flowchart of the reconstruction process in our method.

下载图片 查看所有图片

The principle of the hologram segmentation technique is to identify the target object boundary and remove the background disturbance caused by other overlapped holograms or the twin image. By several back-and-forth propagations between the sample and the sensor plane, the hologram of the target object can be segmented. For each acquired low-resolution in-line hologram intensity H, the process of separating the hologram of target object is summarized below^[27].(1)Reconstruct the whole sample image I by back-propagating H to the sample plane using the angular spectrum method^[29]: I=FT−1{FT{H}exp[−i2πz(1λ)2−fx2−fy2]},(1)where FT{} represents the Fourier transform, z is the sample-to-sensor distance, λ is the wavelength, and fx, fy are the spatial frequencies. Here, the optimal sample-to-sensor distance is estimated by the autofocusing algorithm of the Tenenbaum gradient^[30].(2)Update I by locating the target object and fill its surroundings with the background value Bexp(−i2πz/λ), where B is the average value of the whole sample image: I1(x,y)={I(x,y),(x,y)∈the target objectBexp(−i2πz/λ),otherwise,(2)where (x, y) are the spatial coordinates. The target object is located by finding the object boundary with a thresholding process. In this step, the twin-image background and signals from other objects that are located outside the target object boundary will be suppressed. Those signals inside the target object boundary could be further suppressed by the next few steps^[27].(3)Propagate I1 to the sensor plane and get H1 using the angular spectrum method, similar to Eq. (1).(4)Calculate H2 as follows: H2=H−2Re{H1}+2Bexp(−i2πz/λ).(3)(5)Propagate H2 back to the sample plane to get I2, then update I2 to get I3 by locating the target object, and fill its surroundings with the background value using Eq. (2).(6)Propagate I3 to the sensor plane to get H3, and then the segmented remaining hologram of the target object can be written as H4=H−Re{H2−H3+Bexp(−i2πz/λ)}.(4)

To obtain the enhanced-resolution image of the target object, we assume pure translational motion and common space-invariant blur and use the PSR technique of the fast noniterative algorithm reported by Elad et al.^[18,19], where the low-resolution images Yk(k=1,…,N, where N is the number of acquired holograms) can be written as Yk=DMFkX+Vk,k=1,…,N,(5)where X is the enhanced-resolution image, Fk is the translation operator, M is the blur matrix, D is the decimation operation, and Vk is the unknown additive noise. According to the PSR theory, we can obtain the maximum likelihood estimation of X as X^=M−1(∑k=1NFkTDTDFk)(∑k=1NFkTDTYk).(6)

In our simulation and experiment, the translation operator Fk was estimated by an image registration process, where the relative shifts between the low-resolution images were estimated by computing the cross-correlation among each other. The blur matrix M was assumed to be a circular disk, and the radius was chosen to optimize the image quality. We used 4×4 pixel enhancement of the enhanced-resolution image to define the decimation operator D.

We first did numerical simulations to demonstrate the resolution enhancement effects in LIHM with our method. The sample we used was a two-layer model, where the top layer was part of the USAF target [element 2–6 of group 8, as shown in Fig. 3(a)], and the bottom layer was a bar [as shown in Fig. 3(b)]. The distance between the two layers was set as 400 μm, and the distance between the top layer and the sensor was 2 mm. The transverse distance of the USAF target elements and the bar was close (<50μm) so that their in-line holograms overlapped. In the simulation, we obtained 30 in-line holograms while moving the top layer within the moving range of 44 μm and keeping the bottom layer static. To simulate the experiments, the original sample image in Figs. 3(a) and 3(b) has a pixel size of 0.22 μm, and the holograms were obtained by scalar diffraction propagation of the sample image to the sensor plane and then down-sampled to the experimental sensor pixel size of 2.2 μm.

Fig. 3. Numerical simulation to demonstrate our method with a two-layer sample. (a) The top layer of the sample; (b) the bottom layer of the sample; (c) one of the in-line holograms of the two-layer sample; (d) the segmented hologram of the USAF target elements obtained from (c); (e) the holographic reconstruction result according to (c); (f) PSR reconstruction using multiple low-resolution images of (e); (g) PSR reconstruction using multiple segmented low-resolution images obtained from (d).

下载图片 查看所有图片

Figure 3(c) shows one of the in-line holograms of the two-layer sample, and Fig. 3(d) is the segmented hologram of the USAF target elements. The direct holographic reconstruction of Fig. 3(c) at the top layer is shown in Fig. 3(e), where the USAF target elements cannot be discerned because of the insufficient pixel size of 2.2 μm determined by the imaging sensor. Figure 3(f) shows the PSR reconstruction result using multiple low-resolution sample images shown in Fig. 3(e) without applying the segmentation technique. As expected, we cannot see resolution enhancement because of the disturbance of the bottom layer. In contrast, Fig. 3(g) shows the PSR reconstruction result using multiple segmented low-resolution sample images obtained from Fig. 3(d). We can clearly see the resolution enhancement when the disturbance of the bottom layer was removed.

We then did the experiment with the USAF target for verification. Figure 4(a) shows the two-layer sample for our experiment, where the top layer was the USAF target, and the bottom layer contained the aggregations of microspheres acting as the disturbance. During the experiment, the top layer was translated by hand randomly, and 30 in-line holograms were acquired. One of the acquired in-line holograms is shown in Fig. 4(b), and Fig. 4(c) shows the conventional holographic reconstruction result according to Fig. 4(b). We can see that the disturbance of the bottom layer blocked part of the USAF target. Figure 4(d) shows the enlarged image of the region indicated in Fig. 4(c). Because of the insufficient pixel size (2.2 μm) in our imaging sensor, we cannot discern the elements of groups 8 and 9 of the USAF target, where the line widths were smaller than 2.2 μm. With multiple low-resolution images similar to Fig. 4(d), we can use the PSR technique directly for reconstruction, and the result is shown in Fig. 4(e), where we can get better resolutions. However, the image is disturbed with unwanted background because of the disturbance caused by the bottom layer. To remove the disturbance, we applied the segmentation technique to segment the region shown in Fig. 4(d) and then did PSR reconstruction. Figure 4(f) shows the reconstruction results, where we can clearly observe that the image quality has been significantly improved, and we can discern the element 4 of group 8 with line width of 1.38 μm.

Fig. 4. Experiment with a two-layer sample, where the top layer is a USAF target, and the bottom layer contains aggregations of microspheres. (a) The schematic of the two-layer sample; (b) one of the in-line holograms of the two-layer sample; (c) the conventional holographic reconstruction result according to (b); (d) the enlarged image of the region indicated in (c); (e) PSR reconstruction using multiple low-resolution images of (d); (f) PSR reconstruction using multiple low-resolution images acquired by segmentation and reconstruction of the enlarged region shown in (d).

下载图片 查看所有图片

We next did experiments with the sample of water solution containing the living algae Euglena gracilis. During the experiment, the living algae moved randomly inside the water solution, and we acquired 30 in-line holograms in our LIHM system. To avoid rotation, the algae were paralyzed at low temperatures before the experiment. One of the in-line holograms is shown in Fig. 5(a), and Fig. 5(b) shows the conventional holographic reconstruction of Fig. 5(a), where the reconstruction distance is tuned to focus on the target alga, as indicated in the figures. Figure 5(c) shows the segmentation of the in-line hologram of the target alga. The enlarged image of the target alga indicated in Fig. 5(b) is shown in Fig. 5(d), which has low resolution because of the insufficient sensor pixel size. Figure 5(e) is the PSR reconstruction of the target alga without using the segmentation technique, which shows distorted images caused by disturbance of the adjacent object around the target alga. In contrast, Fig. 5(f) shows the PSR reconstruction of the target alga after applying the segmentation technique, i.e., based on multiple segmented in-line holograms similar to Fig. 5(c). We can clearly see the resolution improvements. For comparison, a microscope image of similar alga observed under a 20× objective is shown in Fig. 5(g), which is comparable to our reconstructed images in Fig. 5(f).

Fig. 5. Experiments with the sample of water solution containing the algae Euglena gracilis. (a) One of the in-line holograms; (b) the conventional holographic reconstruction according to (a), where the target alga and some adjacent objects can be observed; (c) the segmented in-line hologram of the target alga as indicated in (a); (d) the enlarged image of the target alga as indicated in (b); (e) PSR reconstruction of the target alga without using the segmentation technique; (f) PSR reconstruction of the target alga with applying the segmentation technique; (g) microscope image of similar alga observed under a 20× objective.

下载图片 查看所有图片

Finally, to further demonstrate the capability of our method, Fig. 6 shows an atlas of the images of Euglena gracilis inside the water solution obtained by PSR reconstruction after applying the segmentation technique. Notice that the distances between these algae and the imaging sensor were different in the water solution. We can see that enhanced-resolution microscopic imaging can be achieved with our LIHM system.

Fig. 6. Atlas of algae images obtained by PSR reconstruction after the segmentation technique in our LIHM system.

下载图片 查看所有图片

Compared with previous LIHM systems using the PSR technique, the major advantage of our work is the capability to apply the PSR algorithm even when the objects in different layers move in different speeds or directions and result in overlapping and occlusion of the holograms. Furthermore, twin-image background can be partially suppressed with the hologram segmentation technique. Besides the advantages, we should also notice that our method has the following limitations: (1) the method does not work for a pure phase object, which cannot be holographically reconstructed and segmented; (2) for sparse samples where the object’s holograms overlapping is not an issue, the simple PSR method without hologram segmentation works better; (3) the method is not suitable for truly three-dimensional distributed samples because holographic reconstruction does not have satisfactory sectioning capability.

In summary, we propose an enhanced-resolution LIHM method based on the PSR reconstruction and in-line hologram segmentation technique. Our method is suitable for imaging specific target objects in multilayer samples where the overlapping holograms of different objects in the sample may disturb the PSR reconstruction. In our method, the samples are either moved by hand randomly or self-driven for living biological samples. The simple actuation mechanism can reduce the system hardware requirement and thus the system complexity and cost. We believe our method is promising for many biological applications, where low-cost compact in-line holographic microscope solutions are preferred, and enhanced-resolution microscopic images are necessary in the situation of overlapping holograms disturbance.