Optic flaws detection and location based on a plenoptic camera

In this Letter, we propose an on-line inspection method based on a plenoptic camera to detect and locate flaws of optics. Specifically, due to the extended depth of field of the plenoptic camera, a series of optics can be inspected efficiently and simultaneously. Moreover, the depth estimation capability of the plenoptic camera allows for locating flaws while detecting them. Besides, the detection and location can be implemented with a single snapshot of the plenoptic camera. Consequently, this method provides us with the opportunity to reduce the cost of time and labor of inspection and remove the flaw optics, which may lead to performance degradation of optical systems.

In optical systems, different kinds of flaws (such as scratch, pitting, laser-induced damage, etc.) will result in different degrees of scattering, energy absorption, and diffraction patterns. Furthermore, the surface flaws in the optics can also lead to performance degradation of the optical system caused by the diffraction of flaws sites, and even cause damage to optical equipment. Therefore, the optics flaw inspection plays an important role in the research of optics physics and technology, which provides the ability to inspect the initiation and growth of flaws and remove the flaw optics in time^{[13" target="_self" style="display: inline;">–3]}.

The surface flaws in optics have been intensively studied in the last several decades^{[1–5" target="_self" style="display: inline;">–5]}. In 1997, the first off-line optics flaws inspection system was designed by Rainer in Lawrence Livermore National Laboratory^[3]. In the off-line inspection system, Rainer used fiber-optics in a dark background to illuminate the flaws, and built a long working distance microscope to scan the optics. Consequently, the flaws and artifacts in the bulk and on the surface can be inspected and mapped by the image processing method. However, the optics need to be disassembled from the optical system first and mounted in a customized frame before inspection^[6].

Compared to the off-line inspection system, the on-line optics flaw inspection system is required more, because the on-line inspection is free of the heavy workload brought by optics disassembly and re-alignment. Moreover, some on-line inspection methods can provide the ability to inspect the initiation and growth of flaws in the working optical system^[4]. Therefore, in 1998, Thompson et al. designed the first on-line flaw inspection system to measure surface flaws and artifacts for optics^[1]. In this inspection system, to isolate the image planes from components in various parts of the beamline, a spatial filter pinhole is inserted in sequence in the cavity and transports spatial filters. Meanwhile, the CCD $z$ -axis position is adjusted correspondingly to image the optics under inspection. Due to the mechanical adjustment and alignment caused by inserting pinholes and adjusting the CCD, this on-line flaw inspection method is complicated and time consuming^[1].

In 2010, Conder et al. dilated the on-line inspection method of optics flaws, which includes the flaw image acquirement, the optic illumination technique, and the flaw image analysis algorithm^[4]. In this method, a high resolution CCD camera moves in sequence by a linear translation stage to focus on each optic, and all flaw images of optics are collected, respectively. As a result, the image data will be extremely large, since the optical system utilizes a lot of optics and every optic will generate a high resolution image. Furthermore, the large amount of image data will consume considerable time in image processing and flaw characterization. Besides, when detecting flaws by dark-field imaging, it is difficult to discriminate in which optics the flaws occur when two or more optics are in close proximity^[4].

To solve the aforementioned problem, in this Letter, the plenoptic camera is introduced in the optic flaws inspection. The plenoptic camera is a multi-imaging camera that was rapidly developed in recent years^{[7–10" target="_self" style="display: inline;">–10]}. In a single snapshot, the plenoptic camera is able to obtain a large depth of field and estimate the depth map of the scene by making use of the redundancy created by the multi-view geometry^[9,11,12]. Accordingly, when using the plenoptic camera to inspect the optics flaws, a series of optics can be inspected simultaneously without any manual intervention, and the position of the damaged optics can be easily located.

To describe the principles of the optic flaws inspection method based on a plenoptic camera, first, the depth estimation principle of plenoptic camera is explained. As shown in Fig. 1, the typical plenoptic camera consists of three parts: a main lens, a microlens array, and an image sensor^[7,8]. Compared to standard cameras, a microlens array is applied in front of the image sensor, hence, the plenoptic camera can record the direction of rays while acquiring their intensity^[7]. Specifically, in the imaging of the plenoptic camera, the main lens shrinks the object space, and the microlens can be regarded as a relay lens to form the image of the original scene for the second time^[8]. Generally, when the image points are located at the proper position in the object space of the microlens, they can be imaged by several microlenses, which will offer parallax for calculating the depth of the image point via triangulation^[8].

Fig. 1. Schematic diagram of a plenoptic camera.

下载图片查看所有图片

Assuming in the absence of noise, the depth estimation only requires the calculation of the intersection of two rays in space, which is a typical stereo camera approach^[13]. Concretely, Fig. 2 is the magnified diagram of the left part of Fig. 1, which shows the details of microlens imaging. To simplify, only the points located on the line parallel to the optical axes and lying halfway between two microlenses are considered. An image point at distance $x$ from the micro image center intersects the central bisecting line at point $V (x)$ . The relation between $V (x)$ and $x$ is given by $\frac{V (x)}{α \cdot d / 2} = \frac{Q}{x} \Leftrightarrow V (x) = \frac{Q \cdot α \cdot d / 2}{x},$ (1)where $α$ is the distance ratio between the distance of the choosing microlens and the microlens pitch^[14]. Equally, for the image point at distance $x - Δ x$ , the distance $V (x - Δ x)$ can be calculated. If $Δ x$ is the size of a pixel, the distance between $V (x)$ and $V (x - Δ x)$ is regarded as the minimum resolvable distance (depth resolution) in the object space of the microlens. In addition, at present, the image features can be located and matched with sub-pixel accuracy, then, the minimum resolvable distance can be shortened accordingly^[8].

Fig. 2. Depth estimating via triangulation.

下载图片查看所有图片

So when using the plenoptic camera to inspect the optics in optical systems, the depth of the damaged optics can be estimated by this camera. Consequently, it can discriminate in which optics the flaws occur even when two or more optics are in close proximity.

The other advantage of the plenoptic camera is that it has an extended depth of field compared to a standard camera. For standard camera imaging, as shown in Fig. 3, using the similar triangle method, the relation between $s$ and $b_{0}$ can derived from Fig. 3 as $\frac{s}{D} = \frac{Q_{0} - b_{0}}{b_{0}},$ (2)thus, $s = D \cdot (Q_{0} / b_{0} - 1)$ ^[8]. In general, assuming the diameter of the Airy disk is smaller than the pixel size $p$ , i.e., $| s | \leq p$ . Besides, $p ≪ d$ , so the depth of field in the image space of lens $δ_{S}$ can be approximated by $δ_{S} \approx 2 p N_{F},$ (3)where $N_{F} = Q_{0} / D$ is the image $f$ number of the lens^[8].

Fig. 3. Imaging of a lens.

下载图片查看所有图片

For a plenoptic camera, as shown in Fig. 1, the microlens is focused on the image shrunk by the main lens, hence, the depth of field in the image space of the main lens is determined by the depth of field in the object space of the microlens^[15]. Therefore, we can use the depth of field in the object space of the microlens to represent the depth of field in the image space of the main lens; since $p ≪ d$ , then, the depth of field in the object space of microlens $δ_{P}$ can be approximated by $δ_{P} \approx \frac{2 p N_{f}}{{(\frac{Q}{f} - 1)}^{2}}, Q \neq f,$ (4)where $N_{f} = Q / d$ is the image $f$ number of the microlens, $Q$ is the distance between the microlens array and the image sensor, and $f$ is the focus length of the microlens.

In practice, to make the best use of the image sensor, the image $f$ number of the microlens and the main lens should match^[8], i.e., $N_{F} = N_{f}$ . So, the depth of field ratio $γ$ between the plenoptic camera and the standard camera can be derived by Eqs. (3) and (4) as $γ = \frac{δ_{P}}{δ_{s}} = \frac{1}{{(\frac{Q}{f} - 1)}^{2}}, Q \neq f .$ (5)

Note that $γ$ is the image space ratio. By using Eq. (5), we can find that the depth of field of the plenoptic camera can be extended with appropriate parameters. In general, the depth of field of the plenoptic camera can be extended several times compared to the standard camera. With the extended depth of field, more optics can be inspected simultaneously when using a plenoptic camera to detect optic flaws.

In addition, as shown in Fig. 4, assuming the pixel size $p$ is the limit of the spatial resolution of the microlens, the minimum resolvable object size $c$ is $\frac{c}{p} = \frac{a}{Q} \Leftrightarrow c = \frac{a \cdot p}{Q} .$ (6)

Fig. 4. Spatial resolution of a microlens.

下载图片查看所有图片

Then, the minimum resolvable size in the image space of the main lens is $c$ , and the spatial resolution of the object space of the main lens depends on the focal length and the object distance.

The optical arrangement of an optic flaws inspection system based on a plenoptic camera is shown in Fig. 5. To inspect optics in an optical system, the plenoptic camera can be placed in front of a splitter, and the real path of the optical axis is shown by the solid line in Fig. 5. The splitter can reflect the inspection light (green light) from the flaws to the plenoptic camera while transmitting the working light (red light). The images $A^{'}$ , $B^{'}$ , and $C^{'}$ are the virtual image of mirrors A, B, and C in the image space of the splitter, since each mirror can be imaged by the next mirror in the optical path. Hence, in the demonstration, mirrors A, B, and C can be inspected on-line, and the flaws of the mirrors can be detected and located in real time.

Fig. 5. Demonstration of on-line inspection based on a plenoptic camera.

下载图片查看所有图片

Finally, by this method, mirrors A, B, and C are inspected and located simultaneously without any mechanical adjustment and alignment. Consequently, it inspects the optics with less cost of time and labor. With this improvement, the optics inspection is able to be a routine part of optical system operations.

To verify the proposed method, we designed and built a plenoptic camera and did a principal experiment. The experimental prototype of the plenoptic camera is built with a full format CCD, a 160 mm main lens, and a microlens array in which the microlenses are arranged as a hexagonal grid. The plenoptic camera is assembled by the Keplerian mode without removing the cover glass of the image sensor^[16], and the picture of the plenoptic camera is shown in Fig. 6. Specifically, the microlens pitch is $d = 0.3 mm$ , the microlens focal length is $f = 2.7 mm$ , the pixel size is $p = 9 μm$ , and the distance between the microlens array and image sensor is $Q = 3.4 mm$ . So, in this plenoptic camera, the depth of field range boundaries in the object space of the microlens are $a^{-} = 11.75 mm$ , and $a^{+} = 14.83 mm$ .

Fig. 6. Experimental prototype of a plenoptic camera.

下载图片查看所有图片

To simulate an optical system, three circular mirrors are arranged in the principal experiment, and they are placed as the dotted line square shown in Fig. 5. In this principal experiment, the distance between the optimal focus plane and the CCD is set as 1300 mm. Then, the depth of field range boundaries in the object space of the main lens can be calculated as $A_{0}^{+} = 1236.4 mm$ , and $A_{0}^{-} = 1100.7 mm$ , according to the depth of field range boundaries in object space of microlens. Consequently, the depth of field of the plenoptic camera is 135.8 mm, while the depth of field of the standard camera is 29.7 mm with the same main lens, the same CCD, and the same effective pixels. To verify the depth of field of this plenoptic camera, mirrors A and C are placed at $A_{0}^{-}$ and $A_{0}^{+}$ , respectively, in this principal experiment. Besides, to prove the depth resolution, mirrors B and C are placed as close as possible, and the distance between them is about 30 mm. Furthermore, the depth resolution in the object space is 8.92 mm with the camera parameters described above and the image feature location accuracy of 1/10 pixel.

In addition, illumination is critical for the successful performance of the inspection system^[4]. So in this experiment, ring lights are integrated on the mirrors for edge illumination.

For comparison, the three mirrors are inspected on the off-line inspection system. In Fig. 7, the images of the mirrors are shown, and the flaws can be imaged clearly with the short object distance.

Fig. 7. (a) Mirror A, (b) mirror B, and (c) mirror C.

下载图片查看所有图片

When the plenoptic camera is built, the first step is to calibrate it by using the automatic calibration method proposed in Ref. [17]. Then, the plenoptic camera is installed in the flaws inspection system, as shown in Fig. 5. The raw image without any image enhancement is shown in the left part of Fig. 8, and the red box in the right part is the magnified image of the micro image in the corresponding yellow box. In Fig. 8, the flaws on mirrors A and C are imaged clearly compared to mirror B, so it demonstrates that all mirrors are in the depth of field of the plenoptic camera. Moreover, these flaws of different mirrors are recorded in one image with a single snapshot. Therefore, compared to traditional methods, the image data will be small, and the image processing time will be short. Furthermore, in practice, when the optical system is working, while inspecting the growth of flaws, this method provides the ability to obtain the flaw initiation sequence by inspecting the mirrors simultaneously. This will provide more information about flaw initiation and growth; subsequently, it may be helpful to find a method that avoids the generation of flaws.

Fig. 8. Left part: raw image, right part: magnified micro image of flaws on mirrors A, B, and C.

下载图片查看所有图片

On the other hand, from the raw image, we can see that the flaws are imaged several times in different micro images. Since every bright pixel may be a flaw in the optical system, all of them are set as key points, and they are described and matched by the method in Ref. [18]. Subsequently, the depth of the key points is calculated by the triangulation summarized by Ref. [19]. Moreover, the depth map of these flaws is shown in Fig. 9, and the color of the flaw describes its depth (the warm color shows the near mirror, and the cool color shows the far mirror). The flaws on mirrors B and C are different colors, which verify the depth resolution of the plenoptic camera. In practice, the flaws on mirror A are red, the flaws on mirror B are green, and the flaws on mirror C are blue. Therefore, the depth estimation result coincides with the real position of the mirrors, while the outline of the flaws on the depth map is similar to images acquired by the off-line inspection system, and the shape difference is caused by present feature detectors and matchers. With the development of the computer vision, the difference will be smaller.

Fig. 9. Depth map of flaws.

下载图片查看所有图片

To verify the spatial resolution, we measured the flaw on mirror A, and its length is about 26.8 mm. Furthermore, the pixel length of the image of the flaw on mirror A is about 128 pixels, and the pixel size is $p = 9 μm$ , so the calculated length is 26.5 mm, according to Eq. (6) and the parameter of the main lens, which is close to its real length. Therefore, for the object at the position of mirror A, when the image length is within one pixel, its spatial resolution is about 0.21 mm without considering the influence of light intensity.

Compared to the traditional optic flaws inspection method, the proposed method changes the way of inspection. Consequently, the complexity of the inspection system is reduced, and the intelligence level is improved, which will make optics inspection a routine part of optical system operations.

In this Letter, to detect and locate flaws, the plenoptic camera is applied in the on-line optics inspection system. By the properties of the plenoptic camera, i.e., the extended depth of field and the depth estimation, a series of optics can be inspected at the same time. Meanwhile, the optics flaws can be detected and located with a single snapshot. In the principal experiment, the raw image of optics flaws and the depth map of flaws are achieved, which verify the proposed method.

Future studies will focus on characterizing the flaws quantitatively and developing the flaw image analysis system.

References

[1] ThompsonC. E.KnoppC. F.DeckerD. E., Proc. SPIE3492, 921 (1998).

[2] RainerA. F.DicksonR. K.JenningsR. T.KimmonsJ. F.MaricleS. M.MouserR. P.SchwartzS.WeinzapfelC. L., Office of Scientific & Technical Information Technical Reports3492, 556 (1999).

[3] RainerA. F., in Laser-Induced Damage in Optical Materials: 1997 (1997), p. 272.

[4] ConderA.AzevedoS.KegelmeyerL.SpaethM., Proc. SPIE7797, 125 (2010).

[5] ZhangS.BaiJ.WangK.HeF.ZhouB., in Applied Optics and Photonics China (2015).

[6] LiaoZ. M.RaymondB.GaylordJ.FallejoR.BudeJ.WegnerP., Opt. Express22, 28845 (2014).OPEXFF1094-4087

[7] AdelsonE. H.WangJ. Y. A., IEEE Trans. Pattern Anal. Mach. Intell.14, 99 (1992).ITPIDJ0162-8828

[8] PerwassC.WietzkeL., Proc. SPIE8291, 829108 (2012).

[9] DansereauD. G.PizarroO.WilliamsS. B., in IEEE Conference on Computer Vision and Pattern Recognition (CVPR) (2013), p. 1027.

[10] LeeM.-C.InoueK.KimC.-S.ChoA. M., Chin. Opt. Lett.14, 121101 (2016).CJOEE31671-7694

[11] JangJ.-Y.ChoM.KimE.-S., Chin. Opt. Lett.13, 031101 (2015).CJOEE31671-7694

[12] ChoM.ShinD., Chin. Opt. Lett.13, 051101 (2015).CJOEE31671-7694

[13] HartleyR. I.SturmP., Comput. Vision Image Understanding68, 146 (1997).

[14] DrazicV.SacréJ.-J.SchubertA.BertrandJ.BlondéE., J. Electron. Imaging21, 011007 (2012).JEIME51017-9909

[15] OberdörsterA.LenschH. P., in IS&T/SPIE Electronic Imaging (2013), paper 86671R.

[16] GeorgievT.LumsdaineA., in Proceedings of Eurographics (2009), p. 5.

[17] LuanY.HeX.XuB.YangP.TangG., Opt. Eng.55, 043111 (2016).

[18] LoweD. G., Int. J. Comput. Vision60, 91 (2004).IJCVEQ0920-5691

[19] HartleyR.ZissermanA., Multiple View Geometry in Computer Vision (Cambridge University, 2003).

Yinsen Luan, Bing Xu, Ping Yang, Guomao Tang. Optic flaws detection and location based on a plenoptic camera[J]. Chinese Optics Letters, 2017, 15(4): 041102.