首页 > 论文 > 光学学报 > 40卷 > 1期(pp:0111001--1)

计算光场成像 (特邀综述)

Computational Light Field Imaging (Invited)

  • 摘要
  • 论文信息
  • 参考文献
  • 被引情况
  • PDF全文
分享:

摘要

光场为三维世界中光线集合的完备表示。通过记录更高维度的光线数据,光场能够准确感知周围复杂多变的动态环境,支撑智能系统对环境的理解与决策。计算光场成像技术围绕光场及全光函数表示,旨在结合计算、数字传感器、光学系统和智能光照等技术,以及硬件设计、软件计算能力,突破经典成像模型和数字相机的局限性,建立光在空域、视角、光谱和时域等多个维度的关系,实现耦合感知、解耦重建与智能处理,具备面向大范围动态场景的多维多尺度成像能力。光场成像技术正逐渐被应用于生命科学、工业探测、国家安全、无人系统和虚拟现实/增强现实等领域,具有重要的学术研究价值和广阔的产业应用前景。然而,伴随着高维数据的离散化采样,光场成像面临空间分辨率与视角分辨率的维度权衡挑战,如何对稀疏化的采样数据进行光场重建成为计算光场成像及其应用的基础难题。与此同时,受制于光场信号的高维数据感知量,光场处理面临有效数据感知与计算高效性的矛盾。如何用光场这一高维信息采集手段,取代传统二维成像视觉感知方法,并结合智能信息处理技术实现智能化高效感知,是实现光场成像技术产业化应用的巨大挑战。对过去20年来计算光场成像装置与算法的研究进行概述和讨论,内容涵盖光场表示和理论、光场信号采集、空间与视角维度重建等。

Abstract

High performance imaging of large-scale dynamic scenes is substantial to vision intelligence. The light field is a 3D plenoptic function that describes the amount of light flow in every direction through every point in space. By recording the high dimensional light signal, the light field can accurately perceive the complex dynamic environment, supporting the understanding and decision-making of the intelligent system. Computational light field imaging technique, based on the light field and the representation of plenoptic function, aims to combine computation, digital sensors, optical system, and intelligent lighting, thereby combining the hardware design and software computing power. This technique breaks through the limits of classical imaging model and digital camera, establishes the relationship among light in spatial, angular, spectral, and temporal dimensions, realizes coupling perception, decoupling reconstruction, and intelligent processing, and leads to the multi-dimensional and multi-scale imaging ability for large-scale dynamic scenes. Light field imaging technique plays vital role in various fields, including life science, industrial inspection, national security, unmanned system, VR/AR, etc., attracting broad interests from both academia and industry. With the discrete sampling of high dimensional data, light field imaging faces the challenge of dimension trade-off between spatial resolution and angular resolution. How to reconstruct light field for sparse sampled data becomes a fundamental problem in computational light field imaging and its applications. Meanwhile, limited by high dimensional data perception of light field signals, light field process faces the contradiction between effective data perception and computational efficiency. How to replace the traditional two-dimensional imaging visual perception method with light field which is a high-dimensional information acquisition means and how to combine intelligent information processing technique to realize intelligent efficient perception, are huge challenges for industrial applications of the light field imaging technique. In this paper, we conduct a thorough literature review of devices and algorithms of computational light field imaging, including the representation and theory of light field, light field signal sampling, and light field reconstruction with super-resolution in spatial and angular domain.

Newport宣传-MKS新实验室计划
补充资料

中图分类号:O438

DOI:10.3788/AOS202040.0111001

所属栏目:“计算光学成像"专题

收稿日期:2019-11-09

修改稿日期:2019-12-26

网络出版日期:2020-01-01

作者单位    点击查看

方璐:清华大学脑与认知科学研究院, 北京 100084
戴琼海:清华大学脑与认知科学研究院, 北京 100084

联系人作者:戴琼海(daiqionghai@tsinghua.edu.cn)

【1】Gershun A. The light field [J]. Journal of Mathematics and Physics. 1939, 18: 51-151.

【2】Moon P, Spencer D E. The photic field[M]. USA: , 1981.

【3】Adelson H, Bergen J. The plenoptic function and the elements of early vision[M]. USA: , 1991, 3-20.

【4】Levoy M, Hanrahan P. Light field rendering . [C]∥Proceedings of the 23rd annual conference on Computer graphics and interactive techniques-SIGGRAPH''''96, August 4-9, 1996, New Orleans, LA, USA. New York: ACM. 1996, 31-42.

【5】Raytrix, [2019-11-08]. 3D light-field vision [2019-11-08]. https:∥raytrix.de/. 2017.

【6】NGP capital. Pelican imaging[2019-11-08]. https:∥www.ngpcap.cn/companies/pelican-imaging.[2019-11-08]. 0.

【7】Levoy M, Ng R, Adams A, et al. Light field microscopy [J]. ACM Transactions on Graphics. 2006, 25(3): 924-934.

【8】Levoy M. Light fields and computational imaging [J]. Computer. 2006, 39(8): 46-55.

【9】Ihrke I, Restrepo J, Mignard-Debise L. Principles of light field imaging: briefly revisiting 25 years of research [J]. IEEE Signal Processing Magazine. 2016, 33(5): 59-69.

【10】Wetzstein G, Ihrke I, Lanman D, et al. Computational plenoptic imaging [J]. Computer Graphics Forum. 2011, 30(8): 2397-2426.

【11】Chen S E. Quicktime VR: an image-based approach to virtual environment navigation . [C]∥Proceedings of the 22nd annual conference on Computer graphics and interactive techniques, August 6-11, 1995, Los Angeles, CA, USA. New York: ACM. 1995, 29-38.

【12】McMillan L, Bishop G. Plenoptic modeling: an image-based rendering system . [C]∥Proceedings of the 22nd annual conference on Computer graphics and interactive techniques, August 6-11, 1995, Los Angeles, CA, USA. New York: ACM. 1995, 39-46.

【13】Chai J X, Chan S C, Shum H Y, et al. Plenoptic sampling . [C]∥Proceedings of the 27th annual conference on Computer graphics and interactive techniques-SIGGRAPH''''00, July 23-28, 2000, New Orleans, LA, USA. New York: ACM. 2000, 307-318.

【14】Lin Z C, Shum H Y. A geometric analysis of light field rendering [J]. International Journal of Computer Vision. 2004, 58(2): 121-138.

【15】Buehler C, Bosse M. McMillan L, et al. Unstructured lumigraph rendering . [C]∥Proceedings of the 28th annual conference on Computer graphics and interactive techniques-SIGGRAPH''''01, August 12-17, 2001, Los Angeles, California, USA. New York: ACM. 2001, 425-432.

【16】Ihrke I, Wetzstein G, Heidrich W. A theory of plenoptic multiplexing . [C]∥2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, June 13-18, 2010, San Francisco, CA, USA. New York: IEEE. 2010, 483-490.

【17】Wilburn B, Joshi N, Vaish V, et al. High performance imaging using large camera arrays [J]. ACM Transactions on Graphics. 2005, 24(3): 765-776.

【18】Yang J C, Everett M, Buehler C, et al. A real-time distributed light field camera . [C]∥Proceedings of the 13th Eurographics Workshop on Rendering Techniques, June 26-28, 2002, Pisa, Italy. Frontenex: Eurographics Association. 2002, 77-86.

【19】Venkataraman K, Lelescu D, Duparré J, et al. PiCam: an ultra-thin high performance monolithic camera array [J]. ACM Transactions on Graphics (TOG). 2013, 32(6): 166.

【20】Wilburn B S, Smulski M. Lee H H K, et al. Light field video camera [J]. Proceedings of SPIE. 2001, 4674: 29-36.

【21】Wilburn B, Joshi N, Vaish V, et al. High-speed videography using a dense camera array . [C]∥Proceedings of the 2004 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, 2004. CVPR 2004. June 27-July 2, 2004, Washington, DC, USA. New York: IEEE. 2004, 8168976.

【22】Zhang C, Chen T. A self-reconfigurable camera array . [C]∥Proceedings of the 15th Eurographics Workshop on Rendering Techniques, June 21-23, 2004, Nork?ping, Sweden. Frontenex: Eurographics Association. 2004, 243-254.

【23】Chan S C, Ng K T, Gan Z F, et al. The plenoptic video [J]. IEEE Transactions on Circuits and Systems for Video Technology. 2005, 15(12): 1650-1659.

【24】Liu Y B, Dai Q H, Xu W L. A real time interactive dynamic light field transmission system . [C]∥2006 IEEE International Conference on Multimedia and Expo, July 9-12, 2006, Toronto, ON, Canada. New York: IEEE. 2006, 2173-2176.

【25】ViewPLUS Inc. ProFUSION 25 5×5 camera array system[2019-11-08]. http:∥www.viewplus.co.jp/products/profusion25/index-%20e.html,%202009.[2019-11-08]. 0.

【26】The (New) Stanford light field archive [2019-11-08].http:∥lightfield.stanford.edu/lfs.html.[2019-11-08]. 0.

【27】Unger J, Wenger A, Hawkins T, et al. Capturing and rendering with incident light fields . [C]∥Proceedings of the 14th Eurographics workshop on Rendering Techniques, June 25-27, 2003, Leuven, Belgium. Frontenex: Eurographics Association. 2003, 141-149.

【28】Kim C, Zimmer H, Pritch Y, et al. Scene reconstruction from high spatio-angular resolution light fields [J]. ACM Transactions on Graphics. 2013, 32(4): 73.

【29】Dansereau D G, Schuster G, Ford J, et al. A wide-field-of-view monocentric light field camera . [C]∥2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), July 21-26, 2017, Honolulu, HI, USA. New York: IEEE. 2017, 3757-3766.

【30】Liang C K, Lin T H, Wong B Y, et al. Programmable aperture photography: multiplexed light field acquisition [J]. ACM Transactions on Graphics. 2008, 27(3): 55.

【31】Ihrke I, Stich T, Gottschlich H, et al. Fast incident light field acquisition and rendering [J]. Journal of WSCG. 2008, 16(1/2/3): 25-32.

【32】Taguchi Y, Agrawal A, Ramalingam S, et al. Axial light field for curved mirrors: reflect your perspective, widen your view . [C]∥2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, June 13-18, 2010, San Francisco, CA, USA. New York: IEEE. 2010, 499-506.

【33】Lippmann G. La photographie integrale [J]. Academie des Sciences. 1908, 146: 446-451.

【34】Adelson E H. Wang J Y A. Single lens stereo with a plenoptic camera [J]. IEEE Transactions on Pattern Analysis and Machine Intelligence. 1992, 14(2): 99-106.

【35】Ng R, Levoy M, Brédif M, et al. Light field photography with a hand-held plenoptic camera[R] . USA: Computer Science Technical Report CSTR. 2005.

【36】LightField Forum. Lytro[2019-11-08]. http:∥lightfield-forum.com/lytro/.[2019-11-08]. 0.

【37】Wei L Y, Liang C K, Myhre G, et al. Improving light field camera sample design with irregularity and aberration [J]. ACM Transactions on Graphics. 2015, 34(4): 152.

【38】Georgiev T, Lumsdaine A. Superresolution with plenoptic camera 2.0[R] . USA: Adobe Systems Incorporated. 2009.

【39】Horstmeyer R, Euliss G, Athale R. Flexible multimodal camera using a light field architecture . [C]∥2009 IEEE International Conference on Computational Photography (ICCP), April 16-17, 2009, San Francisco, CA, USA. New York: IEEE. 2009, 11499063.

【40】Levin A, Fergus R, Durand F, et al. Image and depth from a conventional camera with a coded aperture [J]. ACM Transactions on Graphics. 2007, 26(3): 70.

【41】Manakov A, Restrepo J F, Klehm O, et al. A reconfigurable camera add-on for high dynamic range, multispectral, polarization, and light-field imaging [J]. ACM Transactions on Graphics. 2013, 32(4): 47.

【42】Lanman D, Crispell D, Wachs M, et al. Spherical catadioptric arrays: construction, multi-view geometry, and calibration . [C]∥Third International Symposium on 3D Data Processing, Visualization, and Transmission (3DPVT''''06), June 14-16, 2006, Chapel Hill, NC, USA. New York: IEEE. 2006, 10285053.

【43】Levoy M, Zhang Z. McDowall I. Recording and controlling the 4D light field in a microscope using microlens arrays [J]. Journal of Microscopy. 2009, 235(2): 144-162.

【44】Prevedel R, Yoon Y G, Hoffmann M, et al. Simultaneous whole-animal 3D imaging of neuronal activity using light-field microscopy [J]. Nature Methods. 2014, 11(7): 727-730.

【45】N?bauer T, Skocek O. Pernía-Andrade A J, et al. Video rate volumetric Ca 2+ imaging across cortex using seeded iterative demixing (SID) microscopy [J]. Nature Methods. 2017, 14(8): 811-818.

【46】Lin X, Wu J M, Zheng G A, et al. Camera array based light field microscopy [J]. Biomedical Optics Express. 2015, 6(9): 3179-3189.

【47】Mignard-Debise L, Ihrke I. Light-field microscopy with a consumer light-field camera . [C]∥2015 International Conference on 3D Vision, October 19-22, 2015, Lyon, France. New York: IEEE. 2015, 335-343.

【48】Georgiev T G, Zheng K C, Curless B, et al. Spatio-angular resolution tradeoffs in integral photography . [C]∥Proceedings of the Eurographics Symposium on Rendering Techniques, Nicosia, Cyprus. Frontenex: Eurographics Association. 2006, 263-272.

【49】Veeraraghavan A, Raskar R, Agrawal A, et al. Dappled photography: mask enhanced cameras for heterodyned light fields and coded aperture refocusing . [C]∥SIGGRAPH''''07 ACM SIGGRAPH 2007 papers, August 5-9, 2007, San Diego, California. New York: ACM. 2007, 69.

【50】Lanman D, Raskar R, Agrawal A, et al. Shield fields: modeling and capturing 3D occluders [J]. ACM Transactions on Graphics. 2008, 27(5): 131.

【51】Amit A, Mark A. Compressive light field imaging [J]. Proceedings of SPIE. 2010, 7690: 76900Q.

【52】Antipa N, Necula S, Ng R, et al. Single-shot diffuser-encoded light field imaging . [C]∥2016 IEEE International Conference on Computational Photography (ICCP), May 13-15, 2016, Evanston, IL, USA. New York: IEEE. 2016.

【53】Pégard N C, Liu H Y, Antipa N, et al. Compressive light-field microscopy for 3D neural activity recording [J]. Optica. 2016, 3(5): 517-524.

【54】Cohen N, Yang S, Andalman A, et al. Enhancing the performance of the light field microscope using wavefront coding [J]. Optics Express. 2014, 22(20): 24817-24839.

【55】Marwah K, Wetzstein G, Bando Y, et al. Compressive light field photography using overcomplete dictionaries and optimized projections [J]. ACM Transactions on Graphics. 2013, 32(4): 46.

【56】Wang Y P, Wang L C, Kong D H, et al. High-resolution light field capture with coded aperture [J]. IEEE Transactions on Image Processing. 2015, 24(12): 5609-5618.

【57】Kopf J, Uyttendaele M, Deussen O, et al. Capturing and viewing gigapixel images [J]. ACM Transactions on Graphics. 2007, 26(3): 93.

【58】Gigapan [2019-11-08].http:∥www.gigapan.com/.[2019-11-08]. 0.

【59】Pirk S, Cohen M F, Deussen O, et al. Video enhanced gigapixel panoramas . [C]∥SIGGRAPH Asia 2012 Technical Briefs on-SA''''12, November 28-December 1, 2012, Singapore, Singapore. New York: ACM. 2012, 7.

【60】Cossairt O S, Miau D, Nayar S K. Gigapixel computational imaging . [C]∥2011 IEEE International Conference on Computational Photography (ICCP), April 8-10, 2011, Pittsburgh, PA, USA. New York: IEEE. 2011, 11962821.

【61】Cossairt O S, Miau D, Nayar S K. Scaling law for computational imaging using spherical optics [J]. Journal of the Optical Society of America A. 2011, 28(12): 2540-2553.

【62】Brady D J, Gehm M E, Stack R A, et al. Multiscale gigapixel photography [J]. Nature. 2012, 486(7403): 386-389.

【63】Kittle D S, Daniel L, David J. Automated calibration and optical testing of the AWARE-2 gigapixel multiscale camera [J]. Proceedings of SPIE. 2013, 8660: 866006.

【64】Kittle D S, Marks D L, Son H S, et al. A testbed for wide-field, high-resolution, gigapixel-class cameras [J]. Review of Scientific Instruments. 2013, 84(5): 053107.

【65】Pang W, Brady D J. Parallel MMS gigapixel imagers . [C]∥Computational Optical Sensing and Imaging, June 26-29, 2017, San Francisco, California, United States. Washington, D. C.: OSA. 2017, CM2B: 3.

【66】Schuster G M, Agurok I P, Ford J E, et al. Panoramic monocentric light field camera . [C]∥International Optical Design Conference, July 9-13, 2017, Denver, Colorado, United States. Washington, D. C.: OSA. 2017, ITh4A: 5.

【67】Wang Y W, Liu Y B, Heidrich W, et al. The light field attachment: turning a DSLR into a light field camera using a low budget camera ring [J]. IEEE Transactions on Visualization and Computer Graphics. 2017, 23(10): 2357-2364.

【68】Yuan X Y, Fang L, Dai Q H, et al. Multiscale gigapixel video: a cross resolution image matching and warping approach . [C]∥2017 IEEE International Conference on Computational Photography (ICCP), May 12-14, 2017, Stanford, CA, USA. New York: IEEE. 2017, 16969274.

【69】Wanner S, Goldluecke B. Variational light field analysis for disparity estimation and super-resolution [J]. IEEE Transactions on Pattern Analysis and Machine Intelligence. 2014, 36(3): 606-619.

【70】Zhang Z, Liu Y B, Dai Q H. Light field from micro-baseline image pair . [C]∥2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 7-12, 2015, Boston, MA, USA. New York: IEEE. 2015, 3800-3809.

【71】Pearson J, Brookes M, Dragotti P L. Plenoptic layer-based modeling for image based rendering [J]. IEEE Transactions on Image Processing. 2013, 22(9): 3405-3419.

【72】Penner E, Zhang L. Soft 3D reconstruction for view synthesis [J]. ACM Transactions on Graphics. 2017, 36(6): 235.

【73】Chaurasia G, Duchene S, Sorkine-Hornung O, et al. Depth synthesis and local warps for plausible image-based navigation [J]. ACM Transactions on Graphics. 2013, 32(3): 30.

【74】Kalantari N K, Wang T C, Ramamoorthi R. Learning-based view synthesis for light field cameras [J]. ACM Transactions on Graphics. 2016, 35(6): 193.

【75】Zhou T, Tucker R, Flynn J, et al. -05-24)[2019-11-08] . https:∥arxiv.xilesou. 2018, top/abs/1805: 09817.

【76】Mildenhall B, Srinivasan P P, Ortiz-Cayon R, et al. -05-02)[2019-11-08] . https:∥arxiv.xilesou. 2019, top/abs/1905: 00889.

【77】Chai J X, Ton X, Chan S C, et al. Plenoptic sampling . [C]∥Proceedings of the 27th annual conference on Computer graphics and interactive techniques-SIGGRAPH''''00, July 23-28, 2000, New Orleans, LA, USA. New York: ACM. 2000, 307-318.

【78】Shi L X, Hassanieh H, Davis A, et al. Light field reconstruction using sparsity in the continuous Fourier domain [J]. ACM Transactions on Graphics. 2014, 34(1): 12.

【79】Vagharshakyan S, Bregovic R, Gotchev A. Light field reconstruction using shearlet transform [J]. IEEE Transactions on Pattern Analysis and Machine Intelligence. 2018, 40(1): 133-147.

【80】Vagharshakyan S, Bregovic R, Gotchev A. Accelerated shearlet-domain light field reconstruction [J]. IEEE Journal of Selected Topics in Signal Processing. 2017, 11(7): 1082-1091.

【81】Yoon Y, Jeon H G, Yoo D, et al. Learning a deep convolutional network for light-field image super-resolution . [C]∥2015 IEEE International Conference on Computer Vision Workshop (ICCVW), December 7-13, 2015, Santiago, Chile. New York: IEEE. 2015, 57-65.

【82】Yeung H W F, Hou J H, Chen J, et al. Fast light field reconstruction with deep coarse-to-fine modeling of spatial-angular clues [M]. ∥Ferrari V, Hebert M, Sminchisescu C, et al. Computer vision-ECCV 2018. Lecture notes in computer science. Cham: Springer. 2018, 11210: 138-154.

【83】Wu G C, Liu Y B, Fang L, et al. Light field reconstruction using convolutional network on EPI and extended applications [J]. IEEE Transactions on Pattern Analysis and Machine Intelligence. 2019, 41(7): 1681-1694.

【84】Dai Q H, Suo J L, Ji X Y, et al. Computational photography: computational capture of plenoptic visual information[M]. Beijing: Tsinghua University Press, 2016, 11.
戴琼海, 索津莉, 季向阳, 等. 计算摄像学: 全光视觉信息的计算采集[M]. 北京: 清华大学出版社, 2016, 11.

【85】Zhu K, Xue Y J, Fu Q, et al. Hyperspectral light field stereo matching [J]. IEEE Transactions on Pattern Analysis and Machine Intelligence. 2019, 41(5): 1131-1143.

【86】Holloway J, Mitra K, Koppal S J, et al. Generalized assorted camera arrays: robust cross-channel registration and applications [J]. IEEE Transactions on Image Processing. 2015, 24(3): 823-835.

【87】Xiong Z W, Wang L Z, Li H Q, et al. Snapshot hyperspectral light field imaging . [C]∥2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), July 21-26, 2017, Honolulu, HI, USA. New York: IEEE. 2017, 6873-6881.

【88】Lin X, Rivenson Y, Yardimci N T, et al. All-optical machine learning using diffractive deep neural networks [J]. Science. 2018, 361(6406): 1004-1008.

【89】Yan T, Wu J M, Zhou T K, et al. Fourier-space diffractive deep neural network [J]. Physical Review Letters. 2019, 123(2): 023901.

引用该论文

Fang Lu,Dai Qionghai. Computational Light Field Imaging[J]. Acta Optica Sinica, 2020, 40(1): 0111001

方璐,戴琼海. 计算光场成像[J]. 光学学报, 2020, 40(1): 0111001

您的浏览器不支持PDF插件,请使用最新的(Chrome/Fire Fox等)浏览器.或者您还可以点击此处下载该论文PDF