首页 > 论文 > 光学学报 > 38卷 > 10期(pp:1010002--1)

基于双通道卷积神经网络的深度图超分辨研究

Depth Map Super-Resolution Based on Two-Channel Convolutional Neural Network

  • 摘要
  • 论文信息
  • 参考文献
  • 被引情况
  • PDF全文
分享:

摘要

目前直接获取的深度图受其成像原理及硬件设备等因素的限制,存在分辨率低、边缘信息丢失等缺点,极大地影响了深度图的应用。针对这一问题,提出基于双通道卷积神经网络的深度图超分辨率重建模型。该模型由深、浅两个通道组成,21层的深层通道通过联合卷积与反卷积,结合跳跃连接与多尺度理论,实现深度图细节特征的快速学习;3层的浅层通道用于学习深度图的轮廓特征;最后融合深、浅两个通道,将细节与轮廓相结合,实现由低分辨率深度图到高分辨率深度图的端到端的学习。该模型充分利用卷积神经网络的学习能力自主提取深度图的有效特征,避免了手工提取特征的不准确性。在Middlebury RGBD数据集上的实验结果表明,本文模型在大采样因子8时仍能取得较好的效果,具有很高的实际应用价值。

Abstract

The depth map obtained directly is limited by the disadvantages such as low resolution and missing edge information, it greatly affects the application of depth map. In order to solve this problem, a two-channel convolutional neural network for depth map super-resolution is proposed. It consists of two channels, deep and shallow, and there are 21 layers in the deep network. Through joint convolution and deconvolution, combining skip connection and multi-scale theory, the deep channels can quickly learn the detailed features of depth map. Shallow network of 3 layers are used to learn the rough features of depth maps. Finally, the two channels are combined with details and outlines to realize end-to-end mapping from low resolution depth map to high resolution one. The model makes full use of the learning ability of the convolutional neural network to independently extract the effective features of the depth map and avoid the inaccuracy of manually extracting features. The experimental results on the Middlebury RGBD dataset show that the proposed model can achieve good results at a large sampling factor of 8, and has a high practical value.

Newport宣传-MKS新实验室计划
补充资料

中图分类号:TP391.4

DOI:10.3788/aos201838.1010002

所属栏目:图像处理

基金项目:国家自然科学基金(61002028)、国家863计划(2012AA011505,2012AA03A301)

收稿日期:2018-02-06

修改稿日期:2018-04-26

网络出版日期:2018-05-16

作者单位    点击查看

李素梅:天津大学电气自动化与信息工程学院, 天津 300072
雷国庆:天津大学电气自动化与信息工程学院, 天津 300072
范如:天津大学电气自动化与信息工程学院, 天津 300072

联系人作者:雷国庆(lgq20051118@163.com)

【1】Kolb A, Barth E, Koch R, et al. Time-of-flight cameras in computer graphics[J]. Computer Graphics Forum, 2010, 29(1): 141-159.

【2】Liu J L, Li S M, Li Y D, et al. High-resolution depth maps based on TOF-Stereo fusion[J]. Information Technology, 2016, 40(12): 190-193.
刘娇丽, 李素梅, 李永达, 等. 基于TOF与立体匹配相融合的高分辨率深度获取[J]. 信息技术, 2016, 40(12): 190-193.

【3】Schuon S, Theobalt C, Davis J, et al. High-quality scanning using time-of-flight depth superresolution[C]∥Proceedings of IEEE Computer Society Conference on Computer Vision and Pattern Recognition Workshops, 2008: 1-7.

【4】Wang Q L, Li J Y, Shen H K. Target tracking system of binocular vision and laser range sensor[J]. Acta Optica Sinica, 2016, 36(9): 0912002.
王琪龙, 李建勇, 沈海阔. 双目视觉-激光测距传感器目标跟踪系统[J]. 光学学报, 2016, 36(9): 0912002.

【5】Li X, Tang R F, Li Z L, et al. Laser ranging data processing based on the analysis of the binary image[J]. Chinese Journal of Lasers, 2014, 41(12): 1208005.
李熙, 汤儒峰, 李祝莲, 等. 基于二值图像的卫星激光测距数据处理[J]. 中国激光, 2014, 41(12): 1208005.

【6】Schuon S, Theobalt C, Davis J, et al. LidarBoost: Depth superresolution for ToF 3D shape scanning[C]∥Proceedings of IEEE Conference on Computer Vision and Pattern Recognition, 2009: 343-350.

【7】Gevrekci M, Pakin K. Depth map super resolution[C]∥Proceedings of 18th IEEE International Conference on Image Processing, 2011: 3449-3452.

【8】Rajagopalan A N, Bhavsar A, Wallhoff F, et al. Resolution enhancement of PMD range maps[C]∥Proceedings of the 30th DAGM Symposium on Pattern Recognition, 2008: 304-313.

【9】Aodha O M, Campbell N D F, Nair A, et al. Patch based synthesis for single depth image super-resolution[C]∥Proceedings of European Conference on Computer Vision, 2012: 71-84.

【10】Timofte R, de Smet V, Gool L V. A+: Adjusted anchored neighborhood regression for fast super-resolution[C]∥Proceedings of Asian Conference on Computer Vision, 2014: 111-126.

【11】Kopf J, Cohen M F, Lischinski D, et al. Joint bilateral upsampling[J]. ACM Transactions on Graphics, 2007, 26(3): 96-101.

【12】He K M, Sun J, Tang X O. Guided image filtering[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2013, 35(6): 1397-1409.

【13】Yang Q X, Yang R G, Davis J, et al. Spatial-depth super resolution for range images[C]∥Proceedings of IEEE Conference on Computer Vision and Pattern Recognition, 2007: 1-8.

【14】Liu M Y, Tuzel O, Taguchi Y. Joint geodesic upsampling of depth images[C]∥Proceedings of IEEE Conference on Computer Vision and Pattern Recognition, 2013: 169-176.

【15】Park J, Kim H, Tai Y W, et al. High quality depth map upsampling for 3D-TOF cameras[C]∥Proceedings of International Conference on Computer Vision, 2011: 1623-1630.

【16】Buades A, Coll B, Morel J M. A review of image denoising algorithms, with a new one[J]. Multiscale Modeling & Simulation, 2005, 4(2): 490-530.

【17】Buades A, Coll B, Morel J M. Imaged enoising methods. A new nonlocal principle[J]. SIAM Review, 2010, 52(1): 113-147.

【18】Ferstl D, Reinbacher C, Ranftl R, et al. Image guided depth upsampling using anisotropic total generalized variation[C]∥Proceedings of IEEE International Conference on Computer Vision, 2013: 993-1000.

【19】Li Y J, Xue T F, Sun L F, et al. Joint example-based depth map super-resolution[C]∥Proceedings of IEEE International Conference on Multimedia and Expo, 2012: 152-157.

【20】Kwon H, Tai Y W, Lin S. Data-driven depth map refinement via multi-scale sparse representation[C]∥Proceedings of IEEE Conference on Computer Vision and Pattern Recognition, 2015: 159-167.

【21】Kiechle M, Hawe S, Kleinsteuber M. A joint intensity and depth co-sparse analysis model for depth map super-resolution[C]∥Proceedings of IEEE International Conference on Computer Vision, 2013: 1545-1552.

【22】Lu J J, Forsyth D. Sparse depth super resolution[C]∥Proceedings of IEEE Conference on Computer Vision and Pattern Recognition, 2015: 2245-2253.

【23】Diebel J, Thrun S. An application of Markov random fields to range sensing[J]. Advances in Neural Information Processing Systems, 2005: 291-298.

【24】Dong C,Loy C C, He K M, et al. Image super-resolution using deep convolutional networks[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2016, 38(2): 295-307.

【25】Wang Z W, Liu D, Yang J C, et al. Deep networks for image super-resolution with sparse prior[C]∥Proceedings of IEEE International Conference on Computer Vision, 2015: 370-378.

【26】He K M, Zhang X Y, Ren S Q, et al. Deep residual learning for image recognition[C]∥Proceedings of IEEE Conference on Computer Vision and Pattern Recognition, 2016: 770-778.

【27】Durbin R, Rumelhart D E. Product units: A computationally powerful and biologically plausible extension to backpropagation networks[J]. Neural Computation, 1989, 1(1): 133-142.

【28】Lu J B, Shi K Y, Min D B , et al. Cross-based local multipoint filtering[C]∥Proceedings of IEEE Conference on Computer Vision and Pattern Recognition, 2012: 430-437.

引用该论文

Li Sumei,Lei Guoqing,Fan Ru. Depth Map Super-Resolution Based on Two-Channel Convolutional Neural Network[J]. Acta Optica Sinica, 2018, 38(10): 1010002

李素梅,雷国庆,范如. 基于双通道卷积神经网络的深度图超分辨研究[J]. 光学学报, 2018, 38(10): 1010002

您的浏览器不支持PDF插件,请使用最新的(Chrome/Fire Fox等)浏览器.或者您还可以点击此处下载该论文PDF