首页 > 论文 > 光学学报 > 38卷 > 7期(pp:710004--1)

结合图像语义分割的增强现实型平视显示系统设计与研究

Design of Augmented Reality Head-up Display System Based on Image Semantic Segmentation

  • 摘要
  • 论文信息
  • 参考文献
  • 被引情况
  • PDF全文
分享:

摘要

为了提高驾驶员在车辆行驶过程中的安全性,设计了一种结合图像语义分割的增强现实型平视显示(AR-HUD)系统。首先,提出一种改进的单发多框检测器网络对道路场景图像进行语义分割,网络前端采用VGG-16提取图像特征,网络后端对获取的特征图进行上采样,从而对特征图进行像素分割。通过对网络的训练,得到场景目标的像素级分类结果,即环境的语义内容信息。随后,通过分析真实场景、光学显示系统、驾驶员之间的关系,将计算机产生的虚拟信息叠加到真实场景,并将显示内容注册到驾驶员视野中,从而提高行车安全。实验结果表明,语义分割算法的准确率能达到77.8%,虚实注册算法处理每帧图像的时间平均为45 ms,约22 frame·s-1。

Abstract

In order to improve the security of drivers, an augmented reality head-up display (AR-HUD) system is designed based on image semantic segmentation. Firstly, we propose an improved single shot multibox detector network for semantic segmentation of road scene images. The front end of the network uses VGG-16 to extract the image features, and the back ends of the network are sampled on the feature maps. Thus, the feature map is segmented. Through the training of the network, the pixel level classification results of the scene objects are obtained, namely, the semantic content information of the environment. Then, with analysis of the relationship among real scene, optical display system, and drivers, the virtual information generated by computer is added to the real scene. In this way, the content is registered into the driver′s view to improve the safety of driving. Experimental results show that the accuracy of the semantic segmentation algorithm can reach 77.8%, and image processing time of the algorithm for each frame is 45 ms, in other words, about 22 frame·s-1.

Newport宣传-MKS新实验室计划
补充资料

中图分类号:TH741

DOI:10.3788/aos201838.0710004

所属栏目:图像处理

基金项目:国家自然科学基金(61605016)

收稿日期:2018-01-25

修改稿日期:2018-03-08

网络出版日期:--

作者单位    点击查看

安喆:长春理工大学光电工程学院, 吉林 长春 130022
徐熙平:长春理工大学光电工程学院, 吉林 长春 130022
杨进华:长春理工大学光电工程学院, 吉林 长春 130022
乔杨:长春理工大学光电工程学院, 吉林 长春 130022
刘洋:长春理工大学光电工程学院, 吉林 长春 130022

联系人作者:徐熙平(xxp@cust.edu.cn)

备注:安喆(1992-),女,博士研究生,主要从事光电检测技术方面的研究。E-mail: 2016200046@mails.cust.edu.cn

【1】Park H S, Min W P, Won K H, et al. In-vehicle AR-HUD system to provide driving-safety information[J]. ETRI Journal, 2013, 35(6):1038-1047.

【2】Yu Y H. Research status and development trend of augmented reality technology[J]. Journal of Hunan Mass Media Vocational Technical College, 2016, 16(1): 55-57.
余艳红. 增强现实技术的研究现状及发展趋势[J]. 湖南大众传媒职业技术学院学报, 2016, 16(1): 55-57.

【3】Gui Z W, Liu Y, Chen J, et al. Online learning of tracking and registration based on natural scenes[J]. Journal of Software, 2016, 27(11): 2929- 2945.
桂振文, 刘越, 陈靖, 等. 基于自然场景在线学习的跟踪注册技术[J]. 软件学报, 2016, 27(11): 2929-2945.

【4】Fiala M. ARTag, afiducial marker system using digital techniques[C]∥Proceedings of 2005 IEEE Computer Society Conference on Computer Vision & Pattern Recognition, 2005,2:590-596.

【5】Zhang G, Chen H S, Ye Y D. A LoG operator based markerless augmented reality algorithm: LoG-PTAMM[J]. Journal of Computer-Aided Design & Computer Graphics, 2016, 28(9):1577-1586.
张格, 陈昊升, 叶阳东. 一种基于LoG算子的无标识增强现实算法: LoG-PTAMM[J]. 计算机辅助设计与图形学学报, 2016, 28(9): 1577-1586.

【6】Gao K J, Sun S Y, Yao G S, et al. Semantic segmentation of night vision images for unmanned vehicls based on deep learning[J]. Journal of Applied Optics, 2017, 38(3): 421-428.
高凯珺, 孙韶媛, 姚广顺, 等. 基于深度学习的无人车夜视图像语义分割[J]. 应用光学, 2017, 38(3): 421-428.

【7】Liu C H, Li Z, Xu C, et al. BRDF model for commonly used materials of space targets based on deep neural network[J]. Acta Optica Sinica, 2017, 37(11): 1129001.
刘程浩, 李智, 徐灿, 等. 基于深度神经网络的空间目标常用材质BRDF模型[J]. 光学学报, 2017, 37(11): 1129001.

【8】Lu Y S, Li Y X, Liu B, et al. Hyperspectral data haze monitoring based on deep residual network[J]. Acta Optica Sinica, 2017, 37(11): 1128001.
陆永帅, 李元祥, 刘波, 等. 基于深度残差网络的高光谱遥感数据霾监测[J]. 光学学报, 2017, 37(11): 1128001.

【9】Huang H, He K, Zheng X L, et al. Spatial-spectral feature extraction of hyperspectral image based on deep learning[J]. Laser & Optoelectronics Progress, 2017, 54(10): 101001.
黄鸿, 何凯, 郑新磊, 等. 基于深度学习的高光谱图像空-谱联合特征提取[J]. 激光与光电子学进展, 2017, 54(10): 101001.

【10】Wu S C, Zhao H T, Sun S Y. Depth estimation from monocular infrared video based on bi-recursive convolutional neural network[J]. Acta Optica Sinica, 2017, 37(12): 1215003.
吴寿川, 赵海涛, 孙韶媛. 基于双向递归卷积神经网络的单目红外视频深度估计[J]. 光学学报, 2017, 37(12): 1215003.

【11】Long J, Shelhamer E, Darrell T. Fully convolutional networks for semantic segmentation[J]. IEEE Transactions on Pattern Analysis & Machine Intelligence, 2017, 39(4): 640-651.

【12】Badrinarayanan V, Kendall A, Cipolla R. SegNet: a deep convolutional encoder-decoder architecture for scene segmentation[J]. IEEE Transactions on Pattern Analysis & Machine Intelligence, 2017, 39(12): 2481-2495.

【13】Chen L C, Papandreou G, Kokkinos I, et al. Semantic image segmentation with deep convolutional nets and fully connected CRFs[J]. Computer Science, 2014, 11(4): 357-361.

【14】Liu W, Anguelov D, Erhan D, et al. SSD: single shot multibox detector[C]∥Proceedings of 2016 European Conference on Computer Vision, 2016, 9905: 21-37.

【15】Zhang K. Research on a rapid fusion method for remote sensing images based on an improved atrous wavelet decompsition[D]. Kaifeng: Henan University, 2016: 66-72.
张凯. 基于改进atrous小波分解的遥感影像快速融合方法的研究[D]. 开封: 河南大学, 2016: 66-72.

【16】Xu L, Zhao H T, Sun S Y. Monocular infrared image depth estimation based on deep convolutional neural networks[J]. Acta Optica Sinica, 2016, 36(7): 0715002.
许路, 赵海涛, 孙韶媛. 基于深层卷积神经网络的单目红外图像深度估计[J]. 光学学报, 2016, 36(7): 0715002.

【17】Besl P J, Mckay N D. Method for registration of 3D shapes[C]∥Proceedings of 1992 Robotics-DL tentative International Society for Optics and Photonics, 1992, 14: 239-256.

【18】Brostow G J, Shotton J, Fauqueur J, et al. Segmentation and recognition using structure from motion point clouds[C]∥Proceedings of 2008 European Conference on Computer Vision. 2008, 5302: 44-57.

【19】Fei T, Liang X H, He Z Y, et al. A registration method based on nature feature with KLT tracking algorithm for wearable computers[C]∥Proceedings of 2009 International Conference on Cyberworlds. 2009, 1: 416-421.

引用该论文

An Zhe,Xu Xiping,Yang Jinhua,Qiao Yang,Liu Yang. Design of Augmented Reality Head-up Display System Based on Image Semantic Segmentation[J]. Acta Optica Sinica, 2018, 38(7): 0710004

安喆,徐熙平,杨进华,乔杨,刘洋. 结合图像语义分割的增强现实型平视显示系统设计与研究[J]. 光学学报, 2018, 38(7): 0710004

您的浏览器不支持PDF插件,请使用最新的(Chrome/Fire Fox等)浏览器.或者您还可以点击此处下载该论文PDF