针对目前基于红外与可见光模态融合的行人检测方法难以自适应外界环境变化的问题, 提出基于多模态信息融合权值学习的行人检测网络。首先, 区别于目前大多数研究采用的两模态直接堆叠融合方法, 权值学习融合网络考虑两种模态在不同环境条件下对行人检测任务的不同贡献比重, 通过双流交互学习二者差异, 然后根据各模态特征的当前特性自主获得各模态特征的相应权重, 进行加权融合得到融合特征, 最后基于融合特征生成新的特征金字塔, 并改变先验框的尺寸和密集度以丰富行人先验信息, 完成行人检测任务。实验结果表明: 在 Kaist多光谱行人检测数据集上获得 26. 96%的平均漏检率, 相比目前采用直接堆叠的最优方法以及 baseline方法分别降低了 2. 77%和 27. 84%, 因此自适应权值融合红外和可见光两种模态的信息可以有效获得互补的模态信息以自适应外界环境变化, 大幅提升行人检测的性能。

A pedestrian detection network based on the weight learning of fusing multimodal information was developed to address the issues of the pedestrian detection method based on infrared and visible modal fusion in adapting to changes in the external environment. First, unlike the fusion method used in several recent studies in which two modalities are stacked directly, the weight learning fusion network reflects dif. ferent contributions of the modalities to the pedestrian detection task under different environmental condi. tions. The differences between the two modalities were determined through dual-stream interaction learn. ing. Next, based on the current characteristics of each modal feature, the weight learning fusion network assigned the corresponding weights to each modal feature to generate the fusion feature by performing weighted fusion autonomously. Finally, a new feature pyramid based on the fusion feature was generated, and previous information about the pedestrian was improved by changing the size and density of prior boxes to complete the pedestrian detection task. The experimental results indicated that the log-average miss rate of the Kaist multispectral pedestrian detection dataset reached 26. 96%, which was 2. 77% and 27. 84% lower than that of the direct stacking method and baseline method, respectively. The adaptive weight fu. sion of infrared and visible modal information could effectively be used to obtain complementary modal in. formation to adapt to external environmental changes and significantly improve pedestrian detection perfor. mance.