As a core technique in the video surveillance and 3D map building fields, the moving object detection and scene re-construction are the foundations of the real-time navigation, obstacle avoidance and path planning. In the meantime, as two important sub-problems of environment perception task, they are not only closely connected with the devel-opment of many fields, such as robot, unmanned aircraft, unmanned vehicles and body feeling game, but also are the part of the lives of human intelligence. The current research is mainly based on single laser sensor or single vision sensor. It is difficult to meet the requirements of real-time multi task scenario due to the limitation of the visual field, the amount of data, the richness of the data, the real-time and the anti-jamming. Based on the mutual supplement and constraint of the laser information and the visual information, an implementation system for fusing laser and vision was designed. In addition, two improved algorithms were proposed to solve the problems of the error foreground detection in the motion detection and the mismatching between the point clouds of different sensors.