Data fusion

Patent metadata

Field	Value
Publication number	US-12510672-B2
Application number	US-202117908921-A
Country	US
Kind code	B2
Filing date	Apr 21, 2021
Priority date	Apr 21, 2020
Publication date	Dec 30, 2025
Grant date	Dec 30, 2025

Abstract

A data fusion method is provided. In one embodiment, the method comprises: acquiring a rotation angle of a laser transmitter of a lidar; selecting, according to a predetermined correspondence between rotation angle intervals and image sensors, an image sensor corresponding to a rotation angle interval in which the obtained rotation angle of the laser emitter is located as a specified image sensor, sending a trigger signal to the specified image sensor, to enable the specified image sensor to acquire an image, receiving the image and a point cloud that is acquired and returned by the lidar within the rotation angle interval in which the obtained rotation angle is located, and fuse information of pixels in the image and information of points in the point cloud according to pose change information of a vehicle in a process of acquiring the image and the point cloud.

First claim

The invention claimed is:

1 . A data fusion method, wherein the method is applied to a system, and the system comprises a vehicle, and a lidar and at least one image sensor that are arranged on the vehicle, the lidar is configured to acquire a point cloud by rotating a laser emitter, and
the method comprises: obtaining a rotation angle of the laser emitter determined by a rotation angle measurer of the lidar; selecting, according to a predetermined correspondence between at least one rotation angle interval and the at least one image sensor, an image sensor corresponding to a rotation angle interval in which the obtained rotation angle of the laser emitter is located as a specified image sensor, wherein for each of the at least one image sensor, a rotation angle interval corresponding to the image sensor is a value interval of the rotation angle of the laser emitter within an acquisition overlap region between the lidar and the image sensor; sending a trigger signal to the specified image sensor, to enable the specified image sensor to acquire an image; receiving the image and a point cloud that is acquired and returned by the lidar within the rotation angle interval in which the obtained rotation angle of the laser emitter is located; and fusing information of pixels in the image and information of points in the point cloud according to pose change information of the vehicle in a process of acquiring the image and the point cloud, for unmanned device self-driving or high-precision map generation; wherein fusing the information of the pixels in the image and information of the points in the point cloud according to pose change information of the vehicle in the process of acquiring the image and the point cloud comprises: obtaining pose information of the vehicle when the image is acquired by the specified image sensor as benchmark pose information, and obtaining pose information of the vehicle when an acquisition for the point cloud is completed by the lidar within an acquisition overlap region corresponding to the specified image sensor as offset pose information; determining compensatory pose information for the offset pose information according to the benchmark pose information; fusing the information of the pixels in the image and the information of the points in the point cloud according to the compensatory pose information; and controlling the unmanned device according to the fused information according to the compensatory pose information.

2 . The method according to claim 1 , wherein sending the trigger signal to the specified image sensor comprises:
using an image sensor corresponding to a rotation angle interval in which a rotation angle of the laser emitter obtained at a previous moment is located as a reference image sensor; and in response to determining that the specified image sensor is not the reference image sensor, sending the trigger signal to the specified image sensor.

3 . The method according to claim 1 , wherein fusing the information of the pixels in the image and the information of the points in the point cloud according to the compensatory pose information comprises:
determining mapping parameter values according to the compensatory pose information and a relative position between the specified image sensor and the lidar; and fusing the information of the pixels in the image and the information of the points in the point cloud according to the mapping parameter values.

4 . The method according to claim 3 , wherein fusing the information of the pixels in the image and the information of the points in the point cloud according to the mapping parameter values comprises:
performing coordinate conversion on spatial position information of the points in the point cloud and/or the information of the pixels in the image according to the mapping parameter values; and for each point in the point cloud, determining a pixel in the image corresponding to the point according to a result of the coordinate conversion, and fusing the spatial position information of the point and a pixel value of the pixel in the image corresponding to the point to obtain fused data.

5 . The method according to claim 1 , wherein the method is performed by a processor configured in the vehicle, and
the processor comprises a field-programmable gate array (FPGA).

6 . A data fusion system, wherein the system comprises a processor, a lidar, at least one image sensor, an inertial measurement unit (IMU), and a vehicle;
the vehicle is provided with the processor, the lidar, the at least one image sensor, and the IMU, wherein the lidar is configured to acquire a point cloud by rotating a laser emitter; the processor is configured to: obtain a rotation angle of the laser emitter determined by a rotation angle measurer of the lidar, select, according to a predetermined correspondence between at least one rotation angle interval and the at least one image sensor, an image sensor corresponding to a rotation angle interval in which the obtained rotation angle of the laser emitter is located as a specified image sensor, wherein for each of the at least one image sensor, a rotation angle interval corresponding to the image sensor is a value interval of the rotation angle of the laser emitter within an acquisition overlap region between the lidar and the image sensor, send a trigger signal to the specified image sensor, to enable the specified image sensor to acquire an image, receive the image and a point cloud that is acquired and returned by the lidar within the rotation angle interval in which the obtained rotation angle of the laser emitter is located, and fuse information of pixels in the image and information of points in the point cloud according to pose change information of the vehicle in a process of acquiring the image and the point cloud, for unmanned device self-driving or high-precision map generation; the lidar is configured to acquire the point cloud and provide the processor with the rotation angle of the laser emitter determined by the rotation angle measurer of the lidar; the at least one image sensor is configured to receive the trigger signal sent by the processor, and acquire and return the image according to the trigger signal; and the IMU is configured to acquire pose information of the vehicle; and, wherein, when fusing information of the pixels in the image and information of the points in the point cloud according to pose change information of the vehicle in the process of acquiring the image and the point cloud, the processor is further configured to: obtain pose information of the vehicle when the image is acquired by the specified image sensor as benchmark pose information, and obtain pose information of the vehicle when an acquisition for the point cloud is completed by the lidar within an acquisition overlap region corresponding to the specified image sensor as offset pose information; determine compensatory pose information for the offset pose information according to the benchmark pose information; fuse the information of the pixels in the image and the information of the points in the point cloud according to the compensatory pose information; and control the unmanned device according to the fused information according to the compensatory pose information.

7 . The data fusion system according to claim 6 , wherein, when sending the trigger signal to the specified image sensor, the processor is further configured to:
use an image sensor corresponding to a rotation angle interval in which a rotation angle of the laser emitter obtained at a previous moment is located as a reference image sensor; and in response to determining that the specified image sensor is not the reference image sensor, send the trigger signal to the specif

Assignees

• Beijing Sankuai Online Tech Co Ltd

Inventors

• Song Shuang

Classifications

• G01S7/4817

  relating to scanning · CPC title

• G01S7/4814

  of transmitters alone · CPC title

• G01S17/86

  Combinations of lidar systems with systems other than lidar, radar or sonar, e.g. with direction finders · CPC title

• G01S17/931 Primary

  of land vehicles · CPC title

• G01C21/1652

  with ranging devices, e.g. LIDAR or RADAR · CPC title