Technologies for fusing data from multiple sensors to improve object detection, identification, and localization

Patent metadata

Field	Value
Publication number	US-11423570-B2
Application number	US-201816232147-A
Country	US
Kind code	B2
Filing date	Dec 26, 2018
Priority date	Dec 26, 2018
Publication date	Aug 23, 2022
Grant date	Aug 23, 2022

Abstract

Technologies for performing sensor fusion include a compute device. The compute device includes circuitry configured to obtain detection data indicative of objects detected by each of multiple sensors of a host system. The detection data includes camera detection data indicative of a two or three dimensional image of detected objects and lidar detection data indicative of depths of detected objects. The circuitry is also configured to merge the detection data from the multiple sensors to define final bounding shapes for the objects.

First claim

The invention claimed is:

1. A compute device comprising:
circuitry to:
obtain detection data indicative of objects detected by two or more sensors of a host system, wherein the detection data includes camera detection data indicative of at least one of a two or three dimensional image of the detected objects and lidar detection data indicative of depths of the detected objects; and
automatically merge the detection data from the two or more sensors to define final bounding shapes for the detected objects without any input from a user of the compute device that is received after obtention of the detection data,

wherein to merge the detection data from the two or more sensors to define the final bounding shapes for the detected objects comprises:
identify, based on the camera detection data, a first initial bounding box shape for each of the detected objects;
identify, based on the lidar detection data, a second initial bounding box shape for each of the detected objects; and
define the final bounding shapes for the detected objects based on the first initial bounding box shape for each of the detected objects and the second initial bounding box shape for each of the detected objects,

wherein to merge the detection data from the two or more sensors comprises to apply a belief function that produces a degree of belief that multiple bounding boxes refer to the same object.

2. The compute device of claim 1 , wherein to merge the detection data from the two or more sensors comprises to apply weights to the detection data as a function of a context in which the detection data was obtained.

3. The compute device of claim 2 , wherein to merge the detection data from the two or more sensors comprises to merge the detection data at least partially based on a range of a detected object from the corresponding sensor.

4. The compute device of claim 2 , wherein to apply weights to the detection data comprises to apply weights as a function of a present weather in an environment of the host system.

5. The compute device of claim 2 , wherein to apply weights to the detection data comprises to apply weights as a function of an amount of light present in an environment of the host system.

6. The compute device of claim 5 , wherein to merge the detection data from the two or more sensors comprises to merge the detection data at least partially based on the amount of light.

7. The compute device of claim 1 , wherein to merge the detection data from the two or more sensors comprises to apply one or more user-defined rules to combine bounding boxes present in the detection data from each sensor.

8. The compute device of claim 1 , wherein to merge the detection data from the two or more sensors comprises to:
determine a class of an object in the detection data; and
apply a rule defined to combine bounding boxes for objects of the determined class.

9. The compute device of claim 1 , wherein to determine the degree of belief comprises to determine the degree of belief as a function of an intersection-over-union between multiple bounding boxes, determine the degree of belief as a function of an intersection-over-minimum between multiple bounding boxes, or determine a degree of belief as a function of a proximity of the objects to each other.

10. The compute device of claim 1 , wherein to merge the detection data comprises to merge the bounding boxes to define the final bounding shape as a polygon.

11. One or more non-transitory machine-readable storage media comprising a plurality of instructions stored thereon that, in response to being executed, cause a compute device to:
obtain detection data indicative of objects detected by each of two or more sensors of a host system, wherein the detection data includes camera detection data indicative of at least one of a two or three dimensional image of the detected objects and lidar detection data indicative of depths of the detected objects; and
merge the detection data from the two or more sensors to define final bounding shapes for the detected objects,
wherein to merge the detection data from the two or more sensors comprises to apply a belief function that produces a degree of belief that multiple bounding boxes refer to the same object.

12. The one or more non-transitory machine-readable storage media of claim 11 , wherein to merge the detection data from the two or more sensors comprises to merge the detection data at least partially based on a range of a detected object from the corresponding sensor.

13. The one or more non-transitory machine-readable storage media of claim 11 , wherein to merge the detection data comprises to merge the detection data as a function of a present weather in an environment of the host system.

14. The one or more non-transitory machine-readable storage media of claim 11 , wherein to merge the detection data comprises to merge the detection data as a function of an amount of light present in an environment of the host system.

15. The one or more non-transitory machine-readable storage media of claim 11 , wherein to merge the detection data from the two or more sensors comprises to apply weights to the detection data as a function of a context in which the detection data was obtained.

16. A method comprising:
obtaining, by a compute device, detection data indicative of objects detected by two or more sensors of a host system, wherein the detection data includes camera detection data indicative of at least one of a two or three dimensional image of the detected objects and lidar detection data indicative of depths of the detected objects; and
merging, by the compute device, the detection data from the two or more sensors to define final bounding shapes for the detected objects,
wherein to merge the detection data from the two or more sensors comprises to apply a belief function that produces a degree of belief that multiple bounding boxes refer to the same object.

17. The method of claim 16 , wherein merging the detection data from the two or more sensors comprises applying weights to the detection data as a function of a context in which the detection data was obtained.

Assignees

• Intel Corp

Inventors

• Kavulya Soila
• Chattopadhyay Rita
• Martinez-Canales Monica

Classifications

• G06T7/73 Primary

  using feature-based methods · CPC title

• G06T2200/04

  involving 3D image data · CPC title

• G06T2207/10028

  Range image; Depth image; 3D point clouds · CPC title

• G06T7/74 Primary

  involving reference images or patches · CPC title