Multi-modal, multi-technique vehicle signal detection

Patent metadata

Field	Value
Publication number	US-11195033-B2
Application number	US-202016803829-A
Country	US
Kind code	B2
Filing date	Feb 27, 2020
Priority date	Feb 27, 2020
Publication date	Dec 7, 2021
Grant date	Dec 7, 2021

Abstract

A vehicle includes one or more cameras that capture a plurality of two-dimensional images of a three-dimensional object. A light detector and/or a semantic classifier search within those images for lights of the three-dimensional object. A vehicle signal detection module fuses information from the light detector and/or the semantic classifier to produce a semantic meaning for the lights. The vehicle can be controlled based on the semantic meaning. Further, the vehicle can include a depth sensor and an object projector. The object projector can determine regions of interest within the two-dimensional images, based on the depth sensor. The light detector and/or the semantic classifier can use these regions of interest to efficiently perform the search for the lights.

First claim

What is claimed is:

1. A method, comprising:
receiving a two-dimensional image of a first three-dimensional object from a camera, wherein the first three-dimensional object is a vehicle;
receiving a first signal indicating first three-dimensional coordinates of the first three-dimensional object;
receiving a second signal indicating second three-dimensional coordinates of a second three-dimensional object;
projecting the first three-dimensional coordinates onto the two-dimensional image;
determining a portion of the first three-dimensional object is occluded in the two-dimensional image by the second three-dimensional object, based on the first three-dimensional coordinates of the first three-dimensional object, and the second three-dimensional coordinates of the second three-dimensional object;
determining a region of interest in the two-dimensional image based on the projection; and
searching for a vehicle signal light of the first three-dimensional object within the region of interest.

2. The method of claim 1 , wherein projecting the first three-dimensional coordinates onto the two-dimensional image comprises applying a matrix transformation to the first three-dimensional coordinates.

3. The method of claim 1 , further comprising:
controlling a vehicle, based on the vehicle signal light of the first three-dimensional object, wherein the controlling is accelerating, braking, or steering the vehicle.

4. The method of claim 1 , further comprising:
in response to determining the portion of the first three-dimensional object is occluded, applying a mask to the two-dimensional image to remove pixels not associated with regions where vehicle signal lights would be present.

5. The method of claim 1 , wherein projecting the first three-dimensional coordinates comprises projecting a surface of a three-dimensional polygon corresponding to the first three-dimensional object.

6. The method of claim 1 , wherein the first signal indicating the first three-dimensional coordinates of the first three-dimensional object includes three-dimensional coordinates of a plurality of surfaces of the first three-dimensional object, and the region of interest is determined based on the three-dimensional coordinates of the plurality of surfaces of the first three-dimensional object.

7. The method of claim 1 , further comprising:
cropping the two-dimensional image to the region of interest to produce a cropped image, wherein the searching is performed on the cropped image.

8. One or more non-transitory, computer-readable media encoded with instructions that, when executed by one or more processing units, perform a method comprising:
receiving a two-dimensional image of a first three-dimensional object from a camera, wherein the first three-dimensional object is a vehicle;
receiving a first signal indicating first three-dimensional coordinates of the first three-dimensional object;
receiving a second signal indicating second three-dimensional coordinates of a second three-dimensional object;
projecting the first three-dimensional coordinates onto the two-dimensional image;
determining a portion of the first three-dimensional object is occluded in the two-dimensional image by the second three-dimensional object, based on the first three-dimensional coordinates of the first three-dimensional object, and the second three-dimensional coordinates of the second three-dimensional object;
determining a region of interest in the two-dimensional image based on the projection; and
searching for a vehicle signal light of the first three-dimensional object within the region of interest.

9. The one or more non-transitory, computer-readable media of claim 8 , wherein projecting the first three-dimensional coordinates onto the two-dimensional image comprises applying a matrix transformation to the first three-dimensional coordinates.

10. The one or more non-transitory, computer-readable media of claim 8 , the method further comprising:
controlling a vehicle, based on the vehicle signal light of the first three-dimensional object, wherein the controlling is accelerating, braking, or steering the vehicle.

11. The one or more non-transitory, computer-readable media of claim 8 , the method further comprising:
in response to determining the portion of the first three-dimensional object is occluded, applying a mask to the two-dimensional image to remove pixels not associated with the first three-dimensional object.

12. The one or more non-transitory, computer-readable media of claim 8 , wherein projecting the first three-dimensional coordinates comprises projecting a surface of a three-dimensional polygon corresponding to the first three-dimensional object.

13. The one or more non-transitory, computer-readable media of claim 8 , wherein the first signal indicating the first three-dimensional coordinates of the first three-dimensional object includes three-dimensional coordinates of a plurality of surfaces of the first three-dimensional object, and the region of interest is determined based on the three-dimensional coordinates of the plurality of surfaces of the first three-dimensional object.

14. The one or more non-transitory, computer-readable media of claim 8 , the method further comprising:
cropping the two-dimensional image to the region of interest to produce a cropped image, wherein the searching is performed on the cropped image.

15. A system, comprising:
a memory including instructions;
a processor to execute the instructions; and
an object projector, encoded in the instructions, to:
receive a two-dimensional image of a first three-dimensional object from a camera, wherein the first three-dimensional object is a vehicle;
receive a first signal indicating first three-dimensional coordinates of the first three-dimensional object;
receive a second signal indicating second three-dimensional coordinates of a second three-dimensional object;
project the first three-dimensional coordinates onto the two-dimensional image;
determine a portion of the first three-dimensional object is occluded in the two-dimensional image by the second three-dimensional object, based on the first three-dimensional coordinates of the first three-dimensional object, and the second three-dimensional coordinates of the second three-dimensional object; and
determine a region of interest in the two-dimensional image based on the projection; and

a vehicle signal detector, encoded in the instructions, to:
search for a vehicle signal light of the first three-dimensional object within the region of interest.

16. The system of claim 15 , wherein the vehicle signal detector further is to:
skip light detection in the portion of the first three-dimensional object in the two-dimensional image occluded by the second three-dimensional object.

17. The system of claim 15 , further comprising:
a vehicle control system to accelerate, brake, or steer a vehicle, based on the vehicle signal light of the first three-dimensional object.

18. The system of claim 15 , wherein the vehicle signal detector further is to:
in response to determining the portion of the first three-dimensional object is occluded, determine any lights within the portion of the first three-dimensional object in the two-dimensional image occluded by the second three-dimensional object do not belong to the first three-dimensional object.

19. The system of claim 15 , wherein projecting the first three-dimensional coordinates comprises projecting a surface of a three-dimensional polygon corresponding to the fir

Assignees

• Gm Cruise Holdings Llc

Inventors

• Creusot Clement
• Thuremella Divya
• Yu Na
• Pu Jia

Classifications

• G06V10/806

  of extracted features · CPC title

• G06V10/811

  the classifiers operating on different input data, e.g. multi-modal recognition · CPC title

• G06V20/584 Primary

  of vehicle lights or traffic lights · CPC title

• G06F18/253

  of extracted features · CPC title

• G06F18/25

  Fusion techniques · CPC title