Multi-modal, multi-technique vehicle signal detection

Patent metadata

Field	Value
Publication number	US-11288527-B2
Application number	US-202016804667-A
Country	US
Kind code	B2
Filing date	Feb 28, 2020
Priority date	Feb 27, 2020
Publication date	Mar 29, 2022
Grant date	Mar 29, 2022

Abstract

A vehicle includes one or more cameras that capture a plurality of two-dimensional images of a three-dimensional object. A light detector and/or a semantic classifier search within those images for lights of the three-dimensional object. A vehicle signal detection module fuses information from the light detector and/or the semantic classifier to produce a semantic meaning for the lights. The vehicle can be controlled based on the semantic meaning. Further, the vehicle can include a depth sensor and an object projector. The object projector can determine regions of interest within the two-dimensional images, based on the depth sensor. The light detector and/or the semantic classifier can use these regions of interest to efficiently perform the search for the lights.

First claim

What is claimed is:

1. A method, comprising:
determining, by an object tracker, a polygon of a three-dimensional object;
generating, by a first vehicle signal detector, geometric information and semantic information for a first light of the three-dimensional object within a first two-dimensional image of the three-dimensional object captured by a camera on a vehicle, wherein geometric information for the first light comprises location of the first light on the three-dimensional object, and semantic information comprises a classification of the first light;
generating, by a second vehicle signal detector, geometric information and semantic information for a second light within a second two-dimensional image of the three-dimensional object, wherein the first light is not captured in the second two-dimensional image, geometric information for the second light comprises location of the second light, and semantic information comprises a classification of the second light;
collecting, over time, a first vector of geometric information and semantic information generated by the first vehicle signal detector;
collecting, over time, a second vector of geometric information and semantic information generated by the second vehicle signal detector;
fusing at least the first vector and a second vector into a matrix;
determining, that the first light and the second light are associated with the polygon based on the geometric information generated by the first and second vehicle signal detectors; and
determining a semantic meaning of the first light and the second light based on the matrix, and the determination that the first light and the second light are both associated with the polygon.

2. The method of claim 1 , further comprising, controlling the vehicle, based on the semantic meaning of the first light and the second light, wherein the controlling is accelerating, braking, or steering the vehicle.

3. The method of claim 1 , wherein generating, by the first vehicle signal detector, comprises:
determining a first location of the first light on the three-dimensional object and a first color of the first light in the first two-dimensional image, wherein the semantic information for the first light is based on the first location and the first color.

4. The method of claim 1 , wherein:
the first two-dimensional image and the second two-dimensional image are different images from different cameras.

5. The method of claim 1 , further comprising:
determining that the first light and the second light are at substantially the same height of the polygon.

6. The method of claim 1 , wherein determining the semantic meaning of the first light and the second light comprises extracting frequency information from the matrix.

7. The method of claim 1 , wherein:
geometric information for the first light comprises a first color of the first light; and
geometric information for the second light comprises a second color of the second light.

8. One or more non-transitory, computer-readable media encoded with instructions that, when executed by one or more processing units, perform a method comprising:
determining, by an object tracker, a polygon of a three-dimensional object;
determining, by a first vehicle signal detector encoded by the instructions, geometric information and semantic information for a first light of the three-dimensional object within a first two-dimensional image of the three-dimensional object captured by a camera on a vehicle, wherein geometric information for the first light comprises location of the first light on the three-dimensional object, and semantic information comprises a semantic label of the first light generated by a first classifier of the first vehicle signal detector;
determining, by a second vehicle signal detector encoded by the instructions, geometric information and semantic information for a second light within a second two-dimensional image, wherein the first light is not present in the second two-dimensional image geometric information for the second light comprises location of the second light, and semantic information comprises a semantic label of the second light generated by a second classifier of the second vehicle signal detector;
accumulating, overtime, a first vector of geometric information and semantic information determined by the first vehicle signal detector;
accumulating, over time, a second vector of geometric information and semantic information determined by the second vehicle signal detector;
forming a matrix with at least the first vector and the second vector; and
determining a semantic meaning of the first light and the second light based on the matrix, and a determination, from the geometric information of the first light, the geometric information of the second light, and the polygon, that the first light and the second light are both associated with the polygon.

9. The one or more non-transitory, computer-readable media of claim 8 , the method further comprising:
controlling the vehicle, based on the semantic meaning of the first light and the second light, wherein the controlling is accelerating, braking, or steering the vehicle.

10. The one or more non-transitory, computer-readable media of claim 8 , the method further comprising:
determining a first location of the first light on the three-dimensional object and a first color of the first light in the first two-dimensional image, wherein the semantic information for the first light is based on the first location and the first color.

11. The one or more non-transitory, computer-readable media of claim 8 , wherein:
determining the semantic meaning comprises applying a filter on the matrix.

12. The one or more non-transitory, computer-readable media of claim 8 , wherein:
determining the semantic meaning comprises applying logic rules on the matrix.

13. The one or more non-transitory, computer-readable media of claim 8 , wherein:
determining the semantic meaning comprises applying a supervised or unsupervised learning technique on the matrix.

14. The one or more non-transitory, computer-readable media of claim 8 , wherein:
geometric information for the first light comprises a first color of the first light; and
geometric information for the second light comprises a second color of the second light.

15. A vehicle, comprising:
one or more memories including instructions;
one or more processors to execute the instructions;
a body including a camera; and
a first vehicle signal detector encoded in the instructions to:
receiving, from an object tracker, a polygon of a three-dimensional object;
determine geometric information and semantic information for a first light of the three-dimensional object within a first two-dimensional image of the three-dimensional object captured by the camera, wherein geometric information for the first light comprises location of the first light, and semantic information comprises a semantic classification of the first light; and
generate a first vector using geometric information and semantic information collected from the first vehicle signal detector over time;

a second vehicle signal detector encoded in the instructions to:
determine geometric information and semantic information for a second light within a second two-dimensional image, wherein the first light is not captured in the second two-dimensional image, geometric information for the second light comprises location of the second light, and semantic information comprises a semantic classification of the second light; and
generate a second vector using geometric information and semantic informati

Assignees

• Gm Cruise Holdings Llc

Inventors

• Creusot Clement
• Thuremella Divya
• Yu Na
• Pu Jia

Classifications

• G06V10/806

  of extracted features · CPC title

• G06V10/811

  the classifiers operating on different input data, e.g. multi-modal recognition · CPC title

• G06F18/256

  of results relating to different input data, e.g. multimodal recognition · CPC title

• G06F18/253

  of extracted features · CPC title

• G06F18/25

  Fusion techniques · CPC title