Multi-modal, multi-technique vehicle signal detection

Patent metadata

Field	Value
Publication number	US-11900692-B2
Application number	US-202117564466-A
Country	US
Kind code	B2
Filing date	Dec 29, 2021
Priority date	Feb 27, 2020
Publication date	Feb 13, 2024
Grant date	Feb 13, 2024

Abstract

A vehicle includes one or more cameras that capture a plurality of two-dimensional images of a three-dimensional object. A light detector and/or a semantic classifier search within those images for lights of the three-dimensional object. A vehicle signal detection module fuses information from the light detector and/or the semantic classifier to produce a semantic meaning for the lights. The vehicle can be controlled based on the semantic meaning. Further, the vehicle can include a depth sensor and an object projector. The object projector can determine regions of interest within the two-dimensional images, based on the depth sensor. The light detector and/or the semantic classifier can use these regions of interest to efficiently perform the search for the lights.

First claim

What is claimed is:

1. A computer-implemented method, comprising:
outputting, by an object tracker, one or more tracked objects;
receiving images by a first vehicle signal detector and a second vehicle signal detector;
determining, by the first vehicle signal detector implementing a first technique, first intermediary vehicle signal light semantic classifications based on the images;
determining, by the second vehicle signal detector implementing a second technique different from the first technique, second intermediary vehicle signal light semantic classifications based on the same images;
receiving further images different from the images by a third vehicle signal detector;
determining, by the third vehicle signal detector, third intermediary vehicle signal light semantic classifications based on the further images;
collecting first time-series data generated by the first vehicle signal detector, wherein the first time-series data comprises the first intermediary vehicle signal light semantic classifications;
collecting second time-series data generated by the second vehicle signal detector, wherein the second time-series data comprises the second intermediary semantic vehicle signal light classifications;
collecting third time-series data generated by the third vehicle signal detector, wherein the third time-series data comprises the third intermediary vehicle signal light semantic classifications;
fusing the first time-series data, the second time-series data, and third time-series data, into a matrix of information; and
deducing a final semantic detection of a vehicle signal light based on the matrix of information, and the one or more tracked objects.

2. The computer-implemented method of claim 1 , wherein:
the images are captured by a first camera; and
the further images are captured by a second camera different from the first camera.

3. The computer-implemented method of claim 1 , wherein the first time-series data includes detection of a first vehicle signal light present in at least one of the images but the second time-series data does not include the detection of the first vehicle signal light the images.

4. The computer-implemented method of claim 1 , wherein the one or more tracked objects are represented by one or more polygons.

5. The computer-implemented method of claim 1 , wherein fusing comprises:
fusing the first time-series data, the second time-series data, and the third time-series data in a voting scheme.

6. The computer-implemented method of claim 1 , wherein the first vehicle signal detector comprises a light detector to geometrically determine one or more of: coordinates, size, shape, and perimeter of one or more vehicle signal lights within the images.

7. The computer-implemented method of claim 1 , further comprising:
providing the one or more tracked objects to the first vehicle signal detector.

8. The computer-implemented method of claim 1 , wherein the second vehicle signal detector comprises a semantic classifier to determine a semantic meaning of one or more vehicle signal lights detected in the images.

9. The computer-implemented method of claim 1 , further comprising:
providing the one or more tracked objects to the second vehicle signal detector.

10. The computer-implemented method of claim 1 , wherein deducing the final semantic detection of the vehicle signal light comprises:
regressing detected vehicle signal lights in one or more of the first time-series data, the second time-series data, and the third time-series data onto a unit cube of a tracked object of the one or more tracked objects.

11. The computer-implemented method of claim 10 , wherein deducing the final semantic detection of the vehicle signal light comprises:
determining that the detected vehicle signal lights in one or more of the first time-series data, the second time-series data, and the third time-series data are at a same height on the unit cube.

12. The computer-implemented method of claim 1 , wherein deducing the final semantic detection of the vehicle signal light comprises:
apply temporal signal interpretation to the matrix of information.

13. The computer-implemented method of claim 1 , wherein deducing the final semantic detection of the vehicle signal light comprises:
generating the final semantic detection based on the first intermediary vehicle signal light semantic classifications in the first time-series data, the second intermediary vehicle signal light semantic classifications in the second time-series data, and the third intermediary vehicle signal light semantic classifications in the third time-series data.

14. The computer-implemented method of claim 1 , further comprising:
causing a vehicle control system of an autonomous vehicle to respond to the final semantic detection of the vehicle signal light.

15. One or more non-transitory, computer-readable media encoded with instructions that, when executed by one or more processing units, perform a method comprising:
outputting, by an object tracker encoded by the instructions, one or more tracked objects;
processing images by a first vehicle signal detector encoded by the instructions and a second vehicle signal detector encoded by the instructions, wherein the first and second vehicle signal detector implement different techniques for detecting vehicle signal lights based on the same images;
outputting, by the first vehicle signal detector, first intermediary vehicle signal light semantic classifications;
outputting, by the second vehicle signal detector, second intermediary vehicle signal light semantic classifications;
processing further images different from the images by a third vehicle signal detector encoded by the instructions;
outputting, by the third vehicle signal detector, third intermediary vehicle signal light semantic classifications;
collecting first time-series data generated by the first vehicle signal detector, wherein the first time-series data comprises the first intermediary vehicle signal light semantic classifications;
collecting second time-series data generated by the second vehicle signal detector, wherein the second time-series data comprises the second intermediary vehicle signal light semantic classifications;
collecting third time-series data generated by the third vehicle signal detector, wherein the third time-series data comprises the third intermediary vehicle signal light semantic classifications; and
deducing a final semantic detection of a vehicle signal light based on the first time-series data, the second time-series data, the third time-series data, and the one or more tracked objects.

16. A vehicle, comprising:
one or more memories including instructions;
one or more processors to execute the instructions;
a first camera and a second camera; and
an object tracker encoded in the instructions to:
output one or more tracked objects;

a first vehicle signal detector encoded in the instructions to:
receive images captured by the first camera and implement a first technique for detecting vehicle signal lights based on the images; and
collect first time-series data, comprising first intermediary vehicle signal light semantic classifications;

a second vehicle signal detector encoded in the instructions to:
receive the images and implement a second technique for detecting vehicle signal lights based on the images, the second technique being different from the first technique; and
collect second time-series data, comprising second intermediary vehicle signal light semantic classifications;

a third vehicle signal detector encode

Assignees

• Gm Cruise Holdings Llc

Inventors

• Creusot Clement
• Thuremella Divya
• Yu Na
• Pu Jia

Classifications

• G06V20/584 Primary

  of vehicle lights or traffic lights · CPC title

• B60W30/09

  Taking automatic action to avoid collision, e.g. braking and steering · CPC title

• G06F18/24

  Classification techniques · CPC title

• G06F18/25

  Fusion techniques · CPC title

• G06V10/806

  of extracted features · CPC title