Fast CNN classification of multi-frame semantic signals

Patent metadata

Field	Value
Publication number	US-11755918-B2
Application number	US-202117519404-A
Country	US
Kind code	B2
Filing date	Nov 4, 2021
Priority date	Nov 15, 2018
Publication date	Sep 12, 2023
Grant date	Sep 12, 2023

Abstract

The present subject matter provides various technical solutions to technical problems facing advanced driver assistance systems (ADAS) and autonomous vehicle (AV) systems. In particular, disclosed embodiments provide systems and methods that may use cameras and other sensors to detect objects and events and identify them as predefined signal classifiers, such as detecting and identifying a red stoplight. These signal classifiers are used within ADAS and AV systems to control the vehicle or alert a vehicle operator based on the type of signal. These ADAS and AV systems may provide full vehicle operation without requiring human input. The embodiments disclosed herein provide systems and methods that can be used as part of or in combination with ADAS and AV systems.

First claim

What is claimed is:

1. A system comprising:
processing circuitry; and
one or more storage devices comprising instructions, which when executed by the processing circuitry, configure the processing circuitry to:
transform a plurality of time sequenced images received from an image capture device to a plurality of vectors stored in a time-sequenced data structure; and
generate a semantic signal based on an application of a convolutional neural network to a temporal image, the temporal image generated based on the plurality of vectors;
wherein each of the plurality of vectors includes a row vector of the same width as each of the plurality of time sequenced images.

2. The system of claim 1 , wherein to transform the plurality of time sequenced images to the plurality of vectors, the processing circuitry is configured to calculate a column value for each of a plurality of columns within each of the plurality of time sequenced images.

3. The system of claim 2 , wherein calculating the column value includes at least one of calculating a mean value, a median value, or a maximal value for each of a plurality of columns within each of the plurality of time sequenced images.

4. The system of claim 1 , wherein the generation of the temporal image includes concatenating the plurality of vectors to form the temporal image.

5. The system of claim 1 , wherein to transform the plurality of time sequenced images to the plurality of vectors, the processing circuitry is configured to use a classifier to obtain each of the plurality of vectors from a respective plurality of images.

6. The system of claim 1 , wherein identifying the semantic signal based on application of the convolutional neural network to the temporal image, the processing circuitry is configured to use a blinking classifier.

7. The system of claim 1 , wherein identifying the semantic signal based on application of the convolutional neural network to the temporal image, the processing circuitry is configured to use a braking classifier.

8. The system of claim 1 , wherein identifying the semantic signal based on application of the convolutional neural network to the temporal image, the processing circuitry is configured to use a braking classifier on a pair of vectors of the plurality of vectors, and to use a blinking classifier on the entire temporal image.

9. The system of claim 8 , wherein the braking classifier is trained for a plurality of braking signals.

10. The system of claim 1 , wherein:
the image capture device is mounted on a vehicle;
the semantic signal indicates a changed path condition for the vehicle; and
the instructions further configure the processing circuitry to:
identify a maneuver for the vehicle in response to the changed path condition; and
send a vehicle control signal for execution of the maneuver.

11. The system of claim 10 , further including a vehicular control device to receive the control signal and execute the maneuver for the vehicle.

12. An autonomous navigation semantic signal method comprising:
mapping each of a plurality of time sequenced images received from an image capture device to each of a plurality of vectors; and
identifying a semantic signal based on applying a convolutional neural network to a temporal image, the temporal image generated based on the plurality of vectors;
wherein each of the plurality of vectors includes a row vector of the same width as each of the plurality of time sequenced images.

13. The method of claim 12 , further including:
capturing the plurality of time sequenced images; and
associating a unique image capture time with each of the captured plurality of time sequenced images.

14. The method of claim 12 , wherein mapping each of the plurality of time sequenced images to each of a plurality of vectors includes calculating a column value for each of a plurality of columns within each of the plurality of time sequenced images.

15. The method of claim 14 , wherein calculating the column value includes at least one of calculating a mean value, a median value, or a maximal value for each of a plurality of columns within each of the plurality of time sequenced images.

16. The method of claim 12 , wherein the temporal image includes a concatenation of the plurality of vectors to form the temporal image.

17. The method of claim 12 , wherein mapping each of the plurality of time sequenced images to the plurality of vectors includes using a classifier to obtain each of the plurality of vectors from a respective plurality of images.

18. The method of claim 12 , wherein identifying the semantic signal based on application of the convolutional neural network to the temporal image includes using a blinking classifier.

19. The method of claim 12 , wherein identifying the semantic signal based on application of the convolutional neural network to the temporal image includes using a braking classifier.

20. The method of claim 12 , wherein identifying the semantic signal based on application of the convolutional neural network to the temporal image includes using a braking classifier on a pair of vectors of the plurality of vectors and using a blinking classifier on the entire temporal image.

21. The method of claim 12 , further including:
identifying a vehicular maneuver based on the semantic signal; and
sending a control signal to execute the vehicular maneuver to a vehicular control device.

22. A non-transitory computer program product that stores instructions that, once executed by a computerized system, cause the computerized system to perform operations comprising:
mapping each of a plurality of time sequenced images received from an image capture device to each of a plurality of vectors;
identifying a semantic signal based on applying a convolutional neural network to a temporal image, the temporal image generated based on the plurality of vectors;
wherein each of the plurality of vectors includes a row vector of the same width as each of the plurality of time sequenced images.

23. The non-transitory computer program product of claim 22 , wherein to transform the plurality of time sequenced images to the plurality of vectors includes using a classifier to obtain each of the plurality of vectors from a respective plurality of images.

24. The non-transitory computer program product of claim 22 , wherein identifying the semantic signal based on application of the convolutional neural network to the temporal image includes using a braking classifier on a pair of vectors of the plurality of vectors, and to use a blinking classifier on the entire temporal image.

25. The non-transitory computer program product of claim 22 , further including:
identifying a vehicular maneuver based on the semantic signal; and
sending a control signal to execute the vehicular maneuver to a vehicular control device.

Assignees

• Mobileye Vision Technologies Ltd

Inventors

• Malach Eran
• Shambik Yaakov
• Bentolila Jacob
• Geller Idan

Classifications

• G06N3/0464

  Convolutional networks [CNN, ConvNet] · CPC title

• G06N3/045

  Combinations of networks · CPC title

• G06N3/09

  Supervised learning · CPC title

• B60R2300/8086

  for vehicle path indication · CPC title

• G05D1/243

  Means capturing signals occurring naturally from the environment, e.g. ambient optical, acoustic, gravitational or magnetic signals (using passive navigation aids external to the vehicle G05D1/244; using signals from positioning sensors located off-board the vehicle G05D1/249) · CPC title