Systems and methods for detecting intersection crossing events using full frame classification techniques

Patent metadata

Field	Value
Publication number	US-12080159-B2
Application number	US-202218288157-A
Country	US
Kind code	B2
Filing date	Jun 8, 2022
Priority date	Jun 9, 2021
Publication date	Sep 3, 2024
Grant date	Sep 3, 2024

Abstract

Disclosed herein are systems and methods for classifying traffic indicators to detect intersection crossings by a vehicle. A computing device can receive a sequence of frames captured by a capture device mounted to the vehicle, and generate an intersection status data structure for each frame in the sequence of frames using a full-frame classification model. The computing device can also classify a set of features detected in each frame using an object detection model, and detect an intersection crossing event based on the intersection status data structure of each frame and the classification of each of the set of features detected in each frame. The systems and methods described herein improve upon existing traffic light classification models by using a full-frame intersection classification model in connection with traditional object tracking techniques, thereby reducing false-positive crossing-event detection and improving the accuracy of classification in a variety of environmental conditions.

First claim

What is claimed is:

1. A method for classifying traffic indicators to detect intersection crossings by a vehicle, the method comprising:
receiving, by one or more processors coupled to a memory, a sequence of frames captured by an image capture device mounted to the vehicle;
generating, by the one or more processors, an intersection status data structure for each frame in the sequence of frames using a full-frame classification model trained on a labeled dataset of frames depicting roadways, wherein each label of the labeled dataset used to train the full-frame classification model is applied to a full image frame;
classifying, by the one or more processors, using an object detection model, a set of features detected in each frame of the sequence of frames; and
detecting, by the one or more processors, an intersection crossing event based on the intersection status data structure of each frame and the classification of each of the set of features detected in each frame.

2. The method of claim 1 , wherein the full-frame classification model comprises at least five outputs, and wherein generating the intersection status data structure for each frame further comprises generating, by the one or more processors, the intersection status data structure to include at least five output values for each frame by providing each frame as input to the full-frame classification model.

3. The method of claim 2 , wherein the at least five outputs comprise an intersection type output, a crossing type output, a left-turn light output, a right-turn light output, and a forward light output.

4. The method of claim 1 , further comprising transmitting, by the one or more processors, a warning signal to an alert device responsive to detecting, based on the intersection status data structure, that the vehicle is approaching an intersection.

5. The method of claim 1 , further comprising:
determining, by the one or more processors, based on the intersection status data structure, that the intersection crossing event is a false positive; and
suppressing, by the one or more processors, a warning signal responsive to determining that the intersection crossing event is a false positive.

6. The method of claim 5 , wherein determining that the intersection crossing event is a false positive further comprises determining, based on the intersection status data structure, that the detected intersection crossing event corresponds to an underpass.

7. The method of claim 1 , wherein detecting the intersection crossing event is further based on data captured by a global-positioning system receiver.

8. The method of claim 1 , wherein detecting the intersection crossing event further comprises:
detecting, by the one or more processors, a potential intersection crossing event based on the classification of each of the set of features detected in each frame; and
validating, by the one or more processors, the potential intersection crossing event using the intersection status data structure.

9. The method of claim 8 , wherein the sequence of frames comprises at least three frames, and wherein detecting the potential intersection crossing event comprises determining the classification of a feature detected in the at least three frames remains constant.

10. The method of claim 1 , wherein classifying the set of features detected in each frame of the sequence of frames further comprises:
detecting, by the one or more processors, one or more bounding regions that each correspond to a respective feature in each frame; and
classifying, by the one or more processors, the set of features by providing the one or more bounding regions of each frame as input to the object detection model.

11. A system for classifying traffic indicators to detect driving violations by a vehicle, the system comprising one or more processors coupled to a non-transitory memory, the one or more processors configured to:
receive a sequence of frames captured by an image capture device mounted to the vehicle;
detect, using an object detection model, a potential intersection crossing event based on a set of features detected in each frame of the sequence of frames;
generate an intersection status data structure for each frame in the sequence of frames, using a full-frame classification model trained on a labeled dataset of frames depicting roadways, wherein each label of the labeled dataset used to train the full-frame classification model is applied to a full image frame; and
validate the potential intersection crossing event based on the intersection status data structure generated using the full-frame classification model.

12. The system of claim 11 , wherein the one or more processors are further configured to detect the potential intersection crossing event by performing operations comprising:
classifying a feature of the set of features detected in the sequence of frames as a traffic light;
determining that the vehicle has passed the feature detected in the sequence of frames; and
detecting the potential intersection crossing event responsive to determining that the vehicle has passed the feature classified as the traffic light.

13. The system of claim 12 , wherein the one or more processors are further configured to:
generate the intersection status data structure to include an indication that the vehicle is approaching an intersection or in the intersection when the traffic light is red; and
determine a severity of the potential intersection crossing event based on the indication that the vehicle is approaching the intersection or in the intersection when the traffic light is red.

14. The system of claim 11 , wherein the one or more processors are further configured to transmit a warning signal to an alert device responsive to validating the potential intersection crossing event.

15. The system of claim 11 , wherein the one or more processors are further configured to determine, based on a second intersection status data structure generated by the full-frame classification model using a second sequence of frames, that a second potential intersection crossing event is a false positive.

16. The system of claim 15 , wherein the one or more processors are further configured to
determine that the second potential intersection crossing event is a false positive by performing operations comprising determining, based on the second intersection status data structure, that the second potential intersection crossing event corresponds to at least one of an underpass, a toll booth, a warning signal, or a railroad crossing.

17. The system of claim 11 , wherein the one or more processors are further configured to detect the potential intersection crossing event further based on data captured by a global-positioning system receiver.

18. The system of claim 11 , wherein the one or more processors are further configured to detect the potential intersection crossing event based on a classification of each of the set of features across each frame in the sequence of frames and a finite-state machine.

19. The system of claim 18 , wherein the sequence of frames comprises at least three frames, and wherein the one or more processors are further configured to detect the potential intersection crossing event based on a transition matrix of the finite-state machine.

20. A method for classifying road images captured by an image capture device mounted to a vehicle, comprising:
receiving, by one or more processors coupled to a memory, a sequence of frames captured by the capture device mounted to th

Assignees

• Netradyne Inc

Inventors

• Debnath Biswarup
• Yanamala Suresh Babu
• Yedla Arvind
• Annapureddy Venkata Sreekanta Reddy

Classifications

• G08G1/0133

  for classifying traffic situation · CPC title

• G06V20/44

  Event detection · CPC title

• G06V10/764

  using classification, e.g. of video objects · CPC title

• G06V10/44

  Local feature extraction by analysis of parts of the pattern, e.g. by detecting edges, contours, loops, corners, strokes or intersections; Connectivity analysis, e.g. of connected components · CPC title

• G06V20/584

  of vehicle lights or traffic lights · CPC title