System and method of determining a curve

Patent metadata

Field	Value
Publication number	US-11688183-B2
Application number	US-202217592264-A
Country	US
Kind code	B2
Filing date	Feb 3, 2022
Priority date	Oct 6, 2017
Publication date	Jun 27, 2023
Grant date	Jun 27, 2023

Abstract

A neural network is configured to process image data captured by a vehicle-mounted camera. The neural network includes common processing layers (a trunk) and separate, parallelizable processing layers (branches). An object detection branch of the neural network is trained to detect objects that may be visible from the vehicle-mounted camera, such as cars, trucks, and traffic signs. A curve determination branch is trained to detect and parameterize salient curves, such as lane boundaries and road boundaries. The curve determination branch itself is configured with a trunk and branch architecture, having both common and separate processing layers. A first branch computes a likelihood that a curve is present in a given location of the image data and a second branch further localizes the curve within the given location if such a curve is present. Training of the different branches of the neural network may be decoupled.

First claim

What is claimed is:

1. A method, comprising:
receiving, from a camera coupled to a vehicle, visual data representing an image captured by the camera;
processing the visual data with a neural network, the neural network comprising:
a curve determination branch,
an object detection branch, and
a common trunk, the common trunk configured to perform one or more processing operations on the visual data to produce a common trunk output;

processing the common trunk output with the curve determination branch of the neural network to produce a curve determination data; and
determining a location of a traffic boundary based on the curve determination data.

2. The method of claim 1 , wherein the traffic boundary is a lane boundary or a road boundary.

3. The method of claim 1 , wherein the traffic boundary is a lane boundary for an adjacent lane relative to the vehicle, and the lane boundary is entirely occluded by other cars in the captured image.

4. The method of claim 1 , further comprising processing the common trunk output with the object detection branch of the neural network to produce object detection data.

5. The method of claim 4 , further comprising determining a location of an object based on the object detection data.

6. The method of claim 5 , wherein the object is a second vehicle, a traffic sign, or a landmark.

7. The method of claim 4 , wherein processing by the curve determination branch and processing by the object detection branch are performed effectively in parallel such that the curve determination data and the object detection data do not depend on each other.

8. The method of claim 1 , further comprising:
providing training data representing a plurality of images captured by one or more cameras coupled to respective vehicles to the neural network; and
training the curve determination branch of the neural network based on the training data.

9. The method of claim 8 , wherein the curve determination branch comprises a first branch and a second branch, wherein the first branch is configured to compute a likelihood that a curve is present in a predetermined portion of the image data, wherein the second branch is configured to compute regression point data, and wherein the location of the traffic boundary is determined based on the likelihood and the regression point data.

10. The method of claim 9 , wherein training the curve determination branch of the neural network further comprises:
setting values of computed regression point data for a given image to zero so that regression guesses associated with lanes that are not present do not flow backwards through the network in a backward pass.

11. The method of claim 8 , further comprising providing, by a labeling system and for a given image represented in the training data prior to providing the training data to the neural network, a label of a training layer indicating that a curve might or might not be present in a given region of the given image and that the neural network should not be penalized regardless of a curve detection data output by the neural network in association with the given region.

12. The method of claim 1 , wherein the image was captured by the camera when the vehicle was at a first vehicle location on a road, and further comprising:
receiving second visual data representing a second image captured by the camera when the vehicle was at a second vehicle location on the road;
determining a second location of the traffic boundary based on the second visual data; and
mapping the traffic boundary into a world-based coordinate frame, wherein the mapping is based on camera calibration parameters, the first vehicle location, the determined location of the traffic boundary, the second vehicle location, and the determined second location of the traffic boundary.

13. An apparatus, comprising:
a memory;
at least one processor operatively coupled to the memory; and
a set of instructions stored on the memory and configured to be executed by the at least one processor to:
receive, from a camera coupled to a vehicle, visual data representing an image captured by the camera;
process the visual data with a neural network, the neural network comprising:
a curve determination branch,
an object detection branch, and
a common trunk, wherein processing operations associated with the common trunk on the visual data produce a common trunk output;

process the common trunk output with the curve determination branch of the neural network to produce a curve determination data; and
determine a location of a traffic boundary based on the curve determination data.

14. The apparatus of claim 13 , wherein the at least one processor is further configured to:
process the common trunk output with the object detection branch of the neural network to produce object detection data.

15. The apparatus of claim 14 , wherein processing by the curve determination branch and processing by the object detection branch are performed effectively in parallel such that the curve determination data and the object detection data do not depend on each other.

16. The apparatus of claim 13 , wherein the at least one processor is further configured to:
provide training data representing a plurality of images captured by one or more cameras coupled to respective vehicles to the neural network; and
train the curve determination branch of the neural network based on the training data.

17. A computer program product, the computer program product comprising a non-transitory computer-readable medium having program code recorded thereon, the program code comprising program code executable by one or more processors to:
receive, from a camera coupled to a vehicle, visual data representing an image captured by the camera;
process the visual data with a neural network, the neural network comprising:
a curve determination branch,
an object detection branch, and
a common trunk, wherein processing operations associated with the common trunk on the visual data produce a common trunk output;

process the common trunk output with the curve determination branch of the neural network to produce a curve determination data; and
determine a location of a traffic boundary based on the curve determination data.

18. The computer program product of claim 17 , wherein the program code is further executable by the one or more processors to:
process the common trunk output with the object detection branch of the neural network to produce object detection data.

19. The computer program product of claim 18 , wherein processing by the curve determination branch and processing by the object detection branch are performed effectively in parallel such that the curve determination data and the object detection data do not depend on each other.

20. The computer program product of claim 17 , wherein the program code is further executable by the one or more processors to:
provide training data representing a plurality of images captured by one or more cameras coupled to respective vehicles to the neural network; and
train the curve determination branch of the neural network based on the training data.

Assignees

• Netradyne Inc

Inventors

• Campos Michael
• Annapureddy Venkata Sreekanta Reddy
• Julian David Jonathan
• Taranalli Veeresh
• Cox Jonathan Albert
• Yedla Arvind

Classifications

• G06N3/0464

  Convolutional networks [CNN, ConvNet] · CPC title

• G06N3/09

  Supervised learning · CPC title

• G06V20/588 Primary

  Recognition of the road, e.g. of lane markings; Recognition of the vehicle driving pattern in relation to the road · CPC title

• G06F18/217

  Validation; Performance evaluation; Active pattern learning techniques · CPC title

• G06N3/08

  Learning methods · CPC title