System and method of determining a curve

Patent metadata

Field	Value
Publication number	US-11244174-B2
Application number	US-201816753606-A
Country	US
Kind code	B2
Filing date	Oct 5, 2018
Priority date	Oct 6, 2017
Publication date	Feb 8, 2022
Grant date	Feb 8, 2022

Abstract

A neural network is configured to process image data captured by a vehicle-mounted camera. The neural network includes common processing layers (a trunk) and separate, parallelizable processing layers (branches). An object detection branch of the neural network is trained to detect objects that may be visible from the vehicle-mounted camera, such as cars, trucks, and traffic signs. A curve determination branch is trained to detect and parameterize salient curves, such as lane boundaries and road boundaries. The curve determination branch itself is configured with a trunk and branch architecture, having both common and separate processing layers. A first branch computes a likelihood that a curve is present in a given location of the image data and a second branch further localizes the curve within the given location if such a curve is present. Training of the different branches of the neural network may be decoupled.

First claim

What is claimed is:

1. A method, comprising:
receiving, from a camera coupled to a vehicle, visual data representing an image captured by the camera;
processing the visual data with a trained neural network to produce:
first curve detection data representing a likelihood that a first curve is present in a first predetermined portion of the image, wherein the first curve is a lane boundary or road boundary; and
first regression point data indicating a point at which the first curve would, if present in the first predetermined portion of the image, intersect a first boundary of the first predetermined portion of the image; wherein the trained neural network comprises:
a first branch including a plurality of nodes collectively configured and trained to output curve detection data associated with received images; and
a second branch including a plurality of nodes collectively configured and trained to output regression point data associated with the received images, wherein processing by nodes in the first branch and processing by nodes in the second branches are performed effectively in parallel such that respective outputs produced by the first and second branches do not depend on each other; and
a common trunk including a plurality of nodes collectively configured and trained to perform one or more image processing operations on the received images prior to processing by the first and second branches; and

determining a parameter of the first curve dependent on the first curve detection data and the first regression point data.

2. The method of claim 1 , wherein:
the camera is a forward-facing camera mounted or attached to a windshield of the vehicle; and
determining the parameter of the first curve comprises determining a location at which the lane boundary or road boundary intersects a bottom edge or a side edge of the image.

3. The method of claim 1 , wherein the first boundary of the first predetermined portion of the image is a straight line.

4. The method of claim 1 , wherein:
the first predetermined portion of the image is one of a plurality of distinct predetermined portions of the image, each comprising a rectangular portion of the image bounded on one side by a bottom edge or a side edge of the image;
the first regression point data indicates the point at which the first curve would, if present in the first predetermined portion of the image, intersect the bottom edge or side edge of the image with which the first predetermined portion of the image is bounded; and
the method further comprises, for each of the plurality of distinct predetermined portions of the image other than the first predetermined portion of the image:
processing the visual data with the trained neural network to produce;
respective curve detection data representing a likelihood that the first curve is present in the predetermined portion of the image; and
respective regression point data indicating a point at which the first curve would, if present in the predetermined portion of the image, intersect the bottom edge or side edge of the image with which the predetermined portion of the image is bounded; and

determining the parameter of the first curve is further dependent on the respective curve detection data and the respective regression point data produced for each of the plurality of distinct predetermined portions of the image other than the first predetermined portion of the image.

5. The method of claim 1 , wherein a width of the first boundary of the first predetermined portion of the image is less than a predetermined expected spacing between lane boundaries.

6. The method of claim 1 , wherein processing the visual data with a trained neural network to produce the first curve detection data comprises determining whether a tangent line extending from an end of the first curve would pass through the first predetermined portion of the image.

7. The method of claim 1 , wherein processing the visual data with the trained neural network further produces second regression point data indicating a point at which the first curve would, if present in the first predetermined portion of the image, intersect a second boundary of the first predetermined portion of the image corresponding to a side of the first predetermined portion of the image opposite a side of the first predetermined portion of the image corresponding to the first boundary.

8. The method of claim 1 , wherein processing the visual data with the trained neural network further produces an estimate of a vanishing point of the image.

9. The method of claim 1 , wherein:
the first branch of the neural network comprises multiple sub-branches, each of which is configured and trained to output curve detection data representing a likelihood that a given curve representing a lane boundary or road boundary is present in a respective one of the plurality of predetermined portions of the image; and
the second branch of the neural network comprises multiple sub-branches, each of which is configured and trained to output regression point data indicating a point at which the given curve would, if present in a respective one of the plurality of predetermined portions of the image, intersect a given boundary of the predetermined portion of the image.

10. The method of claim 1 , wherein the second branch of the neural network further comprises a plurality of nodes collectively configured and trained to output data representing relative positions of one or more control points of a given curve between a point defined by regression point data associated with the given curve and an estimated vanishing point for the given curve.

11. A method comprising:
receiving, from a camera coupled to a vehicle, visual data representing an image captured by the camera;
processing the visual data with a trained neural network to produce:
first curve detection data representing a likelihood that a first curve is present in a first predetermined portion of the image, wherein the first curve is a lane boundary or road boundary; and
first regression point data indicating a point at which the first curve would, if present in the first predetermined portion of the image, intersect a boundary of the first predetermined portion of the image; wherein training the neural network comprises:
providing training data representing a plurality of images captured by one or more cameras coupled to respective vehicles to the neural network;
training a first branch of the neural network to output curve detection data dependent on curves present in the images represented in the training data and dependent on images represented in the training data in which no curves are present; and
training a second branch of the neural network to output regression point data dependent on the curves present in the images represented in the training data and not dependent on the images represented in the training data in which no curves are present, wherein the training of the first branch of the neural network is decoupled from the training of the second branch of the neural network; and

determining a parameter of the first curve dependent on the first curve detection data and the first regression point data.

12. The method of claim 11 , further comprising setting values of regression point data for a given image represented in the training data and associated with a given curve to zero in response to determining that the given curve is not present in the given image.

13. The method of claim 11 , further comprising providing, by a labeling system and for a given image represented in the training data prior to providing the training data to the neural network,

Assignees

• Netradyne Inc

Inventors

• Campos Michael
• Annapureddy Venkata Sreekanta Reddy
• Julian David Jonathan
• Taranalli Veeresh
• Cox Jonathan Albert
• Yedla Arvind

Classifications

• G06V10/752

  Contour matching · CPC title

• G06V10/82

  using neural networks · CPC title

• H04N23/54

  Mounting of pick-up tubes, electronic image sensors, deviation or focusing coils · CPC title

• G06F18/214

  Generating training patterns; Bootstrap methods, e.g. bagging or boosting · CPC title

• G06F18/217

  Validation; Performance evaluation; Active pattern learning techniques · CPC title