Vehicle localization using cameras

Patent metadata

Field	Value
Publication number	US-11216972-B2
Application number	US-201916545862-A
Country	US
Kind code	B2
Filing date	Aug 20, 2019
Priority date	Mar 14, 2017
Publication date	Jan 4, 2022
Grant date	Jan 4, 2022

Abstract

According to one embodiment, a system for determining a position of a vehicle includes an image sensor, a top-down view component, a comparison component, and a location component. The image sensor obtains an image of an environment near a vehicle. The top-down view component is configured to generate a top-down view of a ground surface based on the image of the environment. The comparison component is configured to compare the top-down image with a map, the map comprising a top-down light LIDAR intensity map or a vector-based semantic map. The location component is configured to determine a location of the vehicle on the map based on the comparison.

First claim

What is claimed is:

1. A method comprising:
receiving an image from a camera of a vehicle;
generating a top-down view of a ground plane surrounding the vehicle by projecting the image to approximate the camera facing downward from the vehicle;
extracting a comparison image from a vector-based semantic map by generating a synthetic ground plane, wherein the synthetic ground plane comprises dark pixels for a road surface and light or bright pixels for road markings;
comparing the top-down view of the ground plane with the comparison image comprising the synthetic ground plane; and
determining a location of the vehicle on the vector-based semantic map based on the comparison of the top-down view of the ground plane and the comparison image.

2. The method of claim 1 , further comprising segmenting the ground plane from the image to determine segmented ground plane pixels by analyzing a stereo pair of images providing points in three-dimensional space and applying random sample consensus to determine a set of points in the image for the ground plane.

3. The method of claim 2 , wherein applying the random sample consensus comprises randomly selecting hypothesis points pertaining to the ground plane and evaluating the hypothesis points to determine the set of points in the image making up the ground plane.

4. The method of claim 1 , wherein generating the synthetic ground plane of the comparison image comprises generating an approximation of an aerial view of a roadway comprising lane markings.

5. The method of claim 1 , wherein comparing the top-down view of the ground plane with the comparison image comprising the synthetic ground plane comprises calculating a score for each of a plurality of hypothesis relative positions using mutual information.

6. The method of claim 1 , wherein comparing the top-down view of the ground plane with the comparison image comprising the synthetic ground plane comprises comparing using one or more of:
a mutual information algorithm; or
a best-fit algorithm.

7. The method of claim 1 , wherein comparing the top-down view of the ground plane with the comparison image comprising the synthetic ground plane comprises calculating a score for a plurality of relative positions and selecting a relative position of the vehicle.

8. A system for localizing a vehicle, the system comprising:
an image sensor to obtain an image of an environment near the vehicle; and
a processor that is programmable to execute instructions stored in non-transitory computer readable storage media, the instructions comprising:
receiving an image from a camera of a vehicle;
generating a top-down view of a ground plane surrounding the vehicle by projecting the image to approximate the camera facing downward from the vehicle;
extracting a comparison image from a vector-based semantic map by generating a synthetic ground plane, wherein the synthetic ground plane comprises dark pixels for a road surface and light or bright pixels for road markings;
comparing the top-down view of the ground plane with the comparison image comprising the synthetic ground plane; and
determining a location of the vehicle on the vector-based semantic map based on the comparison of the top-down view of the ground plane and the comparison image.

9. The system of claim 8 , wherein the instructions further comprise segmenting the ground plane from the image to determine segmented ground plane pixels by analyzing a stereo pair of images providing points in three-dimensional space and applying random sample consensus to determine a set of points in the image for the ground plane.

10. The system of claim 9 , wherein the instructions are such that applying the random sample consensus comprises randomly selecting hypothesis points pertaining to the ground plane and evaluating the hypothesis points to determine the set of points in the image making up the ground plane.

11. The system of claim 8 , wherein the instructions are such that generating the synthetic ground plane of the comparison image comprises generating an approximation of an aerial view of a roadway comprising lane markings.

12. The system of claim 8 , wherein the instructions are such that comparing the top-down view of the ground plane with the comparison image comprising the synthetic ground plane comprises calculating a score for each of a plurality of hypothesis relative positions using mutual information.

13. The system of claim 8 , wherein the instructions are such that comparing the top-down view of the ground plane with the comparison image comprising the synthetic ground plane comprises comparing using one or more of:
a mutual information algorithm; or
a best-fit algorithm.

14. The system of claim 8 , wherein the instructions further comprise calculating a score for a plurality of relative positions and selecting a relative position from the plurality of relative positions as the location of the vehicle on the vector-based semantic map.

15. A non-transitory computer readable storage media storing instructions to be executed by one or more processors, the instructions comprising:
receiving an image from a camera of a vehicle;
generating a top-down view of a ground plane surrounding the vehicle by projecting the image to approximate the camera facing downward from the vehicle;
extracting a comparison image from a vector-based semantic map by generating a synthetic ground plane, wherein the synthetic ground plane comprises dark pixels for a road surface and light or bright pixels for road markings;
comparing the top-down view of the ground plane with the comparison image comprising the synthetic ground plane; and
determining a location of the vehicle on the vector-based semantic map based on the comparison of the top-down view of the ground plane and the comparison image.

16. The non-transitory computer readable storage media of claim 15 , wherein the instructions further comprise segmenting the ground plane from the image to determine segmented ground plane pixels by analyzing a stereo pair of images providing points in three-dimensional space and applying random sample consensus to determine a set of points in the image for the ground plane.

17. The non-transitory computer readable storage media of claim 15 , wherein the instructions are such that generating the synthetic ground plane of the comparison image comprises generating an approximation of an aerial view of a roadway comprising lane markings.

18. The non-transitory computer readable storage media of claim 15 , wherein the instructions are such that comparing the top-down view of the ground plane with the comparison image comprising the synthetic ground plane comprises comparing using one or more of:
a mutual information algorithm; or
a best-fit algorithm.

Assignees

• Ford Global Tech Llc

Inventors

• Houts Sarah
• Gurghian Alexandru Mihai
• Nariyambut Murali Vidya
• Smith Tory

Classifications

• G01C21/30 Primary

  Map- or contour-matching · CPC title

• G06V20/588

  Recognition of the road, e.g. of lane markings; Recognition of the vehicle driving pattern in relation to the road · CPC title

• G06T7/11

  Region-based segmentation · CPC title

• G01S17/86

  Combinations of lidar systems with systems other than lidar, radar or sonar, e.g. with direction finders · CPC title

• G06T2207/30256

  Lane; Road marking · CPC title