Vehicle localization using cameras

Patent metadata

Field	Value
Publication number	US-10430968-B2
Application number	US-201715458611-A
Country	US
Kind code	B2
Filing date	Mar 14, 2017
Priority date	Mar 14, 2017
Publication date	Oct 1, 2019
Grant date	Oct 1, 2019

Abstract

According to one embodiment, a system for determining a position of a vehicle includes an image sensor, a top-down view component, a comparison component, and a location component. The image sensor obtains an image of an environment near a vehicle. The top-down view component is configured to generate a top-down view of a ground surface based on the image of the environment. The comparison component is configured to compare the top-down image with a map, the map comprising a top-down light LIDAR intensity map or a vector-based semantic map. The location component is configured to determine a location of the vehicle on the map based on the comparison.

First claim

What is claimed is:

1. A method for localizing a vehicle, the method comprising:
obtaining an image of an environment of the vehicle;
segmenting a ground plane from the image to determine segmented ground plane pixels, wherein the segmenting comprises applying random sample consensus and determining a set of points in the image for the ground plane;
generating a top-down view of the ground plane by projecting the segmented ground plane pixels to approximate a view of a camera facing downward from the vehicle;
comparing the top-down view with a map, the map comprising a top-down light detection and ranging (LIDAR) intensity map or a vector-based semantic map; and
determining a location of the vehicle on the map based on the comparison of the top-down view of the ground plane and the map.

2. The method of claim 1 , wherein segmenting the ground plane from the image comprises analyzing a stereo pair of images providing points in three-dimensional space and applying the random sample consensus by randomly selecting hypothesis points pertaining to the ground plane and evaluating the hypothesis points to determine the set of points in the image that are the good fit for the ground plane.

3. The method of claim 1 , wherein obtaining the image comprises capturing the image using a camera of the vehicle.

4. The method of claim 1 , wherein the map comprises the vector-based semantic map and the method further comprises generating a synthetic image.

5. The method of claim 4 , wherein comparing the top-down view with the map comprises comparing the top-down view with the synthetic image.

6. The method of claim 4 , wherein generating the synthetic image comprises generating a synthetic image comprising dark pixels for a road surface and light or bright pixels for road markings.

7. The method of claim 1 , wherein comparing the top-down view with the map comprises comparing using one or more of:
a mutual information algorithm; or
a best-fit algorithm.

8. The method of claim 1 , wherein comparing the top-down view with the map comprises calculating a score for a plurality of relative positions and selecting a relative position.

9. A system for localizing a vehicle, the system comprising:
an image sensor to obtain an image of an environment near the vehicle; and
a processor that is programmable to execute instructions stored in non-transitory computer readable storage media, the instructions comprising:
receiving the image of the environment of the vehicle from the image sensor;
segmenting a ground plane from the image to determine segmented ground plane pixels, wherein the segmenting comprises applying random sample consensus and determining a set of points in the image for the ground plane;
generating a top-down view of the ground plane by projecting the segmented ground plane pixels to approximate a view of a camera facing downward from the vehicle;
comparing the top-down view with a map, the map comprising a top-down light detection and ranging (LIDAR) intensity map or a vector-based semantic map; and
determining a location of the vehicle on the map based on the comparison of the top-down view of the ground plane and the map.

10. The system of claim 9 , wherein segmenting the ground plane from the image comprises analyzing a stereo pair of images providing points in three-dimensional space and applying the random sample consensus by randomly selecting hypothesis points pertaining to the ground plane and evaluating the hypothesis points to determine the set of points in the image that are the good fit for the ground plane.

11. The system of claim 9 , wherein the image sensor is a monocular camera mounted on the vehicle.

12. The system of claim 9 , wherein the map comprises the vector-based semantic map and the instructions further comprise generating a synthetic image based on the vector-based semantic map.

13. The system of claim 12 , wherein comparing the top-down view with the map comprises comparing the top-down view with the synthetic image.

14. The system of claim 12 , wherein generating the synthetic image comprises generating a synthetic image comprising dark pixels for a road surface and light or bright pixels for road markings.

15. The system of claim 9 , wherein comparing the top-down view with the map comprises comparing using one or more of:
a mutual information algorithm; or
a best-fit algorithm.

16. The system of claim 9 , wherein the instructions further comprise calculating a score for a plurality of relative positions and selecting a relative position from the plurality of relative positions as the location of the vehicle on the map.

17. Non-transitory computer readable storage media storing instructions that, when executed by one or more processors, cause the one or more processors to:
obtain an image of an environment near a vehicle;
segment a ground plane from the image to determine segmented ground plane pixels, wherein the segmenting comprises applying random sample consensus and determining a set of points in the image for the ground plane;
generate a top-down view of the ground plane by projecting the segmented ground plane pixels to approximate a view of a camera facing downward from the vehicle;
compare the top-down view with a map, the map comprising a top-down light detection and ranging (LIDAR) intensity map or a vector-based semantic map; and
determine a location of the vehicle on the map based on the comparison of the top-down view of the ground plane and the map.

18. The non-transitory computer readable storage media of claim 17 , wherein the instructions cause the one or more processors to segment the ground plane from the image by analyzing a stereo pair of images providing points in three-dimensional space and applying the random sample consensus by randomly selecting hypothesis points pertaining to the ground plane and evaluating the hypothesis points to determine the set of points in the image that are the good fit for the ground plane.

19. The non-transitory computer readable storage media of claim 17 , wherein the map comprises the vector-based semantic map and wherein the instructions further cause the one or more processors to:
generate a synthetic image comprising dark pixels for a road surface and light or bright pixels for road markings based on the vector-based semantic map; and
compare the top-down view with the map by comparing the top-down view with the synthetic image.

20. The non-transitory computer readable storage media of claim 17 , wherein the instructions cause the processors to compare the top-down view with the map using one or more of:
a mutual information algorithm; or
a best-fit algorithm.

Assignees

• Ford Global Tech Llc

Inventors

• Houts Sarah
• Gurghian Alexandru Mihai
• Nariyambut Murali Vidya
• Smith Tory

Classifications

• G01S17/90

  using synthetic aperture techniques · CPC title

• G01S17/86

  Combinations of lidar systems with systems other than lidar, radar or sonar, e.g. with direction finders · CPC title

• G01S17/931

  of land vehicles · CPC title

• G06T7/11

  Region-based segmentation · CPC title

• G06T7/74 Primary

  involving reference images or patches · CPC title