Stereo camera-based detection of objects proximate to a vehicle

Patent metadata

Field	Value
Publication number	US-10776636-B2
Application number	US-201615390409-A
Country	US
Kind code	B2
Filing date	Dec 23, 2016
Priority date	Dec 29, 2015
Publication date	Sep 15, 2020
Grant date	Sep 15, 2020

Abstract

A system is disclosed. The system includes one or more processors, and a memory including instructions, which when executed by the one or more processors, cause the one or more processors to perform a method. The method includes receiving first image data from a first image sensor mounted on a vehicle and second image data from a second image sensor mounted on the vehicle, the vehicle including an element configured to open into a space external to the vehicle; generating a depth map from a comparison of the first image data and the second image data, the depth map including an object; in accordance with a determination that the object, in the depth map, is within the space into which the element is configured to open, determining that the object will interfere with opening of the element into the space; and in accordance with a determination that the object, in the depth map, is not within the space into which the element is configured to open, determining that the object will not interfere with opening of the element into the space.

First claim

The invention claimed is:

1. A system comprising:
one or more processors; and
a memory including instructions, which when executed by the one or more processors, cause the one or more processors to perform a method comprising:
receiving first image data from a first image sensor mounted on a vehicle and second image data from a second image sensor mounted on the vehicle, the vehicle including an element configured to open into a space external to the vehicle;
generating a depth map from a comparison of the first image data and the second image data, the depth map including an object;
generating a confidence value associated with the depth map; and comparing the confidence value to a predetermined threshold;
in accordance with the confidence value being above the predetermined threshold and a determination that the object, in the depth map, is within the space into which the element is configured to open, determining that the object will interfere with opening of the element into the space; and
in accordance with the confidence value being above the predetermined threshold and a determination that the object, in the depth map, is not within the space into which the element is configured to open, determining that the object will not interfere with opening of the element into the space;
in accordance with the confidence value being below the predetermined threshold,

locating the object using ultrasonic detection;
in accordance with a determination that the object, as located using ultrasonic detection, is within the space into which the element is configured to open, determining that the object will interfere with opening of the element into the space; and
in accordance with a determination that the object, as located using ultrasonic detection, is not within the space into which the element is configured to open, generating a warning notification and determining that the object will not interfere with opening of the element into the space.

2. The system of claim 1 , wherein the element comprises a door configured to open into an interaction space external to the vehicle.

3. The system of claim 2 , wherein the first image sensor is mounted on the door, and the second image sensor is mounted on an additional door of the vehicle.

4. The system of claim 1 , wherein generating the depth map comprises:
in accordance with a determination that the vehicle will likely stop within a time threshold, generating the depth map; and
in accordance with a determination that the vehicle will not likely stop within the time threshold, forgoing generating the depth map.

5. The system of claim 1 , wherein generating the depth map comprises:
in accordance with a determination that the vehicle is stopped, generating the depth map; and
in accordance with a determination that the vehicle is not stopped, forgoing generating the depth map.

6. The system of claim 1 , wherein the first image sensor is mounted a first distance from the second image sensor, and the depth map is generated based on the first distance.

7. The system of claim 1 , wherein the first and second image sensors comprise a stereo camera.

8. The system of claim 1 , wherein the method further comprises:
in accordance with a determination that the object will interfere with the opening of the element into the space, preventing the opening of the element; and
in accordance with a determination that the object will not interfere with the opening of the element into the space, allowing the opening of the element.

9. The system of claim 1 , wherein the method further comprises:
detecting user input for opening the element;
in response to detecting the user input:
in accordance with a determination that the object will interfere with the opening of the element into the space, partially opening the element into the space to the extent allowed by the object; and
in accordance with a determination that the object will not interfere with the opening of the element into the space, fully opening the element into the space.

10. The system of claim 1 , wherein the method further comprises:
detecting a user approaching the vehicle;
in response to detecting the user approaching the vehicle:
in accordance with a determination that the object will interfere with the opening of the element into the space, partially opening the element into the space to the extent allowed by the object; and
in accordance with a determination that the object will not interfere with the opening of the element into the space, fully opening the element into the space.

11. A non-transitory computer-readable medium including instructions, which when executed by one or more processors, cause the one or more processors to perform a method comprising:
receiving first image data from a first image sensor mounted on a vehicle and second image data from a second image sensor mounted on the vehicle, the vehicle including an element configured to open into a space external to the vehicle;
generating a depth map from a comparison of the first image data and the second image data, the depth map including an object;
generating a confidence value associated with the depth map; and comparing the confidence value to a predetermined threshold;
in accordance with the confidence value being above the predetermined threshold and a determination that the object, in the depth map, is within the space into which the element is configured to open, determining that the object will interfere with opening of the element into the space; and
in accordance with the confidence value being above the predetermined threshold and a determination that the object, in the depth map, is not within the space into which the element is configured to open, determining that the object will not interfere with opening of the element into the space;
in accordance with the confidence value being below the predetermined threshold,

locating the object using ultrasonic detection;
in accordance with a determination that the object, as located using ultrasonic detection, is within the space into which the element is configured to open, determining that the object will interfere with opening of the element into the space; and
in accordance with a determination that the object, as located using ultrasonic detection, is not within the space into which the element is configured to open, generating a warning notification and determining that the object will not interfere with opening of the element into the space.

12. A method comprising:
receiving first image data from a first image sensor mounted on a vehicle and second image data from a second image sensor mounted on the vehicle, the vehicle including an element configured to open into a space external to the vehicle;
generating a depth map from a comparison of the first image data and the second image data, the depth map including an object;
generating a confidence value associated with the depth map; and comparing the confidence value to a predetermined threshold;
in accordance with the confidence value being above the predetermined threshold and a determination that the object, in the depth map, is within the space into which the element is configured to open, determining that the object will interfere with opening of the element into the space; and
in accordance with the confidence value being above the predetermined threshold and a determination that the object, in the depth map, is not within the space into which the element is configured to open, determining that the object will not interfere with opening of the element into the space;
in accordance with the confidence value being below the predetermined threshold,

Assignees

• Faraday&Future Inc

Inventors

• Jeromin Oliver Max
• Theodosis Paul Alan

Classifications

• G06V20/647

  by matching two-dimensional images to three-dimensional objects · CPC title

• G06V40/103

  Static body considered as a whole, e.g. static pedestrian or occupant recognition · CPC title

• G06V20/58 Primary

  Recognition of moving objects or obstacles, e.g. vehicles or pedestrians; Recognition of traffic objects, e.g. traffic signs, traffic lights or roads · CPC title

• H04N13/239 Primary

  using two two-dimensional [2D] image sensors having a relative position equal to or related to the interocular distance (H04N13/243 takes precedence) · CPC title

• G06T2207/10012

  Stereo images · CPC title