System and method for detecting occluded objects based on image processing

Patent metadata

Field	Value
Publication number	US-11714420-B2
Application number	US-202117477047-A
Country	US
Kind code	B2
Filing date	Sep 16, 2021
Priority date	Jun 12, 2017
Publication date	Aug 1, 2023
Grant date	Aug 1, 2023

Abstract

The present invention is related to systems and methods for detecting an occluded object based on the shadow of the occluded object. In some examples, a vehicle of the present invention can capture one or more images while operating in an autonomous driving mode, and detecting shadow items within the captured image. In response to detecting a shadow item moving towards the direction of vehicle travel, the vehicle can reduce its speed to avoid a collision, should an occluded object enter the road. The shadow can be detected using image segmentation or a classifier trained using convolutional neural networks or another suitable algorithm, for example.

First claim

The invention claimed is:

1. A vehicle comprising:
one or more cameras;
one or more actuator systems;
one or more of a LiDAR sensor, an ultrasonic sensor, a radar sensor, and a range sensor; and
a processor operatively coupled to the one or more cameras and the one or more actuator systems, the processor configured to:
identify a shadow in one or more images captured by the one or more cameras;
determine whether the shadow is moving in a direction towards a direction of vehicle travel; and
in accordance with a determination that the shadow is moving in a direction towards the direction of vehicle travel, reducing a speed of the vehicle using the one or more actuator systems;
wherein identifying a plurality of pixels of the one or more images illustrating an image of a ground based on data from the one or more of the LiDAR sensor, the ultrasonic sensor, the radar sensor, and the range sensor; and identifying the shadow within the pixels illustrating the image of the ground.

2. The vehicle of claim 1 , further comprising a location system and a map interface,
wherein the processor is operatively coupled to the location system and the map interface, and the processor is further configured to:
identify a location of the vehicle based on one or more of the location system and the map interface; and
based on a determination that the vehicle location is in a pedestrian heavy zone, enter a shadow detection mode, wherein the shadow detection mode causes the processor to identify the shadow and determine whether the shadow is moving.

3. The vehicle of claim 1 , wherein the shadow is a shadow of an occluded object and the occluded object is not included in the one or images captured by the one or more cameras of the vehicle.

4. The vehicle of claim 1 , wherein the processor is further configured to, in accordance with a determination that the shadow is stationary or moving in a direction away from the direction of vehicle travel, maintain the speed of the vehicle using the one or more actuator systems.

5. The vehicle of claim 1 , wherein identifying the shadow in the one or more images comprises:
segmenting a plurality of pixels of the one or more images into groups based on a darkness of each pixel, wherein pixels within each group have darknesses within a first threshold difference of each other; and
identifying one of the groups having a plurality of dark pixels having a first darkness surrounded by one of the groups having a plurality of light pixels having a second darkness, the first darkness darker than the second darkness by at least a second threshold difference.

6. The vehicle of claim 1 , wherein the radar sensor is configured to detect an occluded object when a radar wave from the radar sensor bounces beneath other objects.

7. The vehicle of claim 1 , wherein identifying the shadow in the one or more images comprises comparing the shadow to an expected shadow shape.

8. The vehicle of claim 1 , wherein identifying the shadow in the one or more images comprises: collecting a plurality of example images; segmenting a plurality of example shadows in the plurality of example images; training a classifier using the plurality of example images; and applying the classifier to the one or more images.

9. The vehicle of claim 1 , wherein the processor is further configured to estimate a position of a horizon and identify ground pixels in response to the estimated position of the horizon.

10. The vehicle of claim 1 , further comprising a storage for storing one or more reference images corresponding to various shadows of objects in a variety of lighting conditions and the processor is further configured to compare a shape of the shadow to one or more stored reference images.

11. A method of operating a vehicle in an autonomous driving mode, the method comprising:
capturing one or more images at one or more cameras of the vehicle;
identifying a plurality of pixels of the one or more images illustrating an image of a ground based on data from one or more of a LiDAR sensor, an ultrasonic sensor, a radar sensor, and a range sensor included in the vehicle;
identifying the shadow within the pixels illustrating the image of the ground;
determining whether the shadow is moving in a direction towards a direction of vehicle travel; and
in accordance with a determination that the shadow is moving in a direction towards the direction of vehicle travel, reducing a speed of the vehicle using one or more actuator systems of the vehicle.

12. The method of claim 11 , further comprising: identifying a location of the vehicle based on one or more of a location system and a map interface of the vehicle; and based on a determination that the vehicle location is in a pedestrian heavy zone, entering a shadow detection mode, wherein the shadow detection mode causes the processor to identify the shadow and determine whether the shadow is moving.

13. The method of claim 11 , wherein the shadow is a shadow of an occluded object and the occluded object is not included in the one or images captured by the one or more cameras of the vehicle.

14. The method of claim 11 , further comprising: in accordance with a determination that the shadow is stationary or moving in a direction away from the direction of vehicle travel, maintaining the speed of the vehicle using the one or more actuator systems.

15. The method of claim 11 , further comprising: segmenting a plurality of pixels of the one or more images into groups based on a darkness of each pixel, wherein pixels within each group have darknesses within a first threshold difference of each other; and identifying one of the groups having a plurality of dark pixels having a first darkness surrounded by one of the groups having a plurality of light pixels having a second darkness, the first darkness darker than the second darkness by at least a second threshold difference.

16. The method of claim 11 , further comprising: detecting an occluded object when a radar wave from the radar sensor bounces beneath other objects.

17. The method of claim 11 , further comprising comparing the shadow to an expected shadow shape.

18. The method of claim 11 , further comprising: collecting a plurality of example images; segmenting a plurality of example shadows in the plurality of example images; training a classifier using the plurality of example images; and applying the classifier to the one or more images.

19. The method of claim 11 , further comprising: estimating a position of a horizon and identifying ground pixels in response to the estimated position of the horizon.

20. The method of claim 11 , further comprising: storing one or more reference images corresponding to various shadows of objects in a variety of lighting conditions and comparing a shape of the shadow to one or more stored reference images.

Assignees

• Faraday & Future Inc

Inventors

• Batur Aziz Umit

Classifications

• G06V20/58 Primary

  Recognition of moving objects or obstacles, e.g. vehicles or pedestrians; Recognition of traffic objects, e.g. traffic signs, traffic lights or roads · CPC title

• G05D1/0223 Primary

  involving speed control of the vehicle (vehicle fittings for automatically controlling, i.e. preventing speed from exceeding an arbitrarily established velocity or maintaining speed at a particular velocity, as selected by the vehicle operator B60K31/00) · CPC title

• G05D1/0088

  characterized by the autonomous decision making process, e.g. artificial intelligence, predefined behaviours (using knowledge based models G06N5/00) · CPC title

• G05D1/0246

  using a video camera in combination with image processing means · CPC title

• G06F18/24

  Classification techniques · CPC title