Photometric stereo object detection for articles left in an autonomous vehicle

Patent metadata

Field	Value
Publication number	US-11100347-B2
Application number	US-201916299510-A
Country	US
Kind code	B2
Filing date	Mar 12, 2019
Priority date	Mar 12, 2019
Publication date	Aug 24, 2021
Grant date	Aug 24, 2021

Abstract

Abandoned articles left by a user departing from an autonomous vehicle are automatically detected by capturing image data including a plurality of diversely-illuminated images of a target area within a passenger cabin of the vehicle. A plurality of normal vectors are determined for respective pixels representing the target area in a normal extractor based on the images. A normal-driven map is stored in a first array in response to the plurality of normal vectors. A baseline map is stored in a second array compiled from baseline images of the target area in a nominal clean state. Differences between the normal-driven map and the baseline map indicative of an object not present in the clean state are detected in a comparator. Difficult to detect objects can be found using a single, fixed camera.

First claim

What is claimed is:

1. An autonomous vehicle for automatically detecting an abandoned article left by a user departing from the vehicle, comprising:
a camera capturing image data including a plurality of diversely-illuminated images of a target area containing the abandoned article within a passenger cabin of the vehicle;
a normal extractor receiving the images to determine a plurality of normal vectors for respective pixels representing the target area;
a first array storing a normal-driven map in response to the plurality of normal vectors;
a second array storing a baseline map compiled from baseline images of the target area in a nominal clean state; and
a comparator coupled to the arrays to detect differences between the normal-driven map and the baseline map indicative of an object not present in the clean state.

2. The vehicle of claim 1 wherein the arrays are comprised of vector values indicating respective orientations of the respective normal vectors.

3. The vehicle of claim 1 wherein the arrays are comprised of depth values, and wherein the vehicle further comprises a map generator receiving the plurality of normal vectors to determine respective depth values for each respective pixel.

4. The vehicle of claim 1 wherein the arrays are comprised of voxel occupancy values, and wherein the vehicle further comprises a map generator receiving the plurality of normal vectors to determine respective voxel occupancy values for a three-dimensional region including the target area.

5. The vehicle of claim 1 further comprising:
a plurality of light sources at diverse locations in the cabin; and
a lighting analyzer responsive to ambient lighting illuminating the cabin in order to select a plurality of lighting configurations respectively produced by the light sources during capture of each of the respective diversely-illuminated images.

6. The vehicle of claim 1 further comprising a classifier responsive to the detected differences to assign one of a plurality of predetermined object classifications to the object.

7. The vehicle of claim 6 further comprising intervention logic selecting a predetermined action according to the assigned object classification.

8. The vehicle of claim 7 wherein the predetermined action is comprised of transmitting a message to the user indicating the presence of the detected object.

9. The vehicle of claim 8 wherein the message is transmitted wirelessly to a portable device of the user.

10. The vehicle of claim 8 wherein the message is transmitted as a visible or audible signal produced by the vehicle.

11. The vehicle of claim 7 wherein the predetermined action is comprised of autonomously driving the vehicle to a cleaning station.

12. A method of automatically detecting an abandoned article left by a user departing from an autonomous vehicle, comprising the steps of:
capturing image data including a plurality of diversely-illuminated images of a target area containing the abandoned article within a passenger cabin of the vehicle;
determining a plurality of normal vectors for respective pixels representing the target area in a normal extractor based on the images;
storing a normal-driven map in a first array in response to the plurality of normal vectors;
storing a baseline map in a second array compiled from baseline images of the target area in a nominal clean state; and
detect differences between the normal-driven map and the baseline map indicative of an object not present in the clean state.

13. The method of claim 12 wherein the maps are comprised of vector values indicating respective orientations of the respective normal vectors.

14. The method of claim 12 wherein the maps are comprised of depth values, and wherein the method further comprises the step of generating the respective depth values for each respective pixel in response to the plurality of normal vectors.

15. The method of claim 12 wherein the maps are comprised of voxel occupancy values for a three-dimensional region including the target area, and wherein the method further comprises the step of generating the voxel occupancy values in response to the plurality of normal vectors.

16. The method of claim 12 further comprising the steps of:
analyzing ambient illumination of the cabin in order to select a plurality of lighting configurations for capturing each of the respective diversely-illuminated images; and
activating a plurality of light sources at diverse locations in the cabin to produce the selected lighting configurations.

17. The method of claim 12 further comprising the step of assigning one of a plurality of predetermined object classifications to the object a classifier in response to the detected differences.

18. The method of claim 17 further comprising the step of selecting a predetermined intervention action according to the assigned object classification.

19. The method of claim 18 wherein the selected action is comprised of transmitting a message to the user indicating the presence of the detected object.

20. The method of claim 18 wherein the transmitted message is comprised of a visible or audible signal produced by the vehicle.

Assignees

• Ford Global Tech Llc

Inventors

• Herman David M

Classifications

• G06T7/254 Primary

  involving subtraction of images · CPC title

• G06V20/59 Primary

  inside of a vehicle, e.g. relating to seat occupancy, driver state or inner lighting conditions · CPC title

• G06V10/751

  Comparing pixel values or logical combinations thereof, or feature values having positional relevance, e.g. template matching · CPC title

• G06V10/44

  Local feature extraction by analysis of parts of the pattern, e.g. by detecting edges, contours, loops, corners, strokes or intersections; Connectivity analysis, e.g. of connected components · CPC title

• G06T2207/10152

  Varying illumination · CPC title