Combined information for object detection and avoidance

Patent metadata

Field	Value
Publication number	US-11377232-B2
Application number	US-201916525271-A
Country	US
Kind code	B2
Filing date	Jul 29, 2019
Priority date	Mar 28, 2016
Publication date	Jul 5, 2022
Grant date	Jul 5, 2022

Abstract

Described is an imaging component for use by an unmanned aerial vehicle (“UAV”) for object detection. As described, the imaging component includes one or more cameras that are configured to obtain images of a scene using visible light that are converted into a depth map (e.g., stereo image) and one or more other cameras that are configured to form images, or thermograms, of the scene using infrared radiation (“IR”). The depth information and thermal information are combined to form a representation of the scene based on both depth and thermal information.

First claim

What is claimed is:

1. A method, comprising:
receiving, from a first camera coupled to an aerial vehicle and having a first orientation, first image data of a scene using visible light;
receiving, from a second camera coupled to the aerial vehicle and having the first orientation, second image data of the scene using visible light;
receiving, from a sensor coupled to the aerial vehicle and having the first orientation, sensor data representative of the scene;
processing the first image data, the second image data, and the sensor data to produce a combined information representative of the scene;
detecting an object represented in the combined information;
determining that an object signature of the object, as represented in the combined information, is an unknown object signature; and
in response to determining that the object signature is an unknown object signature, performing additional processing of at least a portion of the combined information to determine the object;
wherein the first camera, the second camera, and the sensor are positioned in a same plane; and
wherein the sensor is positioned approximately equidistant between the first camera and the second camera.

2. The method of claim 1 , wherein the combined information includes a horizontal dimension, a vertical dimension, and a depth dimension.

3. The method of claim 1 , further comprising:
determining, based at least in part on the first image data, the second image data, and the sensor data, that the object is an object to avoid.

4. The method of claim 1 , wherein the sensor is at least one of an infrared sensor or an ultrasonic sensor.

5. The method of claim 1 , wherein a first field of view of the first camera, a second field of view of the second camera, and a third field of view of the sensor overlap to form an effective field of view.

6. The method of claim 1 , wherein the first orientation is downward with respect to the aerial vehicle and the scene includes an area toward which the aerial vehicle is descending.

7. A method, comprising:
receiving, from a first image capture device at a first position on an aerial vehicle, first image data representative of at least a portion of a scene using a visible light spectrum;
receiving, from a second image capture device at a second position on the aerial vehicle, second image data representative of at least a portion of the scene using the visible light spectrum;
receiving, from a third image capture device at a third position on the aerial vehicle, third image data representative of at least a portion of the scene using a light spectrum other than the visible light spectrum;
detecting, based at least in part on one or more of the first image data, the second image data, or the third image data, an object;
determining that an object signature of the object is an unknown object signature; and
in response to determining that the object signature is an unknown object signature, performing additional processing of at least a portion of one or more of the first image data, the second image data, or the third image data to determine the object;
wherein the first image capture device, the second image capture device, and the third image capture device are positioned in a same plane; and
wherein the third image capture device is positioned approximately equidistant between the first image capture device and the second image capture device.

8. The method of claim 7 , further comprising:
forming, based at least in part on the first image data and the second image data, a depth map corresponding to at least a portion of the scene, wherein each position of the depth map corresponds to a vertical dimension, a horizontal dimension, and a depth dimension.

9. The method of claim 8 , further comprising:
combining the depth map and the third image data to form combined information representative of at least a portion of the scene.

10. The method of claim 7 , further comprising:
sending an instruction to cause the aerial vehicle to descend toward a location represented in the scene.

11. The method of claim 7 , further comprising:
processing at least one of the first image data, the second image data, or the third image data to detect a presence of a living object within an effective field of view.

12. The method of claim 11 , wherein the effective field of view corresponds to at least a portion of the scene within a field of view of the first image capture device, the second image capture device, and the third image capture device.

13. A method, comprising:
receiving, from a first image capture device of a vehicle, first image data;
receiving, from a second image capture device of the vehicle, second image data;
processing the first image data and the second image data to form a depth map wherein each pixel representative of a position includes a horizontal dimension, a vertical dimension, and a depth dimension;
receiving third data from a sensor;
processing the third data to determine, for each corresponding horizontal dimension and vertical dimension, a thermal dimension;
generating combined information representative of a scene that includes for each pixel of the combined information the horizontal dimension, the vertical dimension, the depth dimension, and the thermal dimension;
determining, based at least in part on the thermal dimension, an object represented in the combined information;
determining that an object signature of the object, as represented in the combined information, is an unknown object signature; and
in response to determining that the object signature is an unknown object signature, performing additional processing of at least a portion of the combined information to determine the object;
wherein the first image capture device, the second image capture device, and the sensor are positioned in a same plane; and
wherein the sensor is positioned approximately equidistant between the first image capture device and the second image capture device.

14. The method of claim 13 , wherein the first image capture device and the second image capture device form a stereo pair of image capture devices.

15. The method of claim 13 , wherein the sensor from which the third data is received is a long wave infrared camera.

16. The method of claim 13 , wherein a first field of view of the first image capture device and a second field of view of the second image capture device at least partially overlap to form an effective field of view.

17. The method of claim 13 , further comprising:
determining, based at least in part on one or more of the depth map or the thermal dimension, that the object represented in one or more of the first image data, the second image data, or the third data is a warm-blooded object; and
sending instructions to alter a navigation of the vehicle in response to determining that the object is potentially a warm-blooded object.

18. The method of claim 13 , wherein:
the vehicle is an unmanned aerial vehicle; and
the scene includes a delivery location to which the unmanned aerial vehicle is to descend.

19. The method of claim 13 , further comprising:
determining, based on one or more of the horizontal dimension, the vertical dimension, the depth dimension, the thermal dimension, or the combined information, that the object is potentially a warm blooded object.

20. An unmanned aerial vehicle (“UAV”) apparatus, comprising:
a frame;
a control system;
an imaging component that includes a thermographic camera having a field of view and a pai

Assignees

• Amazon Tech Inc

Inventors

• Harris Scott Raymond
• O'Brien Barry James
• Watson Joshua John

Classifications

• G06V20/17

  taken from planes or by drones · CPC title

• B64U2201/104

  using satellite radio beacon positioning systems, e.g. GPS · CPC title

• B64U2101/60

  for transporting passengers; for transporting goods other than weapons · CPC title

• B64U2101/30

  for imaging, photography or videography · CPC title

• G06V20/13

  Satellite images · CPC title