Combined information for object detection and avoidance

Patent metadata

Field	Value
Publication number	US-2020108946-A1
Application number	US-201916525271-A
Country	US
Kind code	A1
Filing date	Jul 29, 2019
Priority date	Mar 28, 2016
Publication date	Apr 9, 2020
Grant date	—

Abstract

Described is an imaging component for use by an unmanned aerial vehicle (“UAV”) for object detection. As described, the imaging component includes one or more cameras that are configured to obtain images of a scene using visible light that are converted into a depth map (e.g., stereo image) and one or more other cameras that are configured to form images, or thermograms, of the scene using infrared radiation (“IR”). The depth information and thermal information are combined to form a representation of the scene based on both depth and thermal information.

First claim

1 .- 20 . (canceled)

21 . A method, comprising:
receiving, from a first camera coupled to an aerial vehicle and having a first orientation, first image data of a scene using visible light; receiving, from a second camera coupled to the aerial vehicle and having the first orientation, second image data of the scene using visible light; and receiving, from a sensor coupled to the aerial vehicle and having the first orientation, sensor data representative of the scene; and processing the first image data, the second image data, and the sensor data to produce a combined information representative of the scene.

22 . The method of claim 21 , wherein the combined information includes a horizontal dimension, a vertical dimension, and at least one of a depth dimension or a thermal dimension.

23 . The method of claim 21 , further comprising:
detecting, based at least in part on the combined information, an object; determining, based at least in part on the first image data, the second image data, and the sensor data, whether the object is an object to avoid; and sending instructions to alter a navigation of the aerial vehicle in response to determining that the object is an object to avoid.

24 . The method of claim 21 , wherein the sensor is at least one of an infrared sensor or an ultrasonic sensor.

25 . The method of claim 21 , wherein a first field of view of the first camera, a second field of view of the second camera, and a third field of view of the sensor overlap to form an effective field of view.

26 . The method of claim 21 , wherein the first orientation is downward with respect to the aerial vehicle and the scene includes an area toward which the aerial vehicle is descending.

27 . A method, comprising:
receiving, from a first image capture device at a first position on an aerial vehicle, first image data representative of at least a portion of a scene using a visible light spectrum; receiving, from a second image capture device at a second position on the aerial vehicle, second image data representative of at least a portion of the scene using the visible light spectrum; and receiving, from a third image capture device at a third position on the aerial vehicle, third image data representative of at least a portion of the scene using a light spectrum other than the visible light spectrum.

28 . The method of claim 27 , further comprising:
forming, based at least in part on the first image data and the second image data, a depth map corresponding to at least a portion of the scene, wherein each position of the depth map corresponds to a vertical dimension, a horizontal dimension, and a depth dimension; and forming, based at least in part on the third image data, a representation of the scene based on the third image data, wherein each position of the representation corresponds to a vertical dimension, a horizontal dimension, and a signature.

29 . The method of claim 28 , further comprising:
combining the depth map and the representation to form combined information representative of at least a portion of the scene.

30 . The method of claim 27 , wherein the first image capture device, the second image capture device and the third image capture device are positioned in a same plane.

31 . The method of claim 27 , further comprising:
sending an instruction to cause the aerial vehicle to descend toward a location represented in the scene.

32 . The method of claim 27 , further comprising:
processing at least one of the first image data, the second image data, or the third image data to detect a presence of a living object within an effective field of view.

33 . The method of claim 32 , wherein the effective field of view corresponds to at least a portion of the scene within a field of view of the first image capture device, the second image capture device, and the third image capture device.

34 . A method, comprising:
receiving, from a first image capture device of a vehicle, first image data; receiving, from a second image capture device of the vehicle, second image data; processing the first image and the second image to form a depth map wherein each pixel representative of a position includes a horizontal dimension, a vertical dimension, and a depth dimension; receiving third data; processing the third data to determine, for each corresponding horizontal dimension and vertical dimension, a thermal dimension; and generating combined information representative of a scene that includes for each pixel of the combined information the horizontal dimension, the vertical dimension, the depth dimension, and the thermal dimension.

35 . The method of claim 34 , wherein the first image capture device and the second image capture device form a stereo pair of image capture devices.

36 . The method of claim 34 , wherein the third data is received from a long wave infrared camera.

37 . The method of claim 34 , wherein a first field of view of the first image capture device and a second field of view of the second image capture device at least partially overlap to form an effective field of view.

38 . The method of claim 34 , further comprising:
determining, based at least in part on one or more of the depth map or the thermal dimension, that an object represented in one or more of the first image data, the second image data, or the third data is a warm-blooded object; and sending instructions to alter a navigation of the vehicle in response to determining that the object is potentially a warm-blooded object.

39 . The method of claim 34 , wherein:
the vehicle is an unmanned aerial vehicle; and the scene includes a delivery location to which the unmanned aerial vehicle is to descend.

40 . The method of claim 34 , further comprising:
detecting an object; and determining, based on one or more of the horizontal dimension, the vertical dimension, the depth dimension, the thermal dimension, or the combined information, that the object is potentially a warm blooded object.

Assignees

• Amazon Tech Inc

Inventors

• Harris Scott Raymond
• O'Brien Barry James
• Watson Joshua John

Classifications

• H04N13/271

  wherein the generated image signals comprise depth maps or disparity maps · CPC title

• H04N13/243

  using three or more two-dimensional [2D] image sensors · CPC title

• G06T2207/10012

  Stereo images · CPC title

• G06T2207/10048

  Infrared image · CPC title

• H04N13/239

  using two two-dimensional [2D] image sensors having a relative position equal to or related to the interocular distance (H04N13/243 takes precedence) · CPC title