Apparatus and methods for robotic operation using video imagery

Patent metadata

Field	Value
Publication number	US-9713982-B2
Application number	US-201414285466-A
Country	US
Kind code	B2
Filing date	May 22, 2014
Priority date	May 22, 2014
Publication date	Jul 25, 2017
Grant date	Jul 25, 2017

Abstract

Data streams from multiple image sensors may be combined in order to form, for example, an interleaved video stream. The video stream may be encoded using a motion estimation encoder. Output of the video encoder may be processed (e.g., parsed) in order to extract motion information present in the encoded video. The motion information may be utilized in order to determine a depth of visual scene, such as by using binocular disparity between two or more images by an adaptive controller in order to detect one or more objects salient to a given task. In one variant, depth information is utilized during control and operation of mobile robotic devices.

First claim

What is claimed:

1. A method of determining a distance to an object by a robotic device, the method comprising:
interleaving left and right images of the device surroundings to produce an interleaved video stream;
evaluating the video stream to determine a distance to and a motion of an object in a visual scene associated with the surroundings visual scene, the evaluating the video stream comprising encoding the video stream using an encoder process comprising:
motion estimation configured to provide information related to a first displacement of a first representation of the object within a first image of the interleaved video stream relative to a second representation of the object within a second image of the interleaved video stream; and
motion estimation configured to provide information related to a second displacement of the second representation of the object within the second image of the interleaved video stream relative to a third representation of the object within a third image of the interleaved video stream;
wherein the first displacement characterizes the distance to the object, and the second displacement characterizes the motion of the object; and

producing an encoded frame stream resultant from the encoder process, the encoded frame stream comprising a sequence of encoded frames associated with (i) the distance to the object and (ii) the motion of the object;
causing the robotic device to execute an action based at least in part on the distance;
wherein:
the left and the right images are provided by a left and right camera, respectively, the right camera disposed spatially from the left camera on the device; and
the interleaving and the evaluating cooperate to effectuate a reduction in an energy use compared to executing the action without the interleaving.

2. The method of claim 1 , wherein:
the evaluation further comprises determination of a binocular disparity between the left and the right images;
the disparity is based at least in part on the distance and spatial separation between the left and the right cameras; and
the distance is determined based at least on the determination of the disparity.

3. The method of claim 2 , wherein:
individual ones of the first and second images comprise a plurality of pixels;
the encoder process is selected from the group consisting of MPEG-4, H.262, H.263, H.264, and H.265 encoders;
the first object representation comprises a first block of pixels of a plurality of pixels of the given image; and
the displacement information is configured to relate displacement of the first block relative to a second block of pixels.

4. The method of claim 2 , further comprising
identifying the object using an output of the encoder process;
wherein the action execution is based at least in part on the distance and the identity of the object.

5. The method of claim 4 , wherein the action comprises a target approach or obstacle avoidance.

6. The method of claim 4 , wherein:
the robotic device comprises an aerial vehicle;
the object is configured to indicate a reference location; and
the distance determination is configured to enable landing of the aerial vehicle at the location.

7. The method of claim 4 , wherein
the robotic device comprises a mobile vehicle;
the object comprises a target;
the action comprises following of the target by the vehicle within a distance range; and
the distance is configured to enable following of the target by the vehicle within the distance range.

8. The method of claim 2 , further comprising communicating a user command to the robotic device, and maintaining the distance to the object.

9. The method of claim 8 , wherein:
the object comprises a portion of the user body;
the command is configured based on a gesture of the user; and
the gesture is configured to cause the action execution.

10. A robotic apparatus comprising:
a first and a second camera configured to provide a stereo view of a visual scene comprising an object, the stereo view comprising left and right portions;
logic configured to form an interleaved sequence of images by arranging the left and the right portions interleaved with one another within the interleaved sequence;
a video encoder component configured to encode the interleaved sequence of images to produce a sequence of compressed images, the encoding of the interleaved sequence of images comprising:
disparity estimation configured to provide first information related to a displacement of a first representation of the object within a given one of the sequence of compressed images relative to a second representation of the object within another one of the sequence of compressed images;
motion estimation configured to provide second information related to a second displacement of the second representation of the object within the another one of the sequence of compressed images relative to a third representation of the object within a subsequent one of the sequence of compressed images; and
production of an encoded frame stream by the video encoder component based at least on the first, second, and third representations of the object, where the encoded frame stream comprises a series of encoded frames that is associated with (i) a distance to the object and (ii) a motion of the object;
where the first information characterizes the distance between the object and the robotic apparatus, and where the second information characterizes the motion of the object; and

a processing component configured to determine the distance between the object and the robotic apparatus based on the first information encoded in a compressed image within the sequence of compressed images.

11. The apparatus of claim 10 , wherein:
the first camera is disposed spatially from the second camera on the apparatus;
the distance is determined based at least on a binocular disparity associated with the left and the right portions; and
the motion information enables determination of the disparity.

12. The apparatus of claim 11 , further comprising:
an actuator; and
an energy source;
wherein:
the processing component is configured to provide, based on the determination of the distance, an instruction to the actuator, the instruction being configured to cause the robotic apparatus to execute an action; and
the arrangement and the encoding collectively reduce energy use associated with the determination of the distance as compared to a determination of the distance performed in absence of encoding the interleaved sequence.

13. The apparatus of claim 12 , wherein:
the robotic apparatus comprises an unmanned vehicle; and
the action is configured to maintain the distance.

14. The apparatus of claim 12 , wherein:
the robotic apparatus comprises an unmanned aerial vehicle; and
the action is configured to reduce a distance to the object.

15. The robotic apparatus of claim 11 , further comprising:
a third camera configured to provide an image of the visual scene comprising the object;
wherein: the interleaved sequence of images comprises the image interleaved with the first and the second portions; and
the distance determination by the processing component is configured based on another version of binocular disparity associated with the image and at least one of the left or the right portions of the interleaved sequence of images.

16. The apparatus of claim 15 , wherein:
the first camera is disposed spatially from the second camera by a first span;
the third camera is disposed spatially from the second camera by a second span, the second span being smaller tha

Assignees

• Brain Corp

Inventors

• Buibas Marius
• Richert Micah

Classifications

• G06V20/17

  taken from planes or by drones · CPC title

• G06V20/13

  Satellite images · CPC title

• G06K9/00355

  Physics · mapped topic

• G06T2207/10021

  Stereoscopic video; Stereoscopic image sequence · CPC title

• G06K9/0063

  Physics · mapped topic