Apparatus and methods for distance estimation using multiple image sensors

Patent metadata

Field	Value
Publication number	US-10989521-B2
Application number	US-201816214730-A
Country	US
Kind code	B2
Filing date	Dec 10, 2018
Priority date	May 22, 2014
Publication date	Apr 27, 2021
Grant date	Apr 27, 2021

Abstract

Data streams from multiple image sensors may be combined in order to form, for example, an interleaved video stream, which can be used to determine distance to an object. The video stream may be encoded using a motion estimation encoder. Output of the video encoder may be processed (e.g., parsed) in order to extract motion information present in the encoded video. The motion information may be utilized in order to determine a depth of visual scene, such as by using binocular disparity between two or more images by an adaptive controller in order to detect one or more objects salient to a given task. In one variant, depth information is utilized during control and operation of mobile robotic devices.

First claim

What is claimed:

1. A system for instructing a robot to execute a physical task, comprising:
at least one processor; and
a non-transitory computer readable media having computer executable instructions stored therein, which, when executed by the at least processor configure the at least one processor to,
receive a plurality of images associated with a plurality of spatially separated cameras, at least a portion of the plurality of images depicting an object,
determine, from the at least the portion of the plurality of images, a pattern of movement associated with the object based on disparity information derived from two or more of the plurality of images, the pattern of movement within the plurality of images is partitioned into a plurality of clusters wherein a largest area cluster is associated with a prevailing motion and is removed from the frame to obtain residual motion distribution associated with the object, the prevailing motion corresponding to apparent motion caused by at least one of a moving background or motions of the robot,
generate, based on the pattern of movement, a command signal configured to instruct the robot to execute a physical action, and
send the command signal to the robot,

wherein the non-transitory computer readable media comprises an artificial neural network configured to, receive as input the plurality of images and the pattern of movement associated with the object to produce the command signal.

2. The system of claim 1 , wherein the robot is configured to execute the physical action responsive to receiving the command signal.

3. The system of claim 1 , wherein the plurality of images comprise a monocular frame sequence comprising an interleaved frame sequence derived from frame sequences obtained from the plurality of spatially separated cameras.

4. The system of claim 3 , further comprising:
a motion estimation encoder, wherein the monocular frame sequence is encoded in part using the motion estimation encoder.

5. The system of claim 1 , wherein the artificial neural network generates a sparse transformation of the one or more images.

6. The system of claim 1 , wherein the plurality of encoded images are generated by encoding an interleaved sequence of images from three or more cameras of the plurality of spatially separate cameras.

7. The system of claim 6 , wherein the interleaved sequence of images comprises a transition between diagonally opposing cameras.

8. A non-transitory computer readable media having computer executable instructions stored therein, which, when executed by the at least processor coupled to a robot configure the at least one processor to,
receive a plurality of images associated with a plurality of spatially separated cameras, at least a portion of the plurality of images depicting an object;
determine, from the at least the portion of the plurality of images, a pattern of movement associated with the object based on disparity information derived from two or more of the plurality of images, the pattern of movement within the plurality of images is partitioned into a plurality of clusters wherein a largest area cluster is associated with a prevailing motion and is removed from the frame to obtain residual motion distribution associated with the object, the prevailing motion corresponding to apparent motion caused by at least one of a moving background or motions of the robot;
generate, based on the pattern of movement, a command signal configured to instruct the robot to execute a physical action; and
send the command signal to the robot,
wherein the non-transitory computer readable media comprises an artificial neural network configured to, receive as input the plurality of images and the pattern of movement associated with the object to produce the command signal.

9. The non-transitory computer readable media of claim 8 , wherein the robot is configured to execute the physical action responsive to receiving the command signal.

10. The non-transitory computer readable media of claim 8 , wherein the plurality of images comprise a monocular frame sequence comprising an interleaved frame sequence derived from frame sequences obtained from the plurality of spatially separated cameras.

11. The non-transitory computer readable media of claim 10 , wherein the monocular frame sequence is encoded in part using a motion estimation encoder.

12. The non-transitory computer readable media of claim 8 , wherein the artificial neural network generates produces a sparse transformation of the one or more images.

13. The non-transitory computer readable media of claim 8 , wherein the plurality of encoded images are generated by encoding an interleaved sequence of images from three or more cameras of the plurality of spatially separate cameras.

14. The non-transitory computer readable media of claim 13 , wherein the interleaved sequence of images comprises a transition between diagonally opposing cameras.

15. A method for instructing a robot to execute a physical task, comprising:
receiving as input a plurality of images associated with a plurality of spatially separated cameras, at least a portion of the plurality of images depicting an object;
generating a sparse transformation of one or more of the plurality of images;
determining, from the at least the portion of the plurality of images, a pattern of movement associated with the object based on disparity information derived from two or more of the plurality of images wherein the pattern of movement associated with the object is determined by partitioning the plurality of images into a plurality of clusters wherein a largest area cluster is associated with a prevailing motion, removing the largest area cluster from the frame and obtaining residual motion distribution associated with the object, the prevailing motion corresponding to apparent motion caused by at least one of a moving background or motions of the robot;
generating, based on the pattern of movement, a command signal configured to instruct the robot to execute a physical action; and
sending the command signal to the robot;
wherein the non-transitory computer readable media comprises an artificial neural network configured to, receive as input the plurality of images and the pattern of movement associated with the object to produce the command signal.

16. The method of claim 15 , wherein the robot is configured to execute the physical action responsive to receiving the command signal.

17. The method of claim 15 , wherein the plurality of images comprise a monocular frame sequence comprising an interleaved frame sequence derived from frame sequences obtained from the plurality of spatially separated cameras.

18. The method of claim 17 , wherein the monocular frame sequence is encoded in part using a motion estimation encoder.

19. The method of claim 15 , wherein the artificial neural network generates produces a sparse transformation of the one or more images.

20. The method of claim 15 , wherein,
the plurality of encoded images are generated by encoding an interleaved sequence of images from three or more cameras of the plurality of spatially separate cameras, and
the interleaved sequence of images comprises a transition between diagonally opposing cameras.

Assignees

• Brain Corp

Inventors

• Richert Micah
• Buibas Marius
• Polonichko Vadim

Classifications

• G06T7/593 Primary

  from stereo images · CPC title

• G06T2207/10021

  Stereoscopic video; Stereoscopic image sequence · CPC title

• G01B11/14 Primary

  for measuring distance or clearance between spaced objects or spaced apertures (G01B11/26 takes precedence; rangefinders G01C3/00) · CPC title

• H04N19/51

  Motion estimation or motion compensation · CPC title

• G06T2207/30196

  Human being; Person · CPC title