Apparatus and methods for distance estimation using stereo imagery

Patent metadata

Field	Value
Publication number	US-10057593-B2
Application number	US-201414326374-A
Country	US
Kind code	B2
Filing date	Jul 8, 2014
Priority date	Jul 8, 2014
Publication date	Aug 21, 2018
Grant date	Aug 21, 2018

Abstract

Frame sequences from multiple image sensors may be combined in order to form, for example, an interleaved frame sequence. Individual frames of the combined sequence may be configured a by combination (e.g., concatenation) of frames from one or more source sequences. The interleaved/concatenated frame sequence may be encoded using a motion estimation encoder. Output of the video encoder may be processed (e.g., parsed) in order to extract motion information present in the encoded video. The motion information may be utilized in order to determine a depth of visual scene, such as by using binocular disparity between two or more images by an adaptive controller in order to detect one or more objects salient to a given task. In one variant, depth information is utilized during control and operation of mobile robotic devices.

First claim

What is claimed:

1. A non-transitory computer-readable storage medium having computer readable instructions stored thereon, that when executed by at least one processor causes the at least one processor to,
produce a sequence of composite images, each one of a respective sequence of composite images comprising a first image from a first sequence of images and a second image from a second sequence of images, the first image and the second image being joined adjacent to each other; and evaluate the sequence of composite images to determine a depth parameter of a scene, the evaluating of the sequence of composite images comprising encoding the sequence of composite images into an encoded frame sequence, the encoded frame sequence comprising disparity estimates and motion estimates obtained based on (i) a first set of images from the first sequence of images occurring at one or more time frames and (ii) a second set of images from the second sequence of images occurring at the one or more time frames,
wherein the sequence of composite images includes a first composite image and a second composite image, the first composite image is based on a combination of the second image from the second sequence of images and a plurality of replicas of the first image from the first sequence of images, the second composite image subsequent to the first composite image within the sequence of composite images is based on a combination of a third image from the second sequence of images and a plurality of replicas of a fourth image from the first sequence of images, the third image from the second sequence of images is acquired contemporaneously with the first image from the first sequence of images and subsequent to the second image from the second sequence of images, and the fourth image from the first sequence of images is acquired subsequent to the first image from the first sequence of images.

2. The computer readable storage medium of claim 1 , wherein the at least one processor is further configured to execute the computer readable instructions to,
generate the first sequence of images based on input received of the scene by a first image sensor, and
generate the second sequence of images based on input received of the scene by a second image sensor, respectively, the first image sensor is spatially separated from the second image sensor such that a binocular disparity in the first sequence of images and the second sequence of images is achieved, the binocular disparity corresponds to the depth parameter.

3. The computer readable storage medium of claim 2 , wherein, the first image from the first sequence of images comprises a first set of plurality of pixels, the second image from the second sequence of images comprises a second set of plurality of pixels, and the at least one processor is further configured to execute the computer readable instructions to encode the sequence of composite images using a motion estimation encoder.

4. The computer readable storage medium of claim 3 , wherein, the first composite image of the sequence of composite images comprises the first image from the first sequence of images; and the second composite image occurring subsequent to the first composite image within the sequence of composite images configured, the second composite image based on a combination of a fifth image from the first sequence of images with a sixth image from the second sequence of images, the first image from the first sequence of images is acquired contemporaneously with the second image from the second sequence of images and prior to the fifth image from the first sequence of images.

5. The computer readable storage medium of claim 3 , wherein: the motion estimation encoder is configured to, based on encoding the first composite image and the second composite image of the sequence of composite images, determine: a first displacement of at least one pixel in the first set of plurality of pixels within the first image from the first sequence of images relative to the second set of plurality of pixels within the second image from the second sequence of images; and a second displacement of at least one pixel in the first set of plurality of pixels within the fourth image from the first sequence of images relative to the second set of plurality of pixels within the first image from the first sequence of images; the first displacement of pixels corresponds to determination of the disparity; and the second displacement of pixels corresponds to determination of motion associated with one or more objects in the scene.

6. The computer readable storage medium of claim 3 , wherein
the first composite image of the sequence of composite images is based on a combination of the first image from the first sequence of images and a fifth transformed image from the first sequence of images, the fifth transformed image includes a reduced pixel count compared to a pixel count in the first image from the first sequence of images.

7. The computer readable storage medium of claim 4 , wherein, the motion estimation encoder is configured to determine a first displacement of at least one pixel in the first set of plurality of pixels within the image from the first sequence of images relative to the second set of plurality of pixels within the second image from the second sequence of images; and the binocular disparity is determined based on the first displacement.

8. The computer readable storage medium of claim 4 , wherein, the first set of plurality pixels within the first image from the first sequence of images and the second set of plurality of pixels within the second image from the second sequence of images correspond to at least one object within the scene; and the at least one processor is further configured to execute the computer readable instructions to determine distance to the object based on the depth parameter.

9. The computer readable storage medium of claim 8 ,
wherein, the motion estimation encoder is further configured to determine a second displacement of a first pixel in the first set of plurality of pixels within the first image from the first sequence of images and a second pixel in the first set of plurality of pixels within the fourth image from the first sequence of images; and
the second displacement corresponds to information related to motion of the at least one object.

10. An image processing system, comprising: an input interface configured to receive a stereo representation of a visual scene, the stereo representation comprising a first portion and a second portion; a logic component in communication with the input interface and configured to: arrange the first portion of the stereo representation with the second portion of the stereo representation into a concatenated frame; and form a sequence of concatenated frames by arranging first portions of the stereo representation and second portions of the stereo representation within a first concatenated frame in an alternate order relative to a preceding concatenated frame within the sequence, the first concatenated frame comprising a different size from either the first portion or the second portion of the stereo representation; wherein the sequence of concatenated frames further includes a first composite image and a second composite image; the first composite image of the sequence of concatenated frames is based on a combination of an image from the second sequence of images and a plurality of replicas of an image from the first sequence of images; the second composite image subsequent to the first composite image within the sequence of concatenated frames, the second composite image is based on a combination of another image from the second sequence of images and a plurality of repl

Assignees

• Brain Corp

Inventors

• Richert Micah

Classifications

• G06T2207/30252

  Vehicle exterior; Vicinity of vehicle · CPC title

• G06T7/579

  from motion · CPC title

• H04N19/51 Primary

  Motion estimation or motion compensation · CPC title

• G06K9/00355

  Physics · mapped topic

• G06T2207/10021

  Stereoscopic video; Stereoscopic image sequence · CPC title