Computational stereoscopic camera system

Patent metadata

Field	Value
Publication number	US-9237331-B2
Application number	US-201113083483-A
Country	US
Kind code	B2
Filing date	Apr 8, 2011
Priority date	Jan 18, 2011
Publication date	Jan 12, 2016
Grant date	Jan 12, 2016

Abstract

A closed-loop control system for stereoscopic video capture is provided. At least two motorized lenses are positioned in accordance with specified parameters to capture spatially-disparate images of a scene. The motorized lenses focus light on a corresponding one of the at least two sensors, which generate image streams. One or more processors execute instructions to provide a stream analyzer and a control module. The stream analyzer receives the image streams from the sensors and analyzes the image streams and the specified parameters in real time; the stream analyzer then modifies the image streams and generates metadata. The control module then receives and analyzes the image streams and metadata and transmits updated parameters to a control mechanism that is coupled to the at least two motorized lenses. The control mechanism then modifies operation of the at least two motorized lenses in real time in accordance with the updated parameters.

First claim

What is claimed is:

1. A stereo camera, comprising:
at least two lenses, each arranged to focus light on a respective camera sensor, wherein each camera sensor is capable of generating an image stream comprising a sequence of images;
a user interface to receive user input including:
a lower disparity range bracket setting for a target disparity range, and
an upper disparity range bracket setting for the target disparity range;

a stream analyzer comprising at least one processor and configured to:
receive the image streams; and
process the image streams and generate metadata including whether disparities in images of the image streams are outside of the target disparity range;

a control module comprising logic for generating, based on the metadata, control signals for one or more parameters relating to stereo camera control, wherein the one or more parameters relating to stereo camera control include an interaxial distance between the lenses and an angle formed by a principal axis of a lens with a baseline between the lenses; and
a control element that responds to the control signals for the one or more parameters relating to stereo camera control to alter at least one of the parameters relating to stereo camera control while the two or more image streams are being captured.

2. The stereo camera of claim 1 , wherein the user interface includes a touchscreen input device and a stereographic display.

3. The stereo camera of claim 1 wherein principal axes of the lenses are coplanar and form a substantially same angle with the baseline, the angle with the baseline being adjustable.

4. A closed-loop control system for stereoscopic video capture, the system comprising:
at least two motorized lenses, wherein the lenses are positioned in accordance with specified parameters, wherein the specified parameters include an interaxial distance and an angle formed by a principal axis of a lens with a baseline between a baseline between the lenses;
at least two light sensors, wherein each light sensor is coupled to a corresponding one of the at least two motorized lenses and the corresponding lens focuses light onto the light sensor, and wherein each light sensor generates an image stream comprising a sequence of images;
one or more processors;
a user interface to receive user input including:
a lower disparity range bracket setting for a target disparity range, and
an upper disparity range bracket setting for the target disparity range;

a stream analyzer for receiving the image streams from the light sensors, wherein the stream analyzer is operable to analyze the image streams and the specified parameters in real-time, generate metadata, including data related to disparities in the images of the image streams, and wherein the stream analyzer operates using the one or more processors;
a control module that operates using the one or more processors for receiving:
the user input received via the user interface, the image streams, and the metadata from the stream analyzer, wherein the control module is operable to:
analyze the user input, the image streams and the metadata, and
transmit control signals that update one or more of the specified parameters when the analysis of the user input, the image streams, and the metadata determines that disparities in images of the image streams are outside of the target disparity range; and

a control mechanism coupled to the at least two motorized lenses, wherein the control mechanism is operable to receive transmissions of the control signals from the control module and modify operation of the at least two motorized lenses in real-time in accordance with the updated one or more of the specified parameters while the two or more image streams are being captured.

5. The closed-loop control system of claim 4 , wherein a specified parameter comprises a conventional camera parameter or a stereoscopic camera parameter.

6. The closed-loop control system of claim 5 , wherein a conventional camera parameter comprises one or more of focus, zoom, aperture, shutter speed, exposure, frame rate, or region of interest.

7. The closed-loop control system of claim 4 , further comprising a non-transitory computer readable medium coupled to the processor, the computer readable medium comprising code executable by the one or more processors for implementing one or more of fixed pattern noise reduction, pixel response correction, Bayer demosaicing, linear color correction, non-linear color correction, radial distortion correction, tangential distortion correction, gain correction, stereoscopic matching, disparity estimation, camera calibration, image rectification, image warping, feature matching, optical flow, alignment estimation, homography estimation, color correction estimation, or non-linear disparity mapping.

8. The closed-loop control system of claim 4 , wherein metadata generated by the stream analyzer comprises one or more of statistical information, disparity information, or positional information.

9. The closed-loop control system of claim 4 , wherein the control module transmits event data objects to the control mechanism in real-time, wherein an event data object is associated with an event, and wherein an event comprises an update of the one or more specified parameters at a specified time.

10. The closed-loop control system of claim 9 , wherein the event data objects are added to an event queue.

11. The closed-loop control system of claim 9 , wherein the event data objects in the event queue are executed in a specified order.

12. The closed-loop control system of claim 11 , wherein the event data objects in the event queue are executed in an order specified in accordance with a priority of each event data object in the event queue.

13. The closed-loop control system of claim 9 , wherein the specified time for an event is specified in relation to the current clock or in relation to a future exposure starting time.

14. The closed-loop control system of claim 9 , wherein an event has a duration.

15. The closed-loop control system of claim 9 , wherein the control module is operable to process events described in a script, further comprising:
a computing device operable to generate a script for programming camera operations using the control module.

16. The closed-loop control system of claim 4 , further comprising:
a camera rig adjustably coupled to the control mechanism and to the at least two motorized lenses, wherein the camera rig supports the at least two motorized lenses, and wherein portions of the camera rig are movable to adjust the at least two motorized lenses in accordance with the updated specified parameters.

17. The closed-loop control system of claim 16 , wherein portions of the camera rig are movable for rotationally adjusting the angle with the baseline of the principal axes of the at least two motorized lenses in accordance with a specified camera convergence.

18. The closed-loop control system of claim 16 , wherein portions of the camera rig are movable for adjusting the at least two motorized lenses in accordance with a specified interaxial distance.

19. The closed-loop control system of claim 4 , further comprising:
a touch screen for displaying a user interface to control and monitor the closed-loop control system, wherein the user interface provides interaction metaphors that abstract underlying parameters, and wherein the touch screen is communicably connected to the one or more processors; and
input circuitry operable to receive and process input signals from the touch scre

Assignees

• Heinzle Simon
• Greisen Pierre
• Smolic Aljoscha
• Matusik Wojciech
• Gross Markus
• Disney Entpr Inc
• Eth Zürich Eidgenöessische Tech Hochschule Zürich

Inventors

• Heinzle Simon
• Greisen Pierre
• Smolic Aljoscha
• Matusik Wojciech
• Gross Markus

Classifications

• H04N13/239 Primary

  using two two-dimensional [2D] image sensors having a relative position equal to or related to the interocular distance (H04N13/243 takes precedence) · CPC title

• H04N13/00

  Stereoscopic video systems; Multi-view video systems; Details thereof · CPC title

• H04N13/246

  Calibration of cameras · CPC title

• H04N13/20

  Image signal generators · CPC title

• H04N13/243

  using three or more two-dimensional [2D] image sensors · CPC title