Automatic setting of zoom, aperture and shutter speed based on scene depth map

Patent metadata

Field	Value
Publication number	US-9307134-B2
Application number	US-201113072614-A
Country	US
Kind code	B2
Filing date	Mar 25, 2011
Priority date	Mar 25, 2011
Publication date	Apr 5, 2016
Grant date	Apr 5, 2016

Abstract

A Depth Map (DM) is able to be utilized for many parameter settings involving cameras, camcorders and other devices. Setting parameters on the imaging device includes zoom setting, aperture setting and shutter speed setting.

First claim

What is claimed is:

1. A method implemented on a device comprising:
a. generating a depth map using a single-lens camera by:
i. receiving a first image signal with a first blur quantity at an image sensor after the first image signal passes through a lens at a first position;
ii. receiving one or more second image signals with a second blur quantity at the image sensor after the one or more second image signals pass through the lens at a second position;
iii. computing a blur difference using the first blur quantity and the second blur quantity; and
iv. generating the depth map from the blur difference;

b. configuring a parameter setting on the device based on the depth map; and
c. classifying a scene to assist in configuring the parameter setting comprising:
i. computing a histogram of values from the depth map;
ii. smoothing the histogram to eliminate noise;
iii. computing a first derivative of the smoothed histogram using central differences;
iv. detecting a leftmost peak and a rightmost peak in the smoothed histogram;
v. if the leftmost peak and the rightmost peak correspond to depths greater than a landscape threshold, then the scene is classified as a landscape;
vi. otherwise, if a local minimum is not found between the leftmost peak and the rightmost peak, then the scene is classified as other; and
vii. otherwise, a foreground/background threshold is determined by choosing the local minimum between the leftmost peak and the rightmost peak, and pixels are classified as foreground or background based on the foreground/background threshold and a bounding box of foreground pixels determines whether there is an object of interest in the scene.

2. The method of claim 1 wherein the depth map is generated using at least one of: range sensors integrated with the device, double-lens stereoscopic cameras, a single-lens camera with scanning focus area, a single-lens camera utilizing an approximate depth map, a single-lens camera moved horizontally, and a movable lens and an image sensor to acquire multiple images with different blur quantities.

3. The method of claim 1 wherein configuring the parameter setting includes configuring a zoom setting.

4. The method of claim 1 wherein configuring the parameter setting includes configuring an aperture setting.

5. The method of claim 1 wherein configuring the parameter setting includes configuring a shutter speed setting.

6. The method of claim 1 wherein the parameter setting is automatic zoom adjustment comprising:
if the scene is classified as the landscape, incrementally zooming out of the scene and re-classifying the depth map to maintain a landscape classification and reach a widest available zoom setting;
if the scene is classified as containing a main object, automatically zooming in on the object up to object boundaries of the scene;
performing aperture adjustment;
performing shutter speed adjustment; and
updating a focus position.

7. The method of claim 1 wherein the device is selected from the group consisting of a personal computer, a laptop computer, a computer workstation, a server, a mainframe computer, a handheld computer, a personal digital assistant, a cellular/mobile telephone, a smart appliance, a gaming console, a digital camera, a digital camcorder, a camera phone, an iPod®, a video player, a DVD writer/player, a television and a home entertainment system.

8. A system programmed in a controller in a device configured for utilizing a depth map in one or more applications comprising:
a. an aperture setting module configured for setting an aperture size of the device based on the depth map;
b. a shutter speed setting module configured for setting a shutter speed of the device based on the depth map, wherein the depth map is generated using a single-lens camera by:
i. receiving a first image signal with a first blur quantity at an image sensor after the first image signal passes through a lens at a first position;
ii. receiving one or more second image signals with a second blur quantity at the image sensor after the one or more second image signals pass through the lens at a second position;
iii. computing a blur difference using the first blur quantity and the second blur quantity; and
iv. generating the depth map from the blur difference; and

c. a classification module configured for classifying a scene to assist in configuring the aperture size and the shutter speed comprising:
i. computing a histogram of values from the depth map;
ii. smoothing the histogram to eliminate noise;
iii. computing a first derivative of the smoothed histogram using central differences;
iv. detecting a leftmost peak and a rightmost peak in the smoothed histogram;
v. if the leftmost peak and the rightmost peak correspond to depths greater than a landscape threshold, then the scene is classified as a landscape;
vi. otherwise, if a local minimum is not found between the leftmost peak and the rightmost peak, then the scene is classified as other; and
vii. otherwise, a foreground/background threshold is determined by choosing the local minimum between the leftmost peak and the rightmost peak, and pixels are classified as foreground or background based on the foreground/background threshold and a bounding box of foreground pixels determines whether there is an object of interest in the scene.

9. The system of claim 8 wherein the depth map is generated using at least one of: range sensors integrated with the device, double-lens stereoscopic cameras, a single-lens camera with scanning focus area, a single-lens camera utilizing an approximate depth map, a single-lens camera moved horizontally, and a movable lens and an image sensor to acquire multiple images with different blur quantities.

10. The system of claim 8 wherein the device is selected from the group consisting of a personal computer, a laptop computer, a computer workstation, a server, a mainframe computer, a handheld computer, a personal digital assistant, a cellular/mobile telephone, a smart appliance, a gaming console, a digital camera, a digital camcorder, a camera phone, an iPod®, a video player, a DVD writer/player, a television and a home entertainment system.

11. A device comprising:
a. a memory for storing an application, the application configured for:
i. generating a depth map using a single-lens camera by:
(1) receiving a first image signal with a first blur quantity at an image sensor after the first image signal passes through a lens at a first position;
(2) receiving one or more second image signals with a second blur quantity at the image sensor after the one or more second image signals pass through the lens at a second position;
(3) computing a blur difference using the first blur quantity and the second blur quantity; and
(4) generating the depth map from the blur difference; and

ii. configuring a parameter setting on the device based on the depth map;
iii. classifying a scene to assist in configuring the parameter setting comprising:
(1) computing a histogram of values from the depth map;
(2) smoothing the histogram to eliminate noise;
(3) computing a first derivative of the smoothed histogram using central differences;
(4) detecting a leftmost peak and a rightmost peak in the smoothed histogram;
(5) if the leftmost peak and the rightmost peak correspond to depths greater than a landscape threshold, then the scene is classified as a landscape;
(6) otherwise, if a local minimum is not found between the leftmost peak and the rightmost peak, then the scene is classified as other; and
(7) otherwise, a foreground/background threshold is determined by choosing the local minimum between the leftmost peak and the rightmost p

Assignees

• Ciurea Florian
• Ali Gazi
• Berestov Alexander
• Lee Chuen-Chien
• Sony Corp

Inventors

• Ciurea Florian
• Ali Gazi
• Berestov Alexander
• Lee Chuen-Chien

Classifications

• H04N23/62 Primary

  Control of parameters via user interfaces · CPC title

• H04N13/239

  using two two-dimensional [2D] image sensors having a relative position equal to or related to the interocular distance (H04N13/243 takes precedence) · CPC title

• H04N13/207

  using a single two-dimensional [2D] image sensor · CPC title

• H04N13/271

  wherein the generated image signals comprise depth maps or disparity maps · CPC title

• H04N2013/0081

  Depth or disparity estimation from stereoscopic image signals · CPC title