Depth imaging system and method of rendering a processed image to include in-focus and out-of-focus regions of one or more objects based on user selection of an object

Patent metadata

Field	Value
Publication number	US-11019255-B2
Application number	US-201916425090-A
Country	US
Kind code	B2
Filing date	May 29, 2019
Priority date	Nov 29, 2016
Publication date	May 25, 2021
Grant date	May 25, 2021

Abstract

A method of image processing includes obtaining depth information respectively corresponding to one or more objects in an image frame captured by an imaging system based on spatial configuration of the imaging system, calculating one or more rendering parameters associated with one of the one or more objects in the image frame based on at least the depth information of the one or more objects, and rendering the image frame using the one or more rendering parameters to obtain a processed image frame. The processed image frame includes an in-focus region and an out-of-focus region relative to the one of the one or more objects.

First claim

What is claimed is:

1. A method of image processing, the method comprising:
obtaining depth information respectively corresponding to one or more objects in an image frame captured by an imaging system based on spatial configuration of the imaging system;
calculating one or more rendering parameters associated with one of the one or more objects in the image frame based on at least the depth information of the one or more objects, including:
determining a depth range of the image frame, the depth range being between a nearest object that is nearest to the imaging system among the one or more objects and a farthest object that is farthest to the imaging system among the one or more objects;
dividing the depth range into a plurality of layers; and
for each layer of the plurality of layers:
determining corresponding depth information that is associated with one of an average depth of all pixels in each layer or a depth of one or more representative pixels for a distinctive object in each layer; and
calculating a rendering parameter for each respective layer according to the corresponding depth information corresponding to the respective layer; and

rendering the image frame using the one or more rendering parameters to obtain a processed image frame, wherein the processed image frame includes an in-focus region and an out-of-focus region relative to the one of the one or more objects,
wherein the method further comprises:
in response to a user interaction to select an object for including in the in-focus region of the image frame, determining a layer that includes pixels at an on-focus point
based on the calculated one or more rendering parameters, determining layers l i−m , l i−m+1 , . . . . . . , l i−1 , l i in a first in-focus sub-region of the in-focus region and layers l i , l i+1 , . . . . . . , l i+n−1 , l i+n in a second in-focus sub-region of the in-focus region, the first in-focus sub-region being on a side of the on-focus point that is proximal to the imaging system, the second in-focus sub-region being on a side of the on-focus point that is distal to the imaging system, m, i, and n being positive integers, and i>m; and
determining layers l 1 , l 2 , . . . . . ., l i−m−1 in a first out-of-focus sub-region of the out-of-focus region and layers l i+n+1 ,l i+n+2 , . . . . . . , l j−1 , l j , in a second out-of-focus sub-region of the out-of-focus region, the first out-of-focus sub-region being on a side of the in-focus region that is proximal to the imaging system, the second out-of-focus sub-region being on a side of the in-focus region that is distal to the imaging system, j being a positive integer, and j>i+n.

2. The method of claim 1 , wherein obtaining the depth information respectively corresponding to the one or more objects comprises:
obtaining the spatial configuration of the imaging system when the imaging system captures the image frame.

3. The method of claim 2 , wherein the imaging system comprises a monocular imaging device borne on a movable platform, and wherein one or more sensors are configured to measure translational and rotational movement data of the imaging system during a time period when the image frame is captured by the monocular imaging device.

4. The method of claim 3 , wherein the translational and rotational movement data includes:
a first translational matrix and a first rotational matrix representing a first spatial position corresponding to a start time of the time period, and
a second translational matrix and a second rotational matrix representing a second spatial position corresponding to an end time of the time period, the movable platform moving from the first spatial position to the second spatial position during the time period.

5. The method of claim 3 , wherein the one or more sensors comprise at least an inertial measurement unit (IMU) associated with the movable platform.

6. The method of claim 2 , wherein the one or more sensors comprise a depth sensor configured to provide the depth information corresponding to the one or more objects.

7. The method of claim 1 , further comprising:
smoothing transitions of rendering effects to the plurality of layers.

8. The method of claim 1 , further comprising:
stitching a plurality of raw image frames captured by the imaging system into a panorama view as the image frame, wherein the panorama view includes more objects than any one of the raw image frames; and
rendering the panorama view using the one or more rendering parameters.

9. The method of claim 1 , further comprising:
processing an overlapped area of one or more sets of pixels, one or more regions of the image frame, or one or more of the plurality of layers, by an image smoothing algorithm, to obtain a smooth transitional overlapped area.

10. The method of claim 1 , wherein each layer includes at least a complete object in the image.

11. The method of claim 1 ,
wherein the one or more rendering parameters include a standard deviation of a Gaussian distribution used to convolve the plurality of layers in rendering the image frame.

12. The method of claim 1 , further comprising:
determining a first layer of the plurality of layers to be an in-focus layer corresponding to a first rendering parameter; and
processing a second layer of the plurality of layers and a third layer of the plurality of layers to be separate out-of-focus layers, the second layer and the third layer being adjacent to the first layer, the second layer corresponding to a second rendering parameter, the third layer corresponding to a third rendering parameter, and the first rendering parameter, the second rendering parameter, and the third rendering parameter being different from each other.

13. The method of claim 1 , wherein the one or more rendering parameters corresponding to the layers l i−m , l i−m+1 , . . . . . . , l i+n−1 , l i+n are zeros.

14. The method of claim 1 , wherein:
the one or more rendering parameters including a plurality of rendering parameters σ 1 , σ 2 , . . . . . . , σ j corresponding to the layers l 1 ,l 2 , . . . . . . , l j , respectively, and
σ 1 >σ 2 >. . . >σ i−m−1 and σ i+n+1 <. . . <σ j−1 <σ j .

15. The method of claim 1 , wherein following equations are satisfied:



Δ
⁢

⁢

L
1



=



F
⁢

⁢
σ
⁢

⁢

L
2

f
2

+

F

Assignees

• Sz Dji Technology Co Ltd

Inventors

• Hu Pan
• Zhou You
• Bao Linchao
• Feng Xuyang
• Mao Chenglin
• Cao Zisheng
• Zhao Cong
• Yan Jiaqi

Classifications

• G06T7/50 Primary

  Depth or shape recovery · CPC title

• H04N5/2226 Primary

  Determination of depth image, e.g. for foreground/background separation (determining depth by image analysis in general G06T7/50; segmentation by image analysis in general G06T7/10) · CPC title

• H04N23/671

  in combination with active ranging signals, e.g. using light or sound signals emitted toward objects · CPC title

• H04N23/60

  Control of cameras or camera modules · CPC title

• B64U2101/30

  for imaging, photography or videography · CPC title