Apparatus including multiple cameras and image processing method

Patent metadata

Field	Value
Publication number	US-10848746-B2
Application number	US-201916547435-A
Country	US
Kind code	B2
Filing date	Aug 21, 2019
Priority date	Dec 14, 2018
Publication date	Nov 24, 2020
Grant date	Nov 24, 2020

Abstract

An apparatus includes a first camera module providing a first image of an object with a first field of view, a second camera module providing a second image of the object with a second field of view different from the first field of view, a first depth map generator that generates a first depth map of the first image based on the first image and the second image, and a second depth map generator that generates a second depth map of the second image based on the first image, the second image, and the first depth map.

First claim

What is claimed is:

1. An apparatus comprising:
a first camera module configured to obtain a first image of an object with a first field of view;
a second camera module configured to obtain a second image of the object with a second field of view different from the first field of view;
a first depth map generator configured to generate a first depth map of the first image based on the first image and the second image; and
a second depth map generator configured to generate a second depth map of the second image based on the first image, the second image, and the first depth map.

2. The apparatus of claim 1 , wherein the first field of view is a narrow angle and the second field of view is a wider angle.

3. The apparatus of claim 2 , wherein the second image is divided into a primary region and a residual region, and
the second depth map generator comprises:
a relationship estimating module configured to estimate a relationship between the primary region and the residual region based on the first image and the second image; and
a depth map estimating module configured to estimate a depth map of the residual region based on the estimated relationship and the first depth map.

4. The apparatus of claim 3 , wherein at least one of the relationship estimating module and the depth map estimating module performs an estimating operation based on a neural network module.

5. The apparatus of claim 1 , further comprising:
a depth map fusion unit configured to generate a third depth map of the second image by performing a fusion operation based on the first depth map and the second depth map.

6. The apparatus of claim 5 , wherein the depth map fusion unit comprises:
a tone mapping module configured to generate a tone-mapped second depth map to correspond to the first depth map by performing a bias removing operation on the second depth map; and
a fusion module configured to generate the third depth map by fusing the tone-mapped second depth map and the first depth map.

7. The apparatus of claim 6 , wherein the depth map fusion unit further comprises a propagating module configured to generate a propagated first depth map in the second image by iterated propagating of the first depth map based on the first depth map and the second image, and
the fusion module generates the third depth map by fusing the tone-mapped second depth map and the propagated first depth map.

8. The apparatus of claim 6 , wherein the depth map fusion unit further comprises a post-processing module configured to perform a post-processing operation on the third depth map generated by the fusion module to provide the post-processed third depth map.

9. The apparatus of claim 8 , wherein the post-processing module performs the post-processing operation by filtering an interface generated in the third depth map in accordance with fusion of the fusion module.

10. The apparatus of claim 8 , wherein the post-processing module removes artifacts generated in the third depth map in accordance with fusion of the fusion module.

11. The apparatus of claim 1 , wherein the first depth map generator analyses a distance relationship between the first image and the second image, and generates a first depth map of the first image based on the distance relationship.

12. A method of processing an image of an electronic apparatus, the method comprising:
obtaining a first image of an object using a first camera module;
obtaining a second image of the object using a second camera module;
generating a first depth map of the first image based on the first image and the second image;
estimating a relationship between a primary region of the second image and a residual region of the second image based on the first image and the second image; and
generating a second depth map of the second image based on the estimated relationship between the primary region and the residual region, and the first depth map.

13. The method of claim 12 , wherein the electronic apparatus comprises a first camera module including a first lens having a first field of view and a second camera module including a second lens having a second field of view wider than the first field of view.

14. The method of claim 13 , wherein the generating of the second depth map comprises:
estimating a depth map of the residual region based on the estimated relationship between the primary region and the residual region, and the first depth map; and
generating the second depth map based on a depth map of the residual region and the first depth map.

15. The method of claim 12 , wherein the estimating of the relationship between a primary region of the second image is performed using a neural network model.

16. The method of claim 12 , further comprising:
performing a pre-processing operation on the second depth map; and
generating a third depth map of the residual image by fusing the second depth map on which the pre-processing operation is performed and the first depth map.

17. The method of claim 16 , wherein the performing of the pre-processing operation comprises performing a tone mapping operation between a depth map of the primary region and a depth map of the residual region based on the second depth map.

18. An operating method for an electronic apparatus,
the electronic apparatus including; a first camera module providing a first image of an object using a first field of view and a second camera module providing a second image of the object using second field of view wider than the first field of view, and
a processor generating a depth map of the second image based on a primary region of the second image and a residual region of the second image,
the operating method comprising:
generating a first depth map of the primary region by estimating a relationship between the first image and the second image;
estimating a relationship between the primary region and the residual region based on the first image and the second image;
generating a second depth map of the second image by estimating a depth map of the second region based on the estimated relationship between the primary region and the residual region; and
generating a depth map of the second image by fusing the first depth map and the second depth map.

19. The operation method of claim 18 , further comprising:
executing an application that applies an image effect to the second image based on a depth map of the residual image.

20. The operation method of claim 19 , wherein the application applies at least one image effect of auto-focusing, out-focusing, fore/background separation, face recognition, object detection within a frame, and augmented reality to the second image based on a depth map of the second image.

Assignees

• Samsung Electronics Co Ltd

Inventors

• Han Seung-Min
• Guo Kai
• Song Seong-Wook

Classifications

• H04N13/257

  Colour aspects · CPC title

• H04N9/643

  Hue control means, e.g. flesh tone control · CPC title

• H04N23/57

  Mechanical or electrical details of cameras or camera modules specially adapted for being embedded in other devices · CPC title

• H04N13/271 Primary

  wherein the generated image signals comprise depth maps or disparity maps · CPC title

• H04N13/243

  using three or more two-dimensional [2D] image sensors · CPC title