Image signal processor for generating depth map from phase detection pixels and device having the same

Patent metadata

Field	Value
Publication number	US-9824417-B2
Application number	US-201615063109-A
Country	US
Kind code	B2
Filing date	Mar 7, 2016
Priority date	Mar 9, 2015
Publication date	Nov 21, 2017
Grant date	Nov 21, 2017

Abstract

An image signal processor including a CPU is provided. The CPU receives image data and positional information of phase detection pixels from an imaging device, extracts first phase detection pixel data and second phase detection pixel data from the image data using the positional information of phase detection pixels, computes first phase graphs from the first phase detection pixel data based upon moving a first window, computes second phase graphs from the second phase detection pixel data based upon moving a second window, computes disparities of the phase detection pixels using the first phase graphs and the second phase graphs, and generates a depth map using the disparities.

First claim

What is claimed is:

1. An image signal processor comprising:
a central processing unit (CPU) configured to receive image data and positional information of phase detection pixels from an imaging device, extract first phase detection pixel data and second phase detection pixel data from the image data using the positional information of the phase detection pixels, compute first phase graphs from the first phase detection pixel data based upon moving a first window, compute second phase graphs from the second phase detection pixel data based upon moving a second window, compute disparities between the first phase graphs and the second phase graphs, and generate a depth map based upon the disparities, wherein:
a size of the first window is set to be n*m pixels (the n, and m are the natural number more than 2),
using the first window present at a first position at a first time, each first axis value of each of the first phase graph for the first window is computed as a combination of values of each column of the first window or a combination of values of each row of the first window,
a size of the second window is set to be n*m pixels (the n, and m are the natural number more than 2),
using the second window present at the first position at the first time, each first axis value of each of the second phase graph for the second window is computed as a combination of values of each column of the second window or a combination of values of each row of the second window.

2. The image signal processor of claim 1 , wherein the CPU is configured to convert the disparities into gray levels, and generate the depth map based upon the gray levels.

3. The image signal processor of claim 1 , wherein the CPU is configured to set the first window and the second window having a same size as each other, and compute each of the first phase graphs and each of the second phase graphs based upon moving the first window and the second window in a scanning direction.

4. The image signal processor of claim 1 , wherein each size of the first phase detection pixel data and the second phase detection pixel data is greater than a size of the depth map.

5. The image signal processor of claim 1 , wherein the CPU is configured to generate three-dimensional image data using color data included in the image data and the depth map.

6. The image signal processor of claim 1 , wherein the CPU is configured to recognize a gesture of a user corresponding to the image data using the depth map.

7. A mobile computing device comprising:
an imaging device including color pixels and phase detection pixels; and
an application processor including an image signal processor having a first CPU, the application processor including a second CPU, wherein:
at least one of the first CPU and the second CPU is configured to receive image data and positional information of the phase detection pixels from the imaging device, extract first phase detection pixel data and second phase detection pixel data from the image data using the positional information of phase detection pixels, compute first phase graphs from the first phase detection pixel data based upon moving a first window, compute second phase graphs from the second phase detection pixel data based upon moving a second window, compute disparities between the first phase graphs and the second phase graphs, and generate a depth map based upon the disparities,
a size of the first window is set to be n*m pixels (the n, and m are the natural number more than 2),
using the first window present at a first position at a first time, each first axis value of each of the first phase graph for the first window is computed as a combination of values of each column of the first window or a combination of values of each row of the first window,
a size of the second window is set to be n*m pixels (the n, and m are the natural number more than 2),
using the second window present at the first position at the first time, each first axis value of each of the second phase graph for the second window is computed as a combination of values of each column of the second window or a combination of values of each row of the second window.

8. The mobile computing device of claim 7 , further comprising a memory configured to store the image data and the positional information of the phase detection pixels.

9. The mobile computing device of claim 7 , wherein the at least one of the first CPU and the second CPU is configured to convert the disparities into gray levels, and generate the depth map based upon the gray levels.

10. The mobile computing device of claim 7 , wherein the application processor further includes a graphics processing unit (GPU), and the at least one of the first CPU and the second CPU is configured to generate three-dimensional image data using color data included in the image data and the depth map, and the GPU is configured to render the three-dimensional image data.

11. The mobile computing device of claim 7 , wherein the at least one of the first CPU and the second CPU is configured to determine a gesture of a user corresponding to the image data using the depth map.

12. The mobile computing device of claim 7 , wherein the at least one of the first CPU and the second CPU is configured to generate a control signal for auto-focus on a region of interest (ROI) using the depth map, and transmit the control signal to the imaging device.

13. The mobile computing device of claim 7 , wherein the at least one of the first CPU and the second CPU is configured to set the first window and the second window having a same size as each other, and generate each of the first phase graphs and each of the second phase graphs based upon moving the first window and the second window in a scanning direction at a same time.

14. The mobile computing device of claim 7 , wherein each size of the first phase detection pixel data and the second phase detection pixel data is greater than a size of the depth map.

15. An image processing system comprising:
an imaging device including color pixels and phase detection pixels;
a memory configured to store image data and positional information of the phase detection pixels;
a controller including electronic circuitry configured to generate first phase detection pixel data and second phase detection pixel data from the image data using the positional information of the phase detection pixels, generate first phase representations from the first phase detection pixel data based upon moving a first window, generate second phase representations from the second phase detection pixel data based upon moving a second window, determine disparities between the first phase representations and the second phase representations, and generate a depth map based upon the disparities; and
a display configured to display images from the imaging device based upon the controller, wherein:
a size of the first window is set to be n*m pixels (the n, and m are the natural number more than 2),
using the first window present at a first position at a first time, each first axis value of each of the first phase representations for the first window is computed as a combination of values of each column of the first window or a combination of values of each row of the first window,
a size of the second window is set to be n*m pixels (the n, and m are the natural number more than 2),
using the second window present at the first position at the first time, each first axis value of each of the second phase representations for the second window is computed as a combination of values of each column of the second window or a combination of

Assignees

• Samsung Electronics Co Ltd

Inventors

• Kim Dong Hoon
• Cho Ju Hyun

Classifications

• H04N25/704

  Pixels specially adapted for focusing, e.g. phase difference pixel sets · CPC title

• H04N23/672

  based on the phase difference signals · CPC title

• G06T7/557 Primary

  from light fields, e.g. from plenoptic cameras · CPC title

• G06T2200/21

  involving computational photography · CPC title

• G06T2207/10024

  Color image · CPC title