Prism array depth sensing auto-focus

Patent metadata

Field	Value
Publication number	US-9549115-B1
Application number	US-201414492783-A
Country	US
Kind code	B1
Filing date	Sep 22, 2014
Priority date	Sep 22, 2014
Publication date	Jan 17, 2017
Grant date	Jan 17, 2017

Abstract

An depth sensing (AF) module including a depth sensing (DS) lens, DS image sensor, a prism array, and a DS controller is disclosed. The DS module may be configured to provide a control signal to an electromechanical device, such as a voice coil motor (VCM), of an image capture module to move a lens assembly to focus an image of an object to be imaged onto a primary image sensor. The DS module may direct the capture of an image for depth sensing by the DS image sensor and perform a depth sensing (DS) technique on the captured DS image to identify a focus point of the lens assembly. The DS technique may identify a magnitude of a split in segments of the DS image, where the segments correspond to light refracted through different prism elements of the array of prism elements. This determined focus point may be used to displace the lens assembly relative to the primary image sensor to achieve focus of the object onto the primary image sensor.

First claim

That which is claimed:

1. A mobile device, comprising:
a lens to receive light reflected by an object;
a red-green-blue (RGB) image sensor to receive the light via the lens;
a voice coil motor (VCM) mechanically coupled to the lens to move the lens relative to the RGB image sensor;
a prism array having a plurality of prism elements to receive the light reflected by the object;
a panchromatic image sensor to receive light reflected from the object and refracted through the prism array, wherein the prism array is positioned between the object and the panchromatic image sensor;
at least one memory that stores computer-executable instructions; and
at least one processor that accesses the at least one memory, wherein the at least one processor executes the computer-executable instructions to:
receive image data associated with a first image representing the light received by the panchromatic image sensor;
determine a plurality of segments of the first image using the image data by identifying discontinuities in the first image, the plurality of segments including a first image segment and a second image segment;
determine a misalignment distance between the first image segment and the second image segment by determining a level of discontinuity between the first image segment and the second image segment;
determine, using a look-up table, a focus offset distance corresponding to the misalignment distance;
provide a VCM control signal, corresponding to the focus offset distance, to the VCM to place the lens the focus offset distance away from the RGB image sensor at, and

wherein a second image of the object is captured by the RGB image sensor after the lens is displaced.

2. The mobile device of claim 1 , wherein the panchromatic image sensor includes a first plurality of first photosensitive pixels and the RGB image sensor includes a second plurality of second photosensitive pixels, and wherein each of the first photosensitive pixels have an area greater than each of the second photosensitive pixels.

3. The mobile device of claim 1 , wherein the VCM control signal is a second VCM control signal and wherein providing the VCM control signal to the VCM comprises:
generating a first VCM control signal using the focus offset distance;
providing the first VCM control signal to the VCM;
receiving, from a hall sensor coupled to the lens, a first displacement value indicative of a distance of the lens from the RGB image sensor;
determining a first difference value between the focus offset distance and the first displacement value;
determining that the first difference value is greater than a depth of focus (DOF) of the lens;
generating the second VCM control signal corresponding to the first difference;
providing the second VCM control signal to the VCM, wherein the VCM moves the lens by the first difference;
receiving, from the hall sensor, a second displacement value indicative of the distance of the lens from the RGB image sensor;
determining a second difference value between the focus offset distance and the second displacement value; and
determining that the second difference value is less than the DOF of the lens.

4. The mobile device of claim 1 , wherein the misalignment distance is a first misalignment distance, wherein the plurality of segments of the first image further includes a third image segment, and wherein the at least one processor executes the computer-executable instructions to further:
determine a second misalignment distance between the third image segment and the second image segment;
determine a first depth using the first misalignment distance and a look-up table mapping misalignment distances to depth;
determine a second depth using the second misalignment distance and the look-up table; and
generate a depth map including the first depth and the second depth, wherein the depth map indicates depth of features in the first image relative to the lens.

5. A method, comprising:
receiving, by a controller comprising one or more computer processors, image data from a first image sensor, wherein the image data corresponds to light passing through a prism array having a plurality of prism elements;
determining a plurality of segments using the image data;
determining a misalignment distance in the plurality of segments using discontinuities corresponding to the image data;
determining, by the controller and based at least in part on the image data, a focus offset distance corresponding to a lens assembly associated with a second image sensor;
determining, by the controller, an autofocus (AF) control signal based at least in part on the focus offset distance; and
providing, by the controller, the AF control signal to an electromechanical device coupled to the lens assembly, wherein the electromechanical assembly moves the lens assembly, to a distance equal to the focus offset distance, relative to the second image sensor.

6. The method of claim 5 , further comprising:
determining, by the controller, that an image capture has been requested; and
causing, by the controller, acquisition of the image data.

7. The method of claim 5 , wherein the first image sensor includes a plurality of first photosensitive pixels and the second image sensor includes a plurality of second photosensitive pixels, and wherein each of the plurality of first photosensitive pixels have an area greater than each of the plurality of second photosensitive pixels.

8. The method of claim 5 , wherein the misalignment distance is a first misalignment distance, and wherein determining the focus offset distance comprises:
determining, by the controller a first image segment and a second image segment each corresponding to the image data, wherein the first image segment corresponds to a first prism element of the plurality of prism elements and the second image element corresponds to a second prism element of the plurality of prism elements;
determining, by the controller, the first misalignment distance between the first image segment and the second image segment; and
determining, by the controller, the focus offset distance based at least in part on the first misalignment distance and a look-up table.

9. The method of claim 8 , wherein determining the first misalignment distance further comprises:
identifying, by the controller, a first pixel in the first image segment;
identifying a second pixel in the second image segment; and
determine the first misalignment distance as a distance corresponding to a number of pixels of separation between the first pixel and the second pixel.

10. The method of claim 8 , further comprising a second misalignment distance, and wherein the plurality of segments of the image further includes a third image segment, further comprising:
determining, by the controller, a second misalignment distance between the third image segment and the second image segment;
determining, by the controller, a first depth using the first misalignment distance;
determining, by the controller, a second depth using the second misalignment distance; and
generating, by the controller, a depth map including the first depth and the second depth, wherein the depth map indicates depth of features in the image data.

11. The method of claim 10 , wherein the image data is a first image data, and further comprising:
receiving, by the controller, second image data from the second image sensor; and
linking, by the controller, the depth map to the second image data.

12. The method of claim 5 , further comprising causing, by the controller, an illuminator to illuminate an object associated with the image data, wherein the illuminator emits at leas

Assignees

• Amazon Tech Inc

Inventors

• Baldwin Leo Benedict

Classifications

• H04N23/45

  for generating image signals from two or more image sensors being of different type or operating in different modes, e.g. with a CMOS sensor for moving images in combination with a charge-coupled device [CCD] for still images · CPC title

• H04N23/67 Primary

  Focus control based on electronic image sensor signals · CPC title

• H04N23/55

  Optical parts specially adapted for electronic image sensors; Mounting thereof · CPC title

• G02B7/28 Primary

  Systems for automatic generation of focusing signals · CPC title

• H04N23/56

  provided with illuminating means · CPC title