Multi-baseline camera array system architectures for depth augmentation in VR/AR applications

Patent metadata

Field	Value
Publication number	US-11368662-B2
Application number	US-202017068243-A
Country	US
Kind code	B2
Filing date	Oct 12, 2020
Priority date	Apr 19, 2015
Publication date	Jun 21, 2022
Grant date	Jun 21, 2022

Abstract

Embodiments of the invention provide a camera array imaging architecture that computes depth maps for objects within a scene captured by the cameras, and use a near-field sub-array of cameras to compute depth to near-field objects and a far-field sub-array of cameras to compute depth to far-field objects. In particular, a baseline distance between cameras in the near-field subarray is less than a baseline distance between cameras in the far-field sub-array in order to increase the accuracy of the depth map. Some embodiments provide an illumination near-IR light source for use in computing depth maps.

First claim

What is claimed is:

1. A method of estimating distances to objects within a scene from a set of images captured from different cameras in a set of cameras using a processor configured by an image processing application, the method comprising:
generating an initial depth map of a scene;
identify near-field and far-field portions of the scene;
refine the depth map for the near-field portions of the scene using image data captured from a near-field set of cameras; and
refine the depth map for the far-filed portions of the scene using image data captured from a far-field set of cameras, wherein a baseline distance between cameras in the near-field set of cameras is less than a baseline distance between cameras in the far-field set of cameras.

2. The method of claim 1 , further comprising illuminating the scene using an illumination light source positioned within a threshold distance from a reference camera.

3. The method of claim 1 , wherein the near-field portions of the scene are at a distance less than a certain threshold and the far-field portions of the scene are at a distance greater than the certain threshold.

4. The method of claim 1 , wherein each camera in the set of cameras has the same imaging properties.

5. The method of claim 1 , where the set of cameras have different field of views.

6. The method of claim 1 , wherein the set of cameras have different resolutions.

7. The method of claim 1 , wherein the set of cameras image in different portions of a light spectrum.

8. The method of claim 1 , wherein the set of cameras are monochrome cameras with a same field of view and resolution.

9. The method of claim 8 , wherein the monochrome cameras image across at least a portion of the visible spectrum and at least a portion of the near-infrared (near-IR) spectrum.

10. The method of claim 1 , further comprising using the depth map in order to determine where to render an object on a display of a virtual reality (VR) application such that the object is appropriately occluded by a real world object also being displayed within the VR application.

11. The method of claim 1 , further comprising using the depth map in order to determine where to render an object on a display of an augmented reality (AR) application such that the object is appropriately occluded by a real world object visible through a transparent display within the AR application.

12. The method of claim 1 , wherein the set of cameras are mounted within a headset that includes a display via which images can be displayed.

13. The method of claim 12 , wherein a reference camera and the display are part of a removable component that is mounted within the headset.

14. A non-transitory machine readable medium containing processor instructions for estimating distances to objects within a scene from a set of images captured from different cameras in a set of cameras, where execution of the instructions by a processor causes the processor to perform a process that comprises:
generating an initial depth map of a scene;
identify near-field and far-field portions of the scene;
refine the depth map for the near-field portions of the scene using image data captured from a near-field set of cameras; and
refine the depth map for the far-filed portions of the scene using image data captured from a far-field set of cameras, wherein a baseline distance between cameras in the near-field set of cameras is less than a baseline distance between cameras in the far-field set of cameras.

15. The non-transitory machine readable medium of claim 14 , further comprising illuminating the scene using an illumination light source positioned within a threshold distance from a reference camera.

16. T The non-transitory machine readable medium of claim 14 , wherein the near-field portions of the scene are at a distance less than a certain threshold and the far-field portions of the scene are at a distance greater than the certain threshold.

17. The non-transitory machine readable medium of claim 14 , wherein each camera in the set of cameras has the same imaging properties.

18. The non-transitory machine readable medium of claim 14 , where the set of cameras have different field of views.

19. The non-transitory machine readable medium of claim 14 , wherein the set of cameras have different resolutions.

Assignees

• Fotonation Ltd

Inventors

• Venkataraman Kartik
• Lelescu Dan
• Duparre Jacques

Classifications

• H04N23/11

  for generating image signals from visible and infrared light wavelengths · CPC title

• H04N23/56

  provided with illuminating means · CPC title

• H04N23/45

  for generating image signals from two or more image sensors being of different type or operating in different modes, e.g. with a CMOS sensor for moving images in combination with a charge-coupled device [CCD] for still images · CPC title

• H04N23/90

  Arrangement of cameras or camera modules, e.g. multiple cameras in TV studios or sports stadiums · CPC title

• H04N13/254

  in combination with electromagnetic radiation sources for illuminating objects · CPC title