Immersive stereoscopic video acquisition, encoding and virtual reality playback methods and apparatus

Patent metadata

Field	Value
Publication number	US-10750154-B2
Application number	US-201715396765-A
Country	US
Kind code	B2
Filing date	Jan 2, 2017
Priority date	Apr 11, 2013
Publication date	Aug 18, 2020
Grant date	Aug 18, 2020

Abstract

Methods and apparatus for capturing and displaying stereoscopic images are described in a manner that allows a user to obtain a 3d virtual reality experience simulating that of being in a seat at a football game or other event. Rear images are modified, e.g., in luminance intensity, to make them consistent with the luminance intensity of the forward images to avoid or reduce edges or differences in luminance intensity as a users turns his head from viewing a main image area to a side or rear image area. A seamless 3D presentation is made possible through the use of fisheye lenses at capture time and combining of images corresponding to forward and rear image areas as a user turns his or her head requiring a change in the captured image area which is displayed to the user.

First claim

What is claimed is:

1. A method of providing content, comprising:
capturing one or more images corresponding to a first viewing direction, said one or more images corresponding to the first viewing direction including a first captured image corresponding to the first viewing direction;
capturing one or more images corresponding to a second viewing direction which is different from said first viewing direction, said one or more images corresponding to the second viewing direction including a second captured image corresponding to the second viewing direction, and wherein said second captured image corresponding to the second viewing direction is captured at a different time than said first captured image corresponding to the first viewing direction;
generating an image based on at least: i) a portion of the first captured image corresponding to the first viewing direction and ii) a portion of the second captured image corresponding to the second viewing direction, wherein said generating an image includes performing a luminance image adjustment operation on said portion of the second captured image corresponding to the second viewing direction to produce a luminance adjusted portion of the second captured image corresponding to the second viewing direction, and combining the luminance adjusted portion of the second captured image corresponding to the second viewing direction with said portion of the first captured image corresponding to the first viewing direction to produce said generated image, wherein performing the luminance image adjustment operation includes adjusting a luminance of said portion of the second captured image corresponding to the second viewing direction based on a luminance of the portion of the first captured image corresponding to the first viewing direction; and
outputting the generated image to a display device for display.

2. The method of claim 1 , wherein performing said luminance image adjustment operation further includes:
determining the luminance of said portion of the first captured image corresponding to the first viewing direction.

3. The method of claim 1 , wherein capturing one or more images corresponding to a first viewing direction includes capturing one or more images using a first camera corresponding to the first viewing direction, said first camera including a fish eye lens.

4. The method of claim 3 , wherein said fish eye lens captures an image corresponding to a 180 degree viewing area or approximately a 180 degree viewing area.

5. The method of claim 3 , wherein said fish eye lens captures an image corresponding to at least a 120 degree area.

6. The method of claim 1 , wherein the first captured image corresponding to the first viewing direction and the second captured image corresponding to the second viewing direction are captured hours apart; and
wherein the first viewing direction is a forward direction relative to a first camera used to capture one or more images corresponding to the first viewing direction and the second viewing direction is a rear direction relative to the first camera.

7. The method of claim 1 , wherein lighting conditions at the time of capture of at least one of the first captured image corresponding to the first viewing direction and the second captured image corresponding to the second viewing direction are affected by a cloud.

8. The method of claim 7 , further comprising:
capturing additional images corresponding to the first viewing direction at a video frame rate.

9. The method of claim 8 , further comprising:
capturing additional images corresponding to the second viewing direction at a second rate which is lower than said video frame rate.

10. The method of claim 9 , wherein said second rate is less than 1/5 said video frame rate.

11. The method of claim 10 , wherein capturing images corresponding to the second viewing direction is performed using a lower resolution camera than is used to capture images in the first viewing direction.

12. The method of claim 1 , further comprising:
making depth measurements of an area surrounding a mount on which cameras used to capture said one or more images corresponding to the first viewing direction are mounted.

13. The method of claim 12 , wherein making depth measurements of an area surrounding a mount include making LIDAR measurements.

14. The method of claim 12 , further comprising:
generating a depth map for the area surrounding said mount; and
transmitting depth map information with at least some image information corresponding to an image captured by a camera on said mount.

15. The method of claim 14 , further comprising:
using said depth map information when generating an image from one or more images captured by a camera on said mount.

16. A system, comprising:
a first camera for capturing one or more images corresponding to a first viewing direction, said one or more images corresponding to the first viewing direction including a first captured image corresponding to the first viewing direction;
a second camera for capturing one or more images corresponding to a second viewing direction which is different from said first viewing direction, said one or more images corresponding to the second viewing direction including a second captured image corresponding to the second viewing direction and wherein said second captured image corresponding to the second viewing direction is captured at a different time than said first captured image corresponding to the first viewing direction;
a player rendering an image based on at least: i) a portion of the first captured image corresponding to the first viewing direction and ii) a portion of the second captured image corresponding to the second viewing direction, said player performing a luminance image adjustment operation to the portion of the second captured image corresponding to the second viewing direction to produce a luminance adjusted portion of the second captured image corresponding to the second viewing direction, and combining the luminance adjusted portion of the second captured image corresponding to the second viewing direction with the portion of the first captured image corresponding to the first viewing direction to generate said image as part of said rendering, and wherein performing the luminance image adjustment operation includes adjusting a luminance of said portion of the second captured image corresponding to the second viewing direction based on a luminance of the portion of the first captured image corresponding to the first viewing direction; and
a display for receiving the rendered image and displaying the rendered image generated by said rendering.

17. The system of claim 16 , wherein said player includes a luminance controller that controls the luminance of a rear portion of an image, as part of a luminance adjustment operation, based on the luminance of a front portion of an image when combining front and rear image portions as part of the image rendering.

18. The system of claim 17 , wherein said luminance controller limits maximum luminance value of the rear image portion to be equal to or less than the maximum luminance value of the front image portion being included in a rendered image.

19. The system of claim 18 , wherein the luminance of a front image portion is used by said luminance controller to control the luminance of a rear image portion.

20. The system of claim 19 , where front and rear image portions which are combined by said player when rendering an image in response to a user turning his he

Assignees

• Nextvr Inc
• Nevermind Capital Llc

Inventors

• Cole David Michael

Classifications

• H04N13/239 Primary

  using two two-dimensional [2D] image sensors having a relative position equal to or related to the interocular distance (H04N13/243 takes precedence) · CPC title

• H04N13/161

  Encoding, multiplexing or demultiplexing different image signal components (for multi-view video sequence encoding H04N19/597) · CPC title

• H04N13/243

  using three or more two-dimensional [2D] image sensors · CPC title

• H04N13/271

  wherein the generated image signals comprise depth maps or disparity maps · CPC title