Omnistereo capture and render of panoramic virtual reality content

Patent metadata

Field	Value
Publication number	US-2016353090-A1
Application number	US-201514723178-A
Country	US
Kind code	A1
Filing date	May 27, 2015
Priority date	May 27, 2015
Publication date	Dec 1, 2016
Grant date	—

Abstract

Systems and methods are described include defining, at a computing device, a set of images based on captured images, projecting, at the computing device, a portion of the set of images from a planar perspective image plane onto a spherical image plane by recasting a plurality of viewing rays associated with the portion of the set of images from a plurality of viewpoints arranged around a curved path to a viewpoint, determining, at the computing device, a periphery boundary corresponding to the viewpoint and generating updated images by removing pixels that are outside of the periphery boundary, and providing, for display, the updated images within the bounds of the periphery boundary.

First claim

What is claimed is:

1 . A computer-implemented method comprising:
defining, at a computing device, a set of images based on captured images; projecting, at the computing device, a portion of the set of images from a planar perspective image plane onto a spherical image plane by recasting a plurality of viewing rays associated with the portion of the set of images from a plurality of viewpoints arranged around a curved path to a viewpoint; determining, at the computing device, a periphery boundary corresponding to the viewpoint and generating updated images by removing pixels that are outside of the periphery boundary; and providing, for display, the updated images within the bounds of the periphery boundary.

2 . The method of claim 1 , wherein the set of images include encoded video content.

3 . The method of claim 1 , wherein the captured images include content acquired with at least one stereo camera pair configured with a one-hundred eighty degree field of view.

4 . The method of claim 1 , wherein the periphery boundary is defined to a field of view of about 120 degrees for a camera associated with the captured images.

5 . The method of claim 1 , wherein the periphery boundary is defined to a field of view of about 150 degrees for a camera associated with the captured images.

6 . The method of claim 1 , wherein the periphery boundary is a portion of a spherical shape corresponding to about 30 degrees above a viewing plane for a camera associated with the captured images.

7 . The method of claim 1 , wherein the periphery boundary is a portion of a spherical shape corresponding to about 30 degrees below a viewing plane for a camera associated with the captured images.

8 . The method of claim 1 , wherein removing pixels outside of the periphery boundary includes replacing the pixels with a color block, a static image, or a blurred representation of the pixels.

9 . The method of claim 1 , further comprising stitching together at least two frames in the set of images by:
sampling a plurality of columns of pixels from the frames and interpolating, between at least two sampled columns of pixels, additional columns of pixels that are not captured in the frames; blending, at the computing device, the sampled columns and the additional columns together to generate a pixel value; generating a three-dimensional stereoscopic panorama comprising video content by configuring, at the computing device, the pixel value into a left scene and a right scene; and providing the panorama to a head-mounted display.

10 . The method of claim 9 , wherein the blending is performed using a stitching ratio selected based at least in part on a diameter of a circular camera rig used to acquire the captured images.

11 . An imaging system configured to generate stereo panoramas, the system comprising:
an interpolation module configured to provide interpolated images from a set of defined images and interleave the interpolated images into the set of images to generate additional virtual content for a stereo panorama; a projection module configured to project the set of images from a planar projection to a spherical projection; a capture correction module configured to adjust the set of images to compensate for a non-circular camera trajectory; a stitching module configured to:
sample portions of images from the set of images and from the interpolated images,
blend the portions of images together to generate at least one pixel value, and
generate a three-dimensional stereoscopic panorama comprising video content by configuring the pixel value into a left scene and a right scene; and
an image correction module configured to estimate optical flow for the set of images to eliminate distortion.

12 . The system of claim 11 , wherein blending the portions of images together includes:
stitching the portions of images together using the optical flow to generate at least one image from the portions of images; and generating the at least one pixel value includes calculating a value that represents a difference between the portions of images and the at least one image.

13 . The system of claim 11 , wherein the capture correction module is further configured to adjust the set of images to compensate for content captured with multiple cameras in which camera separation is larger than about 30 degrees.

14 . The system of claim 11 , wherein the capture correction module is further configured to adjust the set of images to compensate for camera misalignment due to camera pose errors.

15 . The system of claim 11 , wherein estimating optical flow includes adjusting the set of images by rectifying a portion of the set of images, determining, using the portion, an estimated camera pose associated with the portion and determining a flow between images in the portion.

16 . The system of claim 15 , wherein determining the flow includes:
determining, for at least two pixels in the images in the portion, a magnitude and an orientation; and comparing the magnitude and orientation for each of the at least two pixels.

17 . A computer-implemented method comprising:
defining, at a computing device, a set of images based on captured images; stitching, at the computing device, the set of images into an equirectanglar video stream; rendering, at the computing device, the video stream for playback by projecting the video stream from planar perspective to equirectangular perspective for a first view and a second view; determining, at the computing device, a boundary in which distortion is above a predefined threshold, the distortion being based at least in part on projecting the video stream; generating, at the computing device, an updated video stream by removing image content in the set of images at and outside of an interior defined by the boundary; and providing the updated video stream for display.

18 . The method of claim 17 , wherein stitching the set of images into an equirectanglar video stream includes combining images associated with a leftward camera capture angle with images associated with a rightward facing camera capture angle.

19 . The method of claim 17 , wherein the first view corresponds to a left eye view of a head-mounted display and the second view corresponds to a right eye view of the head-mounted display.

Assignees

• Google Inc

Inventors

• Esteban Carlos Hernandez
• Anderson Robert
• Wu Changchang
• Krainin Michael

Classifications

• H04N13/296

  Synchronisation thereof; Control thereof · CPC title

• H04N13/243

  using three or more two-dimensional [2D] image sensors · CPC title

• H04N13/221

  using the relative movement between cameras and objects · CPC title

• H04N13/161

  Encoding, multiplexing or demultiplexing different image signal components (for multi-view video sequence encoding H04N19/597) · CPC title

• H04N13/239

  using two two-dimensional [2D] image sensors having a relative position equal to or related to the interocular distance (H04N13/243 takes precedence) · CPC title