Capture and render of panoramic virtual reality content

Patent metadata

Field	Value
Publication number	US-10038887-B2
Application number	US-201514723151-A
Country	US
Kind code	B2
Filing date	May 27, 2015
Priority date	May 27, 2015
Publication date	Jul 31, 2018
Grant date	Jul 31, 2018

Abstract

Systems and methods are described for defining a set of images based on captured images, receiving a viewing direction associated with a user of a virtual reality (VR) head mounted display, receiving an indication of a change in the viewing direction. The methods further include configuring, a re-projection of a portion of the set of images, the re-projection based at least in part on the changed viewing direction and a field of view associated with the captured images, and converting the portion from a spherical perspective projection into a planar perspective projection, rendering by the computing device and for display in the VR head mounted display, an updated view based on the re-projection, the updated view configured to correct distortion and provide stereo parallax in the portion, and providing, to the head mounted display, the updated view including a stereo panoramic scene corresponding to the changed viewing direction.

First claim

What is claimed is:

1. A computer-implemented method comprising:
defining, at a computing device, a set of images based on captured images;
receiving, at the computing device, a viewing direction associated with a user of a virtual reality (VR) head mounted display;
receiving, at the computing device, an indication of a change in the viewing direction;
in response to receiving the indication;
configuring, by the computing device, a re-projection of a portion of the set of images, the re-projection based at least in part on the changed viewing direction and a field of view associated with the captured images,
converting the portion, using the re-projection, from a spherical perspective projection into a planar perspective projection to reduce vertical parallax in the portion, and
generating, by the computing device, an updated view based on the re-projection, the updated view configured to correct distortion and provide, in the portion, stereo effects for the user of the head mounted display; and

triggering for display, in the head mounted display, the updated view including a stereo panoramic scene corresponding to the changed viewing direction.

2. The method of claim 1 , wherein the re-projection further comprises recasting a portion of viewing rays associated with the set of images from a plurality of viewpoints arranged around a circular shape.

3. The method of claim 1 , wherein the set of images are captured using a camera rig that is circular and configured to move, during capture, in an arcing motion parallel to a rotatable base of the camera rig, the camera rig including the base, a mount plate, and at least one stereo pair of cameras, the at least one stereo pair including:
a first camera, placed on the mount plate, and configured to point in a viewing direction that is tangential to an edge of the mount plate and arranged to point toward a leftward direction; and
a second camera, placed on the mount plate in a side-by-side fashion to the first camera and placed at an interpupillary distance from the first camera, the second camera arranged to point in a viewing direction that is tangential to an edge of the mount plate and arranged to point toward a rightward direction.

4. The method of claim 3 , wherein the camera rig is configured with a diameter of about 10 centimeters to about 75 centimeters.

5. The method of claim 3 , wherein the interpupillary distance includes about 5 centimeters to about 6.5 centimeters.

6. The method of claim 1 , wherein the field of view associated with the captured images is about one-hundred eighty degrees.

7. The method of claim 1 , wherein the set of images correspond to rendered computer graphics (CG) content.

8. A computer-implemented method for capturing and generating stereo panoramic content, the method comprising:
rotating a camera rig in a first direction while simultaneously capturing, using a stereo pair of cameras arranged on the camera rig, a first set of images associated with a scene surrounding the camera rig;
rotating the camera rig in a second direction while simultaneously capturing, using the stereo pair, a second set of images associated with the scene, the camera rig being circular and configured to move, during capture, in an arcing motion parallel to a base of the camera rig, the stereo pair placed with a viewing direction that is offset at opposite sides of a center of the base of the camera rig, the stereo pair being arranged on the camera rig at approximate human interpupillary distance apart;
generating a combined set of images from the first set of images and the second set of images;
performing a re-projection of a portion of the combined set of images, the re-projection based at least in part on a field of view associated with the first set of images or the second set of images; and
generating, using the re-projected portion, at least one stereoscopic panorama of the scene.

9. The method of claim 8 , wherein the scene surrounding the camera rig comprises a three-hundred sixty degree field and the first set of images and the second set of images depict different portions of degrees within the three-hundred sixty degree field.

10. The method of claim 8 , wherein the camera rig is circular and includes a rotatable base and a mount plate, and the stereo pair of cameras includes a first camera, placed on the mount plate, and configured to point in a viewing direction that is tangential to an edge of the mount plate and arranged to point toward a leftward direction, and a second camera, placed on the mount plate in a side-by-side fashion to the first camera and placed at an interpupillary distance from the first camera, the second camera arranged to point in a viewing direction that is tangential to an edge of the mount plate and arranged to point toward a rightward direction.

11. The method of claim 8 , wherein the re-projection is performed to correct distortion in the portion.

12. The method of claim 8 , wherein the stereo pair comprises at least one pair of a video camera, an image sensor, a stereoscopic camera, an infrared camera, or a mobile device.

13. A computer-implemented system comprising:
at least one processor;
memory storing instructions that, when executed by the at least one processor, cause the system to perform operations including:
defining, at a computing device, a set of images based on captured images;
receiving, at the computing device, a viewing direction associated with a user of a virtual reality (VR) head mounted display;
receiving, at the computing device, an indication of a change in the viewing direction;
in response to receiving the indication:
configuring, by the computing device, a re-projection of a portion of the set of images, the re-projection based at least in part on the changed viewing direction and a field of view associated with the captured images,
converting the portion, using the re-projection, from a spherical perspective projection into a planar perspective projection to reduce vertical parallax in the portion, and
generating, by the computing device, an updated view based on the re-projection, the updated view configured to correct distortion and provide, in the portion, stereo effects for the user of the head mounted display; and

triggering for display, in the head mounted display, the updated view including a stereo panoramic scene corresponding to the changed viewing direction.

14. The system claim 13 , wherein the re-projection further comprises recasting a portion of viewing rays associated with the set of images from a plurality of viewpoints arranged around a circular shape.

15. The system claim 13 , wherein the set of images are captured using a camera rig that is circular and configured to move, during capture, in an arcing motion parallel to a rotatable base of the camera rig, the camera rig including the base, a mount plate, and at least one stereo pair of cameras, the at least one stereo pair including:
a first camera, placed on the mount plate, and configured to point in a viewing direction that is tangential to an edge of the mount plate and arranged to point toward a leftward direction; and
a second camera, placed on the mount plate in a side-by-side fashion to the first camera and placed at an interpupillary distance from the first camera, the second camera arranged to point in a viewing direction that is tangential to an edge of the mount plate and arranged to point toward a rightward direction.

16. The system claim 15 , wherein the camera rig is configured with a diameter of about 10 centimeters to about 75 centimeters.

Assignees

• Google Inc
• Google Llc

Inventors

• Gallup David
• Anderson Robert
• Esteban Carlos Hernandez
• Seitz Steven Maxwell
• Adams Riley

Classifications

• G06T15/205

  Image-based rendering · CPC title

• H04N13/0011

  Electricity · mapped topic

• H04N13/0239

  Electricity · mapped topic

• H04N13/0048

  Electricity · mapped topic

• H04N13/0296

  Electricity · mapped topic