Finite aperture omni-directional stereo light transport

Patent metadata

Field	Value
Publication number	US-2018276879-A1
Application number	US-201815881554-A
Country	US
Kind code	A1
Filing date	Jan 26, 2018
Priority date	Mar 23, 2017
Publication date	Sep 27, 2018
Grant date	—

Abstract

In various embodiments, a finite aperture omni-directional camera is modeled by aligning a finite aperture lens and focal point with the omni-directional part of the projection. For example, each point on an image plane maps to a direction in camera space. For a spherical projection, the lens can be orientated along this direction and the focal point is picked along this direction at focal distance from the lens. For a cylindrical projection, the lens can be oriented along the projected direction on the two dimensional (2D) xz-plane, as the projection is not omni-directional in the y direction. The focal point is picked along the (unprojected) direction so its projection on the xz-plane is at focal distance from the lens. The final outgoing ray can be constructed by sampling of point on this oriented lens and shooting a ray from there through the focal point.

First claim

What is claimed is:

1 . A method of generating an image, said method comprising:
a) accessing a first data model of an environment captured by a virtual reality camera system, said data model representing an image backplate; b) accessing a second data model of objects within a virtualized three dimensional (3D) space, wherein said image backplate is within said space; c) simulating a presence of at least one light source, or camera, within said space and casting a ray from said light source, or said camera into the space, to a point, p, in said space, possibly after interacting with objects in the second data model, using ray tracing techniques; d) simulating a finite aperture with respect to a camera capturing a two dimensional (2D) projection of a 3D scene by projecting a ray from said point p, through a focal point of said camera, to a lens point l within said finite aperture of said camera; e) provided said point p lies within said backplate and is not occluded from a camera performing the following:
e 1 ) based on the point p, performing inverse mapping to obtain an orientation in camera space corresponding to said point p and said lens point l, wherein said orientation is defined by two values;
e 2 ) based on said orientation, performing inverse mapping to obtain a pixel, s, on a capture plane of said camera corresponding to said point p and said lens point l; and
e 3 ) using said pixel s, looking up a color value within the first data model corresponding to pixel s, and using this color value to compute a ray tracing value at said point p; and
f) provided said point p lies within an object of said objects and is not occluded from a camera, performing the following:
f 1 ) based on the point p, performing inverse mapping to obtain a orientation in camera space corresponding to said point p and said lens point l, wherein said orientation is defined by two values;
f 2 ) based on said orientation, performing inverse mapping to obtain a pixel, s, on a capture plane of said camera corresponding to said point p and said lens point l; and
f 3 ) at said pixel s of said capture plane, rendering energy contribution from said point p.

2 . The method as described in claim 1 wherein said virtual reality (VR) camera system is a cylindrical VR 360 camera system and wherein further said two values comprise theta and h and wherein theta is a camera angle within an xz-plane of said space associated with point p and wherein h is a height in a y coordinate in said camera space associated with said point p.

3 . The method as described in claim 1 wherein said virtual reality camera system is a spherical VR 360 camera system and wherein further said two values comprise theta and phi and wherein theta is a camera angle within an xz-plane of said space associated with point p and wherein phi is tilt angle of said camera system with respect to y-axis of said space and associated with said point p.

4 . The method as described in claim 1 wherein said d) simulating said finite aperture with respect to said camera capturing a 2D projection of a 3D scene comprises projecting a plurality of rays from said point p, through said focal point of said camera, to a plurality of lens points l across said finite aperture of said camera and further comprising performing said e) or said f) for each of said plurality of lens points lto determine a corresponding plurality of pixels s.

5 . The method as described in claim 1 wherein said first data model comprises pixels that are referenced by said two values.

6 . The method as described in claim 1 wherein said ray tracing techniques are Monte Carlo ray tracing techniques.

7 . The method as described in claim 1 wherein said method is performed by at least a processor.

8 . The method as described in claim 1 wherein said method is performed by at least a graphics processing unit.

9 . A method of generating an image, said method comprising:
a) accessing a first data model of an environment captured by a virtual reality camera system, said data model representing an image backplate; b) accessing a second data model of objects within a virtualized three dimensional (3D) space, wherein said image backplate is within said space; c) provided a given pixel, s, perform a mapping to a camera orientation, wherein said orientation is defined by two values; d) provided a lens point, I, generating a camera ray through a focal point f, of said oriented camera; e) computing incoming energy along said ray using ray tracing techniques; f) rendering energy contribution at the pixel s of a capture plane; and g) repeating said c)-f) for a second lens position l for each pixel s.

10 . The method as described in claim 9 wherein said virtual reality (VR) camera system is a cylindrical VR 360 camera system and wherein further said two values comprise theta and h and wherein theta is a camera angle within an xz-plane of said space associated with pixel s and wherein h is a height in a y coordinate in said camera space associated with said pixel s.

11 . The method as described in claim 9 wherein said virtual reality camera system is a spherical VR 360 camera system and wherein further said two values comprise theta and phi and wherein theta is a camera angle within an xz-plane of said space associated with pixel s and wherein phi is tilt angle of said camera system with respect to y-axis of said space and associated with said pixel s.

12 . The method as described in claim 9 wherein said first data model comprises pixels that are referenced by said two values.

13 . The method as described in claim 9 wherein said ray tracing techniques are Monte Carlo ray tracing techniques.

14 . The method as described in claim 9 wherein said virtual reality (VR) camera system is a cylindrical VR 360 camera system or a spherical VR 360 camera system..

15 . The method as described in claim 9 wherein said method is performed by at least a processor.

16 . The method as described in claim 9 wherein said method is performed by at least a graphics processing unit.

17 . A computer system comprising:
a processor; and a memory coupled to said processor; wherein said memory comprises instructions for implementing a method of generating an image, said method comprising: a) accessing a first data model of an environment captured by a virtual reality camera system, said data model representing an image backplate; b) accessing a second data model of objects within a virtualized three dimensional (3D) space, wherein said image backplate is within said space; c) simulating a presence of at least one light source, or camera into the space, within said space and casting a ray from said light source, or said camera, to a point, p, in said space, possibly after interacting with objects in the second data model, using ray tracing techniques; d) simulating a finite aperture with respect to a camera capturing a two dimensional (2D) projection of a 3D scene by projecting a ray from said point p, through a focal point of said camera, to a lens point l within said finite aperture of said camera; e) provided said point p lies within said backplate and is not occluded from a camera performing the following:
e 1 ) based on the point p, performing inverse mapping to obtain an orientation in camera space corresponding to said point p and said lens point l, wherein said orientation is defined by two values;
e 2 ) based on said orientation, performing inverse mapping to obtain a pixel, s, on a capture plane of said camera corresponding to said point p and said lens poi

Assignees

• Nvidia Corp

Inventors

• van Antwerpen Dietger

Classifications

• G06T15/80 Primary

  Shading · CPC title

• H04N13/275 Primary

  from three-dimensional [3D] object models, e.g. computer-generated stereoscopic image signals · CPC title

• G06T15/06

  Ray-tracing · CPC title