Free-viewpoint photorealistic view synthesis from casually captured video

Patent metadata

Field	Value
Publication number	US-10893250-B2
Application number	US-201916574652-A
Country	US
Kind code	B2
Filing date	Sep 18, 2019
Priority date	Jan 14, 2019
Publication date	Jan 12, 2021
Grant date	Jan 12, 2021

Abstract

A respective target viewpoint may be rendered for each of a plurality of multiplane images of a three-dimensional scene. Each of the multiplane images may be associated with a respective single plane image of the three-dimensional scene captured from a respective viewpoint. Each of the multiplane images may include a respective plurality of depth planes. Each of the depth planes may include a respective plurality of pixels from the respective single plane image. Each of the pixels in the depth plane may be positioned at approximately the same distance from the respective viewpoint. A weighted combination of the target viewpoint renderings may be determined, where the sampling density of the single plane images is sufficiently high that the weighted combination satisfies the inequality in Equation (7). The weighted combination of the target viewpoint renderings may be transmitted as a novel viewpoint image.

First claim

The invention claimed is:

1. A method comprising:
rendering a plurality of target viewpoint images based on plurality of multiplane images of a three-dimensional scene, each of the plurality of multiplane images corresponding with a respective one of the plurality of target viewpoint images, each of the plurality of multiplane images associated with a respective one of a plurality of single plane images of the three-dimensional scene, each of the plurality of single plane images being captured from a respective viewpoint, each of the plurality of multiplane images including a respective plurality of depth planes, each of the respective plurality of depth planes including a respective plurality of pixels from the respective single plane image, each of the respective plurality of pixels in the respective plurality of depth planes being positioned at approximately a same distance from the respective viewpoint;
determining a weighted combination of the respective target viewpoint image for each of the plurality of multiplane images via a processor at a computing device, wherein a sampling density of the plurality of single plane images is sufficiently high that the weighted combination satisfies an inequality wherein a maximum pixel disparity of any scene point between adjacent ones of the plurality of target viewpoint images is less than or equal to a minimum of: (a) a number of depth layers associated with the plurality of multiplane images and (b) half of a target rendering resolution for a novel viewpoint image; and
transmitting the weighted combination of the respective target viewpoint image for each of the plurality of multiplane images as the novel viewpoint image.

2. The method recited in claim 1 , wherein each respective target viewpoint image for each of the plurality of multiplane images is associated with a respective set of color values, and wherein determining the weighted combination of the respective target viewpoint image for each of the plurality of multiplane images comprises applying one or more blending weights to the respective set of color values.

3. The method recited in claim 1 , wherein each of the plurality of multiplane images is associated with a respective set of alpha values, and wherein each respective target viewpoint image for each of the plurality of multiplane images is determined based on alpha weights determined based on the respective set of alpha values.

4. The method recited in claim 1 , wherein each of the plurality of multiplane images is associated with a respective set of alpha values, and wherein the weighted combination is determined at least in part based on the respective set of alpha values.

5. The method recited in claim 1 , wherein the weighted combination of the respective target viewpoint image for each of the plurality of multiplane images is determined based on a plurality of weights, each of the plurality of weights being determined based on a respective distance between a target viewpoint and a corresponding viewpoint of one of the multiplane images.

6. The method recited in claim 1 , wherein determining the weighted combination of the respective target viewpoint image for each of the plurality of multiplane images involves rasterizing each of the respective plurality of depth planes as a respective texture-mapped rectangle in three-dimensional space.

7. The method recited in claim 6 , wherein the respective plurality of depth planes are combined via a fragment shader.

8. The method recited in claim 1 , wherein the three-dimensional scene includes a non-Lambertian material, and wherein the three-dimensional scene includes a specularity represented at a respective one or more virtual depths in each of the plurality of multiplane images.

9. The method recited in claim 1 , wherein the novel viewpoint image is transmitted to a storage device for storage.

10. The method recited in claim 1 , wherein the novel viewpoint image is transmitted to a display screen for live presentation.

11. The method recited in claim 10 , wherein the novel viewpoint image is displayed on the display screen in real time during a three-dimensional navigation of the three-dimensional scene via the display screen.

12. The method recited in claim 1 , wherein each respective single plane image is a respective photo of the three-dimensional scene.

13. The method recited in claim 12 , wherein each respective photo of the three-dimensional scene is captured via a camera at a mobile computing device.

14. The method recited in claim 1 , wherein the computing device is a smartphone.

15. A computing device comprising a processor and a memory storing instructions for execution by the processor to perform a method, the method comprising:
rendering a plurality of target viewpoint images based on a plurality of multiplane images of a three-dimensional scene, each of the plurality of multiplane images corresponding with a respective one of the plurality of target viewpoint images, each of the plurality of multiplane images associated with a respective one of a plurality of single plane images of the three-dimensional scene, each of the plurality of single plane images being captured from a respective viewpoint, each of the plurality of multiplane images including a respective plurality of depth planes, each of the respective plurality of depth planes including a respective plurality of pixels from the respective single plane image, each of the respective plurality of pixels in the respective plurality of depth planes being positioned at approximately a same distance from the respective viewpoint;
determining a weighted combination of the respective target viewpoint image for each of the plurality of multiplane images via a processor at a computing device, wherei n a sampling density of the plurality of single plane images is sufficiently high that the weighted combination satisfies an inequality wherein a maximum pixel disparity of any scene point between adjacent ones of the plurality of target viewpoint images is less than or equal to a minimum of: (a) a number of depth layers associated with the plurality of multiplane images and (b) half of a target rendering resolution for a novel viewpoint image; and
transmitting the weighted combination of the respective target viewpoint image for each of the plurality of multiplane images as the novel viewpoint image.

16. The computing device recited in claim 15 , wherein each respective target viewpoint image for each of the plurality of multiplane images is associated with a respective set of color values, and wherein determining the weighted combination of the respective target viewpoint image for each of the plurality of multiplane images comprises applying one or more blending weights to the respective set of color values.

17. The computing device recited in claim 15 , wherein each of the plurality of multiplane images is associated with a respective set of alpha values, and wherein each respective target viewpoint image for each of the plurality of multiplane images is determined based on alpha weights determined based on the respective set of alpha values.

18. The computing device recited in claim 15 , wherein each of the plurality of multiplane images is associated with a respective set of alpha values, and wherein the weighted combination is determined at least in part based on the respective set of alpha values.

19. The computing device recited in claim 15 , wherein the weighted combination of the respective target viewpoint image for each of the plurality of multiplane images is determined

Assignees

• Fyusion Inc

Inventors

• Kar Abhishek
• Cayon Rodrigo Ortiz
• Mildenhall Ben
• Holzer Stefan Johannes Josef
• Rusu Radu Bogdan

Classifications

• H04N23/64

  Computer-aided capture of images, e.g. transfer from script file into camera, check of taken image quality, advice or proposal for image composition or decision on when to take image · CPC title

• H04N23/635

  Region indicators; Field of view indicators · CPC title

• H04N23/6811

  based on the image signal · CPC title

• H04N23/632

  for displaying or modifying preview images prior to image capturing, e.g. variety of image resolutions or capturing parameters · CPC title

• G06V10/247

  by affine transforms, e.g. correction due to perspective effects; Quadrilaterals, e.g. trapezoids · CPC title