Parameterizing 3D scenes for volumetric viewing

Patent metadata

Field	Value
Publication number	US-10460509-B2
Application number	US-201816179715-A
Country	US
Kind code	B2
Filing date	Nov 2, 2018
Priority date	Nov 7, 2017
Publication date	Oct 29, 2019
Grant date	Oct 29, 2019

Abstract

A target view to a 3D scene depicted by a multiview image is determined. The multiview image comprising sampled views at sampled view positions distributed throughout a viewing volume. Each sampled view in the sampled views comprises a wide-field-of-view (WFOV) image and a WFOV depth map as seen from a respective sampled view position in the sampled view positions. The target view is used to select, from the sampled views, a set of sampled views. A display image is caused to be rendered on a display of a wearable device. The display image is generated based on a WFOV image and a WFOV depth map for each sampled view in the set of sampled views.

First claim

What is claimed is:

1. A method for selecting sampled views of multiview images, comprising:
determining a target view to a 3D scene depicted by a multiview image, the multiview image comprising a plurality of sampled views at a plurality of sampled view positions distributed throughout a viewing volume, each sampled view in the plurality of sampled views of the multiview image comprising a wide-field-of-view (WFOV) image and a WFOV depth map corresponding to the WFOV image, each sampled view of the multiview image in the plurality of sampled views of the multiview image corresponding to a respective sampled view position in the plurality of sampled view positions;
using the target view to select, from the plurality of sampled views of the multiview image, a set of sampled views, each sampled view in the plurality of sampled views corresponding to a respective viewpoint to the 3D scene;
causing a display image to be rendered on a display of a wearable device of a user, the display image being generated based on one or more portions of the WFOV image and one or more portions of the WFOV depth map for each such sampled view in the set of sampled views.

2. The method of claim 1 , wherein the multiview image is a part of a sequence of multiview images indexed by a sequence of time instants.

3. The method of claim 1 , wherein the target view is determined based on a spatial position and a spatial direction of a wearable device operating in conjunction with the downstream device.

4. The method of claim 1 , wherein the target view has a view position that does not coincide with a sampled view position of any sampled view in the plurality of sampled views.

5. The method of claim 1 , wherein the set of sampled views represents a proper subset of the plurality of sampled views of the multiview image.

6. The method of claim 1 , wherein the set of sampled views comprises neighboring sampled views, relative to the target view, that are selected from the plurality of sampled views based on linear distances of view positions of the neighboring sampled views to a view position of the target view.

7. The method of claim 1 , wherein the viewing volume represents one of: a cubic volume, a spherical volume, a cylindrical volume, a tetrahedral volume, or a non-cubic-non-spherical bounded volume.

8. The method of claim 1 , wherein at least some of the plurality of sampled view positions are distributed in the viewing volume's interior regions.

9. The method of claim 1 , wherein at least some of the plurality of sampled view positions are distributed on an outer surface enclosing the viewing volume.

10. The method of claim 1 , wherein each of the WFOV image and the WFOV depth map covers a field of view that is no less than a hemisphere.

11. The method of claim 1 , wherein each of the WFOV image and the WFOV depth map comprises image or depth data indexed on a spherical surface that is no less than a hemispheric surface.

12. The method of claim 1 , wherein each of the WFOV image and the WFOV depth map comprises image or depth data indexed on one or more non-spherical surfaces that cover a solid angle no less than a solid angle of a hemispheric surface.

13. The method of claim 1 , wherein an overall solid angle covered by each of the WFOV image and the WFOV depth map is greater than a solid angle of a hemispheric surface; and wherein one or more solid angle portions covered by the one or more portions of the WFOV image is no greater than a solid angle of the hemispheric surface.

14. The method of claim 1 , wherein an overall solid angle covered by each of the WFOV image and the WFOV depth map is greater than a solid angle of a hemispheric surface; and wherein the one or more portions of the WFOV image covers a solid angle between a solid angle of the hemispheric surface and a solid angle of a viewport of the display of the wearable device.

15. The method of claim 1 , where the plurality of sampled views of the multiview image comprises real-world multiview content that is generated based at least in part on applying 3D reconstruction and depth estimation techniques on real-world image data acquired by an image acquisition system that comprises a plurality of cameras located at different spatial positions.

16. A method for constructing display images from sampled views of multiview images, comprising:
receiving one or more portions of a wide-field-of-view (WFOV) image and one or more portions of a WFOV depth map for each sampled view in a set of sampled views of a multiview image, each sampled view in the set of sampled views corresponding to a respective viewpoint in a set of viewpoints to a 3D scene;
warping the one or more portions of the WFOV image for each such sampled view in the set of sampled views to a set of warped images of a target view based on the one or more portions of the WFOV depth map for each such sampled view;
blending the set of warped images of the target view into a blended warped image of the target view;
causing a display image derived at least in part from the blended warped image of the target view to be rendered on a display of a wearable device.

17. The method of claim 16 , wherein the method is performed by a downstream device that receives the multiview video signal from an upstream device, and wherein the target view is determined based on device tracking information indicating a spatial position and a spatial direction of the wearable device operating in conjunction with the downstream device.

18. The method of claim 16 , wherein the set of sampled views is selected based on the target view from a plurality of sampled views of the multiview image.

19. The method of claim 16 , further comprising: decoding a multiview video signal into the one or more portions of the WFOV image and the one or more portions of the WFOV depth maps for the set of sampled views.

20. The method of claim 16 , wherein at least one or more portions of the WFOV image and the one or more portions of the WFOV depth maps is decoded based at least in part on one or more of: one or more INTRA prediction methods, one or more INTER prediction methods, or one or more INTER_VIEW prediction methods.

Assignees

• Dolby Laboratories Licensing Corp

Inventors

• Lakshman Haricharan
• Ninan Ajit

Classifications

• G09G3/005

  forming an image using a quickly moving array of imaging elements, causing the human eye to perceive an image which has a larger resolution than the array, e.g. an image on a cylinder formed by a rotating line of LEDs parallel to the axis of rotation · CPC title

• G06T15/205

  Image-based rendering · CPC title

• H04N19/597

  specially adapted for multi-view video sequence encoding · CPC title

• G09G3/003

  to produce spatial visual effects · CPC title

• G09G2340/14

  Solving problems related to the presentation of information to be displayed · CPC title