Multi-layer UV map based texture rendering for free-running FVV applications

Patent metadata

Field	Value
Publication number	US-10453244-B2
Application number	US-201715723021-A
Country	US
Kind code	B2
Filing date	Oct 2, 2017
Priority date	Oct 7, 2016
Publication date	Oct 22, 2019
Grant date	Oct 22, 2019

Abstract

A scalable texture data representation which enables spatio-temporal texture synthesis for occlusion by using a temporally consistent texture UV map, view-dependent texture mapping for seamless/specular color by using an additional multi-layer texture UV map, and view-dependent texture alignment for natural mapping by using an extra flow UV map, is described herein.

First claim

What is claimed is:

1. A method programmed in a memory of a device to render a texture on an object in a free viewpoint video content comprising:
implementing inpainting in a UV map domain;
utilizing a multi-layer texture UV map to combine texture segments; and
using a flow UV map to apply texture to a shape to prevent an incorrect shape, wherein implementing inpainting, utilizing the multi-layer texture UV map and using the flow UV map produce a temporally consistent texture for the object, wherein implementing inpainting in the UV map domain includes filling in any occluded aspects of the texture for the object using the texture captured at a different time.

2. The method of claim 1 wherein utilizing the multi-layer texture UV map includes employing a weighted sum to combine the texture segments.

3. The method of claim 1 wherein using the flow UV map includes generating and storing global alignment information of the texture.

4. The method of claim 1 wherein implementing inpainting, utilizing the multi-layer texture UV map and using the flow UV map are included in processing the free viewpoint video content which further includes geometry processing.

5. The method of claim 1 further comprising transmitting, rendering and displaying the free viewpoint video content including the object with temporally consistent texture.

6. The method of claim 1 further comprising acquiring the free viewpoint video content using at most 10 video cameras.

7. The method of claim 1 further comprising acquiring the free viewpoint video content using video cameras with at least 30 degrees of separation on every axis from other cameras.

8. An apparatus comprising:
a non-transitory memory for storing an application, the application for:
implementing inpainting in a UV map domain;
utilizing a multi-layer texture UV map to combine texture segments; and
using a flow UV map to apply texture to a shape to prevent an incorrect shape, wherein implementing inpainting, utilizing the multi-layer texture UV map and using the flow UV map produce a temporally consistent texture for the object, wherein implementing inpainting in the UV map domain includes filling in any occluded aspects of the texture for the object using the texture captured at a different time; and

a processor coupled to the memory, the processor configured for processing the application.

9. The apparatus of claim 8 wherein utilizing the multi-layer texture UV map includes employing a weighted sum to combine the texture segments.

10. The apparatus of claim 8 wherein using the flow UV map includes generating and storing global alignment information of the texture.

11. The apparatus of claim 8 wherein implementing inpainting, utilizing the multi-layer texture UV map and using the flow UV map are included in processing the free viewpoint video content which further includes geometry processing.

12. The apparatus of claim 8 further comprising transmitting, rendering and displaying the free viewpoint video content including the object with temporally consistent texture.

13. The apparatus of claim 8 further comprising acquiring the free viewpoint video content using at most 10 video cameras.

14. The apparatus of claim 8 further comprising acquiring the free viewpoint video content using video cameras with at least 30 degrees of separation on every axis from other cameras.

15. A system comprising:
a plurality of camera devices each configured for capturing free viewpoint video content;
a first computing device configured for:
implementing inpainting in a UV map domain;
utilizing a multi-layer texture UV map to combine texture segments; and
using a flow UV map to apply texture to a shape to prevent an incorrect shape, wherein implementing inpainting, utilizing the multi-layer texture UV map and using the flow UV map produce a temporally consistent texture for the object in the free viewpoint video content, wherein implementing inpainting in the UV map domain includes filling in any occluded aspects of the texture for the object using the texture captured at a different time; and

a second computing device configured for:
displaying the free viewpoint video content on the second computing device.

16. The system of claim 15 wherein utilizing the multi-layer texture UV map includes employing a weighted sum to combine the texture segments.

17. The system of claim 15 wherein using the flow UV map includes generating and storing global alignment information of the texture.

18. The system of claim 15 wherein implementing inpainting, utilizing the multi-layer texture UV map and using the flow UV map are included in processing the free viewpoint video content which further includes geometry processing.

19. The system of claim 15 further comprising transmitting, rendering and displaying the free viewpoint video content including the object with temporally consistent texture.

20. The system of claim 15 further comprising acquiring the free viewpoint video content using at most 10 video cameras.

21. The system of claim 15 further comprising acquiring the free viewpoint video content using video cameras with at least 30 degrees of separation on every axis from other cameras.

Assignees

• Sony Corp

Inventors

• Hirota Yoichi

Classifications

• G06T2210/36

  Level of detail · CPC title

• H04N13/117

  the virtual viewpoint locations being selected by the viewers or determined by viewer tracking · CPC title

• H04N13/111

  Transformation of image signals corresponding to virtual viewpoints, e.g. spatial image interpolation · CPC title

• H04N13/282

  for generating image signals corresponding to three or more geometrical viewpoints, e.g. multi-view systems · CPC title

• G06T15/205

  Image-based rendering · CPC title