Neural network based patch blending for immersive video

Patent metadata

Field	Value
Publication number	US-10819968-B2
Application number	US-201816050285-A
Country	US
Kind code	B2
Filing date	Jul 31, 2018
Priority date	Jul 31, 2018
Publication date	Oct 27, 2020
Grant date	Oct 27, 2020

Abstract

Embodiments are generally directed to neural network based patch blending for immersive video. An embodiment of a system includes one or more processor cores; a memory to store data for images in immersive video; and a neural network training framework. The neural network training framework is to generate a trained neural network for blending of a view of an object with patches generated for one or more detected occlusions for the object.

First claim

What is claimed is:

1. A system comprising:
one or more processor cores;
a memory to store data for images in immersive video; and
a neural network training framework;
wherein the system is to:
generate a plurality of views of an object for immersive video,
detect one or more occlusions in views of the object for the immersive video,
generate a set of patches for the one or more detected occlusions in views of the object in the immersive video,
generate a trained neural network utilizing the neural network training framework to enable blending of a main view of an object with patches generated for one or more detected occlusions for the object, wherein inputs to the neural network training framework include a first input for main view data for the object, a second input for patch data for the one or more detected occlusions for the object, and a third input for camera data to be utilized as a ground truth in the generation of the trained neural network, the camera data including an actual complementary view of the object captured by a camera, and
provide the trained neural network to a client system to enable generation of blended images for the immersive video based on a main view of the object and the one or more patches, the trained neural network including a first input for data representing a main view of the object and a second input for data representing one or more patches for an occlusion in the main view of the object.

2. The system of claim 1 , wherein generation of the patches includes generation of at least a first patch for a first occlusion, the generation of the first patch including addition of padding data, the padding data being data for an area surrounding or adjacent to an area for the first occlusion.

3. The system of claim 1 , wherein the immersive video is 6DoF (6 Degrees of Freedom) video.

4. The system of claim 1 , wherein the one or more processor cores includes one or more graphics processor cores of a graphical processing unit (GPU).

5. A system comprising:
one or more processor cores; and
a memory to store data for images in immersive video, wherein the system is to:
receive data for immersive video from a server system, the data including one or more views of an object and a set of patches for one or more detected occlusions for the object in the immersive video,
receive a trained neural network from the server system, wherein generation of the trained neural network includes receiving a first input for main view data for the object, a second input for patch data for the one or more detected occlusions for the object, and a third input for camera data being utilized as a ground truth in the generation of the trained neural network, the camera data including an actual complementary view of the object captured by a camera, and
generate immersive video including the object, wherein generating the immersive video includes blending data representing a main view of the object with data representing one or more patches of the set of patches for one or more detected occlusions for the object to generate a blended image for the immersive video, the blending being performed utilizing the trained neural network.

6. The system of claim 5 , wherein the set of patches includes at least a first patch for a first occlusion, the first patch including padding data, the padding data being data for an area surrounding or adjacent to an area for the first occlusion.

7. The system of claim 6 , wherein the set of patches further includes a second patch for a second occlusion, the second patch including padding data, the first patch and second patch each projecting at least in part to an overlapping area, and wherein the system is further to blend the first patch and the second patch utilizing alpha blending in the generation of the immersive video.

8. The system of claim 5 , wherein the immersive video is 6DoF (6 Degrees of Freedom) video.

9. The system of claim 5 , wherein the one or more processor cores includes one or more graphics processor cores of a graphical processing unit (GPU).

10. A non-transitory computer-readable storage medium having stored thereon data representing sequences of instructions that, when executed by one or more processors, cause the one or more processors to perform operations comprising:
generating a plurality of views of an object for immersive video;
detecting one or more occlusions in the views of the object;
generating a set of one or more patches for the one or more occlusions in views of the object;
generating a trained neural network to enable blending of a main view of the object with patches generated for one or more detected occlusions for the object wherein generating the trained neural network includes receiving a first input for main view data for the object, a second input for patch data for the one or more detected occlusions for the object, and a third input for camera data being utilized as a ground truth in the generation of the trained neural network, the camera data including an actual complementary view of the object captured by a camera; and
providing the trained neural network to a client system to enable generation of blended images for the immersive video based on a main view of the object and the one or more patches, the trained neural network including a first input for data representing a main view of the object and a second input for data representing one or more patches for an occlusion in the main view of the object.

11. The medium of claim 10 , wherein generating the one or more patches includes generation of at least a first patch for a first occlusion including adding padding data, the padding data being data for an area surrounding or adjacent to an area for the first occlusion.

12. The medium of claim 10 , wherein the immersive video is 6DoF (6 Degrees of Freedom) video.

13. A non-transitory computer-readable storage medium having stored thereon data representing sequences of instructions that, when executed by one or more processors, cause the one or more processors to perform operations comprising:
receiving data for immersive video from a server system, the data including one or more views of an object and a set of patches for one or more detected occlusions for the object in the immersive video;
receive a trained neural network from the server system, wherein generation of the trained neural network includes receiving a first input for main view data for the object, a second input for patch data for the one or more detected occlusions for the object, and a third input for camera data being utilized as a ground truth in the generation of the trained neural network, the camera data including an actual complementary view of the object captured by a camera, and
generate immersive video including the object, wherein generating the immersive video includes blending data representing a main view of the object with data representing one or more patches for one or more detected occlusions for the object to generate a blended image for the immersive video, the blending being performed utilizing the trained neural network.

14. The medium of claim 13 , wherein the one or more patches include at least a first patch for a first occlusion, the first patch including padding data, the padding data being data for an area surrounding or adjacent to an area for the first occlusion.

15. The medium of claim 14 , wherein the one or more patches further include a second patch for a second occlusion, the second patch including padding data, the first patch and second patch each projecting at least in part to an overlapping area, and further compris

Assignees

• Intel Corp

Inventors

• Raziel Ben
• Shiloni Asaf
• Bauman Gilad
• Boyce Jill
• Ashkenazi Tzach
• Beer Ilan
• Kats Eytan

Classifications

• G06N3/045

  Combinations of networks · CPC title

• G06N3/044

  Recurrent networks, e.g. Hopfield networks · CPC title

• G06N3/09

  Supervised learning · CPC title

• G06N3/0464

  Convolutional networks [CNN, ConvNet] · CPC title

• H04N13/344

  with head-mounted left-right displays · CPC title