Augmenting a view of a real-world environment with a view of a volumetric video object

Patent metadata

Field	Value
Publication number	US-12112426-B2
Application number	US-202017784883-A
Country	US
Kind code	B2
Filing date	Dec 15, 2020
Priority date	Dec 16, 2019
Publication date	Oct 8, 2024
Grant date	Oct 8, 2024

Abstract

A method of augmenting a view of a real-world environment with a view of a volumetric video object on a user device is disclosed. The method includes determining a current pose information (CPI) indicating a current pose of the view of the real-world environment and a desired pose of the volumetric video object in the real-world environment. The method further includes sending the CPI to a remote server. The method further includes receiving a rendered view of the volumetric video object that has been rendered in accordance with the CPI from the remote server. The method also includes augmenting the view of the real-world environment by at least mapping the rendered view of the volumetric video object onto a planar mapping surface arranged according to the desired position of the volumetric video object.

First claim

The invention claimed is:

1. A method of augmenting a view of a real-world environment with a view of a volumetric video object on a user device, the method comprising the steps of:
determining a current pose information (CPI) indicating a current pose of the view of the real-world environment and a desired pose of the volumetric video object in the real-world environment;
sending the current pose information to a remote server;
receiving a rendered view of the volumetric video object that has been rendered in accordance with the current pose information from the remote server; and
augmenting the view of the real-world environment by at least mapping the rendered view of the volumetric video object onto a planar mapping surface arranged according to the desired position of the volumetric video object;
wherein the planar mapping surface is oriented such that the rendered view of the volumetric video object is projected into the view of the real-world environment substantially free of perspective distortion.

2. The method as defined in claim 1 , wherein the rendered view of the volumetric video object comprises at least one background region that does not show the volumetric video object, and wherein the augmenting step comprises setting the at least one background region to be transparent.

3. The method as defined in claim 2 , wherein the rendered view of the volumetric video object comprises a transparency map that indicates the at least one background region is indicated in the rendered view of the volumetric video object by a predefined color, and wherein the augmenting step further comprises determining the at least one background region based on the transparency map or the predefined color.

4. The method as defined in claim 2 , wherein the augmenting step further comprises segmenting the volumetric video object as shown in the rendered view of the volumetric video object and determining the at least one background region based on the segmentation of the volumetric video object.

5. The method as defined in claim 1 , wherein the size and/or shape of the planar mapping surface is dynamically set based on the size and/or shape of the volumetric video object as shown in the rendered view of the volumetric video object.

6. The method as defined in claim 1 , wherein the receiving step further comprises receiving an approximate volume information indicating the approximate volume of the volumetric video object, and wherein the augmenting step further comprises using the approximate volume information for one or more of collision avoidance, correcting a lighting of the rendered view of the volumetric video object, generating at least one shadow of the volumetric video object in the real-world environment, and generating at least one reflection of the volumetric video object in the real-world environment.

7. The method as defined in claim 6 , wherein the approximate volume information comprises first approximate volume information for visible parts of the volumetric video object as shown in the rendered view of the volumetric video object and/or second approximate volume information for non-visible parts of the volumetric video object as shown in the rendered view of the volumetric video object.

8. The method as defined in claim 7 , wherein the first approximate volume information and/or the second approximate volume information comprises at least one of a depth map, mesh data, and one or more geometric primitives that approximate the volume of the volumetric video object.

9. The method as defined in claim 6 , wherein the determining step further comprises determining a reflection information indicating at least one reflection of the volumetric video object in the real-world environment, wherein the sending step further comprises sending the reflection information to the remote server, wherein the receiving step further comprises receiving at least one rendered view of the at least one reflection that has been rendered taking into account the reflection information from the remote server, and wherein the augmenting step further comprises generating the at least one reflection of the volumetric video object in the real-world environment based on the at least one rendered view of the at least one reflection.

10. The method as defined in claim 1 , wherein the determining step further comprises determining a lighting information indicating a lighting of the real-world environment, wherein the sending step further comprises sending the lighting information to the remote server, and wherein the rendered view of the volumetric video object has been rendered taking into account the lighting information.

11. The method as defined in claim 1 , wherein the sending step further comprises sending the view of the real-world environment to the remote server, wherein the rendered view of the volumetric video object has been rendered taking into account a lighting of the real-world environment, and/or wherein the receiving step further comprises receiving at least one rendered view of at least one shadow of the volumetric video object in the real-world environment, and/or wherein the receiving step further comprises receiving at least one rendered view of at least one reflection that has been rendered taking into account at least one reflection of the volumetric video object in the real-world environment, and/or wherein the augmenting step comprises generating the at least one shadow of the volumetric video object in the real-world environment based on the at least one rendered view of the at least one shadow, and/or wherein the augmenting step comprises generating the at least one reflection of the volumetric video object in the real-world environment based on the at least one rendered view of the at least one reflection, wherein the lighting and/or the at least one shadow and/or the at least one reflection has been determined based on the view of the real-world environment.

12. The method as defined in claim 11 , wherein the rendered view of the volumetric video object is received in a packed texture in which it is packed together with a rendered view of a further volumetric video object and/or the at least one rendered view of the at least one shadow and/or the at least one rendered view of the at least one reflection.

13. The method as defined in claim 12 , wherein the receiving further comprises receiving metadata comprising unpack information indicating how to unpack the packed texture into the different rendered views.

14. The method as defined in claim 1 , wherein the view of the real-world environment and the rendered view of the volumetric video object are stereoscopic views comprising two different views for the two eyes of a user, and wherein the augmenting step comprises at least mapping each of the two different views of the rendered view of the volumetric video object onto a planar mapping surface.

15. The method as defined in claim 1 , wherein the view of the real-world environment is a stereoscopic view comprising two different views for the two eyes of a user, wherein the receiving step comprises receiving stereoscopic view generation information for rendering a stereoscopic view of the volumetric video object from the rendered view of the volumetric video object, and wherein the augmenting step further comprises generating the stereoscopic view of the volumetric video object from the rendered view of the volumetric video object in accordance with the stereoscopic view generation information and mapping each of the two different views of the generated stereoscopic view of the volumetric video object onto a planar mapping surface.

Assignees

• Fraunhofer Ges Forschung
• Fraunhofer Gessellschaft Zur Foerderung Der Angewandten Forschung Ev

Inventors

• Hellge Cornelius
• Schierl Thomas
• Eisert Peter
• Hilsmann Anna
• Skupin Robert
• Sanchez Yago
• Son Jangwoo
• Bhullar Gurdeep Singh
• Gül Serhan
• Podborski Dimitri

Classifications

• G06T2215/16

  Using real world measurements to influence rendering · CPC title

• G06T2207/20112

  Image segmentation details · CPC title

• G06T19/006 Primary

  Mixed reality (object pose determination, tracking or camera calibration for mixed reality G06T7/00) · CPC title

• G06T15/60

  Shadow generation · CPC title

• G06T15/08

  Volume rendering · CPC title