Methods and apparatuses for dynamic navigable 360 degree environments

Patent metadata

Field	Value
Publication number	US-11087549-B2
Application number	US-201916653266-A
Country	US
Kind code	B2
Filing date	Oct 15, 2019
Priority date	Oct 15, 2018
Publication date	Aug 10, 2021
Grant date	Aug 10, 2021

Abstract

Systems, methods, apparatuses, and computer program products for creating freely explorable, dynamic and photorealistic virtual environments, reconstructing view dependent holograms in real-time, and inserting 3D virtual objects into 360 camera based navigable environment. A method, may include simultaneously capturing 360 video data and audio data from a plurality of viewpoints within a real-world environment. The method may also include preprocessing and compressing the 360 video data and the audio data into a three-dimensional representation suitable for display. The method may further include rendering a virtual environment of the real-world environment. In addition, the method may include creating a blended virtual environment by combining the captured 360 video data and the audio data with the rendered virtual environment. Further, the method may include displaying the blended virtual environment in a display apparatus of a user.

First claim

We claim:

1. A method, comprising:
simultaneously capturing 360 degree video data and audio data from a plurality of viewpoints within a real-world environment;
preprocessing and compressing the 360 degree video data and the audio data into a three-dimensional representation suitable for display;
rendering a virtual environment of the real-world environment;
creating a blended virtual environment by combining the captured 360 degree video data and the audio data at multiple locations with the rendered virtual environment;
displaying the blended virtual environment in a display apparatus of a user; and
estimating physical properties of a plurality of objects, in multiple 360 degree video data, wherein the estimated physical properties of the plurality of objects in the multiple 360 degree video data are utilized to establish a correspondence between the plurality objects in the multiple 360 degree video data and physical and virtual objects in the blended virtual environment so that the physical and the virtual objects in the blended virtual environment have position, scale, and orientation that are consistent with the multiple 360 degree video data across multiple views,
wherein the captured audio data and the 360 degree video data are in time synchronization when navigating between the multiple locations and during display of the blended virtual environment, and
wherein during the time synchronization of the captured audio data and the 360 degree video data, time progresses continuously when navigating between the multiple locations.

2. The method according to claim 1 , wherein the method further comprises initializing a playback procedure to play back the 360 degree video data and the audio data as a photorealistic navigable virtual environment to the user to recreate the appearance and sound of the real-world environment from a given viewpoint at a given orientation.

3. The method according to claim 1 , wherein the three-dimensional representation is rendered based on interpolated views of captured nearby camera views.

4. The method according to claim 1 , wherein the preprocessing and compressing comprises utilizing multi-stream video encoding to create multi-view video textures for constructing the three-dimensional representation.

5. The method according to claim 1 , wherein the preprocessing and compressing comprises implementing matching parameters including frame resolution and coding format across content files representing each 360 degree viewpoint.

6. An apparatus, comprising:
at least one processor; and
at least one memory comprising computer program code,
the at least one memory and the computer program code are configured, with the at least one processor to cause the apparatus at least to:
simultaneously capture 360 degree video data and audio data from a plurality of viewpoints within a real-world environment;
preprocess and compress the 360 degree video data and the audio data into a three-dimensional representation suitable for display;
render a virtual environment of the real-world environment;
create a blended virtual environment by combining the captured 360 degree video data and the audio data at multiple locations with the rendered virtual environment;
display the blended virtual environment in a display apparatus of a user; and
estimate physical properties of a plurality of objects in multiple 360 degree video data, wherein the estimated physical properties of the plurality of objects in the multiple 360 degree video data are utilized to establish a correspondence between the plurality objects in the multiple 360 degree video data and physical and virtual objects in the blended virtual environment so that the physical and the virtual objects in the blended virtual environment have position, scale, and orientation that are consistent with the multiple 360 degree video data across multiple views,
wherein the captured audio data and the 360 degree video data are in time synchronization when navigating between the multiple locations and during display of the blended virtual environment, and
wherein during the time synchronization of the captured audio data and the 360 degree video data, time progresses continuously when navigating between the multiple locations.

7. The apparatus according to claim 6 , wherein the at least one memory and the computer program code are further configured, with the at least one processor to cause the apparatus at least to initialize a playback procedure to play back the 360 degree video data and the audio data as a photorealistic navigable virtual environment to the user to recreate the appearance and sound of the real-world environment from a given viewpoint at a given orientation.

8. The apparatus according to claim 6 , wherein the three-dimensional representation is rendered based on interpolated views of captured nearby camera views.

9. The apparatus according to claim 6 , wherein the preprocessing and compressing comprises utilizing multi-stream video encoding to create multi-view video textures for constructing the three-dimensional representation.

10. The apparatus according to claim 6 , wherein the preprocessing and compressing comprises implementing matching parameters including frame resolution and coding format across content files representing each 360 degree viewpoint.

11. A computer program, embodied on a non-transitory computer readable medium, the computer program, when executed by a processor, causes the processor to:
simultaneously capture 360 degree video data and audio data from a plurality of viewpoints within a real-world environment;
preprocess and compress the 360 degree video data and the audio data into a three-dimensional representation suitable for display;
render a virtual environment of the real-world environment;
create a blended virtual environment by combining the captured 360 degree video data and the audio data at multiple locations with the rendered virtual environment; and
display the blended virtual environment in a display apparatus of a user; and
estimate physical properties of a plurality of objects in multiple 360 degree video data, wherein the estimated physical properties of the plurality of objects in the multiple 360 degree video data are utilized to establish a correspondence between the plurality objects in the multiple 360 degree video data and physical and virtual objects in the blended virtual environment so that the physical and the virtual objects in the blended virtual environment have position, scale, and orientation that are consistent with the multiple 360 degree video data across multiple views,
wherein the captured audio data and the 360 degree video data are in time synchronization when navigating between the multiple locations and during display of the blended virtual environment, and
wherein during the time synchronization of the captured audio data and the 360 degree video data, time progresses continuously when navigating between the multiple locations.

12. The computer program according to claim 11 , wherein the computer program, when executed by the processor, further causes the processor to initialize a playback procedure to play back the 360 degree video data and the audio data as a photorealistic navigable virtual environment to the user to recreate the appearance and sound of the real-world environment from a given viewpoint at a given orientation.

13. The computer program according to claim 11 , wherein the three-dimensional representation is rendered based on interpolated views of captured nearby camera views.

14. The computer program according to claim 11 , wherein the preprocessing and compre

Assignees

• Univ Maryland

Inventors

• Varshney Amitabh
• Lee Eric C
• Li Sida

Classifications

• G06T19/20

  Editing of three-dimensional [3D] images, e.g. changing shapes or colours, aligning objects or positioning parts · CPC title

• G06T19/006 Primary

  Mixed reality (object pose determination, tracking or camera calibration for mixed reality G06T7/00) · CPC title

• G06F3/04815

  Interaction with a metaphor-based environment or interaction object displayed as three-dimensional [3D], e.g. changing the user viewpoint with respect to the environment or object · CPC title

• G06F3/012

  Head tracking input arrangements · CPC title

• H04S7/304

  For headphones · CPC title