Providing a tele-immersive experience using a mirror metaphor

Patent metadata

Field	Value
Publication number	US-2017201722-A1
Application number	US-201715471709-A
Country	US
Kind code	A1
Filing date	Mar 28, 2017
Priority date	Feb 20, 2013
Publication date	Jul 13, 2017
Grant date	—

Abstract

A tele-immersive environment is described that provides interaction among participants of a tele-immersive session. The environment includes two or more set-ups, each associated with a participant. Each set-up, in turn, includes mirror functionality for presenting a three-dimensional virtual space for viewing by a local participant. The virtual space shows at least some of the participants as if the participants were physically present at a same location and looking into a mirror. The mirror functionality can be implemented as a combination of a semi-transparent mirror and a display device, or just a display device acting alone. According to another feature, the environment may present a virtual object in a manner that allows any of the participants of the tele-immersive session to interact with the virtual object.

First claim

1 .- 20 . (canceled)

21 . A tele-immersive environment for providing interaction among participants of a tele-immersive session, the tele-immersive environment comprising:
a first set-up system that is operating in a first geographical location to provide a first interactive experience of the tele-immersive session to a remote participant, wherein the first set-up system is configured to capture first camera information that represents the remote participant; and a second set-up system that is operating in a second geographical location to provide a second interactive experience of the tele-immersive session to a local participant, wherein the second set-up system comprises:
image capture functionality to capture second camera information that represents the local participant;
a physical semi-transparent mirror for presenting a virtual image of the local participant, wherein the local participant is physically positioned at a front side of the physical semi-transparent mirror; and
a display device for presenting, based at least in part on the first camera information, a virtual space showing the remote participant as if the remote participant is physically positioned at the front side of the physical semi-transparent mirror.

22 . The tele-immersive environment of claim 21 , wherein the second set-up system further comprises a local processing system configured to:
determine three-dimensional object information associated with at least one object; and cause the display device to present three-dimensional scene information based at least in part on the three-dimensional object information.

23 . The tele-immersive environment of claim 22 , wherein the at least one object is a virtual object presented by the display device, and wherein the local processing system includes functionality that enables both the local participant and the remote participant to interact with the virtual object.

24 . The tele-immersive environment of claim 22 , wherein the first camera information further represents the at least one object.

25 . The tele-immersive environment of claim 22 , wherein the local processing system is further configured to:
process depth information associated with associated with the at least one object to create a plurality of three-dimensional meshes of the at least one object; and generate the three-dimensional object information by applying video information to the plurality of three-dimensional meshes.

26 . The tele-immersive environment of claim 21 , wherein the image capture functionality is configured to provide depth information for use in constructing a depth image of the local participant.

27 . The tele-immersive environment of claim 21 , wherein the display device is physically positioned at a rear side of the physical semi-transparent mirror.

28 . The tele-immersive environment of claim 27 , wherein the display device is substantially adjacent to the rear side of the physical semi-transparent mirror.

29 . A method comprising:
receiving local camera information that represents a local participant associated with a local system that is operating in a first geographical location, wherein the local participant is physically positioned at a front side of a physical semi-transparent mirror; receiving remote input information that represents a remote participant associated with a remote system that is operating in a second geographical location that is different from the first geographical location; composing scene information based at least in part on the local camera information and the remote input information; and causing a display device to project imagery from a rear side of the physical semi-transparent mirror to present a virtual space based at least in part on the scene information, the virtual space showing the remote participant as if the remote participant is physically positioned at the front side of the physical semi-transparent mirror.

30 . The method of claim 29 , further comprising transferring, to the remote system, local input information associated with an object identified by the local system.

31 . The method of claim 30 , wherein the local input information comprises one or more of the local camera information or local three-dimensional object information that provides a three-dimensional representation of the object, wherein the object is located in a real space that is adjacent to the local system.

32 . The method of claim 29 , further comprising providing a functionality to enable the local participant to interact with a virtual object presented in the virtual space by the display device.

33 . The method of claim 32 , wherein the virtual object lacks a corresponding counterpart physical object in a real space associated with the local system or the remote system.

34 . The method of claim 29 , further comprising:
tracking a location or movement of the local participant in a real space that is adjacent to the local system to determine tracked location or movement information corresponding to the local participant; and transferring, to the remote system, the tracked location or movement information corresponding to the local participant.

35 . A method implemented by a local set-up system to facilitate a tele-immersive session, the method comprising:
receiving local camera information from a local image capture functionality corresponding to the local set-up system, wherein the local set-up system is operating in a first geographical location; transferring local input information to a remote set-up system that is operating in a second geographical location that is remote from the first geographical location, the remote set-up system being associated with a remote participant of the tele-immersive session, the local input information associated with a first object that is identified by the local set-up system; receiving, from the remote set-up system, remote input information that is associated with a second object that is identified by the remote set-up system; composing scene information based on the local input information and the remote input information; and providing the scene information to a mirror functionality that is configured to present a virtual space based on the scene information, the virtual space showing the remote participant as if the remote participant is physically present at the first geographical location and looking into the mirror.

36 . The method of claim 35 , wherein the local camera information is associated with a local participant that is physically present at the first geographical location, the virtual space further showing the local participant as if the local participant were looking into the mirror.

37 . The method of claim 36 , further comprising providing a functionality to enable the local participant that is physically present at the first geographical location to interact with the second object that is identified by the remote set-up system, wherein the second object is physically located at the second geographical location.

38 . The method of claim 36 , further comprising providing a functionality to enable the local participant and the remote participant to jointly manipulate a virtual object that corresponds to at least one of the first object or the second object.

39 . The method of claim 38 , wherein the virtual object is a counterpart of a physical object that is associated with a workspace table configured to enable the local participant to place physical objects.

40 . The method of

Assignees

• Microsoft Technology Licensing Llc

Inventors

• Wilson Andrew D
• Zhang Zhengyou
• Chou Philip A
• Fishman Neil S
• Gillett Donald M
• Benko Hrvoje

Classifications

• G06F3/011 Primary

  Arrangements for interaction with the human body, e.g. for user immersion in virtual reality (blind teaching G09B21/00) · CPC title

• G06F3/0304

  Detection arrangements using opto-electronic means (constructional details of pointing devices not related to the detection arrangement using opto-electronic means G06F3/033; optical digitisers G06F3/042) · CPC title

• G06F3/005

  Input arrangements through a video camera · CPC title

• G06T17/20

  Finite element generation, e.g. wire-frame surface description, {tesselation} · CPC title

• H04N7/147

  Communication arrangements, e.g. identifying the communication as a video-communication, intermediate storage of the signals (selecting H04Q) · CPC title