Providing a tele-immersive experience using a mirror metaphor

Patent metadata

Field	Value
Publication number	US-9325943-B2
Application number	US-201313772252-A
Country	US
Kind code	B2
Filing date	Feb 20, 2013
Priority date	Feb 20, 2013
Publication date	Apr 26, 2016
Grant date	Apr 26, 2016

Abstract

A tele-immersive environment is described that provides interaction among participants of a tele-immersive session. The environment includes two or more set-ups, each associated with a participant. Each set-up, in turn, includes mirror functionality for presenting a three-dimensional virtual space for viewing by a local participant. The virtual space shows at least some of the participants as if the participants were physically present at a same location and looking into a mirror. The mirror functionality can be implemented as a combination of a semi-transparent mirror and a display device, or just a display device acting alone. According to another feature, the environment may present a virtual object in a manner that allows any of the participants of the tele-immersive session to interact with the virtual object.

First claim

What is claimed is:

1. A tele-immersive environment for providing interaction among a plurality of participants of a tele-immersive session, the tele-immersive environment comprising:
a plurality of set-up systems, each set-up system of the plurality of set-up systems operating in a different geographical location and providing an interactive experience of the tele-immersive session to a participant that is local to the respective set-up system, and each set-up system comprising:
image capture functionality configured to capture local camera information that represents the local participant associated with the set-up system;
a local processing system configured to:
receive the local camera information;
generate local three-dimensional object information based on the local camera information;
transfer local input information to at least one remote processing system associated with at least one remote participant that is local to a remote set-up system, the local input information corresponding to information associated with objects identified by the local processing system;
receive remote input information from said at least one remote processing system, the remote input information corresponding to information associated with objects identified by said at least one remote processing system; and
compose three-dimensional scene information based on the local three-dimensional object information and the remote input information; and

mirror functionality configured to present a three-dimensional virtual space based on the three-dimensional scene information, the three-dimensional virtual space showing multiple remote participants of the plurality of participants without the local participant as if the multiple remote participants were physically present at a same location and looking into a mirror.

2. The tele-immersive environment of claim 1 , wherein the image capture functionality is configured to provide information for use in constructing a depth image of the local participant.

3. The tele-immersive environment of claim 1 , wherein the mirror functionality comprises:
a physical semi-transparent mirror for presenting a virtual image of the local participant at the set-up system; and
a display device configured to receive the three-dimensional scene information, and to present a virtual image of at least one remote participant.

4. The tele-immersive environment of claim 1 , wherein the mirror functionality comprises a display device configured to receive the three-dimensional scene information, and to present a virtual image of the local participant and at least one remote participant.

5. A local set-up system in a tele-immersive environment, the tele-immersive environment for providing interaction among participants of a tele-immersive session, the local set-up system comprising:
image capture functionality configured to capture local camera information that represents a local participant associated with the local set-up system;
a local processing system configured to:
compose three-dimensional scene information based on the local camera information and remote input information, the remote input information being received from at least one remote processing system associated with at least one remote participant that is local to a remote set-up system operating in a different geographical location from the local set-up system, the remote input information corresponding to information associated with objects identified by said at least one remote processing system; and
transfer at least some of local input information to said at least one remote processing system, the local input information corresponding to information associated with objects identified by the local processing system; and

mirror functionality configured to present a three-dimensional virtual space based on the three-dimensional scene information, the three-dimensional virtual space showing at least some of the participants as if the participants were physically present at a same location and looking into a mirror.

6. The local set-up system of claim 5 , wherein the image capture functionality is configured to provide information for use in constructing a depth image of the local participant.

7. The local set-up system of claim 5 , wherein the mirror functionality comprises:
a physical semi-transparent mirror for presenting a virtual image of the local participant at the local set-up system; and
a display device configured to receive the three-dimensional scene information, and to present a virtual image of at least one remote participant.

8. The local set-up system of claim 5 , wherein the mirror functionality comprises a display device configured to receive the three-dimensional scene information, and to present a virtual image of the local participant and at least one remote participant.

9. The local set-up system of claim 5 , wherein the local processing system includes functionality that enables the local participant to interact with a virtual object presented by the mirror functionality.

10. The local set-up system of claim 9 , wherein the local set-up system includes a physical workspace in which the local participant may place a physical object, and wherein the virtual object is a virtual counterpart of the physical object.

11. The local set-up system of claim 10 , wherein the physical workspace includes a workspace table on which the physical object is placed.

12. The local set-up system of claim 9 , wherein the virtual object has no counterpart physical object in a real space associated with any set-up system.

13. The local set-up system of claim 9 , wherein interaction with the virtual object includes pulling or pushing the virtual object in a direction that is approximately orthogonal to the mirror functionality.

14. The local set-up system of claim 5 , further comprising a management module configured to receive information specified by the local participant during the tele-immersive session, to provide participant-specified information,
wherein the local processing system is configured to present the participant-specified information using the mirror functionality.

15. The local set-up system of claim 5 , further comprising a management module configured to provide a control feature and to manage interaction with the control feature,
wherein the local processing system is configured to present the control feature using the mirror functionality.

16. The local set-up system of claim 5 , wherein the mirror functionality includes a display device having a curved display surface for displaying the participants of the tele-immersive session.

17. The local set-up system of claim 5 , wherein the mirror functionality uses a handheld display device for presenting virtual images of two or more participants of the tele-immersive session.

18. The local set-up system of claim 5 , wherein the three-dimensional scene information composed by the local processing system represents:
a virtual-reflected space that includes one or more virtual-reflected objects that are projected from a perspective of reflections in a mirror surface; and
a virtual-actual space that includes one or more virtual-actual objects that are projected from a perspective of entities that are placed before the mirror surface.

19. The local set-up system of claim 18 , wherein the local processing system includes functionality that enables the local participant to interact with any virtual-reflected object and any virtual-actual object.

20. A method implemented b

Assignees

• Microsoft Technology Licensing Llc

Inventors

• Wilson Andrew D
• Chou Philip A
• Gillett Donald M
• Benko Hrvoje
• Zhang Zhengyou
• Fishman Neil S

Classifications

• G06F3/011 Primary

  Arrangements for interaction with the human body, e.g. for user immersion in virtual reality (blind teaching G09B21/00) · CPC title

• G06F3/0304

  Detection arrangements using opto-electronic means (constructional details of pointing devices not related to the detection arrangement using opto-electronic means G06F3/033; optical digitisers G06F3/042) · CPC title

• G06F2203/0381

  Multimodal input, i.e. interface arrangements enabling the user to issue commands by simultaneous use of input devices of different nature, e.g. voice plus gesture on digitizer · CPC title

• H04N7/157 Primary

  defining a virtual conference space and using avatars or agents (computer conference optimisation or adaptation H04L12/1827) · CPC title

• G06T19/006

  Mixed reality (object pose determination, tracking or camera calibration for mixed reality G06T7/00) · CPC title