Virtual reality telepresence

Patent metadata

Field	Value
Publication number	US-10819952-B2
Application number	US-201615290985-A
Country	US
Kind code	B2
Filing date	Oct 11, 2016
Priority date	Oct 11, 2016
Publication date	Oct 27, 2020
Grant date	Oct 27, 2020

Abstract

Implementations generally relate to virtual reality telepresence. In some implementations, a method includes performing projection mapping of a projection area. The method further includes collecting user information associated with a user. The method further includes positioning the user in a virtual environment based on the projection mapping and the user information. The method further includes determining a point of view of the user in the virtual environment. The method further includes projecting the virtual environment onto the projection area based on the point of view of the user.

First claim

What is claimed is:

1. A computer-implemented method comprising:
scanning a projection area;
collecting environment geometry data based on the scanning of the projection area;
performing projection mapping of the projection area based on the environment geometry data;
scanning a user to collect topography information;
collecting user information associated with the user and one or more remote users, wherein the user information comprises the topography information, and wherein the topography information comprises a first floor plane on which the user is standing, a second floor plane in a scene of a virtual environment, and a third floor plane on which each of the one or more remote users is standing;
positioning the user and the one or more remote users in the virtual environment based on the projection mapping and the user information such that the first floor plane, the second floor plane, and the third floor plane are aligned;
tracking head positions of the user and the one or more remote users;
determining head parallax of the user and the one or more remote users based on the tracking of the head positions;
determining a point of view of the user and the one or more remote users in the virtual environment based on the head parallax of the user and the one or more remote users;
computing a virtual distance between the user and one or more objects in the virtual environment, wherein the one or more objects comprises the one or more remote users, and wherein the virtual distance is based on one or more of the projection mapping, the user information, and the one or more objects in the virtual environment; and
projecting the virtual environment onto the projection area based on the point of view of the user, wherein the point of view of the user is substantially equal to a virtual camera associated with the user, and wherein the point of view of the virtual camera tracks the point of view of the user.

2. The method of claim 1 , wherein, to position the user in the virtual environment, the method further comprises positioning the user such that the user is aligned to a floor plane in the scene of the virtual environment.

3. The method of claim 1 , wherein the performing of the projection mapping comprises:
displaying structured light patterns on the projection area; and
collecting depth data associated with the projection area.

4. The method of claim 1 , wherein the user information comprises one or more of red-green-blue video data, depth data, and skeletal tracking information.

5. The method of claim 1 , further comprising rendering a 3-dimensional (3D) object inside the virtual environment based on depth data associated with the projection area.

6. The method of claim 1 , further comprising simulating eye contact between the user and at least one remote user of the one or more remote users in the virtual environment when the user faces the at least one remote user.

7. Software encoded in one or more non-transitory computer-readable storage media for execution by one or more processors and when executed operable to perform operations comprising:
scanning a projection area;
collecting environment geometry data based on the scanning of the projection area;
performing projection mapping of the projection area based on the environment geometry data;
scanning a user to collect topography information;
collecting user information associated with the user and one or more remote users, wherein the user information comprises the topography information, and wherein the topography information comprises a first floor plane on which the user is standing, a second floor plane in a scene of a virtual environment, and a third floor plane on which each of the one or more remote users is standing;
positioning the user and the one or more remote users in the virtual environment based on the projection mapping and the user information such that the first floor plane, the second floor plane, and the third floor plane are aligned;
tracking head positions of the user and the one or more remote users;
determining head parallax of the user and the one or more remote users based on the tracking of the head positions;
determining a point of view of the user and the one or more remote users in the virtual environment based on the head parallax of the user and the one or more remote users;
computing a virtual distance between the user and one or more objects in the virtual environment, wherein the one or more objects comprises the one or more remote users, and wherein the virtual distance is based on one or more of the projection mapping, the user information, and the one or more objects in the virtual environment; and
projecting the virtual environment onto the projection area based on the point of view of the user, wherein the point of view of the user is substantially equal to a virtual camera associated with the user, and wherein the point of view of the virtual camera tracks the point of view of the user.

8. The software of claim 7 , wherein the projection mapping is based on depth data associated with the projection area.

9. The software of claim 7 , wherein, to perform the projection mapping, the software when executed is further operable to perform operations comprising:
displaying structured light patterns on the projection area; and
collecting depth data associated with the projection area.

10. The software of claim 7 , wherein the user information comprises one or more of red-green-blue video data, depth data, and skeletal tracking information.

11. The software of claim 7 , wherein the user information comprises topography information.

12. The software of claim 7 , wherein the software when executed is further operable to perform operations comprising performing head tracking in order to move a virtual camera associated with the user for parallax effect.

13. The software of claim 7 , wherein the software when executed is further operable to perform operations comprising simulating eye contact in the virtual environment.

14. A system comprising:
one or more processors; and
logic encoded in one or more non-transitory computer-readable storage media for execution by the one or more processors and when executed operable to perform operations comprising:
scanning a projection area;
collecting environment geometry data based on the scanning of the projection area;
performing projection mapping of the projection area based on the environment geometry data;
scanning a user to collect topography information;
collecting user information associated with the user and one or more remote users, wherein the user information comprises the topography information, and wherein the topography information comprises a first floor plane on which the user is standing, a second floor plane in a scene of a virtual environment, and a third floor plane on which each of the one or more remote users is standing;
positioning the user and the one or more remote users in the virtual environment based on the projection mapping and the user information such that the first floor plane, the second floor plane, and the third floor plane are aligned;
tracking head positions of the user and the one or more remote users;
determining head parallax of the user and the one or more remote users based on the tracking of the head positions;
determining a point of view of the user and the one or more remote users in the virtual environment based on the head parallax of the user and the one or more remote users;
computing a virtual distance between the user and one or more objects in the virtual environment, wherein the one or more objects comprises the o

Assignees

• Sony Interactive Entertainment Network America Llc
• Sony Interactive Entertainment LLC

Inventors

• Chaney Allison Marlene
• Rogers Anthony

Classifications

• H04N7/157 Primary

  defining a virtual conference space and using avatars or agents (computer conference optimisation or adaptation H04L12/1827) · CPC title

• G06T19/006

  Mixed reality (object pose determination, tracking or camera calibration for mixed reality G06T7/00) · CPC title

• H04N7/147

  Communication arrangements, e.g. identifying the communication as a video-communication, intermediate storage of the signals (selecting H04Q) · CPC title