Systems and methods for video communication using a virtual camera

Patent metadata

Field	Value
Publication number	US-11410331-B2
Application number	US-201916702111-A
Country	US
Kind code	B2
Filing date	Dec 3, 2019
Priority date	Oct 3, 2019
Publication date	Aug 9, 2022
Grant date	Aug 9, 2022

Abstract

A method for using a virtual camera location to display image data to one device from another device associated with a plurality of cameras. The method includes receiving image data from a plurality of cameras associated with a second user device associated with a second user. The method further includes establishing a virtual camera location different from positions of the plurality of cameras. The method further includes providing an image using the image data on a first user display of a first user device based at least on a viewpoint of the virtual camera location.

First claim

What is claimed is:

1. A method for using a virtual camera location to display image data to one device from another device associated with a plurality of cameras, the method comprising:
receiving image data from a plurality of cameras associated with a second user device associated with a second user;
determining a gaze vector of the second user based on received eye tracking data;
establishing a virtual camera location different from positions of the plurality of cameras based on the gaze vector;
setting a first virtual camera angle and a second virtual camera angle based on the gaze vector; and
providing an image using the image data on a first user display of a first user device based at least on a viewpoint of the virtual camera location.

2. The method of claim 1 , further comprising:
receiving position data associated with at least one body part of the second user,
wherein the viewpoint of the virtual camera location is further established based on the position data and the first virtual camera angle and the second virtual camera angle.

3. The method of claim 2 , wherein the position data comprises data indicating a position of a feature of a head of the second user.

4. The method of claim 3 , wherein
determining the gaze vector of the second user includes determining a gaze location, a first gaze angle, and a second gaze angle,
wherein the virtual camera location is established by setting the virtual camera location to correspond to the gaze location, and
wherein the first virtual camera angle is set based at least on the first gaze angle and the second virtual camera angle is set based at least on the second gaze angle.

5. The method of claim 4 , wherein the virtual camera location is established one of continuously or responsive to changes in imagery captured by the plurality of cameras.

6. The method of claim 1 , wherein the viewpoint is determined using the virtual camera location, the first virtual camera angle with respect to a first axis, and the second virtual camera angle with respect to a second axis, the first axis being perpendicular to the second axis.

7. A method for using a virtual camera location to display image data to one device from another device associated with a plurality of cameras, the method comprising:
receiving image data from a plurality of cameras associated with a second user device associated with a second user;
receiving a requested virtual camera position from the second user;
establishing a virtual camera location different from positions of the plurality of cameras based on the requested virtual camera position;
setting a first virtual camera angle and a second virtual camera angle based on the requested virtual camera position.

8. The method of claim 7 , wherein the requested virtual camera position is selectively movable by the second user.

9. The method of claim 1 , wherein the viewpoint of the virtual camera location is at a non-perpendicular angle with respect to a surface of a second user display of the second user device and is established using the first virtual camera angle and the second virtual camera angle.

10. A system comprising:
a receiving user device associated with a receiving user and having a receiving user display;
a sending user device associated with a sending user and in communication with the receiving user device via a network, the sending user device having a sending user display, a plurality of sensors, and processing circuitry configured to:
receive image data from the plurality of sensors;
determine a gaze vector of the sending user based on received eye tracking data;
establish a virtual camera location of a virtual camera different from positions of the plurality of sensors based on the gaze vector;
set a first virtual camera angle and a second virtual camera angle of the virtual camera based on the gaze vector;
render an image based at least on the image data and a position of the virtual camera; and
transmit, via the network, the image to the receiving user device to be displayed on the receiving user display.

11. The system of claim 10 , wherein the processing circuitry is further configured to combine the image data received from the plurality of sensors to render the image from a viewpoint of the virtual camera.

12. The system of claim 10 , wherein the first virtual camera angle is with respect to a first axis, the second virtual camera angle is with respect to a second axis, and the first axis is perpendicular to the second axis and the virtual camera location has a virtual view defined by the first virtual camera angle and second virtual camera.

13. The system of claim 12 , wherein the processing circuitry is further configured to:
receive position data associated with at least one body part of the sending user from at least one of the plurality of sensors indicating a position of a feature of a head of the sending user; and
wherein rendering the image is further based at least partially on the position data.

14. The system of claim 13 , wherein determining the gaze vector of the sending user includes determining a gaze location, a first gaze angle, and a second gaze angle,
wherein the virtual camera location is established by setting the virtual camera location to correspond to the gaze location, and
wherein the first virtual camera angle is set based at least on the first gaze angle and the second virtual camera angle is set based at least on the second gaze angle.

15. A system comprising:
a receiving user device associated with a receiving user and having a receiving user display;
a sending user device associated with a sending user and in communication with the receiving user device via a network, the sending user device having a sending user display, a plurality of sensors, and processing circuitry configured to:
receive image data from the plurality of sensors;
receive a requested virtual camera position from the sending user;
establish a virtual camera location of a virtual camera different from positions of the plurality of sensors based on the requested virtual camera position;
set a first virtual camera angle and a second virtual camera angle of the virtual camera based on the requested virtual camera position;
render an image based at least on the image data and a position of the virtual camera; and
transmit, via the network, the image to the receiving user device to be displayed on the receiving user display.

16. A video communication device associated with a user comprising:
a plurality of sensors;
a display arranged between at least two of the plurality of sensors;
processing circuitry in communication with a network, the processing circuitry configured to:
receive image data from the plurality of sensors;
determine a gaze vector of the user based on received eye tracking data;
establish a virtual camera location of a virtual camera different from positions of the plurality of sensors based on the gaze vector;
set a first virtual camera angle and a second virtual camera angle of the virtual camera based on the gaze vector; and
render an image based on the image data and a position of the virtual camera; and

communications circuitry configured to transmit the image to a receiving user device via the network.

17. The video communication device of claim 16 , wherein the processing circuitry is further configured to render the image by combining the image data received from the plurality of sensors to produce the image from a viewpoint of the virtual camera.

18. The video communication device of claim

Assignees

• Facebook Tech Llc

Inventors

• Kassis Ronit
• Cabral Brian Keith
• Cohen Michael F

Classifications

• G06T2207/30244

  Camera pose · CPC title

• G06F3/013 Primary

  Eye tracking input arrangements (G06F3/015 takes precedence) · CPC title

• G06T2215/16

  Using real world measurements to influence rendering · CPC title

• G06F1/1626

  with a single-body enclosure integrating a flat display, e.g. Personal Digital Assistants [PDAs] · CPC title

• G06T7/73

  using feature-based methods · CPC title