Controlling an augmented call based on user gaze

Patent metadata

Field	Value
Publication number	US-12212705-B2
Application number	US-202217840725-A
Country	US
Kind code	B2
Filing date	Jun 15, 2022
Priority date	Jun 15, 2022
Publication date	Jan 28, 2025
Grant date	Jan 28, 2025

Abstract

Aspects of the present disclosure are directed to controlling a sending side of an augmented call based on a receiving user's gaze. Some implementations provide a hologram moderation system in which a receiving user's gaze can control how the system generates a representation of a sending user on a sending side. For example, some implementations can moderate the capture or generation of hologram data representing the sending user when the receiving user isn't focused on the hologram that results from the data. Such moderations can reduce power consumption, bandwidth, heat production, and/or processing power needed by the artificial reality system when the receiving user is not looking at the hologram of the sending user, such as when the sending user is in the receiving user's periphery or outside the receiving user's field-of-view.

First claim

We claim:

1. A method for controlling a sending side of an augmented call based on a gaze detected on a receiving side of the augmented call, the method comprising:
establishing a communication channel between a sending system associated with a sending call participant and a receiving system associated with a receiving call participant;
obtaining a hologram representing the receiving call participant;
displaying the hologram representing the receiving call participant on the sending system at a world-locked location established for the receiving call participant;
receiving, over the communication channel, an indication that the gaze of the receiving call participant is not focused on a representation of the sending call participant;
selecting a moderated manner for capturing or generating hologram data representing the sending call participant based on the indication that the gaze is not focused on the representation of the sending call participant;
capturing or generating, according to the selected moderated manner, the hologram data representing the sending call participant; and
transmitting the hologram data representing the sending call participant to the receiving system, wherein the receiving system, in response to receiving the hologram data representing the sending call participant, displays a moderated representation of the sending call participant.

2. The method of claim 1 ,
wherein the selected moderated manner specifies a second quality different from a first quality specified when the gaze of the receiving call participant is focused on the representation of the sending call participant, and
wherein the hologram data generated with the second quality requires one or both of less bandwidth to transmit or less computing resources to create or render than hologram data generated with the first quality.

3. The method of claim 1 , wherein the selected moderated manner includes at least one of reducing a frame rate, two-dimensional rendering, reducing a resolution, dimming, desaturating, pausing capture, foveating, blurring, selecting a lower power capture device, or any combination thereof.

4. The method of claim 1 , wherein the indication is a first indication and wherein the method further comprises:
receiving a second indication specifying that the representation of the sending call participant is outside of a field-of-view of the receiving call participant, and
in response to the second indication, pausing capture or generation of the hologram data.

5. The method of claim 1 , wherein the receiving system is a first receiving system, the receiving call participant is a first receiving call participant, and the hologram data is first hologram data:
wherein the establishing the communication channel includes establishing the communication channel between the sending system, the first receiving system, and a second receiving system associated with a second receiving call participant,
wherein the method further includes receiving, over the communication channel, an indicator that the gaze of the second receiving call participant is focused on the representation of the sending call participant, and
wherein the sending system, in response to the indicator, transmits second hologram data generated without the moderated manner.

6. The method of claim 1 , wherein the moderated manner includes pausing execution of a machine learning model used to capture the hologram data.

7. A non-transitory computer-readable storage medium storing instructions that, when executed by a computing system, cause the computing system to perform a process for controlling a sending side of an augmented call based on a gaze detected on a receiving side of the augmented call, the process comprising:
establishing a communication channel between a sending system associated with a sending call participant and a receiving system associated with a receiving call participant;
receiving, over the communication channel, an indication that the gaze of the receiving call participant is not focused on a representation of the sending call participant;
selecting a moderated manner for capturing or generating hologram data representing the sending call participant based on the indication that the gaze is not focused on the representation of the sending call participant;
capturing or generating, according to the selected moderated manner, the hologram data representing the sending call participant; and
transmitting the hologram data representing the sending call participant to the receiving system, wherein the receiving system, in response to receiving the hologram data representing the sending call participant, displays a moderated representation of the sending call participant.

8. The non-transitory computer-readable storage medium of claim 7 , wherein the process further comprises:
obtaining a hologram representing the receiving call participant; and
displaying the hologram representing the receiving call participant on the sending system at a world-locked location established for the receiving call participant.

9. The non-transitory computer-readable storage medium of claim 7 ,
wherein the selected moderated manner specifies a second quality different from a first quality specified when the gaze of the receiving call participant is focused on the representation of the sending call participant, and
wherein the hologram data generated with the second quality requires one or both of less bandwidth to transmit or less computing resources to create or render than hologram data generated with the first quality.

10. The non-transitory computer-readable storage medium of claim 7 , wherein the selected moderated manner includes at least one of reducing a frame rate, two-dimensional rendering, reducing a resolution, dimming, desaturating, pausing capture, foveating, blurring, selecting a lower power capture device, or any combination thereof.

11. The non-transitory computer-readable storage medium of claim 7 , wherein the indication is a first indication and wherein the process further comprises:
receiving a second indication specifying that the representation of the sending call participant is outside of a field-of-view of the receiving call participant, and
in response to the second indication, pausing capture or generation of the hologram data.

12. The non-transitory computer-readable storage medium of claim 7 ,
wherein the receiving system is a first receiving system, the receiving call participant is a first receiving call participant, and the hologram data is first hologram data;
wherein the establishing the communication channel includes establishing the communication channel between the sending system, the first receiving system, and a second receiving system associated with a second receiving call participant;
wherein the process further includes receiving, over the communication channel, an indicator that the gaze of the second receiving call participant is focused on the representation of the sending call participant, and
wherein the sending system, in response to the indicator, transmits second hologram data generated without the moderated manner.

13. The non-transitory computer-readable storage medium of claim 7 , wherein the moderated manner includes pausing execution of a machine learning model used to capture the hologram data.

14. A computing system for controlling a sending side of an augmented call based on a gaze detected on a receiving side of the augmented call, the computing system comprising:
one or more processors; and
one or more memories storing instructions that, when executed by the one or more processors, cause th

Assignees

• Meta Platforms Tech Llc

Inventors

• Ma Jing
• Hoover Paul Armistead
• Vincent Joshuah

Classifications

• G02B27/0172

  characterised by optical features · CPC title

• G02B2027/014

  comprising information/image processing systems · CPC title

• G02B2027/0138

  comprising image capture systems, e.g. camera · CPC title

• G06N20/00

  Machine learning · CPC title

• G02B2027/0174

  holographic · CPC title