User-configurable spatial audio based conferencing system

Patent metadata

Field	Value
Publication number	US-2023008964-A1
Application number	US-202117367979-A
Country	US
Kind code	A1
Filing date	Jul 6, 2021
Priority date	Jul 6, 2021
Publication date	Jan 12, 2023
Grant date	—

Abstract

A client device receives an arrangement of at least a subset of participants of a virtual meeting. The client device additionally receives an audio stream for each participant of the subset of participants of the virtual meeting. For each participant of the subset of participants, the client device determines a location based at least in part on the received arrangement, and modulates the received audio stream of the participant based on the determined location. The client device generates a combined modulated audio stream by combining the modulated audio stream of each of the participants and plays the combined modulated audio stream.

First claim

1 . A method comprising:
joining a virtual meeting having a plurality of participants; receiving an arrangement for at least a subset of participants of the virtual meeting; receiving an audio stream for each participant of the subset of participants of the virtual meeting; assigning two or more of the subset of participants to an audience group; for each participant of the subset of participants not assigned to the audience group:
determining a location for the participant based on the received arrangement, and
modulating the received audio stream of the participant based on the determined location for the participant;
combining the received audio streams for the participants assigned to the audience group; determining a location for the audience group based on the received arrangement; modulating the combined received audio streams of the participants assigned to the audience group based on the determined location for the audience group; generating a combined modulated audio stream by combining the modulated audio stream of each of the participants of the subset of participants and the modulated audio stream of the audience group; and playing the combined modulated audio stream.

2 . The method of claim 1 , wherein the location of the participant is further determined based on sensor data of one or more sensors for determining a pose of a listener.

3 . The method of claim 2 , wherein the one or more sensors are embedded in a head-mounted display.

4 . The method of claim 2 , wherein the one or more sensors are embedded in one of headphones or earphones.

5 . The method of claim 1 , wherein the received audio stream is modulating using a head-related transfer function.

6 . The method of claim 1 , wherein receiving an arrangement for at least a subset of participants of a virtual meeting comprises:
receiving a position within a graphical user interface for each participant of the subset of participants.

7 . The method of claim 6 , wherein the graphical user interface arranges the participants in one of a grid, a circle, a curved segment, and a three-dimensional arrangement.

8 . The method of claim 1 , wherein receiving an arrangement for at least a subset of participants of a virtual meeting comprises:
receiving a classification for each participant of the subset of participants of the virtual meeting; and determining an arrangement for each of the participants based on the received classification for the participant.

9 . The method of claim 8 , wherein the subset of participants includes a first participant having a first classification and a second participant having a second location, and wherein determining an arrangement for each of the participants comprises:
assigning a first position for the first participant within a first region associated with the first classification, and assigning a second position for the second participant within a second region associated with the second classification, the second region different than the first region.

10 . (canceled)

11 . A non-transitory computer-readable storage medium configured to store instructions, the instructions when executed by a processor cause the processor to:
join a virtual meeting having a plurality of participants; receive an arrangement for at least a subset of participants of the virtual meeting; receive an audio stream for each participant of the subset of participants of the virtual meeting; assign two or more of the subset of participants to an audience group; for each participant of the subset of participants not assigned to the audience group:
determine a location for the participant based on the received arrangement, and
modulate the received audio stream of the participant based on the determined location for the participant;
combine the received audio streams for the participants assigned to the audience group; determine a location for the audience group based on the received arrangement; modulate the combined received audio streams of the participants assigned to the audience group based on the determined location for the audience group; generate a combined modulated audio stream by combining the modulated audio stream of each of the participants of the subset of participants and the modulated audio stream of the audience group; and play the combined modulated audio stream.

12 . The non-transitory computer-readable storage medium of claim 11 , wherein the location of the participant is further determined based on sensor data of one or more sensors for determining a pose of a listener.

13 . The non-transitory computer-readable storage medium of claim 12 , wherein the one or more sensors are embedded in a head-mounted display.

14 . The non-transitory computer-readable storage medium of claim 12 , wherein the one or more sensors are embedded in one of headphones or earphones.

15 . The non-transitory computer-readable storage medium of claim 11 , wherein the received audio stream is modulating using a head-related transfer function.

16 . The non-transitory computer-readable storage medium of claim 11 , wherein the instructions for receiving an arrangement for at least a subset of participants of a virtual meeting cause the processor to:
receive a position within a graphical user interface for each participant of the subset of participants.

17 . The non-transitory computer-readable storage medium of claim 16 , wherein the graphical user interface arranges the participants in one of a grid, a circle, a curved segment, and a three-dimensional arrangement.

18 . The non-transitory computer-readable storage medium of claim 11 , wherein the instructions for receiving an arrangement for at least a subset of participants of a virtual meeting cause the processor to:
receive a classification for each participant of the subset of participants of the virtual meeting; and determine an arrangement for each of the participants based on the received classification for the participant.

19 . The non-transitory computer-readable storage medium of claim 18 , wherein the subset of participants includes a first participant having a first classification and a second participant having a second location, and wherein the instruction for determining an arrangement for each of the participants cause the processor to:
assign a first position for the first participant within a first region associated with the first classification, and assign a second position for the second participant within a second region associated with the second classification, the second region different than the first region.

20 . (canceled)

21 . The method of claim 8 , wherein the two or more of the subset of participants are assigned to the audience group based on the classification of the participants.

22 . The non-transitory computer-readable storage medium of claim 18 , wherein the two or more of the subset of participants are assigned to the audience group based on the classification of the participants.

Assignees

• Meta Platforms Inc

Inventors

• Fernandez Guajardo Maria
• Tewes Isabel

Classifications

• H04N7/157

  defining a virtual conference space and using avatars or agents (computer conference optimisation or adaptation H04L12/1827) · CPC title

• H04S2420/01

  Enhancing the perception of the sound image or of the spatial distribution using head related transfer functions [HRTF's] or equivalents thereof, e.g. interaural time difference [ITD] or interaural level difference [ILD] · CPC title

• G06F3/04817

  using icons (graphical or visual programming using iconic symbols G06F8/34) · CPC title

• H04S7/304 Primary

  For headphones · CPC title

• H04L12/1827 Primary

  Network arrangements for conference optimisation or adaptation · CPC title