Eyewear with audio source separation using pose trackers

Patent metadata

Field	Value
Publication number	US-12238495-B2
Application number	US-202217711494-A
Country	US
Kind code	B2
Filing date	Apr 1, 2022
Priority date	Apr 1, 2022
Publication date	Feb 25, 2025
Grant date	Feb 25, 2025

Abstract

Electronic eyewear device providing simplified audio source separation, also referred to as voice/sound unmixing, using alignment between respective device trajectories. Multiple users of electronic eyewear devices in an environment may simultaneously generate audio signals (e.g., voices/sounds) that are difficult to distinguish from one another. The electronic eyewear device tracks the location of moving remote electronic eyewear devices of other users, or an object of the other users, such as the remote user's face, to provide audio source separation using location of the sound sources. The simplified voice unmixing uses a microphone array of the electronic eyewear device and the known location of the remote user's electronic eyewear device with respect to the user's electronic eyewear device to facilitate audio source separation.

First claim

What is claimed is:

1. Electronic eyewear device, comprising:
an inertial measurement unit (IMU);
a pose tracker;
a microphone array configured to receive sounds from audio sources in an environment and generate audio signals;
a head-mounting frame that supports the IMU, the pose tracker, and the microphone array; and
a processor configured to:
generate a trajectory of the electronic eyewear device worn by a user using the pose tracker;
determine a trajectory of a remote electronic eyewear device worn by another user by identifying a fixed point on a symmetry plane of a face of the other user;
align the trajectory of the pose tracker with the trajectory of the remote electronic eyewear device;
determine a location of the remote electronic eyewear device from the determined trajectory;
unmix the received sounds based on the determined location of the remote electronic eyewear device; and
enable the user to select a desired direction to isolate the sounds by interacting with the electronic eyewear device.

2. The electronic eyewear device of claim 1 , wherein the processor is further configured to establish a shared coordinate system between the electronic eyewear device and the remote electronic eyewear device using the trajectories.

3. The electronic eyewear device of claim 2 , wherein the processor is further configured to use the trajectories and the microphone array to unmix the received sounds.

4. The electronic eyewear device of claim 3 , wherein the processor is further configured to determine a direction of arrival of the received sounds relative to the electronic eyewear device.

5. The electronic eyewear device of claim 3 , further comprising:
at least one speaker supported by the head-mounting frame to present unmixed voices to a user of the electronic eyewear device.

6. The electronic eyewear device of claim 1 , wherein the electronic eyewear device is configured to have an orientation calibrated to the remote electronic eyewear device.

7. The electronic eyewear device of claim 2 , wherein the processor is configured to:
use the aligned trajectories to transform poses of the electronic eyewear device into the shared coordinate system.

8. The electronic eyewear device of claim 1 , wherein the electronic eyewear device further comprises a beamformer configured to separate the audio signals based on the determined location of the remote electronic device.

9. An interactive augmented reality method for use of an electronic eyewear device having an inertial measurement unit (IMU), a pose tracker, a microphone array configured to receive sounds from an environment and generate audio signals, a head-mounting frame that supports the IMU, the pose tracker, and the microphone array, and a processor, the processor:
using the pose tracker to generate a trajectory of the electronic eyewear device;
determining a trajectory of a remote electronic eyewear device worn by another user by identifying a fixed point on a symmetry plane of a face of the other user;
aligning the trajectory of the pose tracker with the trajectory of the remote electronic eyewear device;
determining a location of the remote electronic eyewear device from the received trajectory;
unmixing the received sounds based on the determined location of the remote electronic eyewear device; and
enabling the user to select a desired direction to isolate the sounds by interacting with the electronic eyewear device.

10. The method of claim 9 , wherein the processor establishes a shared coordinate system between the electronic eyewear device and the remote electronic eyewear device using the trajectories.

11. The method of claim 10 , wherein the processor uses the trajectories and the microphone array to unmix the received sounds.

12. The method of claim 11 , wherein the processor determines a direction of arrival of the received sounds relative to the electronic eyewear device.

13. The method of claim 11 , further comprising at least one speaker supported by the head-mounting frame and presenting unmixed voices to a user of the electronic eyewear device.

14. The method of claim 9 , wherein the electronic eyewear device has an orientation calibrated to the remote electronic eyewear device.

15. The method of claim 11 , wherein the processor:
uses the aligned trajectories to transform poses of the electronic eyewear device into the shared coordinate system.

16. The method of claim 9 , wherein the electronic eyewear device further comprises a beamformer configured to separate the audio signals based on the determined location of the remote electronic device.

17. A non-transitory computer-readable medium storing program code which, when executed, is operative to cause an electronic processor of an electronic eyewear device having an inertial measurement unit (IMU), a pose tracker, a microphone array configured to receive sounds from an environment and generate audio signals, and a head-mounting frame that supports the IMU, pose tracker, and microphone array, to perform the steps of:
using the pose tracker to generate a trajectory of the electronic eyewear device;
determining a trajectory of a remote electronic eyewear device worn by another user by identifying a fixed point on a symmetry plane of a face of the other user;
aligning the trajectory of the pose tracker with the trajectory of the remote electronic eyewear device;
determining a location of the remote electronic eyewear device from the received trajectory;
unmixing the received sounds based on the determined location of the remote electronic eyewear device; and
enabling the user to select a desired direction to isolate the sounds by interacting with the electronic eyewear device.

18. The non-transitory computer-readable medium storing program code of claim 17 , further comprising code for the processor to establish shared coordinate system between the electronic eyewear device and the remote electronic eyewear device using the trajectories.

19. The non-transitory computer-readable medium storing program code of claim 18 , further comprising code for:
the processor to use the trajectories and the microphone array to unmix the received sounds.

20. The non-transitory computer-readable medium storing program code of claim 18 , further comprising code for:
the processor to determine a direction of arrival of the received sounds relative to the electronic eyewear device.

Assignees

• Evangelidis Georgios
• Arya Ashwani
• Pounds Jennica
• Rybin Andrei
• Snap Inc

Inventors

• Evangelidis Georgios
• Arya Ashwani
• Pounds Jennica
• Rybin Andrei

Classifications

• H04R5/027

  Spatial or constructional arrangements of microphones, e.g. in dummy heads · CPC title

• H04R1/1091

  Details not provided for in groups H04R1/1008 - H04R1/1083 · CPC title

• H04R1/105

  Earpiece supports, e.g. ear hooks (for stereophonic headphones H04R5/0335) · CPC title

• G06F3/165

  Management of the audio stream, e.g. setting of volume, audio stream path · CPC title

• G06F3/012

  Head tracking input arrangements · CPC title