Audio augmentation using environmental data

Patent metadata

Field	Value
Publication number	US-10595149-B1
Application number	US-201816208596-A
Country	US
Kind code	B1
Filing date	Dec 4, 2018
Priority date	Dec 4, 2018
Publication date	Mar 17, 2020
Grant date	Mar 17, 2020

Abstract

The disclosed computer-implemented method for performing directional beamforming according to an anticipated position may include accessing environment data indicating a sound source within an environment. The device may include various audio hardware components configured to generate steerable audio beams. The method may also include identifying the location of the sound source within the environment based on the accessed environment data, and then steering the audio beams of the device to the identified location of the sound source within the environment. Various other methods, systems, and computer-readable media are also disclosed.

First claim

We claim:

1. A computer-implemented method comprising:
providing a device;
accessing, at the device, one or more portions of environment data that includes an indication of at least one sound source within the environment, the device including one or more audio hardware components configured to generate one or more steerable audio beams, wherein the device comprises an artificial reality (AR) device and wherein the environment data comprises simultaneous location and mapping (SLAM) sensor data acquired by the AR device;
identifying the location of the sound source within the environment based on the accessed environment data including the SLAM sensor data; and
steering the one or more audio beams of the device to the identified location of the sound source within the environment.

2. The computer-implemented method of claim 1 , wherein the environment includes a plurality of AR devices, each AR device recording its own location.

3. The computer-implemented method of claim 1 , wherein the environment includes a plurality of AR devices, each AR device recording the location of other AR devices using sensor data captured by the AR devices.

4. The computer-implemented method of claim 1 , further comprising accessing one or more portions of historical device movement data and implementing the accessed historical device movement data to identify a future sound source location where the sound source is likely to move.

5. The computer-implemented method of claim 1 , further comprising determining future sound source locations on a continually updated basis, such that the one or more audio beams of the device are continually steered to the updated future sound source location.

6. The computer-implemented method of claim 1 , further comprising:
detecting that a reverberated signal was received at the device at a higher signal level than a direct-path signal;
identifying at least one potential path traveled by the reverberated signal; and
steering the one or more audio beams to travel along the identified path traveled by the reverberated signal.

7. The computer-implemented method of claim 6 , further comprising transitioning the audio beam steering back to a direct path as the device moves between the current device location and the future device location.

8. The computer-implemented method of claim 1 , wherein the device comprises at least one of a smartphone, tablet, laptop or wearable device.

9. A device comprising:
at least one physical processor;
physical memory comprising computer-executable instructions that, when executed by the physical processor, cause the physical processor to:
access, at the device, one or more portions of environment data that includes an indication of at least one sound source within the environment, the device including one or more audio hardware components configured to generate one or more steerable audio beams, wherein the device comprises an artificial reality (AR) device and wherein the environment data comprises simultaneous location and mapping (SLAM) sensor data acquired by the AR device;
identify the location of the sound source within the environment based on the accessed environment data including the SLAM sensor data; and
steer the one or more audio beams of the device to the identified location of the sound source within the environment.

10. The device of claim 9 , further comprising:
accessing an audio signal that is to be reproduced using audio signals received via the one or more audio beams;
identifying the location of a second device; and
modifying the accessed audio signal to spatially re-render the audio signal to sound as if coming from the second device.

11. The device of claim 9 , further comprising receiving, at the device, pre-generated environment data or historical environmental data from a remote source and implementing the received environment data or historical environmental data to identify a future sound source location.

12. The device of claim 9 , further comprising providing, by one or more other devices in the environment, one or more portions of environment data to a server or to another local or remote device.

13. The device of claim 12 , further comprising augmenting, by the server, the environment information to account for delay and constraints of a target device.

14. The device of claim 9 , further comprising generating steering control signals upon determining that beamforming is needed to raise a signal level to a specified minimum level.

15. The device of claim 9 , further comprising using the accessed portions of environment data to perform selective active noise cancellation in a specified direction.

16. The device of claim 15 , further comprising adjusting one or more active noise cancellation parameters to selectively remove sounds from a specified a direction, or to selectively allow sounds from a specified direction.

17. The device of claim 9 , further comprising combining a dry audio signal with one or more effects such that the modified dry audio signal sounds as if the modified dry audio signal originated in the environment.

18. A non-transitory computer-readable medium comprising one or more computer-executable instructions that, when executed by at least one processor of an artificial reality (AR) device, cause the computing device to:
access one or more portions of environment data that includes an indication of at least one sound source within the environment, the AR device including one or more audio hardware components configured to generate one or more steerable audio beams, wherein the environment data comprises simultaneous location and mapping (SLAM) sensor data acquired by the AR device;
identify the location of the sound source within the environment based on the accessed environment data including the SLAM sensor data; and
generate steering control signals that steer the one or more audio beams of the device to the identified location of the sound source within the environment.

19. The computer-implemented method of claim 1 , wherein the AR device comprises a pair of artificial reality glasses.

20. The computer-implemented method of claim 1 , wherein the SLAM data is received via a wireless data transfer from a data store to the AR device.

Assignees

• Facebook Tech Llc

Inventors

• Lovitt Andrew
• Selfon Scott Phillip
• Miller Antonio John

Classifications

• H04R5/027

  Spatial or constructional arrangements of microphones, e.g. in dummy heads · CPC title

• H04S2420/01

  Enhancing the perception of the sound image or of the spatial distribution using head related transfer functions [HRTF's] or equivalents thereof, e.g. interaural time difference [ITD] or interaural level difference [ILD] · CPC title

• H04S2400/11

  Positioning of individual sound objects, e.g. moving airplane, within a sound field (H04S2420/13 takes precedence) · CPC title

• H04S7/304 Primary

  For headphones · CPC title

• H04R2430/23

  Direction finding using a sum-delay beam-former · CPC title