Gesture interactive wearable spatial audio system

Patent metadata

Field	Value
Publication number	US-10585486-B2
Application number	US-201415107774-A
Country	US
Kind code	B2
Filing date	Dec 31, 2014
Priority date	Jan 3, 2014
Publication date	Mar 10, 2020
Grant date	Mar 10, 2020

Abstract

One embodiment of the present invention sets forth a technique for modifying an audio parameter based on a gesture. The technique includes acquiring sensor data associated with a hand of a user and analyzing the sensor data to determine at least one hand position. The technique further includes detecting a hand gesture based on the at least one hand position and, in response to the hand gesture, modifying a spatial audio parameter associated with an audio stream to generate a modified audio stream. The technique further includes causing the modified audio stream to be reproduced for output to the user.

First claim

What is claimed is:

1. A method for modifying an audio parameter based on a gesture, the method comprising:
acquiring, via at least one sensor included in a head-mounted device, sensor data associated with a hand of a user;
analyzing, via a processor, the sensor data to determine at least one hand position;
causing haptic feedback to be generated in response to detecting an intersection between the at least one hand position and a virtual object associated with an audio event included in an audio stream;
detecting a hand gesture based on the at least one hand position;
in response to the hand gesture, modifying, via a processor, a spatial audio parameter associated with the audio event to generate a modified audio stream by moving a first audio event included in the audio stream towards a first side of a sound space and moving a second audio event included in the audio stream towards a second side of the sound space; and
causing the modified audio stream to be reproduced for output to the user.

2. The method of claim 1 , wherein analyzing the sensor data comprises determining a first hand position at which the hand gesture was initiated, and determining a second hand position at which the hand gesture was completed.

3. The method of claim 2 , wherein modifying the spatial audio parameter comprises moving the audio event from a first location in a sound space associated with the first hand position towards a second location in the sound space associated with the second hand position.

4. The method of claim 3 , wherein the hand gesture comprises a grasping gesture.

5. The method of claim 3 , wherein modifying the spatial audio parameter further comprises determining, via a physics model, an interaction between a first virtual object that corresponds to the audio event and a second virtual object that corresponds to the hand of the user.

6. The method of claim 1 , wherein the sensor data comprises at least one of image data and depth data.

7. A system for modifying an audio parameter based on a gesture, comprising:
a head-mounted device;
one or more sensors included in the head-mounted device and configured to acquire sensor data associated with a hand of a user;
a processor coupled to the one or more sensors and configured to:
analyze the sensor data to determine at least one hand position;
cause haptic feedback to be generated in response to detecting an intersection between the at least one hand position and a virtual object associated with an audio event included in an audio stream;
detect a hand gesture based on the at least one hand position; and
in response to the hand gesture, move a first audio event included in an audio stream from a first location in a sound space towards a second location in the sound space, wherein the second location corresponds to a first side of the sound space, and move a second audio event included in the audio stream towards a second side of the sound space, to generate a modified audio stream; and

speakers coupled to the processor and configured to reproduce the modified audio stream.

8. The system of claim 7 , wherein the hand gesture comprises a splitting gesture.

9. The system of claim 7 , wherein the processor is configured to analyze the sensor data to determine a first hand position at which the hand gesture was initiated and a second hand position at which the hand gesture was completed, and wherein the first location in the sound space is associated with the first hand position, and the second location in the sound space is associated with the second hand position.

10. The system of claim 9 , wherein the hand gesture comprises at least one of a pushing gesture and a swiping gesture.

11. The system of claim 9 , wherein the processor is configured to move the first audio event by determining a distance between the first hand position and the second hand position.

12. The system of claim 7 , wherein the one or more sensors comprise a first sensor and a second sensor, and wherein the processor is configured to determine the at least one hand position by determining a first position of the first sensor relative to a second position of the second sensor.

13. The system of claim 7 , wherein the processor is configured to move the first audio event from the first location in the sound space towards the second location in the sound space based on a head-related transfer function.

14. The system of claim 7 , further comprising a feedback device configured to produce haptic feedback in response to detecting an intersection between the at least one hand position and a virtual object associated with the first audio event.

15. The system of claim 7 , wherein the processor is further configured to cause the speakers to produce auditory feedback in response to detecting an intersection between the at least one hand position and a virtual object associated with the first audio event.

16. A non-transitory computer-readable storage medium including instructions that, when executed by a processor, cause the processor to modify an audio parameter based on a gesture, by performing the steps of:
acquiring, via at least one sensor included in a head-mounted device, sensor data associated with a hand of a user;
analyzing the sensor data to determine a first hand position and a second hand position;
detecting a hand gesture based on the first hand position and the second hand position, wherein the hand gesture comprises a splitting gesture;
in response to the splitting gesture, modifying a spatial audio parameter associated with an audio stream to generate a modified audio stream by moving a first audio event included in the audio stream towards a first side of a sound space and moving a second audio event included in the audio stream towards a second side of the sound space; and
causing the modified audio stream to be reproduced for output to the user.

17. The method of claim 1 , wherein the head-mounted device comprises at least one of a set of headphones and a pair of eyeglasses.

18. The method of claim 1 , wherein the at least one sensor comprises a first sensor and a second sensor, and the first sensor and the second sensor are configured to acquire a stereo image of the hand of the user.

19. A method for modifying an audio parameter based on a gesture, the method comprising:
acquiring, via at least one sensor included in a head-mounted device, sensor data associated with a hand of a user;
analyzing, via a processor, the sensor data to determine at least one hand position by:
determining a first hand position at which a hand gesture was initiated, and
determining a second hand position at which the hand gesture was completed;

causing haptic feedback to be generated in response to detecting an intersection between the at least one hand position and a virtual object associated with an audio event included in an audio stream;
detecting the hand gesture based on the at least one hand position;
in response to the hand gesture, modifying, via a processor, a spatial audio parameter associated with the audio event to generate a modified audio stream by:
moving the audio event from a first location in a sound space associated with the first hand position towards a second location in the sound space associated with the second hand position, and
determining, via a physics model, an interaction between a first virtual object that corresponds to the audio event and a second virtual object that corresponds to the hand of the user; and

causing the modified audio stream to be reproduced for output to

Assignees

• Harman Int Ind

Inventors

• Coleman Tim
• Marti Stefan
• Di Censo Davide

Classifications

• G06F3/0304

  Detection arrangements using opto-electronic means (constructional details of pointing devices not related to the detection arrangement using opto-electronic means G06F3/033; optical digitisers G06F3/042) · CPC title

• G06F3/04842

  Selection of displayed objects or displayed text elements (G06F3/0482 takes precedence) · CPC title

• G06F3/016

  Input arrangements with force or tactile feedback as computer generated output to the user · CPC title

• G06F3/165

  Management of the audio stream, e.g. setting of volume, audio stream path · CPC title

• H04S7/304

  For headphones · CPC title