Sensor driven enhanced visualization and audio effects

Patent metadata

Field	Value
Publication number	US-10102835-B1
Application number	US-201715581957-A
Country	US
Kind code	B1
Filing date	Apr 28, 2017
Priority date	Apr 28, 2017
Publication date	Oct 16, 2018
Grant date	Oct 16, 2018

Abstract

Systems and methods may be used to provide effects corresponding to movement of instrument objects or other objects. A method may include receiving sensor data from an object based on movement of the object, recognizing a gesture from the sensor data, and determining an effect, such as a visualization or audio effect corresponding to the gesture. The method may include causing the effect to be output in response to the determination.

First claim

What is claimed is:

1. A server in communication with a pair of drum sticks, the server comprising:
a processor to:
receive sensor data from a sensor of at least one drum stick of the pair of drum sticks, the sensor data based on movement of the at least one drum stick;
recognize a gesture from the sensor data;
determine, from the gesture, a visualization effect corresponding to the gesture and an audio effect including a drum sound corresponding to the gesture; and
cause the visualization effect and the audio effect to be output in response to the determination, wherein the visualization effect includes a virtual drum set and is output to a display to be displayed in coordination with the audio effect.

2. The server of claim 1 , wherein to determine the visualization effect, the processor is to use a visualization engine.

3. The server of claim 1 , wherein the sensor data includes data from sensors of both of the pair of drum sticks and wherein the gesture includes movement of the pair of drum sticks in coordination with each other.

4. The server of claim 1 , wherein to cause the visualization effect to be output, the processor is to send the visualization effect to a virtual reality headset of a user controlling the pair of drum sticks to be displayed on the virtual reality headset.

5. The server of claim 1 , wherein to cause the audio effect to be output, the processor is to send the audio effect to a speaker to play the audio effect.

6. The server of claim 1 , wherein the output to the display includes captured video of a person performing the movement with the at least one drumstick.

7. The server of claim 1 , wherein the processor is further to receive data from the sensor indicating an initial position of the at least one drum stick, and wherein to recognize the gesture, the processor is to determine a final position of the at least one drum stick.

8. The server of claim 7 , wherein the drum sound is determined based on the initial position and the final position.

9. The server of claim 1 , wherein the visualization effect corresponding to the gesture and the audio effect including the drum sound corresponding to the gesture are determined based on an orientation of the at least one drum stick identified in the sensor data.

10. The server of claim 1 , wherein to determine the visualization effect, the processor is to determine the visualization effect based on a series of previously recognized gestures.

11. The server of claim 1 , wherein the processor is further to:
receive additional sensor data from a second sensor attached to an ankle or a foot of a user controlling the at least one drum stick;
recognize a second gesture from the additional sensor data;
determine, from the second gesture, a second audio effect including a second drum sound corresponding to the second gesture; and
cause the second audio effect to be output with the visualization effect and the audio effect.

12. The server of claim 1 , wherein the processor is further to:
receive wearable sensor data from a plurality of wearable devices within a predetermined proximity of the at least one drum stick; and
modify the visualization effect based on the wearable sensor data.

13. The server of claim 1 , wherein the visualization effect includes a lighting effect, and wherein to cause the visualization effect to be output, the processor is to send the lighting effect to a plurality of wearable devices within a predetermined proximity of the at least one drum stick to be displayed at the plurality of wearable devices.

14. The server of claim 1 , wherein the gesture includes at least one of a linear movement, a tapping movement, a sweeping movement, a minimum acceleration, or a minimum deceleration.

15. The server of claim 1 , wherein the audio effect includes Multidimensional Polyphonic Expression instructions for a musical instrument digital interface (MIDI) player.

16. A method for providing effects corresponding to movement of drum sticks, the method comprising:
receiving sensor data from a sensor of at least one drum stick of a pair of drum sticks, the sensor data based on movement of the at least one drum stick;
recognizing a gesture from the sensor data;
determining, from the gesture, a visualization effect corresponding to the gesture and an audio effect including a drum sound corresponding to the gesture; and
causing the visualization effect and the audio effect to be output in response to the determination, wherein the visualization effect includes a virtual drum set and is output to a display to be displayed in coordination with the audio effect.

17. The method of claim 16 , wherein determining the visualization effect includes using a visualization engine.

18. The method of claim 16 , wherein the sensor data includes data from sensors of both of the pair of drum sticks and wherein the gesture includes movement of the pair of drum sticks in coordination with each other.

19. The method of claim 16 , wherein causing the visualization effect to be output includes sending the visualization effect to a virtual reality headset of a user controlling the pair of drum sticks to be displayed on the virtual reality headset.

20. At least one non-transitory machine-readable medium including instructions for providing effects corresponding to movement of drum sticks, which when executed by a machine, cause the machine to:
receive sensor data from a sensor of at least one drum stick of a pair of drum sticks, the sensor data based on movement of the at least one drum stick;
recognize a gesture from the sensor data;
determine, from the gesture, a visualization effect corresponding to the gesture and an audio effect including a drum sound corresponding to the gesture; and
cause the visualization effect and the audio effect to be output in response to the determination, wherein the visualization effect includes a virtual drum set and is output to a display to be displayed in coordination with the audio effect.

21. The at least one non-transitory machine-readable medium of claim 20 , further comprising instructions to:
receive wearable sensor data from a plurality of wearable devices within a predetermined proximity of the at least one drum stick; and
modify the visualization effect based on the wearable sensor data.

22. The at least one non-transitory machine-readable medium of claim 20 , wherein the visualization effect includes a lighting effect, and wherein the instructions to cause the visualization effect to be output include instructions to send the lighting effect to a plurality of wearable devices within a predetermined proximity of the at least one drum stick to be displayed at the plurality of wearable devices.

23. A virtual drum set system comprising:
a pair of drum sticks each including:
a sensor to provide data based on movement of a drum stick of the pair of drum sticks; and
a transceiver to transmit the sensor data;

a device including a processor to:
recognize a gesture from the sensor data; and
determine, from the gesture, a visualization effect corresponding to the gesture and an audio effect including a drum sound corresponding to the gesture; and

a display device to display the visualization effect, wherein the visualization effect includes a virtual drum set and is output to a display to be displayed in coordination with the audio effect; and
a speaker to play the audio effect.

24. The virtual drum set system of claim

Assignees

• Intel Corp

Inventors

• Shah Saurin
• Kar Swarnendu
• Krishnamurthy Lakshman
• Webb Arthur D
• Goel Manan

Classifications

• G06F3/165

  Management of the audio stream, e.g. setting of volume, audio stream path · CPC title

• G06F3/167

  Audio in a user interface, e.g. using voice commands for navigating, audio feedback · CPC title

• G10H2220/201

  for movement interpretation, i.e. capturing and recognizing a gesture or a specific kind of movement, e.g. to control a musical instrument · CPC title

• G06F3/017

  Gesture based interaction, e.g. based on a set of recognized hand gestures (interaction based on gestures traced on a digitiser G06F3/04883) · CPC title

• G10H2220/365

  Bow control in general, i.e. sensors or transducers on a bow; Input interface or controlling process for emulating a bow, bowing action or generating bowing parameters, e.g. for appropriately controlling a specialised sound synthesiser · CPC title