Sound to haptic effect conversion system using multiple actuators

Patent metadata

Field	Value
Publication number	US-9715276-B2
Application number	US-201213439241-A
Country	US
Kind code	B2
Filing date	Apr 4, 2012
Priority date	Apr 4, 2012
Publication date	Jul 25, 2017
Grant date	Jul 25, 2017

Abstract

A haptic conversion system is provided that analyzes an audio signal, generates a plurality of haptic signals based on the analysis of the audio signal, and plays the generated plurality of haptic signals through a plurality of actuators to produce one or more haptic effects. The haptic conversion system maps the generated plurality of haptic signals to the plurality of actuators based on one or more audio characteristics of the audio signal. Each generated haptic signal includes one or more haptic parameters, and is played at its mapped actuator to generate the one or more haptic effects.

First claim

We claim:

1. A non-transitory computer-readable medium having instructions stored thereon that, when executed by a processor, cause the processor to convert an audio signal into haptic effects, the converting comprising:
determining sets of values of audio characteristics of the audio signal, each of the sets of values of audio characteristics comprising at least a frequency, a magnitude, or a duration;
mapping the sets of values of audio characteristics to actuators, wherein each of the sets of values of audio characteristics is mapped to a different one of the actuators;
generating haptic signals based on the sets of values of audio characteristics, wherein the haptic signals include a haptic signal generated for each of the sets of values of audio characteristics;
sending the haptic signals to a respective one of the actuators based on the mapping of the sets of values of audio characteristics to the actuators; and
playing the haptic signals at the actuators to generate the haptic effects;
wherein the haptic effects are generated based on audio effects, wherein at least two of the audio effects are output from different parts of a surface of a device, wherein the haptic effects are played at the actuators included in different regions of the device, wherein at least two of the haptic effects are output from different parts of the surface of the device, wherein the actuators play the haptic effects and the audio effects at the same time, wherein at least one haptic effect is generated based on an audio effect being played at a location where a finger of a user is interacting with the surface of the device.

2. The non-transitory computer-readable medium of claim 1 , the converting further comprising:
selecting one or more pre-defined haptic signals from a universal haptic layer library based on the sets of values of audio characteristics of the audio signal;
mapping the one or more pre-defined haptic signals to the actuators, wherein each pre-defined haptic signal is mapped to a corresponding actuator; and
sending each pre-defined haptic signal to its mapped actuator.

3. The non-transitory computer-readable medium of claim 1 , the converting further comprising:
generating one or more secondary haptic signals based on a stored definition, wherein the definition comprises a conditional statement that comprises a condition and a statement, wherein the condition is the generation of one or more primary haptic signals that are sent to a primary actuator, and the statement is an instruction to generate the one or more secondary haptic signals that are sent to a secondary actuator;
mapping the one or more secondary haptic signals to the secondary actuator; and
sending each secondary haptic signal to the secondary actuator based on the stored definition.

4. The non-transitory computer-readable medium of claim 1 , the converting further comprising:
dividing at least one haptic signal into haptic signal components;
mapping the haptic signal components to the actuators, wherein each haptic signal component is mapped to a corresponding actuator; and
sending the haptic signal components to their mapped actuators.

5. The non-transitory computer-readable medium of claim 1 , the converting further comprising:
combining at least two of the haptic signals into a combined haptic signal;
mapping the combined haptic signal to a mapped actuator; and
sending the combined haptic signal to the mapped actuator.

6. The non-transitory computer-readable medium of claim 1 , wherein haptic parameters of the haptic signals comprise at least one of: an amplitude parameter, a duration parameter, or a frequency parameter.

7. The non-transitory computer-readable medium of claim 1 , wherein the determining comprises identifying amplitude peaks of the audio signal.

8. The non-transitory computer-readable medium of claim 7 , wherein the sets of values of audio characteristics comprise at least one of: an amplitude of an audio data frame, a frequency of an audio data frame, or a duration of an audio data frame.

9. The non-transitory computer-readable medium of claim 1 , wherein the mapping is system-defined.

10. The non-transitory computer-readable medium of claim 1 , wherein the mapping is user-defined.

11. The non-transitory computer-readable medium of claim 1 , wherein characteristics of each actuator comprise an ability to generate a haptic effect with at least one of: a specified frequency; a specified amplitude; or a specified duration.

12. The non-transitory computer-readable medium of claim 1 , wherein a user experiences synchronized playback of the haptic effects.

13. The non-transitory computer-readable medium of claim 1 ,
wherein the audio signal comprises channels;
wherein the generating of the haptic signals further comprises generating distinct haptic signals for the channels of the audio signal;
wherein the sending of the haptic signals to the respective one of the actuators sends a distinct haptic signal to a respective distinct actuator.

14. A computer-implemented method for converting an audio signal into haptic effects, the computer-implemented method comprising:
determining sets of values of audio characteristics of the audio signal, each of the sets of values of audio characteristics comprising at least a frequency, a magnitude or a duration;
mapping the sets of values of audio characteristics to actuators, wherein each of the sets of values of audio characteristics is mapped to a different one of the actuators;
generating haptic signals based on the sets of values of audio characteristics, wherein the haptic signals include a haptic signal generated for each of the sets of values of audio characteristics;
sending the haptic signals to a respective one of the actuators based on the mapping of the sets of values of audio characteristics to the actuators; and
playing the haptic signals at the actuators to generate the haptic effects;
wherein the haptic effects are generated based on audio effects, wherein at least two of the audio effects are output from different parts of a surface of a device, wherein the haptic effects are played at the actuators included in different regions of the device, wherein at least two of the haptic effects are output from different parts of the surface of the device, wherein the actuators play the haptic effects and the audio effects at the same time, wherein at least one haptic effect is generated based on an audio effect being played at a location where a finger of a user is interacting with the surface of the device.

15. The computer-implemented method of claim 14 , further comprising:
selecting one or more pre-defined haptic signals from a universal haptic layer library based on the sets of values of audio characteristics of the audio signal;
mapping the one or more pre-defined haptic signals to the actuators, wherein each pre-defined haptic signal is mapped to a corresponding actuator; and
sending each pre-defined haptic signal to its mapped actuator.

16. The computer-implemented method of claim 14 , further comprising:
generating one or more secondary haptic signals based on a stored definition, wherein the definition comprises a conditional statement that comprises a condition and a statement, wherein the condition is the generation of one or more primary haptic signals that are sent to a primary actuator, and the statement is an instruction to generate the one or more secondary haptic signals that are sent to a secondary actuator;
mapping the one or more secondary haptic signals to the secondary actuator; and
sending each secondary hapti

Assignees

• Da Costa Henry
• Timone Herve
• Heubel Robert
• Rank Steve
• Cruz-Hernandez Juan Manuel
• Grant Danny
• Immersion Corp

Inventors

• Da Costa Henry
• Timone Herve
• Heubel Robert
• Rank Steve
• Cruz-Hernandez Juan Manuel
• Grant Danny

Classifications

• G06F3/016 Primary

  Input arrangements with force or tactile feedback as computer generated output to the user · CPC title

• G06F3/16

  Sound input; Sound output (speech processing G10L) · CPC title

• G08B6/00 Primary

  Tactile signalling systems, e.g. tactile personal calling systems · CPC title