Method and system for detecting users' emotions when experiencing a media program

Patent metadata

Field	Value
Publication number	US-9251406-B2
Application number	US-201213528427-A
Country	US
Kind code	B2
Filing date	Jun 20, 2012
Priority date	Jun 20, 2012
Publication date	Feb 2, 2016
Grant date	Feb 2, 2016

Abstract

A system and method for aggregating emotions of users for a media program. A server stores reference audio signal fingerprints, each associated with a reference audio signal of the program. For each user, the server computer: receives a first audio signal fingerprint from a client device operated by a user, the first audio signal fingerprint associated with a first audio signal comprising ambient sound associated with the user and an audio signal of the program; searches the stored reference audio signal fingerprints to determine one that is related to the first audio signal fingerprint; determines an ambient sound signal by obtaining a difference between the stored reference audio signal fingerprint and the first audio signal fingerprint; and determines, using the ambient sound signal, an emotion of the user for a program segment. The server computer aggregates the emotions to determine a representative emotion of the users for the segment.

First claim

What is claimed is:

1. A method comprising:
storing, by a server computer, a plurality of reference audio signal fingerprints, each reference audio signal fingerprint associated with a reference audio signal of a media program;
for each user in a plurality of users,
receiving, by the server computer, a first audio signal fingerprint from a client device operated by a user, the first audio signal fingerprint associated with a first audio signal comprising ambient sound associated with the user and an audio signal of the media program;
searching, by the server computer, the stored plurality of reference audio signal fingerprints to determine one of the stored plurality of reference audio signal fingerprints that matches the first audio signal fingerprint above a given threshold;
extracting, by the server computer, an ambient sound signal comprising the ambient sound from the first audio signal fingerprint by obtaining a difference between the one of the stored plurality of reference audio signal fingerprints and the first audio signal fingerprint, the obtaining a difference comprising obtaining a frequency range of the first audio signal;
determining, by the server computer, using the ambient sound signal, an emotion of the user for a segment of the media program; and
aggregating, by the server computer, the emotion of each user to determine a representative emotion of the plurality of users for the segment of the media program.

2. The method of claim 1 , wherein the determining of the emotion of the user further comprises determining an amplitude of the ambient sound signal.

3. The method of claim 1 , wherein the determining of the emotion of the user further comprises performing a spectrum analysis of the ambient sound signal to classify the ambient sound signal.

4. The method of claim 1 , wherein the aggregating of the emotion of each user to determine a representative emotion of the plurality of users further comprises comparing the ambient sound signal of the user with a plurality of other ambient sound signals associated with other users in different locations.

5. The method of claim 1 , wherein the determining, from the ambient sound signal, of an emotion of the user further comprises determining, from the ambient sound signal, an emotion selected from a group of emotions consisting of laughing, crying, cheering, screaming, clapping, booing, stomping, singing, and sound made by an instrument.

6. The method of claim 1 , wherein the receiving of the first audio signal fingerprint comprising ambient sound associated with the user further comprises receiving the first audio signal fingerprint comprising ambient sound associated with the user and with other people in an area associated with the user.

7. The method of claim 1 , further comprising transmitting the determined emotion of the user to the client device.

8. The method of claim 7 , wherein the transmitting of the determined emotion further comprises transmitting an advertisement to the client device.

9. The method of claim 1 , further comprising updating a status on a social networking site based on the emotion.

10. The method of claim 1 , wherein the aggregating of the emotion of each user to determine a representative emotion of the plurality of users further comprises identifying that the segment of the media program should be a highlight of the media program.

11. The method of claim 1 , further comprising transmitting a recommendation that the media segment should be a highlight of the media program to a third party.

12. The method of claim 1 , wherein the aggregating further comprises identifying a team that the user is cheering for in the media program.

13. The method of claim 1 , wherein the aggregating further comprises providing show synopsis of the media program based on the representative emotion of the plurality of users.

14. A computing device comprising:
a processor;
a storage medium for tangibly storing thereon program logic for execution by the processor, the program logic comprising:
storing logic executed by the processor for storing a plurality of reference audio signal fingerprints, each reference audio signal fingerprint associated with a reference audio signal of a media program;
for each user in a plurality of users,
receiving logic executed by the processor for receiving a first audio signal fingerprint from a client device operated by a user, the first audio signal fingerprint associated with a first audio signal comprising ambient sound associated with the user and an audio signal of the media program;
searching logic executed by the processor for searching the stored plurality of reference audio signal fingerprints to determine one of the stored plurality of reference audio signal fingerprints that matches the first audio signal fingerprint above a given threshold;
first extracting logic executed by the processor for extracting an ambient sound signal comprising the ambient sound from the first audio signal fingerprint by obtaining a difference between the one of the stored plurality of reference audio signal fingerprints and the first audio signal fingerprint, the obtaining a difference comprising obtaining a frequency range of the first audio signal;
second determining logic executed by the processor for determining, using the ambient sound signal, an emotion of the user for a segment of the media program; and
aggregating logic executed by the processor for aggregating the emotion of each user to determine a representative emotion of the plurality of users for the segment of the media program.

15. The computing device of claim 14 , wherein the second determining logic for the determining of the emotion of the user further comprises logic to determine an amplitude of the ambient sound signal.

16. The computing device of claim 14 , wherein the second determining logic for the determining of the emotion of the user further comprises performing logic executed by the processor for performing a spectrum analysis of the ambient sound signal to classify the ambient sound signal.

17. The computing device of claim 14 , wherein the aggregating logic for aggregating the emotion of each user to determine a representative emotion of the plurality of users further comprises comparing logic executed by the processor for comparing the ambient sound signal of the user with a plurality of other ambient sound signals associated with other users in different locations.

18. The computing device of claim 14 , wherein the second determining logic for determining the emotion of the user further comprises determining logic executed by the processor for determining, from the ambient sound signal, an emotion selected from a group of emotions consisting of laughing, crying, cheering, screaming, clapping, booing, stomping, singing, and sound made by an instrument.

19. The computing device of claim 14 , wherein the receiving logic for receiving the first audio signal fingerprint comprising ambient sound associated with the user further comprises logic for receiving the first audio signal fingerprint comprising ambient sound associated with the user and with other people in an area associated with the user.

20. The computing device of claim 14 , further comprising transmitting logic executed by the processor for transmitting the determined emotion of the user to the client device.

21. The computing device of claim 20 , wherein the transmitting logic for transmitting the determined emotion further comprises logic for transmitting an ad

Assignees

• Lempel Ronny
• Bortnikov Edward
• Golbandi Nadav
• Bilobrov Sergiy
• Yahoo Inc

Inventors

• Lempel Ronny
• Bortnikov Edward
• Golbandi Nadav
• Bilobrov Sergiy

Classifications

• H04H60/58

  of audio {(determination or detection of speech characteristics in general G10L25/00; speech recognition in general G10L15/00)} · CPC title

• G06K9/00335

  Physics · mapped topic

• G06Q30/0255 Primary

  based on user history · CPC title

• H04H60/33

  Arrangements for monitoring the users' behaviour or opinions · CPC title

• G06K9/00711

  Physics · mapped topic