Identification of concurrently broadcast time-based media

Patent metadata

Field	Value
Publication number	US-9871606-B1
Application number	US-201414277040-A
Country	US
Kind code	B1
Filing date	May 13, 2014
Priority date	May 13, 2013
Publication date	Jan 16, 2018
Grant date	Jan 16, 2018

Abstract

A real time messaging platform identifies an audio snippet of a time-based media (TBM) event. The messaging platform maintains a real time repository of concurrently broadcasting TBM events as well as a historical repository of previously broadcast TBM events. These repositories contain acoustic fingerprints of their respective TBM events. The messaging platform matches an acoustic fingerprint of the audio snippet with one of the stored acoustic fingerprints to identify the TBM event in the recorded snippet. To identify the TBM event, the messaging platform matches multiple overlapping reference audio segments of the reference audio stream with multiple test audio segments of the audio snippet. This allows the platform to account for time delays between the test and reference audio segments that would otherwise hinder the matching process.

First claim

What is claimed is:

1. A method comprising:
receiving, at a real-time messaging platform comprising a plurality of accounts, a reference audio stream of a time-based media (TBM) event;
identifying reference audio segments of the reference audio stream, having different start times and end times;
generating a plurality of reference audio features based on a plurality of inclusion conditions for each of the reference audio segments, each reference audio feature associated with a reference feature time and comprising a plurality of reference audio properties;
receiving, at the real-time messaging platform, an audio snippet for identification from a client device, the client device associated with one of the accounts;
identifying, by a computer processor, test audio segments of the audio snippet for identification, the test audio segments at least partially overlapping each other in time;
generating, by the computer processor, a plurality of test audio features based on the inclusion conditions for each of the test audio segments, each test audio feature associated with a test feature time and comprising a plurality of test audio properties;
determining, that one or more test audio features of a first one of the test audio segments matches one or more reference audio features of a first one of the reference audio segments, the determination comprising;
comparing the test feature time of a first one of the test audio features and the reference feature time of a first one of the reference audio features, and
comparing a first of the test audio properties of the first test audio feature to the a first of the reference audio properties of the first test audio feature;

determining that the first reference audio segment including at least the first reference audio feature matches the first test audio segment including at least the first test audio feature; and
storing, in response to determining that the test audio segment matches the reference audio segment, an association between the account and TBM event.

2. The method of claim 1 further comprising:
maintaining a repository of reference audio segments comprising reference audio segments having one of: start times after a threshold time and end times after the threshold time, the threshold time determined relative to a current time, the reference audio segment contained in the repository of reference audio segments.

3. The method of claim 1 , wherein determining that the first test audio segment matches the first reference audio segment comprises:
comparing the first test audio segment with a plurality of additional reference audio segments identified from a plurality of additional reference audio streams of a plurality of additional concurrently broadcasting TBM events; and
selecting a final reference audio segment based on a best match between the first reference audio segment and the first test audio segment over matches between the first test audio segment and the plurality of additional reference audio segments.

4. The method of claim 3 , wherein determining that the first test audio segment matches the final reference audio segment comprises:
identifying a plurality of matching candidate reference audio segments from the plurality of additional reference audio segments; and
selecting the final reference audio segment based on the best match between the matching candidate reference audio segments and the first test audio segment based on at least one of:
audio features of the plurality of additional reference audio segments having more finely quantized features than the generated reference audio features and the generated test audio features, and
audio features of the plurality of additional references audio segments are generated by applying a lower energy threshold than an energy threshold applied to generate the first reference audio features and the first test audio features.

5. The method of claim 1 , further comprising:
transmitting, to the client device, an identification of the concurrently broadcasting TBM event corresponding to the reference audio stream.

6. The method of claim 1 , wherein the plurality of reference audio features correspond to at least one of: amplitudes of the reference audio stream at the reference feature times, and frequencies of the reference audio stream at the reference feature times.

7. The method of claim 1 , wherein the plurality of test audio features correspond to at least one of: amplitudes of the audio snippet at the test feature times, and frequencies of the audio snippet at the test feature times.

8. The method of claim 1 , wherein determining that the reference audio segment matches the test audio segment comprises:
determining scores for matches between reference audio segments and test audio segments based at least in part on a number of the one or more test audio features matching the one or more reference audio features; and
selecting the reference audio segment matching the test audio segment based on the determined scores.

9. The method of claim 8 , wherein determining that the reference audio segment matches the test audio segment comprises:
obtaining interests of the account, the interests obtained from at least one of: interests inferred from content of messages associated with the account, interests inferred from interests of accounts connected to the account, and interests explicitly associated with the account; and
modifying the determined scores based on correlations between interests associated with currently broadcasting TBM events and the obtained interests.

10. The method of claim 1 , further comprising:
obtaining a geographic location associated with the client device, the geographic location obtained from at least one of: an inferred geographic location of the client device based on the associated account, and a geographic location transmitted from the client device; and
applying a filter to the reference audio stream to ensure that the reference audio stream does not correspond to a TBM event unavailable to client devices in the obtained geographic location.

11. The method of claim 1 , wherein each reference audio segment of the reference audio segments has a same reference audio segment duration and each test audio segment has a same test audio segment duration.

12. The method of claim 1 , wherein the stored association between the account and the concurrently broadcasting TBM event is a connection from the account to another account associated with the concurrently broadcasting TBM event.

13. The method of claim 1 wherein
the reference audio stream is received at a server from a currently broadcasting television show, live concert, or radio show.

14. The method of claim 1 wherein
the reference audio segments are generated as the reference audio stream is received.

15. The method of claim 1 wherein
the reference audio features are generated as the reference audio stream is segmented.

16. The method of claim 1 wherein
the reference and inclusion conditions correspond to at least one of: an audio property exceeding a threshold, the audio property falling below the threshold, or the audio property deviating from an average value by a threshold amount.

17. A system comprising:
a processor; and
a non-transitory computer readable storage medium configured to store instructions, the instructions executable by the processor and that when executed cause the processor to:
receive, at a real-time messaging platform comprising a plurality of accounts, a reference audio stream of a time-based media (TBM) event;
receive, at th

Assignees

• Twitter Inc

Inventors

• Carroll Andrew John
• Rishel Jeremy
• Whitcomb Jr Richard Douglas
• Watabe Mark
• Vihinen Noah
• Deshmukh Indranrita
• Adib Artur B
• Fleischman Michael Ben
• Roy Deb Kumar

Classifications

• H04H60/372 Primary

  Programme · CPC title

• H04H60/52

  of users · CPC title

• H04H60/56 Primary

  Arrangements characterised by components specially adapted for monitoring, identification or recognition covered by groups H04H60/29-H04H60/54 · CPC title

• H04H60/58

  of audio {(determination or detection of speech characteristics in general G10L25/00; speech recognition in general G10L15/00)} · CPC title