Content recognition and synchronization using local caching

Patent metadata

Field	Value
Publication number	US-9292894-B2
Application number	US-201313804314-A
Country	US
Kind code	B2
Filing date	Mar 14, 2013
Priority date	Mar 14, 2012
Publication date	Mar 22, 2016
Grant date	Mar 22, 2016

Abstract

A content fingerprint based recognition system employs local caching of portions of a fingerprint database to manage network services for identifying which programs a user's mobile device is exposed to and the timing of events within the program. The system enables background recognition and synchronization of network services in a way that consumes less device power and bandwidth.

First claim

We claim:

1. A method comprising:
from a processor within a device, querying a content recognition database to determine program identification;
obtaining through a network connection and caching locally in a memory of the device a local fingerprint database for a program associated with the program identification, the local fingerprint database including audio fingerprints for segments of the program and program times for associated audio fingerprints in the local fingerprint database;
with a processor operating on an electronic audio signal sensed from ambient audio by a microphone of the device, extracting audio features from incoming segments of the electronic audio signal, and forming fingerprints for the incoming segments from audio features extracted from corresponding segments;
with a processor, matching the fingerprints for the incoming segments with audio fingerprints for the program in the local fingerprint database in the memory of the device, and responsive to detecting matches between fingerprints for incoming segments and matching audio fingerprints in the local fingerprint database, obtaining from the local fingerprint database program times corresponding to the matching audio fingerprints; and
with a processor, tracking the program times relative to a time reference in the device to keep track of where a user is within the program as the device is exposed to audio of the program.

2. The method of claim 1 wherein the tracking comprises tracking the program times relative to elapsed time from a first program time corresponding to a first matching audio fingerprint.

3. The method of claim 2 including:
with a processor, synchronizing auxiliary services on the device with events in the program as determined from the elapsed time within the device.

4. The method of claim 1 wherein the matching is used to detect a discontinuity in a user's exposure to a program.

5. The method of claim 1 wherein the matching is used to detect time shifting, and with a processor, downloading supplemental material that is configured for time shifted mode, and synchronizing the supplemental material with program events based on the tracking.

6. The method of claim 4 wherein the discontinuity comprises a switch to a different program.

7. The method of claim 1 wherein the content recognition database comprises a fingerprint database of fingerprints having a first temporal granularity, wherein the local fingerprint database comprises fingerprints having a second temporal granularity, the second temporal granularity being higher than the first temporal granularity.

8. A device comprising:
a microphone;
a memory;
a radio for connecting to a network service through a cellular or wi-fi network;
one or more processors, programmed to:
query through the cellular or wi-fi network a content recognition database of the network service to determine program identification;
obtain through a network connection to the network service and cache locally in the memory of the device a local fingerprint database for a program associated with the program identification, the local fingerprint database including audio fingerprints for segments of the program and program times for associated audio fingerprints in the local fingerprint database;
operate on an electronic audio signal sensed from ambient audio by the microphone of the device to extract audio features from incoming segments of the electronic audio signal, and to form fingerprints for the incoming segments from audio features extracted from corresponding segments;
match the fingerprints for the incoming segments with audio fingerprints for the program in the local fingerprint database in the memory to determine matching audio fingerprints;
obtain from the local fingerprint database program times corresponding to the matching audio fingerprints; and
track the program times relative to a time reference in the device to keep track of where a user is within the program as the device is exposed to audio of the program.

9. The device of claim 8 including a processor configured to track the program times relative to elapsed time from a first program time corresponding to a first matching audio fingerprint.

10. The device of claim 9 including a processor configured to synchronize auxiliary services on the device with events in the program as determined from the elapsed time within the device.

11. The device of claim 9 including a processor configured to detect a discontinuity in a user's exposure to a program based on data associated with the matching audio fingerprints in the local fingerprint database.

12. The device of claim 11 wherein the discontinuity comprises a switch to a different program.

13. The device of claim 8 including a processor configured to detect time shifting from data associated with the matching audio fingerprints, configured to download supplemental material that is configured for time shifted mode, and configured to synchronize the supplemental material with program events based on the tracking.

14. The device of claim 8 wherein the content recognition database comprises a fingerprint database of fingerprints having a first temporal granularity, wherein the local fingerprint database comprises fingerprints having a second temporal granularity, the second temporal granularity being higher than the first temporal granularity.

Assignees

• Digimarc Corp

Inventors

• Macintosh Brian T
• Rodriguez Tony F
• Sharma Ravi K
• Weaver Matthew M

Classifications

• H04N21/4307

  Synchronising the rendering of multiple content streams or additional data on devices, e.g. synchronisation of audio on a mobile phone with the video output on the TV screen · CPC title

• H04N21/44008 Primary

  involving operations for analysing video streams, e.g. detecting features or characteristics in the video stream (arrangements characterised by components specially adapted for monitoring, identification or recognition of video in broadcast systems H04H60/59) · CPC title

• G06T1/0021 Primary

  Image watermarking · CPC title

• H04N21/4622

  Retrieving content or additional data from different sources, e.g. from a broadcast channel and the Internet (web site content organization and management for information retrieval from the Internet G06F16/958; transmission by internet of broadcast information H04H60/82; stock exchange data over packet-switching network H04L12/1804; push services including data channel over packet-switching network H04L12/1859) · CPC title

• H04N21/43072 Primary

  of multiple content streams on the same device · CPC title