Using audio cues to improve object retrieval in video

Patent metadata

Field	Value
Publication number	US-10108617-B2
Application number	US-201314067923-A
Country	US
Kind code	B2
Filing date	Oct 30, 2013
Priority date	Oct 30, 2013
Publication date	Oct 23, 2018
Grant date	Oct 23, 2018

Abstract

A method of object retrieval from visual data is provided that includes annotating at least one portion of the visual data with a context keyword corresponding to an object, wherein the annotating is performed responsive to recognition of the context keyword in audio data corresponding to the at least one portion of the visual data, receiving a query to retrieve the object, wherein the query includes a query keyword associated with both the object and the context keyword, identifying the at least one portion of the visual data based on the context keyword, and searching for the object in the at least one portion of the visual data using an appearance model corresponding to the query keyword.

First claim

What is claimed is:

1. A method of object retrieval from visual data, the method comprising:
annotating at least one portion of the visual data with a context keyword corresponding to an object, wherein the annotating is performed responsive to recognition of the context keyword by performing speech recognition in audio data corresponding to the at least one portion of the visual data;
receiving a query to retrieve the object, wherein the query comprises a query keyword associated with both the object and the context keyword;
identifying the at least one portion of the visual data based on the context keyword; and
searching for the object in the at least one portion of the visual data using an appearance model corresponding to the query keyword.

2. The method of claim 1 , wherein the appearance model, the corresponding query keyword, and the context keyword are predetermined.

3. A digital system configured to perform object retrieval from visual data, the digital system comprising one or more processors configured to:
capture the visual data and corresponding audio data;
annotate at least one portion of the visual data with a context keyword that is obtained from audio data corresponding to the at least one portion of the visual data;
receive a query to retrieve an object, wherein the query comprises a query keyword associated with both the object and the context keyword;
identify the at least one portion of the visual data based on the context keyword; and
search for the object in the at least one portion of the visual data using an appearance model corresponding to the query keyword.

4. The digital system of claim 3 , wherein the appearance model, the corresponding query keyword, and the context keyword are predetermined.

5. The digital system of claim 3 , wherein the context keyword is obtained by speech recognition from the audio data corresponding to the at least one portion of the visual data.

Assignees

• Texas Instruments Inc

Inventors

• Jain Eakta
• Barnum Peter Charles

Classifications

• G06F16/43 Primary

  Querying · CPC title

• G06F16/7834 Primary

  using audio features · CPC title

• G06F17/30787

  Physics · mapped topic

• G06F17/30023 Primary

  Physics · mapped topic

• H04N5/77 Primary

  between a recording apparatus and a television camera · CPC title