Video indexing with viewer reaction estimation and visual cue detection

Patent metadata

Field	Value
Publication number	US-9247225-B2
Application number	US-201213626161-A
Country	US
Kind code	B2
Filing date	Sep 25, 2012
Priority date	Sep 25, 2012
Publication date	Jan 26, 2016
Grant date	Jan 26, 2016

Abstract

Generally, this disclosure provides methods and systems for video indexing systems with viewer reaction estimation based on visual cue detection. The method may include detecting visual cues generated by a user, the visual cues generated in response to the user viewing the video; mapping the visual cues to an emotion space associated with the user; estimating emotion events of the user based on the mapping; and indexing the video with metadata, the metadata comprising the estimated emotion events and timing data associated with the estimated emotion events. The method may further include summarization, partitioning and searching of videos based on the video index.

First claim

What is claimed is:

1. A system for video indexing, said system comprising:
a visual cue detection module configured to detect visual cues generated by a user, said visual cues generated in response to said user viewing said video;
an emotion estimation module configured to map said visual cues to an emotion space having at least two dimensions, each of said at least two dimensions representing a different emotional state property, wherein said emotion space is associated with said user and said emotion estimation module estimates emotion events of said user based on an existence of at least one grouping of said visual cues on said mapping, wherein each distinct grouping of said visual cues represents a distinct estimated emotional event; and
a video indexing module configured to index said video with metadata, said metadata comprising said estimated emotion events and timing data associated with said estimated emotion events.

2. The system of claim 1 , wherein said video indexing module is further configured to identify video frame time-stamps associated with said emotion events, said identifying based on said timing data.

3. The system of claim 1 , further comprising a video summarization module configured to extract frames of said video based on a density of emotion events in said extracted frames exceeding a threshold, wherein said density is determined from said indexing.

4. The system of claim 1 , further comprising a video partitioning module configured to segment said video at frame locations based on a density of emotion events in said frames falling below a threshold, wherein said density is determined from said indexing.

5. The system of claim 1 , further comprising an intra-video search module configured to search for frames in said video associated with requested emotion events, said searching based on said indexing.

6. The system of claim 1 , further comprising a user profile generation module configured to create and maintain a database of user profiles, said user profiles comprising said emotion spaces associated with said user and one or more other users.

7. The system of claim 1 , further comprising an indexed video database configured to store said indexed videos comprising metadata associated with said user and said one or more other users.

8. The system of claim 7 , further comprising an inter-video search module configured to search for videos associated with requested emotion events from said user, said searching based on said indexed video database.

9. A method for video indexing, said method comprising:
detecting visual cues generated by a user, said visual cues generated in response to said user viewing said video;
mapping said visual cues to an emotion space having at least two dimensions, each of said at least two dimensions representing a different emotional state property, wherein said emotion space is associated with said user;
estimating emotion events of said user based on an existence of at least one grouping of said visual cues on said mapping, wherein each distinct grouping of said visual cues represents a distinct estimated emotional event; and
indexing said video with metadata, said metadata comprising said estimated emotion events and timing data associated with said estimated emotion events.

10. The method of claim 9 , further comprising identifying video frame time-stamps associated with said emotion events, said identifying based on said timing data.

11. The method of claim 9 , further comprising extracting frames of said video to generate a summary of said video, said extracting based on a density of emotion events in said frames exceeding a threshold, wherein said density is determined from said indexing.

12. The method of claim 9 , further comprising partitioning said video at frame locations based on a density of emotion events in said frames falling below a threshold, wherein said density is determined from said indexing.

13. The method of claim 9 , further comprising searching for frames in said video associated with requested emotion events, said searching based on said indexing.

14. The method of claim 9 , further comprising estimating a genre of said video based on frequency, duration and types of said emotion events.

15. The method of claim 9 , further comprising:
maintaining a database of user profiles, said user profiles comprising said emotion spaces associated with said user and one or more other users; and
maintaining a database of said indexed videos comprising metadata associated with said user and said one or more other users.

16. The method of claim 15 , further comprising searching for videos associated with requested emotion events from said user, said searching based on said database of indexed videos.

17. The method of claim 15 , further comprising recommending videos for said user based on comparisons between:
said emotion space associated with said user;
said emotion space associated with said other users in said user profile database; and
said metadata in said indexed video database.

18. One or more non-transitory computer-readable storage memories having instructions stored thereon which when executed by a processor result in the following operations for video indexing, said operations comprising:
detecting visual cues generated by a user, said visual cues generated in response to said user viewing said video;
mapping said visual cues to an emotion space having at least two dimensions, each of said at least two dimensions representing a different emotional state property, wherein said emotion space is associated with said user;
estimating emotion events of said user based on an existence of at least one grouping of said visual cues on said mapping, wherein each distinct grouping of said visual represents a distinct estimated emotional event; and
indexing said video with metadata, said metadata comprising said estimated emotion events and timing data associated with said estimated emotion events.

19. The one or more computer-readable storage memories of claim 18 , further comprising the operation of identifying video frame time-stamps associated with said emotion events, said identifying based on said timing data.

20. The one or more computer-readable storage memories of claim 18 , further comprising the operation of extracting frames of said video to generate a summary of said video, said extracting based on a density of emotion events in said frames exceeding a threshold, wherein said density is determined from said indexing.

21. The one or more computer-readable storage memories of claim 18 , further comprising the operation of partitioning said video at frame locations based on a density of emotion events in said frames falling below a threshold, wherein said density is determined from said indexing.

22. The one or more computer-readable storage memories of claim 18 , further comprising the operation of searching for frames in said video associated with requested emotion events, said searching based on said indexing.

23. The one or more computer-readable storage memories of claim 18 , further comprising the operation of estimating a genre of said video based on frequency, duration and types of said emotion events.

24. The one or more computer-readable storage memories of claim 18 , further comprising the operations of:
maintaining a database of user profiles, said user profiles comprising said emotion spaces associated with said user and one or more other

Assignees

• Intel Corp

Inventors

• Yehezkel Raanan
• Stanhill David

Classifications

• G06F16/78

  Retrieval characterised by using metadata, e.g. metadata not derived from the content or metadata generated manually · CPC title

• G06F17/30817

  Physics · mapped topic

• H04N9/8205 Primary

  involving the multiplexing of an additional signal and the colour video signal · CPC title

• H04N21/25891

  being end-user preferences (retrieval of video data in a video database based on user preferences G06F16/739; arrangements for recognizing users' preferences H04H60/46; user profiles in network data switching protocols H04L67/306; processing of user preferences or user profiles in wireless networks H04W8/18) · CPC title

• H04N21/8549

  Creating video summaries, e.g. movie trailer {(retrieval in video databases by using presentations in form of a video summary G06F16/739)} · CPC title