Associating still images and videos

Patent metadata

Field	Value
Publication number	US-10108620-B2
Application number	US-201113098362-A
Country	US
Kind code	B2
Filing date	Apr 29, 2011
Priority date	Apr 29, 2010
Publication date	Oct 23, 2018
Grant date	Oct 23, 2018

Abstract

Methods, systems, and apparatus, including computer programs encoded on computer storage media, for associating still images and videos. One method includes receiving a plurality of images and a plurality of videos and determining whether the images are related to the videos. The determining includes, for an image and a video, extracting features from the image and extracting features frames of the video, and comparing the features to determine whether the image is related to the video. The method further includes maintaining a data store storing data associating each image with each video determined to be related to the image.

First claim

What is claimed is:

1. A system, comprising:
one or more computers including one or more storage devices storing instructions that when executed by the one or more computers cause the one or more computers to perform operations comprising:
receiving a digital image and a digital video;
extracting one or more features from the digital image;
identifying one or more representative scenes in the digital video;
selecting, for each representative scene, a representative frame from a portion of the digital video that corresponds to the representative scene;
extracting one or more respective features from each representative frame;
comparing the one or more features extracted from the digital image to the one or more respective features extracted from each representative frame from the portion of the digital video that corresponds to each representative scene; and
classifying the digital image as related to the digital video based on the comparison of the one or more features extracted from the image to the one or more respective features extracted from each representative frame from the portion of the digital video that corresponds to each representative scene.

2. The system of claim 1 , wherein the digital image is a digital still image.

3. The system of claim 1 , wherein classifying the digital image as related to the digital video comprises:
determining a strength of relationship between the digital image and the digital video based on the comparison of the one or more features extracted from the image to the one or more respective features extracted from each representative frame from the portion of the digital video that corresponds to each representative scene; and
classifying the digital image as related to the digital video if the strength of relationship satisfies a threshold.

4. The system of claim 3 , wherein the strength of relationship between the digital image and the digital video is an estimate of visual similarity between the digital image and the digital video.

5. The system of claim 1 , wherein the operations comprise determining a category for each of the digital image and the digital video; and
wherein extracting one or more features from the digital image, and extracting one or more respective features from each representative frame, respectively, comprise extracting features based on the determined category of the digital image or the determined category of the digital video.

6. The system of claim 5 , wherein determining the category for the digital image comprises determining a category from text associated with the digital image, and determining the category for the digital video comprises determining a category from text associated with the digital video.

7. The system of claim 6 , wherein the text associated with the digital image comprises query text associated with the digital image and the text associated with the digital video comprises query text associated with the digital video.

8. The system of claim 1 , wherein the operations comprise:
storing, at a data store, association data associating the digital image with the digital video.

9. The system of claim 8 , wherein the data store stores association data associating one or more other digital images with the digital video.

10. The system of claim 9 , wherein the operations comprise:
determining that the digital image and another digital image are both associated with the digital video by the association data stored at the data store; and
storing, at the data store, data associating the digital image and the other digital image in response to the determination.

11. The system of claim 9 , wherein the digital image has associated given metadata, and wherein the operations comprise:
associating, by the association data stored at the data store, the given metadata with the one or more other digital images associated with the digital video.

12. The system of claim 1 , wherein identifying one or more representative scenes in the digital video comprises:
merging a plurality of shots in the digital video into one or more candidate scenes based on a pairwise similarity between shots, the similarity being determined based on a recursive formula that evaluates a first and second shot using respective lengths of the first and second shots and respective frame sequences of the first and second shots; and
determining, from the one or more candidate scenes, one or more representative scenes, wherein each representative scene is distinct from other representative scenes.

13. A non-transitory computer-readable storage medium encoded with a computer program, the computer program comprising instructions, that when executed by data processing apparatus, cause the data processing apparatus to perform operations comprising:
receiving a digital image and a digital video;
extracting one or more features from the digital image;
identifying one or more representative scenes in the digital video;
selecting, for each representative scene, a representative frame from a portion of the digital video that corresponds to the representative scene;
extracting one or more respective features from each representative frame;
comparing the one or more features extracted from the digital image to the one or more respective features extracted from each representative frame from the portion of the digital video that corresponds to each representative scene; and
classifying the digital image as related to the digital video based on the comparison of the one or more features extracted from the image to the one or more respective features extracted from each representative frame from the portion of the digital video that corresponds to each representative scene.

14. The computer-readable storage medium of claim 13 , wherein the digital image is a digital still image.

15. The computer-readable storage medium of claim 13 , wherein classifying the digital image as related to the digital video comprises:
determining a strength of relationship between the digital image and the digital video based on the comparison of the one or more features extracted from the image to the one or more respective features extracted from each representative frame from the portion of the digital video that corresponds to each representative scene; and
classifying the digital image as related to the digital video if the strength of relationship satisfies a threshold.

16. The computer-readable storage medium of claim 15 , wherein the strength of relationship between the digital image and the digital video is an estimate of visual similarity between the digital image and the digital video.

17. The computer-readable storage medium of claim 13 , wherein the operations comprise determining a category for each of the digital image and the digital video; and
wherein extracting one or more features from the digital image, and extracting one or more respective features from each representative frame, respectively, comprise extracting features based on the determined category of the digital image or the determined category of the digital video.

18. The computer-readable storage medium of claim 17 , wherein determining the category for the digital image comprises determining a category from text associated with the digital image, and determining the category for the digital video comprises determining a category from text associated with the digital video.

19. The computer-readable storage medium of claim 18 , wherein the text associated with the digital image comprises query text associated with the di

Assignees

• Zhao Ming
• Song Yang
• Adam Hartwig
• Gargi Ullas
• Jing Yushi
• Rowley Henry A
• Google Llc

Inventors

• Zhao Ming
• Song Yang
• Adam Hartwig
• Gargi Ullas
• Jing Yushi
• Rowley Henry A

Classifications

• G06V20/47 Primary

  Detecting features for summarising video content · CPC title

• G06F16/438

  Presentation of query results · CPC title

• G06F18/253

  of extracted features · CPC title

• G06F16/7837

  using objects detected or recognised in the video content · CPC title

• G06F16/41

  Indexing; Data structures therefor; Storage structures · CPC title