Joint photo stream and blog post summarization and exploration

Patent metadata

Field	Value
Publication number	US-9928442-B2
Application number	US-201615076803-A
Country	US
Kind code	B2
Filing date	Mar 22, 2016
Priority date	Mar 22, 2016
Publication date	Mar 27, 2018
Grant date	Mar 27, 2018

Abstract

Systems, methods, and computer program products to perform an operation comprising assigning each of a plurality of images in a blog post and each of a plurality of images in a collection of images to a respective node in a graph, computing an adjacency matrix for the graph, wherein the adjacency matrix defines relationships between images in the blog post and images in the collections of images, and determining a first subset of the images in the collection of images that summarize the images in the image collection, wherein the subset is determined based on the adjacency matrix, wherein the adjacency matrix is computed based on the subset of the images in the collection of images.

First claim

What is claimed is:

1. A method, comprising:
assigning each of a plurality of images in a blog post and each of a plurality of images in a collection of images to a respective node in a graph; computing an adjacency matrix for the graph, wherein the adjacency matrix defines relationships between images in the blog post and images in the collections of images; and determining a first subset of the images in the collection of images that summarize the images in the image collection, wherein the first subset is determined based on the adjacency matrix and the adjacency matrix is computed based on the first subset of the images in the collection of images.

2. The method of claim 1 , further comprising:
determining a location where a first image in the blog post was taken based on a text of the blog post associated with the first image in the blog post, wherein the adjacency matrix associates the first image of the blog post with a first image in the collection of images; and
assigning the location of the first image in the blog post as a location of the first image.

3. The method of claim 2 , further comprising:
extracting a first phrase from the text of the blog post associated with the first image in the blog post; and
assigning the first phrase as a title of the first image.

4. The method of claim 1 , further comprising:
determining a second subset of images in the collection of images based on the adjacency matrix and the first subset of images; and
returning the second subset of images as an interpolation between a first image in the blog post and a second image in the blog post.

5. The method of claim 4 , wherein the collection of images comprises a plurality of collections of images from different users, wherein the second subset of images is from a first collection of images of the plurality of collections of images.

6. The method of claim 1 , wherein the adjacency matrix and the first subset are based on an initial set of images of the collection of images, wherein the initial set is determined by applying K-means clustering on a set of image descriptors of the collection of images, wherein the initial set of images in the collection of images is provided as input to a first support vector machine (SVM) configured to compute the adjacency matrix.

7. The method of claim 6 , wherein the first SVM is configured to compute a first adjacency matrix based on the initial set of images in the collection of images, wherein a second SVM is configured to determine the first set of images in the collection of images based on the first adjacency matrix, wherein the first SVM is configured to operate on the output of the second SVM, wherein the second SVM is configured to operate on the output of the first SVM, wherein the first and second SVMs iteratively execute until the adjacency matrix and the first subset of images in the collection of images converge.

8. The method of claim 1 , wherein the first subset of images in the collection of images is further determined based at least in part on a number of relationships between the plurality of images in the blog post and each image in the collection of images.

9. A computer program product, comprising: a non-transitory computer-readable storage medium having computer-readable program code embodied therewith, the computer-readable program code executable by a processor to perform an operation comprising: assigning each of a plurality of images in a blog post and each of a plurality of images in a collection of images to a respective node in a graph; computing an adjacency matrix for the graph, wherein the adjacency matrix defines relationships between images in the blog post and images in the collections of images; and determining a first subset of the images in the collection of images that summarize the images in the image collection, wherein the first subset is determined based on the adjacency matrix, wherein the adjacency matrix is computed based on the first subset of the images in the collection of images.

10. The computer program product of claim 9 , the operation further comprising:
determining a location where a first image in the blog post was taken based on a text of the blog post associated with the first image in the blog post, wherein the adjacency matrix associates the first image of the blog post with a first image in the collection of images; and
assigning the location of the first image in the blog post as a location of the first image.

11. The computer program product of claim 10 , the operation further comprising:
extracting a first phrase from the text of the blog post associated with the first image in the blog post; and
assigning the first phrase as a title of the first image.

12. The computer program product of claim 9 , the operation further comprising:
determining a second subset of images in the collection of images based on the adjacency matrix and the first subset of images; and
returning the second subset of images as an interpolation between a first image in the blog post and a second image in the blog post.

13. The computer program product of claim 12 , wherein the collection of images comprises a plurality of collections of images from different users, wherein the second subset of images is from a first collection of images of the plurality of collections of images.

14. The computer program product of claim 9 , wherein the adjacency matrix and the first subset are based on an initial set of images of the collection of images, wherein the initial set is determined by applying K-means clustering on a set of image descriptors of the collection of images, wherein the initial set of images in the collection of images is provided as input to a first support vector machine (SVM) configured to compute the adjacency matrix.

15. The computer program product of claim 14 , wherein the first SVM is configured to compute a first adjacency matrix based on the initial set of images in the collection of images, wherein a second SVM is configured to determine the first set of images in the collection of images based on the first adjacency matrix, wherein the first SVM is configured to operate on the output of the second SVM, wherein the second SVM is configured to operate on the output of the first SVM, wherein the first and second SVMs iteratively execute until the adjacency matrix and the first subset of images in the collection of images converge.

16. The computer program product of claim 9 , wherein the first subset of images in the collection of images is further determined based at least in part on a number of relationships between the plurality of images in the blog post and each image in the collection of images.

17. A system, comprising: a computer processor; and a memory containing a program which when executed by the processor performs an operation comprising: assigning each of a plurality of images in a blog post and each of a plurality of images in a collection of images to a respective node in a graph; computing an adjacency matrix for the graph, wherein the adjacency matrix defines relationships between images in the blog post and images in the collections of images; and determining a first subset of the images in the collection of images that summarize the images in the image collection, wherein the first subset is determined based on the adjacency matrix, wherein the adjacency matrix is computed based on the first subset of the images in the collection of images.

18. The system of claim 17 , the operation further comprising:
determining a location where a first image in the blog p

Assignees

• Disney Entpr Inc

Inventors

• Sigal Leonid
• Kim Gunhee
• Moon Seungwhan

Classifications

• G06V20/70 Primary

  Labelling scene content, e.g. deriving syntactic or semantic representations · CPC title

• G06V10/7635

  based on graphs, e.g. graph cuts or spectral clustering · CPC title

• G06F18/2323

  based on graph theory, e.g. minimum spanning trees [MST] or graph cuts · CPC title

• G06F18/23

  Clustering techniques · CPC title

• G06K9/52 Primary

  Physics · mapped topic