Event image curation

Patent metadata

Field	Value
Publication number	US-9940544-B2
Application number	US-201615177197-A
Country	US
Kind code	B2
Filing date	Jun 8, 2016
Priority date	Jun 8, 2016
Publication date	Apr 10, 2018
Grant date	Apr 10, 2018

Abstract

In embodiments of event image curation, a computing device includes memory that stores a collection of digital images associated with a type of event, such as a digital photo album of digital photos associated with the event, or a video of image frames and the video is associated with the event. A curation application implements a convolutional neural network, which receives the digital images and a designation of the type of event. The convolutional neural network can then determine an importance rating of each digital image within the collection of the digital images based on the type of the event. The importance rating of a digital image is representative of an importance of the digital image to a person in context of the type of the event. The convolutional neural network generates an output of representative digital images from the collection based on the importance rating of each digital image.

First claim

The invention claimed is:

1. A method for event image curation, the method comprising:
receiving a collection of digital images as an input to a convolutional neural network, the digital images being associated with a type of event;
determining, using the convolutional neural network, an importance rating of each digital image within the collection of the digital images based on the type of the event, the importance rating of a digital image representative of an importance of the digital image to a person in context of the type of the event; and
generating an output of representative digital images from the collection based on the importance rating of each digital image.

2. The method as recited in claim 1 , wherein:
the collection of the digital images is a digital photo album of digital photos that are associated with the type of the event; and
the representative digital images are a set of the digital photos that are representative of important moments during the event.

3. The method as recited in claim 2 , further comprising:
determining a diversity of the set of the digital photos to identify one or more of the digital photos that represent the important moments during the event;
removing duplicate ones of the set of the digital photos based on the determined diversity of the set of the digital photos; and
adding another of the digital photos to the set of the digital photos for an important moment of the event that is not represented by the set of the digital photos.

4. The method as recited in claim 1 , wherein:
the collection of the digital images is a video of image frames, the video being associated with the type of the event; and
the representative digital images are a set of the image frames of the video that are representative of important moments during the event.

5. The method as recited in claim 1 , further comprising receiving a designation of the type of the event as an additional input to the convolutional neural network.

6. The method as recited in claim 1 , wherein:
the digital images are associated with different types of events;
the method further comprising receiving probability designations of the different types of the events; and
wherein said determining the importance rating of each digital image is based at least in part on the probability designations of the different types of the events.

7. The method as recited in claim 1 , further comprising:
receiving digital image metadata corresponding to each of the respective digital images as an additional input to the convolutional neural network, the digital image metadata that corresponds to a digital image indicating the importance of the digital image in the context of the type of the event; and
wherein said determining the importance rating of each digital image is based at least in part on the digital image metadata that corresponds to each of the respective digital images.

8. The method as recited in claim 1 , further comprising:
detecting one or more faces in each of the digital images that include at least one face;
generating a face heat map for each of the digital images that are detected having the at least one face, the face heat map of a digital image including representations of the one or more faces emphasized based on an importance of a person in the context of the type of the event;
receiving the face heat maps as additional input to the convolutional neural network; and
wherein said determining the importance rating of each digital image is based at least in part on the face heat map for each of the respective digital images.

9. The method as recited in claim 8 , further comprising:
receiving digital image metadata corresponding to each of the respective digital images as an additional input to the convolutional neural network, the digital image metadata that corresponds to a digital image designating the importance of the person in the digital image; and
wherein said determining the importance rating of each digital image is based at least in part on the digital image metadata corresponding to each of the respective digital images.

10. The method as recited in claim 8 , further comprising:
receiving a user input to one of emphasize the importance of the person in the digital image or deemphasize the importance of the person in the digital image; and
wherein said determining the importance rating of each digital image is based at least in part on the user input as it pertains to one or more of the digital images that include the person.

11. The method as recited in claim 1 , further comprising:
detecting physical features of one or more persons in one or more of the digital images;
generating representations of one or more of the physical features emphasized based on an importance of a person in the context of the type of the event;
receiving the representations of the one or more physical features as additional input to the convolutional neural network; and
wherein said determining the importance rating of each digital image is based at least in part on the representations of the one or more physical features for each of the respective digital images.

12. A computing device implemented for event image curation, the computing device comprising:
memory configured to maintain a collection of digital images associated with a type of event;
a curation application executed by a processor system, the curation application implementing a convolutional neural network configured to:
receive the digital images;
receive a designation of the type of the event;
determine an importance rating of each digital image within the collection of the digital images based on the type of the event, the importance rating of a digital image representative of an importance of the digital image to a person in context of the type of the event; and
generate an output of representative digital images from the collection based on the importance rating of each digital image.

13. The computing device as recited in claim 12 , wherein:
the collection of the digital images is a digital photo album of digital photos that are associated with the type of the event; and
the representative digital images are a set of the digital photos that are representative of important moments during the event.

14. The computing device as recited in claim 13 , wherein the curation application is configured to:
determine a diversity of the set of the digital photos to identify one or more of the digital photos that represent the important moments during the event;
remove duplicate ones of the set of the digital photos based on the determined diversity of the set of the digital photos; and
add another of the digital photos to the set of the digital photos for an important moment of the event that is not represented by the set of the digital photos.

15. The computing device as recited in claim 12 , wherein:
the collection of the digital images is a video of image frames, the video being associated with the type of the event; and
the representative digital images are a set of the image frames of the video that are representative of important moments during the event.

16. The computing device as recited in claim 12 , wherein:
the digital images are associated with different types of events; and
the convolutional neural network is configured to:
receive designations of the different types of the events;
receive probability designations of the different types of the events; and
said determine the importance rating of each digital image based at least in part on the probability designations of the

Assignees

• Adobe Systems Inc

Inventors

• Lin Zhe
• Wang Yufei
• Mech Radomir
• Shen Xiaohui
• Miller Gavin Stuart Peter

Classifications

• G06V10/82 Primary

  using neural networks · CPC title

• G06F18/23

  Clustering techniques · CPC title

• G06F16/583

  using metadata automatically derived from the content · CPC title

• G06F16/58

  Retrieval characterised by using metadata, e.g. metadata not derived from the content or metadata generated manually · CPC title

• G06N3/084

  Backpropagation, e.g. using gradient descent · CPC title