Automatic generation of events using a machine-learning model

Patent metadata

Field	Value
Publication number	US-12056929-B2
Application number	US-202117404773-A
Country	US
Kind code	B2
Filing date	Aug 17, 2021
Priority date	May 11, 2021
Publication date	Aug 6, 2024
Grant date	Aug 6, 2024

Abstract

A media application segments a library of media associated with a user account into episodes, wherein each episode is associated with a corresponding time period. The media application generates, using an event machine-learning model, an event signal that indicates a likelihood that an event occurred in each episode, wherein the event machine-learning model is a classifier that receives the media as input. The media application generates an event significance score for each episode. The media application determines one or more events from the episodes based on the event signal and a corresponding event significance score exceeding a threshold event significance value. The media application provides a user interface that includes corresponding media from the one or more events.

First claim

What is claimed is:

1. A computer-implemented method comprising:
segmenting a library of media associated with a user account into episodes, wherein each episode is associated with a corresponding time period;
generating, using an event machine-learning model, an event signal that indicates a likelihood that an event occurred in each episode, wherein the event machine-learning model is a classifier that receives the media as input;
generating an event significance score for each episode;
determining one or more events from the episodes based on the event signal and a corresponding event significance score exceeding a threshold event significance value;
providing a user interface that includes an event item with corresponding media from a particular event of the one or more events;
generating a confidence score that indicates a likelihood that a corresponding event is a type of event that is accurately recognized; and
responsive to the confidence score meeting a threshold confidence value, adding an automatically generated title descriptive of the type of event to the corresponding event.

2. The method of claim 1 , wherein the user interface is part of a reverse-chronological grid of media that includes the corresponding media, and further comprising:
responsive to a user selecting the event item from the reverse-chronological grid, replacing the reverse-chronological grid with a display of the corresponding media from the particular event for a predetermined time duration; and
responsive to completing the display of the corresponding media, displaying the reverse-chronological grid.

3. The method of claim 1 , wherein the event item is displayed with a size that is based on one or more of the event significance score, a number of media items for the particular event, a total number of events in a period of time, or an event type.

4. The method of claim 1 , further comprising:
receiving a request from a user to remove depictions of a person from the media in the library of media that include depictions of the person; and
filtering the depictions of the person from the media, wherein the filtering is performed before the event significance score is generated.

5. The method of claim 1 , wherein the user interface includes an option to hide or an option to add one or more of a media item from the one or more events, a date associated with the one or more events, or a person or pet depicted in media items from the one or more events.

6. The method of claim 1 , wherein the user interface includes an option to hide the event item.

7. The method of claim 1 , wherein generating the event significance score is based on one or more of at least a threshold number of media items in a corresponding episode, at least a threshold number of face clusters in the corresponding episode, a quality indicator of the media items in the corresponding episode, at least one face cluster of a threshold rank, or a presence of a rare face cluster.

8. The method of claim 1 , further comprising combining multiple episodes into a single event based on one or more of the multiple episodes being associate with respective time periods that all fall within a 24-hour period, the single event being a type of event that occurs over multiple days, celebrations on different days that relate to the single event, the multiple episodes involving a same location, or the multiple episodes involving a same set of face clusters.

9. The method of claim 1 , further comprising:
responsive to the confidence score failing to meet the threshold confidence value, adding a title to the corresponding event based on a template phrase.

10. The method of claim 1 , wherein a title machine-learning model receives the corresponding media from the one or more events as input and the title machine-learning model generates a title as output.

11. The method of claim 1 , wherein the user interface includes an option to edit the corresponding media from the particular event.

12. The method of claim 1 , wherein determining the one or more events comprises determining events such that a number of the events is less than or equal to a predetermined number each month and further comprising:
receiving new media to associate with the library of media; and
replacing the particular event with a new event, responsive to the new event being associated with a new event significance score that is higher than the event significance score for the particular event.

13. The method of claim 1 , wherein the user interface includes an option to change a title of the event item.

14. The method of claim 1 , further comprising generating audio for the corresponding media that is based on a type of the particular event.

15. A computing device comprising:
a processor; and
a memory coupled to the processor, with instructions stored thereon that, when executed by the processor, cause the processor to perform operations comprising:
segmenting a library of media associated with a user account into episodes, wherein each episode is associated with a corresponding time period;
generating, using an event machine-learning model, an event signal that indicates a likelihood that an event occurred in each episode, wherein the event machine-learning model is a classifier that receives the media as input;
generating an event significance score for each episode;
determining one or more events from the episodes based on the event signal and a corresponding event significance score exceeding a threshold event significance value;
providing a user interface that includes an event item with corresponding media from a particular event of the one or more events, wherein the user interface is part of a reverse-chronological grid of media that includes the corresponding media;
responsive to a user selecting the event item from the reverse-chronological grid, replacing the reverse-chronological grid with a display of the corresponding media from the particular event for a predetermined time duration; and
responsive to completing the display of the corresponding media, displaying the reverse-chronological grid.

16. The computing device of claim 15 , wherein the operations further comprise:
receiving a request from a user to remove depictions of a person from the media in the library of media that include depictions of the person; and
filtering the depictions of the person from the media, wherein the filtering is performed before the event significance score is generated.

17. The computing device of claim 15 , wherein the event item is displayed with a size that is based on one or more of the event significance score, a number of media items for the particular event, a total number of events in a period of time, or an event type.

18. A non-transitory computer-readable medium with instructions stored thereon that, when executed by one or more computers, cause the one or more computers to perform operations, the operations comprising:
segmenting a library of media associated with a user account into episodes, wherein each episode is associated with a corresponding time period;
generating, using an event machine-learning model, an event signal that indicates a likelihood that an event occurred in each episode, wherein the event machine-learning model is a classifier that receives the media as input;
generating an event significance score for each episode;
determining events from the episodes based on the event signal and corresponding event significance scores exceeding a threshold event significance value such that a number of the events is le

Assignees

• Google Llc

Inventors

• Bohl Kristina
• Agajanian Joe
• Berg Lily
• Potetz Brian
• Mosley Keegan
• Cheng Shinko

Classifications

• G06V20/30 Primary

  in albums, collections or shared content, e.g. social network photos or video · CPC title

• G06F18/2178

  based on feedback of a supervisor · CPC title

• G06V20/44 Primary

  Event detection · CPC title

• G06F18/2193

  based on specific statistical tests · CPC title

• G06F18/2148

  characterised by the process organisation or structure, e.g. boosting cascade · CPC title