Generating smart reminders by assistant systems

Patent metadata

Field	Value
Publication number	US-11636438-B1
Application number	US-201916659070-A
Country	US
Kind code	B1
Filing date	Oct 21, 2019
Priority date	Oct 18, 2019
Publication date	Apr 25, 2023
Grant date	Apr 25, 2023

Abstract

In one embodiment, a method includes receiving a user request including an ambiguous mention to create a reminder from a client system associated with a user, disambiguating the mention to identify a first entity referenced in the mention, wherein the first entity is identified based on user profile data associated with the user, determining an activation condition associated with the user request, wherein the activation condition is based on one or more of a time or a location referenced in the user request, wherein the time and/or location are determined based on contextual information associated with the user request, generating the reminder based on the first entity and the activation condition, and sending instructions to the client system for presenting the reminder when the activation condition is satisfied.

First claim

What is claimed is:

1. A method comprising, by a mobile client system associated with a user:
receiving, at the mobile client system, initial sensory data captured by one or more cameras of the mobile client system, wherein the initial sensory data is visual data;
proactively generating, by the mobile client system responsive to proactively identifying a first entity based on a visual analysis of the visual data and correlating the first entity with knowledge about the user, a reminder associated with the first entity for the user, wherein the knowledge about the user is obtained by analyzing one or more of a routine of the user related to the first entity or an episodic memory of the user referencing the first entity;
determining, by the mobile client system, an activation condition associated with the reminder, wherein the activation condition is based on one or more of a time or a location, wherein the time and/or location are determined based on the analysis of the visual data and the knowledge about the user;
storing, at a data store of the mobile client system, an intent associated with the reminder in an intent stack on the data store based on a priority of the intent;
determining, by the mobile client system, the priority of the intent meets a criteria;
determining, by the mobile client system responsive to determining the priority of the intent meets the criteria, whether the activation condition is satisfied; and
presenting, at the mobile client system, the reminder when the activation condition is satisfied, wherein the reminder comprises an activatable prompt for a user to execute a task referenced in the reminder.

2. The method of claim 1 , wherein the activation condition is based on a location, the method further comprising:
applying one or more scene recognition algorithms to current sensory data captured by the mobile client system;
recognizing that the user is at the location; and
determining the activation condition is satisfied.

3. The method of claim 2 , wherein the current sensory data comprises one or more of an image or a video clip captured by the one or more cameras.

4. The method of claim 1 , wherein the first entity is identified further based on a social graph comprising a plurality of nodes and a plurality of edges connecting the nodes, wherein the plurality of nodes comprise a node corresponding to the user and a node corresponding to the first entity.

5. The method of claim 1 , wherein the first entity is identified further based on prior user requests by the user.

6. The method of claim 1 , wherein the initial sensory data comprises one or more of an image or a video clip, wherein the initial sensory data is based on a field of view of the one or more cameras, and wherein the visual analysis of the visual data is based on one or more machine-learning algorithms.

7. The method of claim 6 , wherein the one or more machine-learning algorithms are based on one or more of facial recognition, gait recognition, or object recognition.

8. The method of claim 1 , wherein determining the activation condition is further based on one or more of routines of the user or episodic memories of the user.

9. The method of claim 1 , further comprising:
determining contextual information associated with the initial sensory data; and
accessing a plurality of episodic memories of the user;
wherein the reminder comprises one or more references to one or more other users, respectively, the referenced users being based on the contextual information and one or more of the accessed episodic memories of the user.

10. The method of claim 9 , wherein the reminder further comprises information associated with the referenced users from the one or more of the accessed episodic memories of the user.

11. The method of claim 9 , further comprising:
retrieving one or more content objects associated with the one or more of the accessed episodic memories of the user, wherein each content object comprises one or more of a post, a comment, an image, or a video clip;
wherein the reminder further comprises one or more of the retrieved content objects.

12. The method of claim 1 , wherein the reminder is generated based on one or more reminder-templates.

13. The method of claim 1 , wherein the mobile client system is associated with an assistant system, and wherein the reminder is generated by a response-execution module of the assistant system.

14. The method of claim 1 , wherein the reminder comprises a social summary associated with the first entity.

15. The method of claim 1 , wherein the reminder comprises a social recommendation referencing one or more other users.

16. One or more computer-readable non-transitory storage media embodying software that is operable when executed to:
receive, at the mobile client system, initial sensory data captured by one or more cameras of the mobile client system, wherein the initial sensory data is visual data;
proactively generate, by the mobile client system responsive to proactively identifying a first entity based on a visual analysis of the visual data and correlating the first entity with knowledge about the user, a reminder associated with the first entity for the user, wherein the knowledge about the user is obtained by analyzing one or more of a routine of the user related to the first entity or an episodic memory of the user referencing the first entity;
determine, by the mobile client system, an activation condition associated with the reminder, wherein the activation condition is based on one or more of a time or a location, wherein the time and/or location are determined based on the analysis of the visual data and the knowledge about the user;
store, at a data store of the mobile client system, an intent associated with the reminder in an intent stack on the data store based on a priority of the intent;
determine, by the mobile client system, the priority of the intent meets a criteria;
determine, by the mobile client system responsive to determining the priority of the intent meets the criteria, whether the activation condition is satisfied; and
present, at the mobile client system, the reminder when the activation condition is satisfied, wherein the reminder comprises an activatable prompt for a user to execute a task referenced in the reminder.

17. The media of claim 16 , wherein the first entity is identified further based on a social graph comprising a plurality of nodes and a plurality of edges connecting the nodes, wherein the plurality of nodes comprise a node corresponding to the user and a node corresponding to the first entity.

18. The media of claim 16 , wherein determining the activation condition is further based on one or more of routines of the user or episodic memories of the user.

19. A system comprising: one or more processors; and a non-transitory memory coupled to the processors comprising instructions executable by the processors, the processors operable when executing the instructions to:
receive, at the mobile client system, initial sensory data captured by one or more cameras of the mobile client system, wherein the initial sensory data is visual data;
proactively generate, by the mobile client system responsive to proactively identifying a first entity based on a visual analysis of the visual data and correlating the first entity with knowledge about the user, a reminder associated with the first entity for the user, wherein the knowledge about the user is obtained by analyzing one or more of a routine of the user related to the first entity or an e

Assignees

• Meta Platforms Tech Llc

Inventors

• Huang Lisa Xiaoyi
• Xiao Eric
• Penov Francislav P

Classifications

• G06Q10/40 Primary

  Business processes related to social networking or social networking services · CPC title

• G06N3/09

  Supervised learning · CPC title

• G06N3/098

  Distributed learning, e.g. federated learning · CPC title

• G06Q10/1093

  Calendar-based scheduling for persons or groups · CPC title

• G06V40/174

  Facial expression recognition · CPC title