Generating Smart Reminders by Assistant Systems

Patent metadata

Field	Value
Publication number	US-2023214781-A1
Application number	US-202318182070-A
Country	US
Kind code	A1
Filing date	Mar 10, 2023
Priority date	Oct 18, 2019
Publication date	Jul 6, 2023
Grant date	—

Abstract

In one embodiment, a method includes receiving initial sensory data of visual data captured by cameras of a head-mounted client device at the head-mounted client device, proactively generating a reminder associated with a first entity responsive to proactively identifying the first entity based on a visual analysis of the visual data and correlating the first entity with knowledge about the user, wherein the knowledge about the user comprises one or more of a routine of the user related to the first entity or an episodic memory of the user referencing the first entity, determining an activation condition associated with the reminder, which is based on one or more of a time or a location, wherein the time and/or location are determined based on the analysis of the visual data and the knowledge about the user, and presenting the reminder when the activation condition is satisfied at the head-mounted client device.

First claim

What is claimed is:

1 . A method comprising, by a head-mounted client device associated with a user:
receiving, at the head-mounted client device, initial sensory data captured by one or more cameras of the head-mounted client device, wherein the initial sensory data is visual data; proactively generating, by the head-mounted client device responsive to proactively identifying a first entity based on a visual analysis of the visual data and correlating the first entity with knowledge about the user, a reminder associated with the first entity for the user, wherein the knowledge about the user comprises one or more of a routine of the user related to the first entity or an episodic memory of the user referencing the first entity; determining, by the head-mounted client device, an activation condition associated with the reminder, wherein the activation condition is based on one or more of a time or a location, wherein the time and/or location are determined based on the analysis of the visual data and the knowledge about the user; and presenting, at the head-mounted client device, the reminder when the activation condition is satisfied.

2 . The method of claim 1 , wherein the activation condition is based on a location, wherein the method further comprises:
applying one or more scene recognition algorithms to current sensory data captured by the one or more cameras of the head-mounted client device; recognizing that the user is at the location; and determining the activation condition is satisfied.

3 . The method of claim 2 , wherein the current sensory data comprises one or more of an image or a video clip captured by the one or more cameras.

4 . The method of claim 1 , wherein the first entity is identified further based on a social graph comprising a plurality of nodes and a plurality of edges connecting the nodes, wherein the plurality of nodes comprise a node corresponding to the user and a node corresponding to the first entity.

5 . The method of claim 1 , wherein the first entity is identified further based on prior user requests by the user.

6 . The method of claim 1 , wherein the initial sensory data comprises one or more of an image or a video clip, wherein the initial sensory data is based on a field of view of the one or more cameras, and wherein the visual analysis of the visual data is based on one or more machine-learning algorithms.

7 . The method of claim 6 , wherein the one or more machine-learning algorithms are based on one or more of facial recognition, gait recognition, or object recognition.

8 . The method of claim 1 , wherein determining the activation condition is further based on one or more of the routine of the user or the episodic memory of the user.

9 . The method of claim 1 , further comprising:
determining contextual information associated with the initial sensory data; and accessing a plurality of episodic memories of the user; wherein the reminder comprises one or more references to one or more other users, respectively, the referenced users being based on the contextual information and one or more of the accessed episodic memories of the user.

10 . The method of claim 9 , wherein the reminder further comprises information associated with the referenced users from the one or more of the accessed episodic memories of the user.

11 . The method of claim 9 , further comprising:
retrieving one or more content objects associated with the one or more of the accessed episodic memories of the user, wherein each content object comprises one or more of a post, a comment, an image, or a video clip; wherein the reminder further comprises one or more of the retrieved content objects.

12 . The method of claim 1 , wherein the reminder is generated based on one or more reminder-templates.

13 . The method of claim 1 , wherein the head-mounted client device is associated with an assistant system, and wherein the reminder is generated by a response-execution module of the assistant system.

14 . The method of claim 1 , wherein the reminder comprises a social summary associated with the first entity.

15 . The method of claim 1 , wherein the reminder comprises a social recommendation referencing one or more other users.

16 . One or more computer-readable non-transitory storage media embodying software that is operable when executed to:
receive, at a head-mounted client device, initial sensory data captured by one or more cameras of the head-mounted client device, wherein the initial sensory data is visual data; proactively generate, by the head-mounted client device responsive to proactively identifying a first entity based on a visual analysis of the visual data and correlating the first entity with knowledge about the user, a reminder associated with the first entity for the user, wherein the knowledge about the user comprises one or more of a routine of the user related to the first entity or an episodic memory of the user referencing the first entity; determine, by the head-mounted client device, an activation condition associated with the reminder, wherein the activation condition is based on one or more of a time or a location, wherein the time and/or location are determined based on the analysis of the visual data and the knowledge about the user; and present, at the head-mounted client device, the reminder when the activation condition is satisfied.

17 . The media of claim 16 , wherein the first entity is identified further based on a social graph comprising a plurality of nodes and a plurality of edges connecting the nodes, wherein the plurality of nodes comprise a node corresponding to the user and a node corresponding to the first entity.

18 . The media of claim 16 , wherein determining the activation condition is further based on one or more of the routine of the user or the episodic memory of the user.

19 . A system comprising: one or more processors; and a non-transitory memory coupled to the processors comprising instructions executable by the processors, the processors operable when executing the instructions to:
receive, at a head-mounted client device, initial sensory data captured by one or more cameras of the head-mounted client device, wherein the initial sensory data is visual data; proactively generate, by the head-mounted client device responsive to proactively identifying a first entity based on a visual analysis of the visual data and correlating the first entity with knowledge about the user, a reminder associated with the first entity for the user, wherein the knowledge about the user comprises one or more of a routine of the user related to the first entity or an episodic memory of the user referencing the first entity; determine, by the head-mounted client device, an activation condition associated with the reminder, wherein the activation condition is based on one or more of a time or a location, wherein the time and/or location are determined based on the analysis of the visual data and the knowledge about the user; and present, at the head-mounted client device, the reminder when the activation condition is satisfied.

20 . The system of claim 19 , wherein the first entity is identified further based on a social graph comprising a plurality of nodes and a plurality of edges connecting the nodes, wherein the plurality of nodes comprise a node corresponding to the user and a node corresponding to the first entity

Assignees

• Meta Platforms Tech Llc

Inventors

• Huang Lisa Xiaoyi
• Xiao Eric
• Penov Francislav P

Classifications

• G06Q10/40 Primary

  Business processes related to social networking or social networking services · CPC title

• G06N3/09

  Supervised learning · CPC title

• G06N3/098

  Distributed learning, e.g. federated learning · CPC title

• G06Q10/1093

  Calendar-based scheduling for persons or groups · CPC title

• H04L67/306

  User profiles · CPC title