Determining device context

Patent metadata

Field	Value
Publication number	US-12443633-B1
Application number	US-202318143285-A
Country	US
Kind code	B1
Filing date	May 4, 2023
Priority date	Apr 4, 2023
Publication date	Oct 14, 2025
Grant date	Oct 14, 2025

Abstract

A system may be configured to receive and process various signals to generate a natural language description of a user's environment, called situational context data. The signals may include sensor data, device status, user activity, user input, and/or inferences made using such data. The situational context data may express a user-centric description of the user's environment; for example: “User is taking a walk in the park on a sunny afternoon” or “activity: driving location: highway”, etc. The system may send the situational context data to various system components that may, for example, process speech, select applications/skills for handling user inputs, and/or that implement those applications/skills. The applications/skills may use the situational context data to provide recommendations, generate responses, and/or perform actions that are more relevant to the user's current environment.

First claim

What is claimed is:

1. A computer-implemented method comprising:
receiving, by a first system component, first context data representing a current interaction between a user and a first user device, the current interaction corresponding to a first type of activity;
receiving second context data representing first sensor data generated by the first user device;
processing the first context data and the second context data using a first neural network encoder to generate first embedding data representing a situational context of the user;
receiving, from a first data storage component, user profile data corresponding to the user;
receiving, from a second data storage component, first data representing factual information about the first type of activity;
processing the user profile data and the first data using a second neural network encoder to generate second embedding data;
processing the first embedding data and the second embedding data using a neural network decoder to generate fourth data representing a natural language description of the user's situational context;
receiving, from the first user device, first input data representing a first utterance of the user;
processing the first input data to determine first natural language understanding (NLU) data representing a user request;
processing using the first NLU data and the fourth data, first response data; and
causing the first user device to output the first response data.

2. The computer-implemented method of claim 1 , further comprising:
determining, using the first context data and the second context data, first tensor data corresponding to a first category of factual information associated with one or more of the first context data and the second context data;
determining, using the first tensor data, second tensor data stored in a graph neural network in the second data storage component, wherein the first data includes the second tensor data and the user profile data includes third tensor data; and
inputting the second tensor data and the third tensor data into the neural network decoder to generate the fourth data.

3. The computer-implemented method of claim 1 , further comprising:
determining, using the first NLU data, a first action to be performed and a first skill for handling the first action;
determining second NLU data representing a second action to be performed and a second skill for handling the second action;
sending, based on the fourth data, the first NLU data to a second system component corresponding to the second skill; and
receiving, from the second system component, the first response data.

4. The computer-implemented method of claim 1 , further comprising:
sending, to the first user device prior to receiving the first context data, first model data representing an untrained model;
receiving, from the first user device, second model data representing a model trained based on first context signals received by the first user device;
receiving third model data representing models trained based on second context signals received by a second user device;
determining, using the second model data and the third model data, fourth model data representing a global model for processing context signals;
sending, to the first user device and at least a second user device, the fourth model data; and
causing the first user device to generate the first context data using the fourth model data.

5. A computer-implemented method comprising:
receiving first data representing a first user activity corresponding to a first user device;
receiving second data representing sensor data generated by the first user device;
determining, using the first data and the second data, first encoded data;
receiving user profile data corresponding to a user of the first user device;
receiving factual data associated with one or more of the first data or the second data;
determining, using the user profile data and the factual data, second encoded data;
processing the first encoded data and the second encoded data using a first neural network decoder to generate third data representing a natural language description of the user's situational context;
receiving, from the first user device, first input data;
processing the first input data to determine first natural language understanding (NLU) data representing a user request;
determining, using the first NLU data and the third data, fourth data representing a response to the user request; and
causing the first user device to output the fourth data.

6. The computer-implemented method of claim 5 , further comprising:
receiving fifth data representing user feedback to the output of the fourth data; and
determining, using the fifth data, parameters for a second neural network decoder, the second neural network decoder representing an update of the first neural network decoder.

7. The computer-implemented method of claim 5 , further comprising:
determining, using the first NLU data, a first action to be performed and a first system component for handling the first action;
determining second NLU data indicating a second action to be performed and a second system component for handling the second action;
sending, based on the third data, the first NLU data to the first system component; and
receiving the fourth data from the first system component.

8. The computer-implemented method of claim 5 , further comprising:
receiving, from a system component, fifth data representing a system-initiated action to perform on behalf of the user;
causing the first user device to output a request for user confirmation that the system-initiated action is to be performed;
receiving second input data representing user confirmation that the system-initiated action is to be performed; and
in response to receiving the second input data, causing the first user device to perform the system-initiated action.

9. The computer-implemented method of claim 5 , further comprising:
determining, using the first data and the second data, a first category of factual data;
receiving, from a first data storage component, fifth data representing structured factual data corresponding to the first category;
receiving, from a second data storage component, sixth data representing unstructured data corresponding to the first category; and
determining the factual data using the fifth data and the sixth data.

10. The computer-implemented method of claim 5 , further comprising:
receiving, from the first user device, second input data representing an utterance of the user;
processing the second input data to determine automatic speech recognition (ASR) data representing a transcript of the utterance;
processing the ASR data and the third data using an NLU component to generate second NLU data representing a user request; and
causing the first user device to perform an action associated with the user request.

11. The computer-implemented method of claim 5 , further comprising:
sending, to the first user device prior to receiving the first data, first model data representing an untrained model;
receiving, from the first user device, second model data representing a model trained based on first context signals received by the first user device;
receiving third model data representing models trained based on second context signals received by a second user device;
determining, using the second model data and the third model data, fourth model data representing a global model for processing context signals;
sending, to the first user device and at least the second user device, the fourth model data; and
causing the first user device to generate the first third data

Assignees

• Amazon Tech Inc

Inventors

• Fan Xing
• Radostev Vasiliy
• Bao Jie
• Chintha Muddu Krishna
• Huang Xiaojiang
• Luo Yi
• Guo Chenlei
• Strom Nikko
• Smith Casey Stuart
• Matsoukas Spyridon
• Bisaria Priti

Classifications

• G10L13/033

  Voice editing, e.g. manipulating the voice of the synthesiser · CPC title

• G10L15/16

  using artificial neural networks · CPC title

• G10L15/1822

  Parsing for meaning understanding · CPC title

• G10L2015/228

  of application context · CPC title

• G10L15/22

  Procedures used during a speech recognition process, e.g. man-machine dialogue · CPC title