Determining device context

Patent metadata

Field	Value
Publication number	US-2026050619-A1
Application number	US-202519332153-A
Country	US
Kind code	A1
Filing date	Sep 18, 2025
Priority date	Apr 4, 2023
Publication date	Feb 19, 2026
Grant date	—

Abstract

A system may be configured to receive and process various signals to generate a natural language description of a user's environment, called situational context data. The signals may include sensor data, device status, user activity, user input, and/or inferences made using such data. The situational context data may express a user-centric description of the user's environment; for example: “User is taking a walk in the park on a sunny afternoon” or “activity: driving location: highway”, etc. The system may send the situational context data to various system components that may, for example, process speech, select applications/skills for handling user inputs, and/or that implement those applications/skills. The applications/skills may use the situational context data to provide recommendations, generate responses, and/or perform actions that are more relevant to the user's current environment.

First claim

What is claimed is:

1 . A computer-implemented method, comprising:
receiving first data representing a first user activity corresponding to a first user device; receiving second data representing sensor data generated by the first user device; receiving user profile data corresponding to a user of the first user device; processing the first data and the second data to generate third data representing a natural language description of a situational context of the user; receiving first input data representing a natural language input captured by the first user device; performing natural language processing using the third data, the user profile data, and the first input data to determine fourth data representing a response to the natural language input; and causing the first user device to output the fourth data.

2 . The computer-implemented method of claim 1 , wherein the natural language processing comprises processing the third data, the user profile data, and the first input data using a language model to determine the fourth data.

3 . The computer-implemented method of claim 1 , further comprising:
determining, using the first input data, a first action to be performed and a first system component for handling the first action; determining, using the first input data, a second action to be performed and a second system component for handling the second action; sending, based on the third data, data representing the second action to the second system component; and receiving, from the second system component, the fourth data.

4 . The computer-implemented method of claim 1 , further comprising:
receiving, from a first system component, fifth data representing a system-initiated action to perform in response to the natural language input, wherein the fourth data represents a request for user confirmation that the system-initiated action is to be performed; receiving input data representing user confirmation that the system-initiated action is to be performed; and in response to receiving the input data, causing the first user device to perform the system-initiated action.

5 . The computer-implemented method of claim 1 , wherein the natural language processing further comprising using a first machine learning model:
receiving fifth data representing user feedback to the output of the fourth data; and determining, using the fifth data, parameters for updating the first machine learning model.

6 . The computer-implemented method of claim 1 , further comprising:
determining, using the first data and the second data, a first category of factual data; receiving, from a first data storage component, fifth data representing structured factual data corresponding to the first category; receiving, from a second data storage component, sixth data representing unstructured data corresponding to the first category; and determining factual data using the fifth data and the sixth data, wherein the natural language processing is based at least in part on the factual data.

7 . The computer-implemented method of claim 1 , wherein the first input data comprises audio data representing a user utterance captured by the first user device.

8 . The computer-implemented method of claim 1 , wherein the first input data represents a transcript of a user utterance captured by the first user device.

9 . The computer-implemented method of claim 1 , further comprising:
sending, to the first user device prior to receiving the first input data, first model data representing an untrained model; receiving, from the first user device, second model data representing a model trained based on first context signals received by the first user device; receiving third model data representing models trained based on second context signals received by a second user device; determining, using the second model data and the third model data, fourth model data representing a global model for processing context signals; sending, to the first user device and at least a second user device, the fourth model data; and causing the first user device to generate the first data using the fourth model data.

10 . The computer-implemented method of claim 1 , wherein processing the first data and the second data to generate the third data comprises:
processing the first data and the second data to determine first encoded data; processing the user profile data to determine second encoded data; and processing the first encoded data and second encoded data to determine the third data.

11 . A system comprising:
at least one processor; and at least one memory comprising instructions that, when executed by the at least one processor, cause the system to perform operations comprising:
receiving first data representing a first user activity corresponding to a first user device;
receiving second data representing sensor data generated by the first user device;
receiving user profile data corresponding to a user of the first user device;
processing the first data and the second data to generate third data representing a natural language description of a situational context of the user;
receiving first input data representing a natural language input captured by the first user device;
performing natural language processing using the third data, the user profile data, and the first input data to determine fourth data representing a response to the natural language input; and
causing the first user device to output the fourth data.

12 . The system of claim 11 , wherein the natural language processing comprises processing the third data, the user profile data, and the first input data using a language model to determine the fourth data.

13 . The system of claim 11 , wherein the at least one memory further comprises instructions that, when executed by the at least one processor, further cause the system to perform further operations comprising:
determining, using the first input data, a first action to be performed and a first system component for handling the first action; determining, using the first input data, a second action to be performed and a second system component for handling the second action; sending, based on the third data, data representing the second action to the second system component; and receiving, from the second system component, the fourth data.

14 . The system of claim 11 , wherein the at least one memory further comprises instructions that, when executed by the at least one processor, further cause the system to perform further operations comprising:
receiving, from a first system component, fifth data representing a system-initiated action to perform in response to the natural language input, wherein the fourth data represents a request for user confirmation that the system-initiated action is to be performed; receiving input data representing user confirmation that the system-initiated action is to be performed; and in response to receiving the input data, causing the first user device to perform the system-initiated action.

15 . The system of claim 11 , wherein the natural language processing further comprising using a first machine learning model:
receiving fifth data representing user feedback to the output of the fourth data; and determining, using the fifth data, parameters for updating the first machine learning model.

16 . The system of claim 11 , wherein the at least one memory further comprises instructions that, when executed by the at least one processor, further cause the system to perform further operations comprising:
deter

Assignees

• Amazon Tech Inc

Inventors

• Fan Xing
• Radostev Vasiliy
• Bao Jie
• Chintha Muddu Krishna
• Huang Xiaojiang
• Luo Yi
• Guo Chenlei
• Strom Nikko
• Smith Casey Stuart
• Matsoukas Spyridon
• Bisaria Priti

Classifications

• G10L15/08

  Speech classification or search · CPC title

• G10L13/033

  Voice editing, e.g. manipulating the voice of the synthesiser · CPC title

• G10L15/16

  using artificial neural networks · CPC title

• G10L15/1822

  Parsing for meaning understanding · CPC title

• G10L2015/228

  of application context · CPC title