Appending assistant server requests, from a client assistant, with proactively-aggregated peripheral device data

Patent metadata

Field	Value
Publication number	US-12505841-B2
Application number	US-202418620628-A
Country	US
Kind code	B2
Filing date	Mar 28, 2024
Priority date	Aug 16, 2021
Publication date	Dec 23, 2025
Grant date	Dec 23, 2025

Abstract

Implementations relate to proactively aggregating client device data to append to client assistant data that is communicated to a server device in response to a user request to a client automated assistant. When a user request that is associated with, for example, a peripheral client device, is received at a client device, the client device can communicate, to a server device, data that embodies the user request (e.g., audio data and/or local speech recognition data), along with peripheral device data that was received before the client device received the user request. In this way, the client automated assistant can bypass expressly soliciting peripheral device data each time a user request is received at another client device. Instead, a peripheral device can proactively communicate device data to a client device so that the device data can be appended to request data communicated to the server device from a particular client device.

First claim

We claim:

1 . A method implemented by one or more processors, the method comprising:
receiving, at a first computing device, second device data that characterizes a status of a second computing device,
wherein the first computing device provides access to an automated assistant and the second computing device is in communication with the first computing device;

subsequent to receiving the second device data from the second computing device:
determining, at the first computing device, that a user provided an input at the first computing device that is directed to the automated assistant;
providing, via the first computing device and in response to the input from the user, client assistant data to a server device that is separate from the first computing device and the second computing device,
wherein the client assistant data is based on both:
the second device data received at the first computing device prior to the input being provided at the first computing device, and
the input from the user;

receiving, from the server device, responsive data that characterizes an assistant response to the input and that is generated based on the client assistant data; and
causing, based on the responsive data, the first computing device or the second computing device to render the assistant response.

2 . The method of claim 1 ,
wherein the second device data is received from the second computing device in a binary message format, and wherein the client assistant data is provided to the server device from the first computing device in a non-binary message format.

3 . The method of claim 2 ,
wherein the input is a spoken utterance that embodies a request for the automated assistant to modify the status of the second computing device, and wherein the client assistant data includes:
textual content generated based on audio processing of the spoken utterance performed at the first computing device, and
non-binary data that characterizes the status of the second computing device.

4 . The method of claim 3 , wherein determining that the user provided the input that is directed to the automated assistant includes:
determining that the second computing device detected an invocation phrase for initializing the automated assistant.

5 . The method of claim 1 , wherein the status of the second device characterizes multiple different states of the second computing device, and the method further comprises:
generating, based on the input from the user, the client assistant data, wherein the client assistant data identifies less than a total number of states of the multiple different states of the second computing device.

6 . The method of claim 5 , further comprising:
determining, based on the second device data, that a particular state of the multiple different states does not satisfy a condition for communicating the particular state to the server device, wherein the client assistant data identifies less than the total number of states of the multiple different states based on the particular state of the multiple different states not satisfying the condition.

7 . The method of claim 6 ,
wherein the particular state corresponds to a battery charge level of the second computing device, and wherein the condition for communicating the particular state to the server device is satisfied when the battery charge level satisfies a battery charge threshold.

8 . The method of claim 6 ,
wherein the second computing device includes a pair of wearable audio devices, and wherein the condition for communicating the particular state to the server device is satisfied when none of the wearable audio devices of the pair of wearable audio devices are being worn by the user.

9 . The method of claim 1 , further comprising:
converting the second device data into a non-binary format that consumes less memory than the second device data,
wherein the client assistant data embodies a portion of the non-binary format of the second device data.

10 . A method implemented by one or more processors, the method comprising:
receiving, at a server device, client assistant data that characterizes:
an assistant input provided by a user to an automated assistant that is accessible via a first computing device, and
a status of a second computing device that is in communication with the first computing device,
wherein the assistant input is provided to the first computing device by the user subsequent to the second computing device providing status data to the first computing device for indicating the status of the second computing device;

generating, based on the client assistant data, assistant response data that characterizes a response for the automated assistant to provide to the user via the first computing device or the second computing device; providing, by the server device, the assistant response data to the first computing device; and causing, based on the assistant response data, the automated assistant at the first computing device to render natural language content that is responsive to the assistant input and is based on the status of the second computing device.

11 . The method of claim 10 ,
wherein the client assistant data is received by the server device in a non-binary message format, and wherein the status data is communicated from the second computing device to the first computing device in a binary message format.

12 . The method of claim 11 ,
wherein the assistant input is a spoken utterance that embodies a request for the automated assistant to modify the status of the second computing device, and wherein the client assistant data includes:
textual content generated based on audio processing of the spoken utterance performed at the first computing device, and
non-binary data that characterizes the status of the second computing device.

13 . The method of claim 10 ,
wherein the status characterized by the client assistant data identifies less than a total number of states of multiple different states of the second computing device, and wherein the status data identifies more states than a number of states identified by the status received by the server device.

14 . The method of claim 10 , wherein the status characterized by the client assistant data indicates a noise cancelation setting of the second computing device, and the method further comprises:
causing, based on the assistant response data, the first computing device to modify the noise cancelation setting of the second computing device.

15 . A computing device comprising:
memory storing instructions; one or more processors operable to execute the instructions to:
receive second device data that characterizes a status of a second computing device,
wherein the computing device provides access to an automated assistant and the second computing device is in communication with the computing device;

subsequent to receiving the second device data from the second computing device:
determine that a user provided an input, at the computing device, that is directed to the automated assistant;
provide, via the computing device and in response to the input from the user, client assistant data to a server device that is separate from the computing device and the second computing device,
wherein the client assistant data is based on both:
the second device data received at the computing device prior to the input being provided at the computing device, and
the input from the user;

receive, from the server device, responsive data that characterizes an assistant response to the input and

Assignees

• Google Llc

Inventors

• Ohlendorf Max
• Derkalousdian Moses
• Shaporev Aleksey
• Kutzman Blair
• Kartoz Michael
• Pathak Sumit Kumar

Classifications

• G10L15/30

  Distributed recognition, e.g. in client-server systems, for mobile phones or network applications · CPC title

• G10L2015/223

  Execution procedure of a spoken command · CPC title

• G10L2015/088

  Word spotting · CPC title

• G10L15/22 Primary

  Procedures used during a speech recognition process, e.g. man-machine dialogue · CPC title