Regulating assistant responsiveness according to characteristics of a multi-assistant environment

Patent metadata

Field	Value
Publication number	US-11037562-B2
Application number	US-201816343934-A
Country	US
Kind code	B2
Filing date	Aug 23, 2018
Priority date	Aug 23, 2018
Publication date	Jun 15, 2021
Grant date	Jun 15, 2021

Abstract

Implementations set forth herein relate to employing dynamic regulations for governing responsiveness of multiple automated assistant devices, and specifically the responsiveness an automated assistant to a given spoken utterance that has been acknowledged by two or more of the assistant devices. The dynamic regulations can be context-dependent and adapted over time in order that the automated assistant can accommodate assistant interaction preferences that may vary from user to user. For instance, a spoken utterance such as “stop,” may be intended to affect different assistant actions based on a context in which the user provided the spoken utterance. The context can refer to a location of the user relative to other rooms in a home, a time of day, a user providing the spoken utterance, an arrangement of the assistant devices within a home, and/or a state of each device in the home.

First claim

We claim:

1. A method implemented by one or more processors, the method comprising:
receiving audio data that captures a spoken utterance of a user, wherein the spoken utterance embodies a request for a modification action to be performed via an automated assistant and is received by at least one of two or more computing devices that are each capable of performing the modification action via the automated assistant;
determining, using the audio data that captures the spoken utterance, that the modification action is capable of modifying each of:
a first ongoing action being performed at a first device of the two or more computing devices, and
a second ongoing action being performed at a second device of the two or more computing devices;

identifying, based on determining that the modification action is capable of modifying both the first ongoing action and the second ongoing action, a stored regulation that corresponds to the modification action, wherein the stored regulation characterizes a contextual dependency of performance of the modification action;
accessing, based on identifying the stored regulation, contextual data that is associated with the stored regulation, wherein the contextual data characterizes a first type of the first ongoing action and a second type of the second ongoing action;
determining, based on the contextual data and the stored regulation, a target computing device, of the two or more computing devices, at which the modification action is to be performed via the automated assistant; and
causing, based on determining the target computing device at which the action is to be controlled, the modification action to be performed at the target computing device via the automated assistant.

2. The method of claim 1 , wherein determining the stored regulation includes identifying the stored regulation from multiple different regulations accessible to the automated assistant, and wherein the multiple different regulations are determined based on one or more previous instances of the user requesting the automated assistant perform the modification action.

3. The method of claim 2 , wherein the stored regulation is determined based on one or more corrective spoken utterances previously received by the automated assistant from the user, each of the one or more corrective spoken utterances being provided by the user subsequent to a corresponding previous instance of the modification action being incorrectly executed via the automated assistant, as indicated by the corrective spoken utterance.

4. The method of claim 1 , wherein the first ongoing action includes providing audible sound via a speaker of the first device.

5. The method of claim 1 , wherein the contextual data includes occupancy data that characterizes an occupancy of one or more rooms, of a building in which the user provided the spoken utterance, and the stored regulation indicates a preference of the user for limiting at least one action with respect to a particular room of the one or more rooms of the building.

6. The method of claim 5 , wherein the contextual data further indicates a location of the user within the particular room of the one or more rooms of the building, and the stored regulation indicates another preference of the user for the target computing device to control the at least one action via the automated assistant.

7. The method of claim 1 , wherein causing the modification action to be performed at the target computing device includes limiting performance of a particular ongoing action at the target computing device of the two or more computing devices.

8. The method of claim 1 , wherein causing the at least one action to be performed at the target computing device includes modifying an ongoing rendering of audio data or visual data, and the contextual data indicates a time at which the ongoing rendering of the audio data or visual data was initialized.

9. The method of claim 1 , wherein the contextual data identifies an operating status for each computing device of the two or more computing devices, and each computing device of the two or more computing devices are configured to provide respective operating status data to a common co-located cluster of server devices for processing the operating status.

10. A method implemented by one or more processors, the method comprising:
determining, at a server device that is in communication with multiple client devices, status information corresponding to activity states of multiple client devices, which are disposed about a location of a user, wherein each client device of the multiple client devices is accessible to an automated assistant;
receiving data that is based on a spoken utterance, provided by the user, to at least one client device of the multiple client devices that is operating according to the determined status information, wherein the spoken utterance embodies a request for an action to be performed by one or more client devices of the multiple client devices;
accessing, in response to receiving the data that is based on the spoken utterance, a stored set of dynamic regulations associated with the multiple client devices, wherein at least one stored regulation of the stored set of dynamic regulations characterizes a contextual dependency of execution of the action, by the one or more client devices, on the status information;
identifying, based on the at least one stored regulation of the stored set of dynamic regulations and the status information, a targeted client device to perform the action, wherein the contextual dependency characterized by the at least one stored regulation includes at least one volume-agnostic condition for designating the targeted client device to perform the action; and
causing the targeted client device to perform the action via the automated assistant.

11. The method of claim 10 , wherein the location includes multiple rooms characterized by a stored topology that is accessible to the server device, and the method further includes:
accessing current contextual data that characterizes the contextual dependency as the user being located within a first room that includes the at least one client device, and the targeted client device being located within a second room of the multiple rooms.

12. The method of claim 10 , further comprising:
determining, based on receiving the data, whether the spoken utterance is as at least one type of utterance selected from an ambiguous type of utterance and a specific type of utterance, wherein particular operations of accessing the stored set of dynamic regulations associated with the multiple client devices, identifying a targeted client device to perform the action, and causing the targeted computing device to perform the action are performed when the spoken utterance is determined to be the ambiguous type of utterance.

13. The method of claim 10 , further comprising:
when the spoken utterance is determined to be a specific type of utterance:
identifying a designated client device, specifically identified by the user via the spoken utterance, and
causing the designated client to perform the action while bypassing accessing the stored set of dynamic regulations associated with the multiple client devices.

14. The method of claim 13 , wherein characterizing the received spoken utterance as at least one type of utterance selected from the ambiguous type of utterance and the specific type of utterance is based on previous interactions between the user and the automated assistant.

15. A method implemented by one or more processors, the method comprising:
receiving status information from mul

Assignees

• Google Llc

Inventors

• Shah Raunaq
• Konzelmann Jaclyn
• Takehana Lisa
• Davies Ruxandra
• Diaconu Adrian

Classifications

• G10L2015/223

  Execution procedure of a spoken command · CPC title

• G10L2015/221

  Announcement of recognition results · CPC title

• G06F3/167

  Audio in a user interface, e.g. using voice commands for navigating, audio feedback · CPC title

• G10L15/22 Primary

  Procedures used during a speech recognition process, e.g. man-machine dialogue · CPC title

• G10L15/04 Primary

  Segmentation; Word boundary detection · CPC title