Customized interface based on vocal input

Patent metadata

Field	Value
Publication number	US-11929844-B2
Application number	US-202217709077-A
Country	US
Kind code	B2
Filing date	Mar 30, 2022
Priority date	Oct 8, 2018
Publication date	Mar 12, 2024
Grant date	Mar 12, 2024

Abstract

Various arrangements for using captured voice to generate a custom interface controller are presented. A vocal recording from a user may be captured in which a spoken command and multiple smart-home devices are indicated. One or more common functions that map to the multiple smart-home devices may be determined. A custom interface controller may be generated that controls the one or more common functions of each smart-home device of the multiple smart-home devices.

First claim

What is claimed is:

1. A method for using captured voice to generate a custom interface controller, the method comprising:
capturing, via a mobile device, a vocal recording, wherein the vocal recording comprises a spoken command directed to a plurality of smart-home devices;
transmitting the vocal recording to a cloud-based server system to perform natural language processing on the spoken command, wherein the spoken command indicates one or more locations;
receiving, by an application executed by the mobile device, a response from the cloud-based server system that comprises a plurality of smart-home device identifiers corresponding to the one or more locations, wherein:
the plurality of smart-home device identifiers were identified by the cloud-based server system based on the natural language processing performed on the spoken command as corresponding to the one or more locations;

determining, by accessing a home device function database, a common function that maps to each smart-home device of the plurality of smart-home devices; and
generating, by the application executed by the mobile device, a control element for the one or more locations, which is presented as part of the custom interface controller, that controls only the common function of each smart-home device of the plurality of smart-home devices for which smart-home device identifiers were received from the cloud-based server system.

2. The method for using the captured voice to generate the custom interface controller of claim 1 , further comprising:
receiving, by the application executed by the mobile device, user input via the custom interface controller and a touchscreen of the mobile device.

3. The method for using the captured voice to generate the custom interface controller of claim 2 , further comprising:
determining, by the application executed by the mobile device, a command to be performed by each of the plurality of smart-home devices based on the user input received via the custom interface controller; and
transmitting, by the application executed by the mobile device, the command and the plurality of smart-home device identifiers to the cloud-based server system.

4. The method for using the captured voice to generate the custom interface controller of claim 3 , further comprising:
in response to receiving the command and the plurality of smart-home device identifiers from the application executed by the mobile device, transmitting, to a second cloud-based server system, an indication of the command and a smart-home device identifier of the plurality of smart-home device identifiers, wherein the second cloud-based server system communicates with a second smart-home device mapped to the smart-home device identifier.

5. The method for using the captured voice to generate the custom interface controller of claim 3 , further comprising:
in response to receiving the command and the plurality of smart-home device identifiers from the application executed by the mobile device, transmitting, a second indication of the command to a second smart-home device mapped to a second smart-home device identifier of the plurality of smart-home device identifiers.

6. The method for using the captured voice to generate the custom interface controller of claim 1 , wherein:
the vocal recording additionally indicates a type of smart-home device; and
the cloud-based server system causes the spoken command to be performed at the plurality of smart-home devices based on the plurality of smart-home devices matching the type of smart-home device indicated in the vocal recording.

7. The method for using the captured voice to generate the custom interface controller of claim 1 , further comprising:
analyzing, by the cloud-based server system, the vocal recording to determine the spoken command and the plurality of smart-home devices.

8. The method for using the captured voice to generate the custom interface controller of claim 1 , wherein the plurality of smart-home device identifiers correspond to multiple different types of smart-home devices.

9. A system for using captured voice to generate a custom interface controller, the system comprising a smart-home application executed by a mobile device and a cloud-based host server system, configured to:
capture, via one or more microphones of the mobile device, a vocal recording from a user, wherein the vocal recording comprises a spoken command directed to a plurality of smart-home devices;
transmit, via a network interface of the mobile device, the vocal recording to the cloud-based host server system to perform natural language processing on the spoken command, wherein the spoken command indicates one or more locations;
receive, by the mobile device, a response from the cloud-based host server system that comprises a plurality of smart-home device identifiers corresponding to the one or more locations, wherein:
the plurality of smart-home device identifiers were identified by cloud-based host server system based on the natural language processing performed on the spoken command as corresponding to the one or more locations;

determine, by accessing a home device function database, a common function that maps to each smart-home device of the plurality of smart-home devices; and
generate a control element for the one or more locations, which is presented as part of the custom interface controller, that controls only the common function of each smart-home device of the plurality of smart-home devices for which smart-home device identifiers were received from the cloud-based host server system.

10. The system for using the captured voice to generate the custom interface controller of claim 9 , wherein the smart-home application further causes the mobile device to:
receive, via a touchscreen of the mobile device, user input via the custom interface controller.

11. The system for using the captured voice to generate the custom interface controller of claim 10 , wherein the smart-home application further causes the mobile device to:
determine a command to be performed by each of the plurality of smart-home devices based on the user input received via the custom interface controller; and
transmit the command and the plurality of smart-home device identifiers to the cloud-based host server system.

12. The system for using the captured voice to generate the custom interface controller of claim 11 , wherein the smart-home application further causes the mobile device to:
transmit, to the cloud-based host server system, an indication of the command and a smart-home device identifier of the plurality of smart-home device identifiers in response to receiving the command and the plurality of smart-home device identifiers from the smart-home application executed by the mobile device, wherein the cloud-based host server system communicates with a second smart-home device mapped to the smart-home device identifier.

13. The system for using the captured voice to generate the custom interface controller of claim 12 , wherein the smart-home application further causes the mobile device to:
transmit, a second indication of the command to the second smart-home device mapped to a second smart-home device identifier of the plurality of smart-home device identifiers in response to receiving the command and the plurality of smart-home device identifiers from the smart-home application executed by the mobile device.

14. The system for using the captured voice to generate the custom interface controller of claim 9 , wherein the custom interface controller comprises a first control to control a first function of the plurality of smart-home

Assignees

• Google Llc

Inventors

• Brown Benjamin
• Huang Da
• Conover Christopher
• Williams Lisa
• Chung Henry

Classifications

• H04L12/2814 Primary

  Exchanging control software or macros for controlling appliance services in a home automation network (arrangements for maintenance or administration involving configuration of the network and network elements H04L41/08) · CPC title

• G06F16/24573

  using data annotations, e.g. user-defined metadata · CPC title

• H04L12/2809

  indicating that an appliance service is present in a home automation network (monitoring functionality H04L43/0817; discovery or management thereof, e.g. service location protocol [SLP] or web services, H04L67/51) · CPC title

• H04L12/282 Primary

  based on user interaction within the home (receiver circuitry for displaying additional information being controlled by a remote control apparatus H04N21/42204) · CPC title

• H04L2012/2847

  characterised by the type of home appliance used · CPC title