Intelligent assistant for home automation

Patent metadata

Field	Value
Publication number	US-11699448-B2
Application number	US-202117540942-A
Country	US
Kind code	B2
Filing date	Dec 2, 2021
Priority date	May 30, 2014
Publication date	Jul 11, 2023
Grant date	Jul 11, 2023

Abstract

This relates to systems and processes for using a virtual assistant to control electronic devices. In one example process, a user can speak an input in natural language form to a user device to control one or more electronic devices. The user device can transmit the user speech to a server to be converted into a textual representation. The server can identify the one or more electronic devices and appropriate commands to be performed by the one or more electronic devices based on the textual representation. The identified one or more devices and commands to be performed can be transmitted back to the user device, which can forward the commands to the appropriate one or more electronic devices for execution. In response to receiving the commands, the one or more electronic devices can perform the commands and transmit their current states to the user device.

First claim

What is claimed is:

1. A non-transitory computer-readable storage medium storing one or more programs, the one or more programs comprising instructions, which when executed by one or more processors of a user device, cause the user device to:
receive, by the user device, an audio input comprising a user speech;
transmit data corresponding to the audio input and a state of each of a plurality of electronic devices to one or more servers;
receive, from the one or more servers:
a textual representation of the user speech including a configuration; and
an indication that the state of each of the plurality of electronic devices is not updated;

determine, by the user device and based on the configuration, a plurality of commands to set the state of each of the plurality of electronic devices; and
transmit the plurality of commands to the plurality of electronic devices based on identifications associated with each of the plurality of commands identified from the configuration.

2. The non-transitory computer-readable storage medium of claim 1 , wherein the one or more programs include further instructions, which when executed by the one or more processors, cause the user device to:
after transmitting the plurality of commands, receive an updated state of each of the plurality of electronic devices from the user device; and
update a state of one or more of the plurality of electronic devices in a database based on the received updated state of each of the plurality of electronic devices.

3. The non-transitory computer-readable storage medium of claim 2 , wherein at least one of the updated states of each of the plurality of electronic devices is an error state, and wherein the error state represents an unavailable or undetermined state of an electronic device.

4. The non-transitory computer-readable storage medium of claim 2 , wherein the updated state of each of the plurality of electronic devices comprises at least one or an ON/OFF state, a dimmable state, a color state, an ACTIVE/INACTIVE state, a LOCKED/UNLOCKED state, an OPEN/CLOSED state, or a temperature state.

5. The non-transitory computer-readable storage medium of claim 2 , wherein the one or more programs include further instructions, which when executed by the one or more processors, cause the user device to:
transmit the updated state of each of the plurality of electronic devices to the one or more servers.

6. The non-transitory computer-readable storage medium of claim 5 , wherein the one or more programs include further instructions, which when executed by the one or more processors, cause the user device to:
output an indication of a result of each of the plurality of commands based on the updated state of each of the plurality of electronic devices received from the plurality of electronic devices.

7. The non-transitory computer-readable storage medium of claim 6 , wherein the indication of the result of each of the plurality of commands is an audio indication, a visual indication, or both an audio and a visual indication.

8. The non-transitory computer-readable storage medium of claim 1 , wherein the user device is a mobile phone, a desktop computer, a laptop computer, a tablet computer, a portable media player, a television, a television set-top box, or a wearable electronic device.

9. The non-transitory computer-readable storage medium of claim 1 , wherein the plurality of electronic devices includes at least one of a light bulb, an electrical outlet, a switch, a door lock, a garage door, and a thermostat.

10. The non-transitory computer-readable of claim 1 , wherein the one or more programs include further instructions, which when executed by the one or more processors, cause the user device to:
query each of the plurality of electronic devices to determine a current state of each of the plurality of electronic devices.

11. The non-transitory computer-readable of claim 1 , wherein the plurality of commands are transmitted to the plurality of electronic devices in response to a determination that a predetermined condition is satisfied.

12. The non-transitory computer-readable of claim 1 , wherein the determination that a predetermined condition is satisfied comprises determining that the user device enters a predetermined area.

13. The non-transitory computer-readable of claim 1 , wherein the determination that a predetermined condition is satisfied comprises determining that the user device exits a predetermined area during a predetermined window of time.

14. A method for controlling a plurality of electronic devices using a virtual assistant on a user device having one or more processors and a memory, the method comprising:
at the user device:
receiving, by the user device, an audio input comprising a user speech;
transmitting data corresponding to the audio input and a state of each of a plurality of electronic devices to one or more servers;
receiving, from the one or more servers:
a textual representation of the user speech including a configuration; and
an indication that the state of each of the plurality of electronic devices is not updated;

determining, by the user device and based on the configuration, a plurality of commands to set the state of each of the plurality of electronic devices; and
transmitting the plurality of commands to the plurality of electronic devices based on identifications associated with each of the plurality of commands identified from the configuration.

15. The method of claim 14 , further comprising:
after transmitting the plurality of commands, receiving an updated state of each of the plurality of electronic devices from the user device; and
updating a state of one or more of the plurality of electronic devices in a database based on the received updated state of each of the plurality of electronic devices.

16. The method of claim 15 , wherein at least one of the updated states of each of the plurality of electronic devices is an error state, and wherein the error state represents an unavailable or undetermined state of an electronic device.

17. The method of claim 15 , wherein the updated state of each of the plurality of electronic devices comprises at least one or an ON/OFF state, a dimmable state, a color state, an ACTIVE/INACTIVE state, a LOCKED/UNLOCKED state, an OPEN/CLOSED state, or a temperature state.

18. The method of claim 15 , further comprising:
transmitting the updated state of each of the plurality of electronic devices to the one or more servers.

19. The method of claim 18 , further comprising:
outputting an indication of a result of each of the plurality of commands based on the updated state of each of the plurality of electronic devices received from the plurality of electronic devices.

20. The method of claim 19 , wherein the indication of the result of each of the plurality of commands is an audio indication, a visual indication, or both an audio and a visual indication.

21. The method of claim 14 , wherein the user device is a mobile phone, a desktop computer, a laptop computer, a tablet computer, a portable media player, a television, a television set-top box, or a wearable electronic device.

22. The method of claim 14 , wherein the plurality of electronic devices includes at least one of a light bulb, an electrical outlet, a switch, a door lock, a garage door, and a thermostat.

23. The method of claim 14 , further comprising:
querying each of the plurality of electronic devices to determine a curr

Assignees

• Apple Inc

Inventors

• Orr Ryan M
• Nell Garett R
• Brumbaugh Benjamin L

Classifications

• H04L12/282

  based on user interaction within the home (receiver circuitry for displaying additional information being controlled by a remote control apparatus H04N21/42204) · CPC title

• G05B15/02 Primary

  electric · CPC title

• G05B2219/2642

  Domotique, domestic, home control, automation, smart house · CPC title

• H04L12/2816

  Controlling appliance services of a home automation network by calling their functionalities (arrangements in telecontrol or telemetry systems for selectively calling a substation from a main station; in which substation desired apparatus is selected for applying a control signal thereto or for obtaining measured values therefrom H04Q9/00) · CPC title

• G05B2219/23373

  Interactive guidance by voice message · CPC title