Intelligent assistant for home automation

Patent metadata

Field	Value
Publication number	US-10170123-B2
Application number	US-201414503105-A
Country	US
Kind code	B2
Filing date	Sep 30, 2014
Priority date	May 30, 2014
Publication date	Jan 1, 2019
Grant date	Jan 1, 2019

Abstract

This relates to systems and processes for using a virtual assistant to control electronic devices. In one example process, a user can speak an input in natural language form to a user device to control one or more electronic devices. The user device can transmit the user speech to a server to be converted into a textual representation. The server can identify the one or more electronic devices and appropriate commands to be performed by the one or more electronic devices based on the textual representation. The identified one or more devices and commands to be performed can be transmitted back to the user device, which can forward the commands to the appropriate one or more electronic devices for execution. In response to receiving the commands, the one or more electronic devices can perform the commands and transmit their current states to the user device.

First claim

What is claimed is:

1. A method for controlling electronic devices using a virtual assistant on a user device having a memory and at least one processor, the method comprising: at the user device:
receiving, an audio input comprising user speech;
transmitting data corresponding to the audio input to one or more servers;
receiving, from the one or more servers:
an identification of a first electronic device determined by the one or more servers based on the data corresponding to the audio input; and
a first command to be performed by the first electronic device determined by the one or more servers based on the data corresponding to the audio input;

transmitting the first command to the first electronic device;
receiving, after transmitting the first command to the first electronic device, a second audio input;
transmitting a state query to the first electronic device in accordance with the second audio input;
receiving, from the first electronic device, a current state of the first electronic device; and
transmitting the current state of the first electronic device to the one or more servers to be stored as at least a portion of a configuration, wherein the configuration defines a plurality of states of a plurality of corresponding electronic devices to use in response to a command that references the configuration.

2. The method of claim 1 , further comprising:
receiving, after transmitting the first command to the first electronic device and prior to receiving the second audio input, a current state of the first electronic device from the first electronic device; and
transmitting the current state of the first electronic device to the one or more servers.

3. The method of claim 2 , wherein the first electronic device comprises a light bulb.

4. The method of claim 3 , wherein the first command comprises a command to set an ON/OFF state, dimmable state, or color state of the light bulb.

5. The method of claim 4 , wherein the current state of the first electronic device comprises the ON/OFF state, dimmable state, or color state of the light bulb after transmitting the first command to set the ON/OFF state, dimmable state, or color state of the light bulb.

6. The method of claim 2 , wherein the first electronic device comprises an electrical outlet.

7. The method of claim 6 , wherein the first command comprises a command to set an ACTIVE/INACTIVE state of the electrical outlet.

8. The method of claim 7 , wherein the current state of the first electronic device comprises the ACTIVE/INACTIVE state of the electrical outlet after transmitting the command to set the ACTIVE/INACTIVE state of the electrical outlet.

9. The method of claim 2 , wherein the first electronic device comprises a switch.

10. The method of claim 9 , wherein the first command comprises a command to set an ON/OFF state of the switch.

11. The method of claim 10 , wherein the current state of the first electronic device comprises the ON/OFF state of the electrical outlet after transmitting the command to set the ON/OFF state of the switch.

12. The method of claim 2 , wherein the first electronic device comprises a door lock.

13. The method of claim 12 , wherein the first command comprises a command to set a LOCKED/UNLOCKED state of the door lock.

14. The method of claim 13 , wherein the current state of the first electronic device comprises the LOCKED/UNLOCKED state of the door lock after transmitting the command to set the LOCKED/UNLOCKED state of the door lock.

15. The method of claim 2 , wherein the first electronic device comprises a garage door.

16. The method of claim 15 , wherein the first command comprises a command to set an OPEN/CLOSED state of the garage door.

17. The method of claim 16 , wherein the current state of the first electronic device comprises the OPEN/CLOSED state of the garage door after transmitting the command to set the OPEN/CLOSED state of the garage door.

18. The method of claim 2 , wherein the first electronic device comprises a thermostat.

19. The method of claim 18 , wherein the first command comprises a command to set a numerical value of a temperature setting of the thermostat.

20. The method of claim 19 , wherein the current state of the first electronic device comprises the numerical value of the temperature setting of the thermostat after transmitting the first command to set the numerical value of the temperature setting of the thermostat.

21. The method of claim 2 , further comprising outputting a notification associated with the current state of the first electronic device in response to determining that a predetermined condition has been satisfied.

22. The method of claim 2 , wherein the user device comprises a database for storing a state of each of a plurality of electronic devices, the plurality of electronic devices comprising the first electronic device, and wherein the method further comprises updating a state of the first electronic device stored in the database based at least in part on the current state of the first electronic device received from the first electronic device.

23. The method of claim 2 , wherein the method further comprises outputting an indication of a result of the first command based on the first command and the current state of the first electronic device received from the first electronic device, wherein the indication of the result is an audio, visual, or both an audio and visual indication.

24. The method of claim 1 , wherein the user device comprises a mobile phone, desktop computer, laptop computer, tablet computer, portable media player, television, television set-top box, or wearable electronic device.

25. The method of claim 1 , further comprising:
receiving, prior to receiving the second audio input, from the one or more servers:
an identification of a second electronic device determined by the one or more servers based on the data corresponding to the audio input; and
a second command to be performed by the second electronic device determined by the one or more servers based on the data corresponding to the audio input;

transmitting the second command to the second electronic device;
receiving, after transmitting the second command to the second electronic device, a current state of the second electronic device from the second electronic device; and
transmitting the current state of the second electronic device to the one or more servers.

26. The method of claim 1 , wherein the user device comprises a microphone.

27. The method of claim 1 , wherein transmitting the state query to the first electronic device in accordance with the second audio input comprises:
transmitting data corresponding to the second audio input to the one or more servers;
receiving, from the one or more servers, an instruction to query a state of the first electronic device determined by the one or more servers based on the data corresponding to the second audio input; and
transmitting the state query to the first electronic device based on the instruction to query the state of the first electronic device.

28. A non-transitory computer-readable storage medium comprising one or more programs for execution by one or more processors of a user device, the one or more programs including instructions which, when executed by the one or more processors, cause the user device to perform the method of:
receiving an audio input co

Assignees

• Apple Inc

Inventors

• Orr Ryan M
• Nell Garett R
• Brumbaugh Benjamin L

Classifications

• G05B19/418

  Total factory control, i.e. centrally controlling a plurality of machines, e.g. direct or distributed numerical control [DNC], flexible manufacturing systems [FMS], integrated manufacturing systems [IMS] or computer integrated manufacturing [CIM] · CPC title

• G05B15/02 Primary

  electric · CPC title

• G10L17/22 Primary

  Interactive procedures; Man-machine interfaces · CPC title

• H04L12/282

  based on user interaction within the home (receiver circuitry for displaying additional information being controlled by a remote control apparatus H04N21/42204) · CPC title

• G05B2219/2642

  Domotique, domestic, home control, automation, smart house · CPC title