Intelligent assistant for home automation

Patent metadata

Field	Value
Publication number	US-10699717-B2
Application number	US-201816175208-A
Country	US
Kind code	B2
Filing date	Oct 30, 2018
Priority date	May 30, 2014
Publication date	Jun 30, 2020
Grant date	Jun 30, 2020

Abstract

This relates to systems and processes for using a virtual assistant to control electronic devices. In one example process, a user can speak an input in natural language form to a user device to control one or more electronic devices. The user device can transmit the user speech to a server to be converted into a textual representation. The server can identify the one or more electronic devices and appropriate commands to be performed by the one or more electronic devices based on the textual representation. The identified one or more devices and commands to be performed can be transmitted back to the user device, which can forward the commands to the appropriate one or more electronic devices for execution. In response to receiving the commands, the one or more electronic devices can perform the commands and transmit their current states to the user device.

First claim

What is claimed is:

1. A user device comprising:
one or more processors;
memory; and
one or more programs, wherein the one or more programs are stored in the memory and configured to be executed by the one or more processors, the one or more programs including instructions for:
receiving an audio input representing a user request for an electronic device to perform an action,
wherein the audio input is in natural language form, and
wherein the audio input is ambiguous with respect to defining the electronic device for performing the action;

determining one or more possible electronic device characteristics corresponding to the audio input;
retrieving a data structure representing a set of electronic devices of a predetermined location;
determining, based on the data structure, one or more candidate electronic devices from the set of electronic devices, the one or more candidate electronic devices corresponding to the audio input;
determining, based on the one or more possible electronic device characteristics and one or more actual electronic device characteristics of the one or more candidate electronic devices, a user intent corresponding to the audio input; and
providing a command that causes an electronic device of the one or more candidate electronic devices to perform the action corresponding to the user intent.

2. The user device of claim 1 , wherein the audio input is in natural language form, and wherein the audio input is ambiguous with respect to defining the action.

3. The user device of claim 1 , wherein the one or more possible electronic device characteristics comprise one or more words associated with electronic devices that are controllable by the user device, or associated with one or more states corresponding to the electronic devices that are controllable by the user device.

4. The user device of claim 1 , wherein the data structure defines one or more actual device characteristics associated with each electronic device in the set of electronic devices.

5. The user device of claim 4 , wherein the one or more actual electronic device characteristics comprise one or more words associated with each electronic device in the set of electronic devices, or associated with one or more states corresponding to each electronic device in the set of electronic devices.

6. The user device of claim 1 , wherein the data structure defines a current state of each electronic device in the set of electronic devices.

7. The user device of claim 6 , wherein the one or more programs comprise further instructions for:
determining the user intent further based on the current state of each of the one or more candidate electronic devices.

8. The user device of claim 6 wherein the one or more programs comprise further instructions for:
after providing the command, receiving an updated state from the electronic device; and
updating the current state of the electronic device based on the updated state received from the electronic device.

9. The user device of claim 8 , wherein the one or more programs comprise further instructions for:
outputting an indication of a result of the command based on the updated state of the electronic device received from the electronic device.

10. The user device of claim 9 , wherein the indication of the result of the command is an audio indication, a visual indication, or both an audio and a visual indication.

11. The user device of claim 1 , wherein the one or more programs comprise further instructions for:
determining a location corresponding to the audio input; and
determining, based on the determined location, the predetermined location from a plurality of predetermined locations.

12. The user device of claim 1 , wherein determining the one or more candidate electronic devices from the set of electronic devices comprises:
determining one or more alternative terms associated with a term in the audio input, wherein the one or more candidate electronic devices is determined based on the one or more alternative terms.

13. The user device of claim 1 , wherein determining the user intent further comprises:
determining the electronic device from the one or more candidate electronic devices based on the one or more possible electronic device characteristics and the one or more actual electronic device characteristics of the one or more candidate electronic devices.

14. The user device of claim 1 , wherein the one or more programs comprise further instructions for:
determining a relative location of the user device with respect to an area within the predetermined location, wherein the user intent is determined based on the relative location.

15. A method for controlling an electronic device using a virtual assistant on a user device having one or more processors and a memory, the method comprising:
at the user device:
receiving an audio input representing a user request for an electronic device to perform an action,
wherein the audio input is in natural language form, and
wherein the audio input is ambiguous with respect to defining the electronic device for performing the action;

determining one or more possible electronic device characteristics corresponding to the audio input;
retrieving a data structure representing a set of electronic devices of a predetermined location;
determining, based on the data structure, one or more candidate electronic devices from the set of electronic devices, the one or more candidate electronic devices corresponding to the audio input;
determining, based on the one or more possible electronic device characteristics and one or more actual electronic device characteristics of the one or more candidate electronic devices, a user intent corresponding to the audio input; and
providing a command that causes an electronic device of the one or more candidate electronic devices to perform the action corresponding to the user intent.

16. The method of claim 15 , wherein the audio input is in natural language form, and wherein the audio input is ambiguous with respect to defining the action.

17. The method of claim 15 , wherein the data structure defines one or more actual device characteristics associated with each electronic device in the set of electronic devices.

18. The method of claim 17 , wherein the one or more actual electronic device characteristics comprise one or more words associated with each electronic device in the set of electronic devices, or associated with one or more states corresponding to each electronic device in the set of electronic devices.

19. The method of claim 15 , further comprising:
determining a location corresponding to the audio input; and
determining, based on the determined location, the predetermined location from a plurality of predetermined locations.

20. The method of claim 15 , wherein determining the one or more candidate electronic devices from the set of electronic devices comprises:
determining one or more alternative terms associated with a term in the audio input, wherein the one or more candidate electronic devices is determined based on the one or more alternative terms.

21. The method of claim 15 , further comprising:
determining a relative location of the user device with respect to an area within the predetermined location, wherein the user intent is determined based on the relative location.

22. A non-transitory computer-readable storage medium comprising instructions for one or more programs for execution by one or more proce

Assignees

• Apple Inc

Inventors

• Orr Ryan M
• Nell Garett R
• Brumbaugh Benjamin L

Classifications

• H04L12/282

  based on user interaction within the home (receiver circuitry for displaying additional information being controlled by a remote control apparatus H04N21/42204) · CPC title

• H04L12/2816

  Controlling appliance services of a home automation network by calling their functionalities (arrangements in telecontrol or telemetry systems for selectively calling a substation from a main station; in which substation desired apparatus is selected for applying a control signal thereto or for obtaining measured values therefrom H04Q9/00) · CPC title

• G10L17/22 Primary

  Interactive procedures; Man-machine interfaces · CPC title

• G05B2219/2642

  Domotique, domestic, home control, automation, smart house · CPC title

• G05B2219/23386

  Voice, vocal command or message · CPC title