Troubleshooting voice-enabled home setup

Patent metadata

Field	Value
Publication number	US-10212040-B2
Application number	US-201815881052-A
Country	US
Kind code	B2
Filing date	Jan 26, 2018
Priority date	Apr 17, 2017
Publication date	Feb 19, 2019
Grant date	Feb 19, 2019

Abstract

A home assistant device assisting with the setup and troubleshooting of devices is described. An assistant device can determine setup instructions for devices. The setup instructions for one device can be determined the technical aptitude of the user and assistant the user with setting up devices within their own technical abilities. The setup instructions can then be provided.

First claim

We claim:

1. A method for a home assistant device with artificial intelligence (AI) to facilitate setup of a device within a home environment, comprising:
determining, by a processor, that the device is to be set up within the home environment;
receiving a first image data depicting a user in the home environment;
receiving a first audio data representing speech of the user in the home environment;
determining a technical aptitude of the user related to setting up the device based on the first image data and the first image data;
identifying a set of instructions for setting up the device within the home environment based on the device and the technical aptitude, wherein the set of instructions include a first setup instruction and a second setup instruction representing a corresponding sequence of steps to be performed to set up the device to operate within the home environment;
providing the first setup instruction of the set of instructions to the user representing steps to be performed by the user to set up the device;
receiving a second image data depicting the user in the home environment representing the user performing the first setup instruction of the set of instructions;
receiving a second audio data representing speech of the user associated with performing the first setup instruction of the set of instructions;
determining that the first setup instruction is completed;
updating the technical aptitude of the user based on the second image data and the second audio data, wherein updating includes:
if the second image data and the second audio data indicates frustration with the first setup instruction being too difficult, setting the technical aptitude of the user to a novice technical aptitude;
if the second image data and the second audio data indicates no frustration with the first setup instruction, setting the technical aptitude of the user to an average technical aptitude; and
if the second image data and the second audio data indicates frustration with the first setup instruction being too easy, setting the technical aptitude of the user to an advanced technical aptitude;

providing the second setup instruction of the set of instructions to the user representing steps to be performed by the user to set up the device, the second setup instruction being associated with the set technical aptitude, wherein each the novice technical aptitude, the advanced technical aptitude, and the average technical aptitude are associated with different second setup instructions; and
determining that the second setup instruction is completed and the device is connected to the home assistant device resulting in the home assistant device being configured to control the device to provide a response to voice instructions receive by the home assistant device.

2. The method of claim 1 , wherein the Second image data and the second audio data indicate frustration associated with the user performing the first setup instruction.

3. The method of claim 2 , wherein the second setup instruction is associated with the novice technical aptitude, and is provided to the user at a slower speed than the second setup instruction associated with the average technical aptitude.

4. The method of claim 2 , wherein frustration is determined by identifying specific words representing frustration, or muttering in the second audio data.

5. The method of claim 2 , wherein frustration is determined by identifying hand waving, or face rubbing in the second image data.

6. The method of claim 2 , wherein frustration is determined by identifying specific words representing frustration in the second image data, and hand waving in the second audio data.

7. The method of claim 2 , wherein frustration is determined by identifying a quick physical movement, or a pattern of movements in the second image data.

8. The method of claim 1 , wherein the technical aptitude of the user is based on the first image data depicting the user interacting with the home assistant device, and the first audio data representing the user communicating with the home assistant device.

9. A system comprising:
a processor; and
a memory storing instructions, execution of which by the processor causes the system to:
determine that a device is to be set up within an environment;
receive a first image data depicting a user in the environment;
receive a first audio data representing speech of the user in the environment;
determine a technical aptitude of the user related to setting up the device based on the first image data and the first image data;
identify a set of instructions for setting up the device within the environment based on the device and the technical aptitude, wherein the set of instructions include a first setup instruction and a second setup instruction representing a corresponding sequence of steps to be performed to set up the device to operate within the environment;
provide the first setup instruction of the set of instructions to the user representing steps to be performed by the user to set up the device;
receive a second image data depicting the user in the environment representing the user performing the first setup instruction of the set of instructions;
receive a second audio data representing speech of the user associated with performing the first setup instruction of the set of instructions;
determine that the first setup instruction is completed;
update the technical aptitude of the user based on the second image data and the second audio data, wherein updating includes:
if the second image data and the second audio data indicates frustration with the first setup instruction being too difficult, setting the technical aptitude of the user to a novice technical aptitude;
if the second image data and the second audio data indicates no frustration with the first setup instruction, setting the technical aptitude of the user to an average technical aptitude; and
if the second image data and the second audio data indicates frustration with the first setup instruction being too easy, setting the technical aptitude of the user to an advanced technical aptitude;
provide the second setup instruction of the set of instructions to the user representing steps to be performed by the user to set up the device, the second setup instruction being associated with the set technical aptitude, wherein each the novice technical aptitude, the advanced technical aptitude, and the average technical aptitude are associated with different second setup instructions; and
determine that the second setup instruction is completed and the device is connected to an assistant device resulting in the assistant device being configured to control the device to provide a response to voice instructions received by the assistant device.

10. The system of claim 9 , wherein the second image data and the second audio data indicate frustration associated with the user performing the first setup instruction.

11. The system of claim 10 , wherein the second setup instruction is associated with the novice technical aptitude, and is provided to the user at a slower speed than the second setup instruction associated with the average technical aptitude.

12. The system of claim 10 , wherein frustration is determined by identifying specific words representing frustration, or muttering in the second audio data.

13. The system of claim 10 , wherein frustration is determined by identifying hand waving, or face rubbing in the second image data.

14. The system of claim 10 , wherein frustration is determined by identifying specific words representing frustration in the second image data, and hand wa

Assignees

• Essential Products Inc

Inventors

• Roman Manuel
• Segal Mara Clair
• Desai Dwipal
• Rubin Andrew E

Classifications

• H04L12/282 Primary

  based on user interaction within the home (receiver circuitry for displaying additional information being controlled by a remote control apparatus H04N21/42204) · CPC title

• G06K9/00315

  Physics · mapped topic

• H04L41/0879 Primary

  Manual configuration through operator · CPC title

• G06K9/00288

  Physics · mapped topic

• G10L15/183

  using context dependencies, e.g. language models · CPC title