Connecting assistant device to devices

Patent metadata

Field	Value
Publication number	US-10353480-B2
Application number	US-201715604325-A
Country	US
Kind code	B2
Filing date	May 24, 2017
Priority date	Apr 17, 2017
Publication date	Jul 16, 2019
Grant date	Jul 16, 2019

Abstract

The present disclosure contemplates a variety of improved methods and systems for enabling set up of a variety of disparate IoT devices coupled to the framework and/or ambient operating system of an assistant device. The described solution includes a voice-driven assistant device setup process. An assistant device can determine the IoT devices in its physical environment and provide setup instructions to the user. The setup including determining voice activatable commands, device information and the adapter to allow the assistant device to operate the one or more IoT devices in response to user instructions.

First claim

We claim:

1. An assistant device, comprising:
a display screen;
a microphone;
a speaker;
a camera;
one or more processors; and
memory storing instructions, wherein the one or more processors are configured to execute instructions such that the one or more processors and memory are configured to:
receive a first image data depicting one or more devices in an environment of the assistant device from the camera;
identify the one or more devices depicted in the first image data using a visual recognition algorithm;
determine that the one or more devices in the received first image data were not previously connected to the assistant device;
determine device information representing identifying information associated with the one or more devices not previously connected to the assistant device;
determine a device type of the one or more devices based on the device information using one or more of local resources and/or one or more of cloud servers;
determine first audio installation instructions for the one or more devices based on the device type, the first audio installation instructions providing steps to install the one or more devices;
provide the first audio installation instructions via the speaker;
receive a second image data depicting a user performing the first audio installation instructions;
determine characteristics of the user performing the first audio installation instructions based on the second image data;
determine second audio installation instructions for the one or more devices based on the device type and the characteristics of the user, the second audio installation instructions providing the steps to install the one or more devices differently than the providing of the steps by the first audio installation instructions;
provide the second audio installation instructions via the speaker;
receive a user response via the microphone and the camera;
determine descriptive information associated with the one or more devices based on the user response;
set up the assistant device to operate the one or more devices, the set up including determining one or more of voice activatable commands and an adapter associated with the one or more devices;
receive a user instruction, wherein speech recognition of the user instruction includes one or more voice activatable commands and one or more descriptive information;
identify the one or more devices associated with the descriptive information; and
transmit a request to the one or more devices to perform an action associated with the one or more voice activatable commands.

2. The assistant device of claim 1 , wherein determining the device type includes analyzing an image frame of the image using the visual recognition algorithm.

3. The assistant device of claim 2 , comprising determination a location of the one or more devices by analyzing the image frame of the image.

4. The assistant device of claim 1 , wherein determining characteristics of the user performing the first audio installation instructions includes determining physical movements of the user while performing the first audio installation instructions.

5. The assistant device of claim 1 , wherein providing the steps to install the one or more devices differently than the providing of the steps by the first audio installation instructions includes providing the second audio installation instructions at a slower speed than the first audio installation instructions.

6. A method, comprising:
receiving a first image data depicting one or more devices in an environment of an assistant device from a camera;
identifying the one or more devices depicted in the first image data using a visual recognition algorithm;
determining that the one or more devices in the received first image data were not previously connected to the assistant device;
determining device information representing identifying information associated with the one or more devices not previously connected to the assistant device;
determining a device type of the one or more devices based on the device information using one or more of local resources or one or more of cloud servers;
determining first audio installation instructions for the one or more devices based on the device type, the first audio installation instructions providing steps to install the one or more devices;
providing the first audio installation instructions via a speaker;
receiving a second image data depicting a user performing the first audio installation instructions;
determining characteristics of the user performing the first audio installation instructions based on the second image data;
determining second audio installation instructions for the one or more devices based on the device type and the characteristics of the user, the second audio installation instructions providing the steps to install the one or more devices differently than the providing of the steps by the first audio installation instructions;
providing the second audio installation instructions via the speaker;
receiving a user response via a microphone and the camera;
determining descriptive information associated with the one or more devices based on the user response;
setting up the assistant device to operate the one or more devices, the set up including determining one or more of voice activatable commands and an adapter associated with the one or more devices;
receiving a user instruction, wherein speech recognition of the user instruction includes one or more voice activatable commands and one or more descriptive information;
identifying the one or more devices associated with the descriptive information; and
transmitting a request to the one or more devices to perform an action associated with the one or more voice activatable commands.

7. The method of claim 6 , wherein determining the device type includes analyzing an image frame of the image using the visual recognition algorithm.

8. The method of claim 7 , comprising determination a location of the one or more devices by analyzing the image frame of the image.

9. The method of claim 6 , wherein determining characteristics of the user performing the first audio installation instructions includes determining physical movements of the user while performing the first audio installation instructions.

10. The method of claim 6 , wherein providing the steps to install the one or more devices differently than the providing of the steps by the first audio installation instructions includes providing the second audio installation instructions at a slower speed than the first audio installation instructions.

11. A computer program product including one or more non-transitory computer-readable media storing computer program instructions, execution of which by a processor causes a processing system to perform operations comprising:
receiving a first image data depicting one or more devices in an environment of an assistant device from a camera;
identifying the one or more devices depicted in the first image data using a visual recognition algorithm;
determining that the one or more devices in the received first image data were not previously connected to the assistant device;
determining device information representing identifying information associated with the one or more devices not previously connected to the assistant device;
determining a device type of the one or more devices based on the device information using one or more of local resources or one or more of cloud servers;
determining first audio installation instructions for the one or more devices based on the device type, the first audio installation instructions providing steps to install the one or more devices;

Assignees

• Essential Products Inc

Inventors

• Roman Manuel
• Segal Mara Clair
• Desai Dwipal
• Rubin Andrew E

Classifications

• G06F3/017 Primary

  Gesture based interaction, e.g. based on a set of recognized hand gestures (interaction based on gestures traced on a digitiser G06F3/04883) · CPC title

• H04W4/70

  Services for machine-to-machine communication [M2M] or machine type communication [MTC] · CPC title

• G10L2015/226

  using non-speech characteristics · CPC title

• H04L67/306

  User profiles · CPC title

• H04L12/2803

  Home automation networks · CPC title