System and method for generating machine-curated scenes

Patent metadata

Field	Value
Publication number	US-10380493-B2
Application number	US-201715604226-A
Country	US
Kind code	B2
Filing date	May 24, 2017
Priority date	Apr 17, 2017
Publication date	Aug 13, 2019
Grant date	Aug 13, 2019

Abstract

The present disclosure contemplates a variety of improved methods and systems for creating a unique user experience using an ambient operating system connected to a variety of disparate IoT devices. The described solution includes a curated scene or set of actions initiated by an assistant device. For example, the assistant device detects it is a weekday morning and initiates actions associated with the wakeup routine such as opening the blinds, making coffee and notifying the user of the current traffic report.

First claim

We claim:

1. A method for automatically generating a scene for an assistant device to cause one or more devices to perform functionality automatically within an environment, comprising:
receiving, by a processor, an image frame depicting a user interaction with a first device of one or more devices within the environment, and receiving audio information indicating the user interaction with the first device of the one or more devices within the environment, the one or more devices communicatively coupled with the assistant device via a wireless local area network (WLAN);
using the image frame and the audio information to identify the first device of the one or more devices with which a user is interacting;
using the image frame or the audio information identifying the functionality the user interaction causes the first device to perform;
identifying a previous user interaction of the user causing a second device of the one or more devices to perform a second functionality which is related to the user interaction with the first device;
determining using fuzzy matching that the previous user interaction with the second device and the user interaction with the first device matches a scene representing a standardized set of user interactions with the one or more devices in the environment;
generating the scene associated with the scene template, and comprising a trigger representing an instruction for initiating the scene, and causing the first device, to perform the functionality associated with the user interaction, and the second device to perform the second functionality associated with the previous user interaction; and
receiving a second image frame or a second audio information indicating the user providing the trigger associated with the scene;
using the second image frame or the second audio information to identify the trigger associated with the scene;
performing automatically the functionality of the first device associated with the user interaction and the second functionality of the second device associated with the previous user interaction associated with the scene responsive to receiving the trigger.

2. A method, comprising:
receiving, by a processor, an image frame depicting a user interaction with a first device of one or more devices within the environment, and receiving audio information indicating the user interaction with the first device of the one or more devices within an environment, the one or more devices communicatively coupled with an assistant device via a wireless local area network (WLAN);
using the image frame and the audio information to identify the first device of the one or more devices with which a user is interacting;
using the image frame or the audio information to identify a functionality the user interaction causes the first device to perform;
identifying a previous user interaction of the user causing a second device of the one or more devices to perform a second functionality which is related to the user interaction with the first device;
determining a scene template by matching the scene template to the previous user interaction with the second device and the user interaction with the first device to a template associated with the scene, the first device, the functionality associated with the user interaction, the second device, and the second functionality associated with the previous user interaction;
generating the scene associated with the scene template, and comprising a trigger representing an instruction for initiating the scene causing the first device, to perform the functionality associated with the user interaction, and the second device to perform the second functionality associated with the previous user;
receiving the trigger representing the instruction for initiating the scene;
and
performing the functionality of the first device associated with the user interaction and the second functionality of the second device associated with the previous user interaction responsive to receiving the trigger associated with the scene.

3. The method of claim 2 , wherein the scene template is determined using a fuzzy matching algorithm.

4. The method of claim 3 , wherein initiating the scene causes a third device to perform a third functionality.

5. The method of claim 4 , comprises receiving configuration information from a user; and wherein generating the scene includes the configuration information.

6. The method of claim 5 , wherein the one or more devices are connected on a wireless network with the assistant device.

7. The method of claim 6 , wherein one or more users are identified in the image frame.

8. The method of claim 7 , wherein generating the scene includes associating the trigger with the one or more users who have authority to cause the scene to be performed.

9. The method of claim 8 , wherein a location of the first device is determined using the image frame, and the location of the first device is used to determine the scene template.

10. An electronic device, comprising:
one or more processors;
memory storing instructions, wherein the processor is configured to execute the instructions such that the processor and memory are configured to:
receive an image frame depicting a user interaction with a first device of one or more devices within the environment, and receiving audio information indicating the user interaction with the first device of the one or more devices within an environment, the one or more devices communicatively coupled with an assistant device via a wireless local area network (WLAN);
use the image frame and the audio information to identify the first device of the one or more devices with which a user is interacting;
identify using the image frame or the audio information a functionality which the user interaction causes the first device to perform;
identify a previous user interaction of the user causing a second device of the one or more devices to perform a second functionality which is related to the user interaction with the first device;
determine a scene template by matching the scene template to the previous user interaction with the second device and the user interaction with the first device to a template associated with the scene, the first device, the functionality associated with the user interaction, the second device, and the second functionality associated with the previous user interaction;
generate the scene associated with the scene template, and comprising a trigger representing an instruction for initiating the scene causing the first device, to perform the functionality associated with the user interaction, and the second device to perform the second functionality associated with the previous user interaction;
receiving the trigger representing the instruction for initiating the scene; and
performing the functionality of the first device associated with the user interaction and the second functionality of the second device associated with the previous user interaction responsive to receiving the trigger associated with the scene.

11. The electronic device of claim 10 , wherein the scene template is determining using a fuzzy matching algorithm.

12. The electronic device of claim 11 , wherein initiating the scene includes causing a third device to perform a third functionality.

13. The electronic device of claim 12 , further comprising the processor and memory are configured to receive configuration information from a user and generate the scene including the configuration information.

14. The electronic device of claim 13 , wherein the one or more devices are connected on a wireless network with the assistant d

Assignees

• Essential Products Inc

Inventors

• Segal Mara Clair
• Roman Manuel
• Desai Dwipal
• Rubin Andrew E

Classifications

• G06N5/048

  Fuzzy inferencing · CPC title

• H04L67/22

  Electricity · mapped topic

• H04L67/12

  specially adapted for proprietary or special-purpose networking environments, e.g. medical networks, sensor networks, networks in vehicles or remote metering networks · CPC title

• G06N7/023 Primary

  Learning or tuning the parameters of a fuzzy system · CPC title

• H04W84/12

  WLAN [Wireless Local Area Networks] · CPC title