System and method for context-based interaction for electronic devices

Patent metadata

Field	Value
Publication number	US-11221823-B2
Application number	US-201715857301-A
Country	US
Kind code	B2
Filing date	Dec 28, 2017
Priority date	May 22, 2017
Publication date	Jan 11, 2022
Grant date	Jan 11, 2022

Abstract

A method includes receiving a voice input at an electronic device. An ambiguity of the voice input is determined. The ambiguity is resolved based on contextual data. The contextual data includes at least one of: an image, a non-voice input comprising a gesture, a pointer of a pointing device, a touch, or a combination thereof.

First claim

What is claimed is:

1. A method, comprising:
receiving a voice input at an electronic device;
determining an ambiguity of the voice input;
resolving, by the electronic device, the ambiguity based on contextual data and determining identification of an object that the voice input applies to, wherein the electronic device is configured for resolving the ambiguity within a display device of the electronic device and within display devices of respective other electronic devices, the contextual data includes a captured image by a camera and user history information, and the ambiguity relates to the identification of the object that the voice input applies to; and
correlating the contextual data with the voice input for determining, based on the identification of the object, a selection of content, and for performing an action based on content type of the selected content,
wherein a combination of objects and their relative sizes are detected in the captured image, context of the voice input are used for determining the object that the voice input is directed to, resulting properties are transferable to at least one other selected content and comprise content settings that are applied to the at least one other selected content, the electronic device adjusts an interface corresponding to the voice input with the ambiguity resolved, the ambiguity includes a demonstrative determiner, and the contextual data further includes information affecting the action applicable to the object.

2. The method of claim 1 , further comprising:
mapping the content with a set of action commands relevant to the content type for the selected content;
wherein the ambiguity relates to identification of a position that the voice input applies to, the contextual data further comprises a non-voice input that includes a non-touch hand gesture, the selection of the content comprises determining a selection for the content from a multi-hierarchical menu including a plurality of content, and the interface filters and reconfigures itself, using dynamic refinement, to elements that fit within a context of an executed action command from the set of action commands.

3. The method of claim 2 , further comprising:
determining, by the electronic device, whether the voice input matches an action command from the set of action commands; and
selecting an app from a plurality of apps for performing the action;

wherein:
the app supports performing the action on the content;
the camera is one of coupled to the electronic device or attached to the display device that is coupled to the electronic device; and
the non-voice input further comprises at least one of: a pointer of a pointing device, a touch gesture, or a combination thereof.

4. The method of claim 3 , wherein the non-voice input is sensed by at least one sensor coupled to the electronic device, and the contextual data further comprises locally available items for a user and user location information.

5. The method of claim 3 , wherein the determining of the identification of the object is based on the captured image containing the object or the non-voice input indicating the object.

6. The method of claim 3 , further comprising:
using output from the camera upon a determination that no non-voice input occurred with the voice input.

7. The method of claim 2 , wherein the object is the electronic device.

8. The method of claim 1 , wherein the action applicable to the object comprises one of: receiving information, assisting with a purchase, calendaring an event, applying features to content, selecting at least one content associated with the object, moving the object on the display device of the electronic device, moving the object on a particular display device of the display devices of the other electronic devices, or a combination thereof.

9. An electronic device comprising:
a memory storing instructions; and
at least one processor executing the instructions including a process configured to:
receive a voice input;
determine an ambiguity of the voice input;
resolve the ambiguity based on contextual data and determining identification of an object that the voice input applies to, wherein the electronic device is configured for resolving the ambiguity within a display of the electronic device and within displays of respective other electronic devices, the contextual data includes a captured image by a camera and user history information, and the ambiguity relates to the identification of the object that the voice input applies to; and
correlate the contextual data with the voice input for determining, based on the identification of the object, a selection of content, and for performing an action based on content type of the selected content,

wherein a combination of objects and their relative sizes detected in the captured image, context of the voice input are used for determining the object that the voice input is directed to, resulting properties are transferable to at least one other selected content and comprise content settings that are applied to the at least one other selected content, the electronic device adjusts an interface corresponding to the voice input with the ambiguity resolved, and the object is the electronic device.

10. The electronic device of claim 9 , wherein:
the process is further configured to map the content with a set of action commands relevant to the content type for the selected content;
the ambiguity relates to identification of a position that the voice input applies to;
the contextual data further comprises a non-voice input that includes a non-touch hand gesture;
the camera is one of coupled to the electronic device or attached to the display device that is coupled to the electronic device;
the selection of the content comprises determining a selection for the content from a multi-hierarchical menu including a plurality of content; and
the interface filters and reconfigures itself, using dynamic refinement, to elements that fit within a context of an executed action command from the set of action commands.

11. The electronic device of claim 10 , wherein:
the process is further configured to:
determine whether the voice input matches an action command from the set of action commands; and
select an app from a plurality of apps for performing the action;

the app supports performing the action on the content;
the non-voice input is sensed by at least one sensor coupled to the electronic device;
the non-voice input further comprises at least one of: a pointer of a pointing device, a touch gesture, or a combination thereof; and
the contextual data further comprises locally available items for a user and user location information.

12. The electronic device of claim 11 , wherein:
the determining of the identification of the object is based on the captured image containing the object or the non-voice input indicating the object;
the ambiguity includes a demonstrative determiner; and
the contextual data further includes information affecting the action applicable to the object.

13. The electronic device of claim 11 , wherein:
the process is further configured to:
using output from the camera upon a determination that no non-voice input occurred with the voice input.

14. The electronic device of claim 12 , wherein the action applicable to the object comprises one of: receiving information, assisting with a purchase, calendaring an event, applying features to content, selecting at least one content associated with the object, moving the object on the display device of the electronic device, moving the object on a particular display device of

Assignees

• Samsung Electronics Co Ltd

Inventors

• Yoganandan Arun Rakesh
• Akiyoshi Kumi
• Mauk Tais C
• ZHU JIN Chang Long
• Hong Jung Won

Classifications

• G10L15/08

  Speech classification or search · CPC title

• G06F3/0346

  with detection of the device orientation or free movement in a three-dimensional [3D] space, e.g. 3D mice, 6-DOF [six degrees of freedom] pointers using gyroscopes, accelerometers or tilt-sensors · CPC title

• G06F2203/0381

  Multimodal input, i.e. interface arrangements enabling the user to issue commands by simultaneous use of input devices of different nature, e.g. voice plus gesture on digitizer · CPC title

• G06F3/04842

  Selection of displayed objects or displayed text elements (G06F3/0482 takes precedence) · CPC title

• G06F3/0304

  Detection arrangements using opto-electronic means (constructional details of pointing devices not related to the detection arrangement using opto-electronic means G06F3/033; optical digitisers G06F3/042) · CPC title