Control method of electronic device using a plurality of sensors and electronic device thereof

Patent metadata

Field	Value
Publication number	US-11630639-B2
Application number	US-202117511841-A
Country	US
Kind code	B2
Filing date	Oct 27, 2021
Priority date	Dec 8, 2020
Publication date	Apr 18, 2023
Grant date	Apr 18, 2023

Abstract

An electronic device and method are disclosed. The electronic device may include a display, a sensor module, a processor, and a memory operatively connected to the processor. The memory stores one or more instructions that, when executed, cause the processor implements the method, including: setting an initial value by obtaining inertial information including position information, tilt information, or movement information of the electronic device, distance information between the electronic device and a user, and angle information of a face of the user, through the sensor module, determining a first virtual coordinate system mapped onto a screen of the display, based on the initial value, displaying a cursor at a position on the screen that corresponds to coordinates of one of within the first virtual coordinate system, detecting a change in the distance information, the angle information, or the inertial information through the sensor module after setting the initial value, and correcting a second virtual coordinate system changed from the first virtual coordinate system, based on an amount of change in the first virtual coordinate system such that the second virtual coordinate system is mapped onto the screen.

First claim

The invention claimed is:

1. An electronic device, comprising:
a display;
a sensor module;
a processor operatively connected to the display and the sensor module; and
a memory operatively connected to the processor,
wherein the memory stores one or more instructions that, when executed, cause the processor to:
set an initial value by obtaining inertial information including at least one of position information, tilt information, or movement information of the electronic device, distance information between the electronic device and a user, and angle information of a face of the user, through the sensor module;
determine a first virtual coordinate system mapped onto a screen of the display, based on the initial value;
display a cursor at a position on the screen that corresponds to coordinates of one of the user's body parts within the first virtual coordinate system;
detect a change in one or more of the distance information, the angle information, or the inertial information through the sensor module after setting the initial value; and
correct a second virtual coordinate system changed from the first virtual coordinate system, based on an amount of change in the first virtual coordinate system such that the second virtual coordinate system is mapped onto the screen.

2. The electronic device of claim 1 , wherein the sensor module includes an image sensor, and
wherein the distance information or the angle information is obtained using the image sensor.

3. The electronic device of claim 1 , wherein the memory stores a plurality of virtual coordinate systems, each corresponding to different respective values for the distance information, the angle information, and the inertial information, and
wherein the instructions cause the processor to:
select the first virtual coordinate system from among the plurality of virtual coordinate systems based on the initial value.

4. The electronic device of claim 1 , wherein setting the initial value further includes:
displaying the cursor at a specified position on the screen,
determining a body part of the user as a focal point,
wherein the body part is a prespecified part, or set according to the distance information,

synchronizing the cursor and the focal point according to the determination that a position on the screen, corresponding to the focal point, is adjacent to the specified position of the cursor; and
moving the cursor displayed on the screen according to detected movement of the focal point.

5. The electronic device of claim 4 , wherein the focal point includes a midpoint between the centers of two pupils of the user, or a part of the user's face closest to the electronic device.

6. The electronic device of claim 4 , wherein the sensor module includes an image sensor, and
wherein the distance information is obtained by measuring a distance from the image sensor to the focal point.

7. The electronic device of claim 4 , wherein the instructions cause the processor to:
output a user interface prompting movement of the focal point such that the cursor traces a movement path along a periphery of the display; and
determine a two-dimensional coordinate system of the first virtual coordinate system, based on the movement path of the cursor.

8. The electronic device of claim 7 , wherein the two-dimensional coordinate system includes feature information, including an aspect ratio, lengths of four sides, and four angles, and
wherein the instructions cause the processor to:
correct the second virtual coordinate system by compensating for the feature information by an amount of change in the feature information.

9. The electronic device of claim 1 , wherein the amount of change in the first virtual coordinate system is determined based on the amount of detected change in the distance information, the angle information, or the inertial information.

10. The electronic device of claim 1 , wherein the instructions cause the processor to provide, to the user, a user interface configured to guide states of the electronic device and the user that correspond to the initial value, based on a change in the distance information, the angle information, or the inertial information.

11. The electronic device of claim 1 , wherein the electronic device further comprises a voice input module, and
wherein the instructions cause the processor to:
receive a first voice command requesting execution of a cursor mode through the voice input module, and display at least one cursor on the screen of the display in response to receiving the first voice command.

12. The electronic device of claim 11 , wherein the instructions cause the processor to receive a second voice command requesting a control of the at least one cursor through the voice input module, and adjust a size, an angle, or a position of the at least one cursor based on the received second voice command.

13. The electronic device of claim 11 , wherein the instructions cause the processor to:
display, on the screen of the display, a virtual movement map including a plurality of cells on which different identifiers are respectively mapped; and
receive a third voice command including designation of one identifier through the voice input module, and

based on the received third voice command, move a position of the at least one cursor to a cell onto which the designated identifier is mapped.

14. The electronic device of claim 13 , wherein the instructions cause the processor to:
receive a fourth voice command requesting expansion of the cell having the designated identifier mapped thereon, with the third voice command, through the voice input module; and
display, on the display, an enlarged screen representing the cell mapped to the designated identifier,
wherein the enlarged screen includes a plurality of sub-cells onto which local identifiers of the cell are mapped.

15. The electronic device of claim 11 , wherein the screen includes at least one object having an identifier mapped thereon, and
wherein the instructions cause the processor to:
receive a second voice command designating the identifier through the voice input module, and
move a position of the at least one cursor to be adjacent to the object having the designated identifier mapped thereon.

16. The electronic device of claim 11 , wherein the instructions cause the processor to:
display execution screens of a plurality of applications on the screen;
select an application from among the plurality of applications, based on a position of the at least one cursor; and
perform an operation corresponding to at least one voice command input through the voice input module for the selected application.

17. The electronic device of claim 1 , wherein the instructions cause the processor to:
track a gesture of a specific part of the user's body using the sensor module; and
perform a control command mapped to the gesture.

18. A method of an electronic device, comprising:
by at least one processor, setting an initial value by obtaining inertial information including at least one of position information, tilt information, or movement information of the electronic device, distance information between the electronic device and a user, and angle information of a face of the user through one or more sensors;
determining a first virtual coordinate system mapped onto a screen of a display, based on the initial value;
displaying a cursor at a position on the screen corresponds to coordinates of one of the user's body parts within the first virtual coordinate system;
after set

Assignees

• Samsung Electronics Co Ltd

Inventors

• Shin Sunghee
• Yeo Hyungsok
• Prushinskiy Valeriy

Classifications

• G06F3/013

  Eye tracking input arrangements (G06F3/015 takes precedence) · CPC title

• G06F3/167 Primary

  Audio in a user interface, e.g. using voice commands for navigating, audio feedback · CPC title

• G06F3/0346

  with detection of the device orientation or free movement in a three-dimensional [3D] space, e.g. 3D mice, 6-DOF [six degrees of freedom] pointers using gyroscopes, accelerometers or tilt-sensors · CPC title

• G06F3/017

  Gesture based interaction, e.g. based on a set of recognized hand gestures (interaction based on gestures traced on a digitiser G06F3/04883) · CPC title

• G06F3/012

  Head tracking input arrangements · CPC title