Electronic device for recognizing gesture of user using a plurality of sensor signals

Patent metadata

Field	Value
Publication number	US-11790698-B2
Application number	US-202017084410-A
Country	US
Kind code	B2
Filing date	Oct 29, 2020
Priority date	Nov 1, 2019
Publication date	Oct 17, 2023
Grant date	Oct 17, 2023

Abstract

Disclosed is an electronic device. The electronic device includes a sensor unit that includes a first sensor and a second sensor, and a processor that is operatively connected with the sensor unit. The processor determines whether a given condition is satisfied, by using a first sensor signal of a user sensed by the first sensor, controls the first sensor to sense the first sensor signal every first cycle and controls the second sensor to sense a second sensor signal from the user every second cycle longer than the first cycle when the given condition is not satisfied, controls the second sensor to sense the second sensor signal every third cycle shorter than the second cycle when the given condition is satisfied, and recognize a gesture of the user by using the first sensor signal sensed every first cycle and the second sensor signal sensed every third cycle.

First claim

What is claimed is:

1. An electronic device comprising:
a housing;
a display disposed in at least a portion of the housing;
a sensor unit including a first sensor and a second sensor; and
a processor operatively connected with the sensor unit,
wherein the processor is configured to:
determine whether to change a screen of the display from an off state to an on state, by using a first sensor signal of a user sensed by the first sensor;
while the screen of the display maintains the off state, control the first sensor to sense the first sensor signal at a first cycle and control the second sensor to sense a second sensor signal from the user at a second cycle longer than the first cycle of the first sensor;
in response to identifying that the screen of the display changes to the on state, control the second sensor to sense the second sensor signal at a third cycle shorter than the second cycle;
identify a start point of a gesture signal interval for the first sensor signal and a start point of a gesture signal interval for the second sensor signal, wherein the start point of the gesture signal interval for the first sensor signal is different from the start point of the gesture signal interval for the second sensor signal;
synchronize the first sensor signal and the second sensor signal temporally in a manner that start points of gesture signal intervals respectively included in the first sensor signal and the second sensor signal are identical;
sense a pose of the user by using at least one of the first sensor signal and the second sensor signal;
detect correction-required signal intervals respectively included in the first sensor signal and the second sensor signal based on amplitudes of the first sensor signal and the second sensor signal, depending on the sensed pose of the user; and
exclude the correction-required signal intervals respectively detected in the first sensor signal and the second sensor signal from a gesture recognition target; and
recognize a gesture of the user by using the first sensor signal sensed every first cycle and the second sensor signal sensed every third cycle.

2. The electronic device of claim 1 , wherein the processor is further configured to:
while the screen of the display maintains the off state, recognize the gesture of the user by using only the first sensor signal.

3. The electronic device of claim 1 , wherein the processor is further configured to:
when the screen of the display maintains the on state, maintain the second sensor in a state of sensing the second sensor signal every third cycle, and
at a time when the screen of the display changes from the on state to the off state, again change a cycle, at which the second sensor senses the second sensor signal, from the third cycle to the second cycle.

4. The electronic device of claim 1 , wherein the sensor unit includes at least one of an acceleration sensor, a photoplethysmography (PPG) sensor, a gyro sensor, a geomagnetic sensor, and an electrocardiogram sensor.

5. The electronic device of claim 1 , wherein:
at least one of the first sensor signal and the second sensor signal includes a heartbeat signal of the user, and
the processor is further configured to filter the heartbeat signal to remove a DC component indicating a linear characteristic of the heartbeat signal.

6. The electronic device of claim 1 , wherein the processor is further configured to:
sense a pose of the user by using at least one of the first sensor signal and the second sensor signal;
detect correction-required signal intervals respectively included in the first sensor signal and the second sensor signal based on amplitudes of the first sensor signal and the second sensor signal, depending on the sensed pose of the user; and
perform correction on the correction-required signal intervals detected.

7. The electronic device of claim 1 , wherein the processor is further configured to:
determine whether it is possible to recognize the gesture, with regard to each of the first sensor signal and the second sensor signal;
determine a validity for recognition of the gesture with respect to each of the first sensor signal and the second sensor signal, based on whether it is possible to recognize the gesture; and
exclude a signal, in which the determined validity is less than a reference value, from among the first sensor signal and the second sensor signal from a gesture recognition target.

8. The electronic device of claim 7 , wherein the processor is further configured to determine whether it is possible to recognize the gesture, by using at least one of a signal-to-noise ratio (SNR), a result of analyzing patterns of the first sensor signal and the second sensor signal, and a result of comparing the first sensor signal and the second sensor signal.

9. The electronic device of claim 7 , wherein:
at least one of the first sensor signal and the second sensor signal includes an acceleration signal of the user, and
the processor is further configured to segment the acceleration signal based on an x-axis, a y-axis, and a z-axis and determines the validity of each of the segmented acceleration signals, independently.

10. The electronic device of claim 1 , wherein the processor is further configured to independently set end points of the gesture signal intervals respectively included in the first sensor signal and the second sensor signal.

11. The electronic device of claim 1 , wherein the processor is further configured to:
select feature value lists respectively corresponding to the first sensor and the second sensor from among feature value lists stored in advance, and
extract feature values associated with the gesture of the user from the first sensor signal and the second sensor signal by using the selected feature value lists.

12. The electronic device of claim 11 , wherein the processor is further configured to:
input the extracted feature values to a gesture classification model learned in advance to recognize the gesture of the user, and
when the extracted feature values are not values not learned by the gesture classification model, output a guidance message indicating a fail in recognition of the gesture of the user.

13. The electronic device of claim 1 , wherein the processor is paired with at least one external electronic device.

14. The electronic device of claim 13 , wherein, when the processor is paired with the at least one external electronic device, the processor is further configured to perform at least one of operations for recognizing the gesture of the user by using the at least one external electronic device paired.

15. The electronic device of claim 13 , wherein, when the processor is paired with the at least one external electronic device, the processor is further configured to set a gesture set for recognizing the gesture of the user depending on a kind of the at least one external electronic device paired.

16. The electronic device of claim 13 , wherein:
at least one of the first sensor signal and the second sensor signal includes an acceleration signal,
the at least one external electronic device is an in-vehicle infotainment system, and
when variations in the acceleration signal exceed a threshold value in a state of being paired with the in-vehicle infotainment system, the processor is further configured to exclude the acceleration signal from a gesture recognition target.

17. The electronic device of claim 1 , wherein:
the sensor unit further includes a position sensor configured to sense position information of the user, and
the processor is fur

Assignees

• Samsung Electronics Co Ltd

Inventors

• Park Sungjin
• Kim Cheolo
• Kim Taeyoon
• Cho Wonjoon
• Ahn Jinyoup
• Lim Chaiman

Classifications

• G06V40/20 Primary

  Movements or behaviour, e.g. gesture recognition (recognition of facial expressions G06V40/16) · CPC title

• G06F3/015

  Input arrangements based on nervous system activity detection, e.g. brain waves [EEG] detection, electromyograms [EMG] detection, electrodermal response detection · CPC title

• G06F3/017 Primary

  Gesture based interaction, e.g. based on a set of recognized hand gestures (interaction based on gestures traced on a digitiser G06F3/04883) · CPC title

• G06F18/213

  Feature extraction, e.g. by transforming the feature space; Summarisation; Mappings, e.g. subspace methods · CPC title

• G06F18/241

  relating to the classification model, e.g. parametric or non-parametric approaches · CPC title