Device and method with trained neural network to identify touch input

Patent metadata

Field	Value
Publication number	US-12045458-B2
Application number	US-202318112255-A
Country	US
Kind code	B2
Filing date	Feb 21, 2023
Priority date	Aug 21, 2020
Publication date	Jul 23, 2024
Grant date	Jul 23, 2024

Abstract

An electronic device includes a touch screen and a processor configured to: based on a touch input of a user being acquired through the touch screen, acquire an image corresponding to the acquired touch input of the user; identify a type of the acquired touch input of the user by inputting, to a neural network model for identifying the type of the touch input of the user, the acquired image, a first image corresponding to a first type touch input obtained by touching the touch screen with a pressure smaller than a preconfigured pressure, and a second image corresponding to a second type touch input obtained by touching the touch screen with a pressure greater than the preconfigured pressure; and perform a function based on the identified type of the touch input.

First claim

What is claimed is:

1. A control method of an electronic device, the control method comprising:
acquiring, for a plurality of fingers of a user, a plurality of first images corresponding to a first type touch input for touching a touch screen of the electronic device with a pressure less than a predetermined pressure, and a plurality of second images corresponding to a second type touch input for touching the touch screen with a pressure greater than the predetermined pressure;
storing the plurality of first images and the plurality of second images;
training a neural network model for identifying a type of a touch input of the user based on the plurality of first images and the plurality of second images, to obtain a trained neural network model;
based on acquiring an acquired touch input of the user through the touch screen, acquiring a plurality of acquired images corresponding to the acquired touch input of the user;
identifying a type of the acquired touch input of the user by inputting, to the trained neural network model, the plurality of acquired images, the plurality of first images, and the plurality of second images; and
performing a function corresponding to the acquired touch input of the user based on the type of the acquired touch input.

2. The control method of claim 1 , wherein the acquiring the plurality of first images and the plurality of second images comprises:
displaying, on the touch screen, a user interface (UI) element for guiding the touch input of the user corresponding to each of the first type touch input and the second type touch input; and
based on detecting the touch input of the user for touching the UI element, generating the plurality of first images and the plurality of second images.

3. The control method of claim 2 , wherein the displaying comprises displaying, on the touch screen, a first UI element for guiding the first type touch input and a second UI element for guiding the second type touch input, and
wherein a size of the second UI element is greater than a size of the first UI element.

4. The control method of claim 1 , further comprising:
based on a first function corresponding to the first type touch input being performed based on the acquired touch input of the user, allocating a first label corresponding to the first type touch input to the plurality of acquired images and storing the plurality of acquired images;
based on a second function corresponding to the second type touch input being performed based on the acquired touch input of the user, allocating a second label corresponding to the second type touch input to the plurality of acquired images and storing the plurality of acquired images; and
retraining the neural network model based on the plurality of acquired images to which the first label is allocated and the plurality of acquired images to which the second label is allocated.

5. The control method of claim 1 , further comprising:
acquiring a first similarity between the plurality of acquired images and the plurality of first images and a second similarity between the plurality of acquired images and the plurality of second images; and
acquiring a plurality of first input images having the first similarity of at least a predetermined value among the plurality of first images, and acquiring a plurality of second input images having the second similarity of at least the predetermined value among the plurality of second images,
wherein the identifying the type of the acquired touch input of the user comprises inputting, to the trained neural network model, the plurality of acquired images, the plurality of first input images, and the plurality of second input images.

6. The control method of claim 5 , wherein each of the first similarity and the second similarity is acquired based on at least one of an area, a shape, and a change over time of a region corresponding to the touch input included in each of the plurality of acquired images, the plurality of first images, and the plurality of second images.

7. The control method of claim 5 , further comprising retraining the neural network model based on the plurality of acquired images, the plurality of first images, and the plurality of second images.

8. The control method of claim 1 , wherein a second function corresponding to the second type touch input comprises a function of displaying at least one user interface (UI) element on the touch screen, and
wherein the at least one UI element comprises at least one of a first UI element for adjusting a volume of a speaker and a second UI element for powering off the electronic device.

9. The control method of claim 1 , wherein the identifying the type of the acquired touch input of the user comprises:
allocating the plurality of acquired images to a plurality of first channels, the plurality of first images to a plurality of second channels, and the plurality of second images to a plurality of third channels, respectively; and
acquiring input data for the trained neural network model by merging the plurality of first channels, the plurality of second channels, and the plurality of third channels with one another.

10. The control method of claim 1 , wherein the identifying the type of the acquired touch input of the user comprises:
acquiring a first probability that the acquired touch input of the user corresponds to the first type touch input, and a second probability that the acquired touch input of the user corresponds to the second type touch input by inputting, to the trained neural network model, the plurality of acquired images, the plurality of first images, and the plurality of second images, and
wherein the type of the acquired touch input of the user is identified as the first type touch input based on the first probability being greater than a predetermined probability, and the type of the acquired touch input of the user is identified as the second type touch input when the second probability being greater than the predetermined probability.

11. An electronic device comprising:
a touch screen;
at least one memory storing at least one instruction; and
at least one processor operatively connected to the at least one memory and configured to execute the at least one instruction to:
acquire, for a plurality of fingers of a user, a plurality of first images corresponding to a first type touch input for touching the touch screen with a pressure less than a predetermined pressure, and a plurality of second images corresponding to a second type touch input for touching the touch screen with a pressure greater than the predetermined pressure,
store the plurality of first images and the plurality of second images in the at least one memory,
train a neural network model for identifying a type of a touch input of the user based on the plurality of first images and the plurality of second images, to obtain a trained neural network model,
based on acquiring an acquired touch input of the user through the touch screen, acquire a plurality of acquired images corresponding to the acquired touch input of the user,
identify a type of the acquired touch input of the user by inputting, to the trained neural network model, the plurality of acquired images, the plurality of first images, and the plurality of second images, and
perform a function corresponding to the acquired touch input of the user based on the type of the acquired touch input.

12. The electronic device of claim 11 , wherein the at least one processor is further configured to execute the at least one instruction to:
display, on the touch screen, a user interface (UI) element for guiding the touch input of the user corresponding to e

Assignees

• Samsung Electronics Co Ltd

Inventors

• Kim Dongchan
• Hwang Jinyoung

Classifications

• G06N3/09

  Supervised learning · CPC title

• G06N3/0464

  Convolutional networks [CNN, ConvNet] · CPC title

• G06F2203/04808

  Several contacts: gestures triggering a specific function, e.g. scrolling, zooming, right-click, when the user establishes several contacts with the surface simultaneously; e.g. using several fingers or a combination of fingers and pen · CPC title

• G06N3/045

  Combinations of networks · CPC title

• G06N3/044

  Recurrent networks, e.g. Hopfield networks · CPC title