Device and method with trained neural network to identify touch input

Patent metadata

Field	Value
Publication number	US-12596471-B2
Application number	US-202418746420-A
Country	US
Kind code	B2
Filing date	Jun 18, 2024
Priority date	Aug 21, 2020
Publication date	Apr 7, 2026
Grant date	Apr 7, 2026

Abstract

An electronic device includes a touch screen and a processor configured to: based on a touch input of a user being acquired through the touch screen, acquire an image corresponding to the acquired touch input of the user; identify a type of the acquired touch input of the user by inputting, to a neural network model for identifying the type of the touch input of the user, the acquired image, a first image corresponding to a first type touch input obtained by touching the touch screen with a pressure smaller than a preconfigured pressure, and a second image corresponding to a second type touch input obtained by touching the touch screen with a pressure greater than the preconfigured pressure; and perform a function based on the identified type of the touch input.

First claim

What is claimed is:

1 . A control method of an electronic device, the control method comprising:
acquiring, for a plurality of fingers of a user, a plurality of first images corresponding to a first type touch input for touching a touch screen of the electronic device with a pressure less than a predetermined pressure, and a plurality of second images corresponding to a second type touch input for touching the touch screen with a pressure greater than the predetermined pressure, the second type touch input having a greater area of a region corresponding to a touch input than the first type touch input, wherein the acquiring the plurality of first images and the plurality of second images comprises:
displaying, on the touch screen, a first user interface (UI) element for guiding the first type touch input and a second UI element for guiding the second type touch input, wherein a size of the second UI element is greater than a size of the first UI element;
based on detecting a user touch input for touching the first UI element, generating the plurality of first images; and
based on detecting the user touch input for touching the second UI element, generating the plurality of second images;
training a neural network model for identifying a type of the user touch input of the user based on the plurality of first images and the plurality of second images to obtain a trained neural network model;
based on acquiring the user touch input through the touch screen, acquiring a plurality of input images corresponding to the user touch input;
identifying the type of the user touch input by inputting, to the trained neural network model, the plurality of input images, the plurality of first images, and the plurality of second images;
based on the type of the user touch input being the first type touch input, performing a first function; and
based on the type of the user touch input being the second type touch input, performing a second function different from the first function.

2 . The control method of claim 1 , further comprising:
based on the first function corresponding to the first type touch input being performed based on the user touch input, allocating a first label corresponding to the first type touch input to the plurality of input images and storing the plurality of input images; based on the second function corresponding to the second type touch input being performed based on the user touch input, allocating a second label corresponding to the second type touch input to the plurality of input images and storing the plurality of input images; and retraining the neural network model based on the plurality of input images to which the first label is allocated and the plurality of input images to which the second label is allocated.

3 . The control method of claim 1 , further comprising:
acquiring a first similarity between the plurality of input images and the plurality of first images and a second similarity between the plurality of input images and the plurality of second images; and acquiring a plurality of first input images having the first similarity of at least a predetermined value among the plurality of first images, and acquiring a plurality of second input images having the second similarity of at least the predetermined value among the plurality of second images, wherein the identifying the type of the user touch input comprises inputting, to the trained neural network model, the plurality of input images, the plurality of first input images, and the plurality of second input images.

4 . The control method of claim 3 , wherein each of the first similarity and the second similarity is acquired based on at least one of the area, a shape, and a change over time of the region corresponding to the touch input included in each of the plurality of input images, the plurality of first images, and the plurality of second images.

5 . The control method of claim 3 , further comprising retraining the neural network model based on the plurality of input images, the plurality of first images, and the plurality of second images.

6 . The control method of claim 1 , wherein the second function corresponding to the second type touch input comprises a function of displaying at least one user interface (UI) element on the touch screen, and
wherein the at least one UI element comprises at least one of a first function UI element for adjusting a volume of a speaker and a second function UI element for powering off the electronic device.

7 . The control method of claim 1 , wherein the identifying the type of the user touch input comprises:
allocating the plurality of input images to a plurality of first channels, the plurality of first images to a plurality of second channels, and the plurality of second images to a plurality of third channels, respectively; and acquiring input data for the trained neural network model by merging the plurality of first channels, the plurality of second channels, and the plurality of third channels with one another.

8 . The control method of claim 1 , wherein the identifying the type of the user touch input comprises:
acquiring a first probability that the user touch input corresponds to the first type touch input, and a second probability that the user touch input corresponds to the second type touch input by inputting, to the trained neural network model, the plurality of input images, the plurality of first images, and the plurality of second images, and wherein the type of the user touch input is identified as the first type touch input based on the first probability being greater than a predetermined probability, and the type of the user touch input is identified as the second type touch input when the second probability being greater than the predetermined probability.

9 . An electronic device comprising:
a touch screen; at least one memory storing at least one instruction; and at least one processor operatively connected to the at least one memory and configured to execute the at least one instruction to: acquire, for a plurality of fingers of a user, a plurality of first images corresponding to a first type touch input for touching the touch screen with a pressure less than a predetermined pressure, and a plurality of second images corresponding to a second type touch input for touching the touch screen with a pressure greater than the predetermined pressure, the second type touch input having a greater area of a region corresponding to a touch input than the first type touch input, wherein the acquiring the plurality of first images and the plurality of second images comprises:
displaying, on the touch screen, a first user interface (UI) element for guiding the first type touch input and a second UI element for guiding the second type touch input, wherein a size of the second UI element is greater than a size of the first UI element;
based on detecting a user touch input for touching the first UI element, generating the plurality of first images; and
based on detecting the user touch input for touching the second UI element, generating the plurality of second images;
train a neural network model for identifying a type of the user touch input of the user based on the plurality of first images and the plurality of second images, to obtain a trained neural network model; based on acquiring the user touch input through the touch screen, acquire a plurality of input images corresponding to the user touch input; identify the type of the user touch input by inputting, to the trained neural network model, the plurality of input images, the plurality of first images, and the plurality of second images; based on the type of the user touch input being the first type t

Assignees

• Samsung Electronics Co Ltd

Inventors

• Kim Dongchan
• Hwang Jinyoung

Classifications

• G06T2207/20081

  Training; Learning · CPC title

• G06V10/82

  using neural networks · CPC title

• G06V10/761

  Proximity, similarity or dissimilarity measures · CPC title

• G06V20/70

  Labelling scene content, e.g. deriving syntactic or semantic representations · CPC title

• G06T7/50

  Depth or shape recovery · CPC title