Electronic apparatus and controlling method thereof

Patent metadata

Field	Value
Publication number	US-2023364465-A1
Application number	US-202318328252-A
Country	US
Kind code	A1
Filing date	Jun 2, 2023
Priority date	May 10, 2022
Publication date	Nov 16, 2023
Grant date	—

Abstract

An electronic apparatus is disclosed. The electronic apparatus includes: a memory including at least one instruction, a processor coupled with the memory and configured to control the electronic apparatus, and the processor is configured, by executing the at least one instruction, to: obtain a moving image, identify a person and pose data of the person from a plurality of frames in the moving image, obtain exercise pattern information corresponding to the plurality of frames using pose data of the identified person, recognize an exercise motion of the identified person by inputting at least one frame of the moving image into at least one neural network model, and obtain exercise feature information corresponding to the recognized exercise motion, identify, based on the exercise feature information and the exercise pattern information, a first frame interval and a second frame interval different from the first frame interval from among the plurality of frames in the moving image, and provide information on an exercise motion corresponding to the second frame interval by comparing the first frame interval with the second frame interval.

First claim

What is claimed is:

1 . An electronic apparatus, comprising:
a memory including at least one instruction; and a processor coupled with the memory and configured to control the electronic apparatus, wherein the processor is configured, by executing the at least one instruction, to: obtain a moving image, identify a person and pose data of the person from a plurality of frames in the moving image, obtain exercise pattern information corresponding to the plurality of frames using pose data of the identified person, recognize an exercise motion of the identified person by inputting at least one frame of the moving image into at least one neural network model, and obtain exercise feature information corresponding to the recognized exercise motion, identify, based on the exercise feature information and the exercise pattern information, a first frame interval and a second frame interval different from the first frame interval from among the plurality of frames in the moving image, and provide information on an exercise motion corresponding to the second frame interval by comparing the first frame interval with the second frame interval.

2 . The electronic apparatus of claim 1 , wherein
the processor is configured to: input at least one frame of the moving image into a first neural network model configured to identify whether an exercise motion of a person is a symmetrical motion, and identify whether the exercise motion of the person is a symmetrical motion, and input at least one frame of the moving image into a second neural network model configured to identify whether an exercise motion of a person is a hold motion, and identify whether the exercise motion of the person is a hold motion.

3 . The electronic apparatus of claim 2 , wherein:
the first neural network model is configured to be trained based on a learning moving image which captured an exercise motion of a person and information on whether the exercise motion of the person corresponding to the learning moving image is a symmetrical motion, and the second neural network model is configured to be trained based on a learning moving image which captured an exercise motion of a person and information on whether the exercise motion of the person corresponding to the learning moving image is a hold motion.

4 . The electronic apparatus of claim 1 , wherein
the processor is configured to: identify, based on an exercise motion of the person being a symmetrical motion and a hold motion, an interval from a start frame to a first time as the first frame interval from among the plurality of frames in the moving image.

5 . The electronic apparatus of claim 1 , wherein
the processor is configured to: identify, based on an exercise motion of the person being a symmetrical motion and a non-hold motion, each exercise repetition interval from the plurality of frames in the moving image based on the exercise pattern information, and identify, based on the identified exercise repetition intervals, an interval from a start frame to a frame corresponding to an exercise repetition interval of a first number of times as the first frame interval from among the plurality of frames in the moving image.

6 . The electronic apparatus of claim 1 , wherein
the processor is configured to: identify, based on an exercise motion of the person being a non-symmetrical motion and a hold motion exercise, a frame corresponding to an interval from a frame in which a first symmetrical exercise motion is first started to a second time and an interval from a frame in which a second symmetrical exercise motion is first started to a second time as the first frame interval from among the plurality of frames in the moving image based on the exercise pattern information.

7 . The electronic apparatus of claim 1 , wherein
the processor is configured to: identify, based on an exercise motion of the person being a non-symmetrical motion and a non-hold motion, each frame corresponding to a first symmetrical exercise motion and a second symmetrical exercise motion from the plurality of frames in the moving image based on the exercise pattern information, and identify a frame corresponding to the first symmetrical exercise motion of a second number of times and a frame corresponding to the second symmetrical exercise motion of a second number of times from a start frame as the first frame interval from among the plurality of frames in the moving image.

8 . The electronic apparatus of claim 1 , wherein
the processor is configured to: identify, based on the moving image being played, a frame of a specified interval from the frame that is being played as the second frame.

9 . The electronic apparatus of claim 1 , wherein
the processor is configured to: identify a physically depleted state based on a difference value of exercise pattern information corresponding to the first frame interval and exercise pattern information of the second frame interval being greater than or equal to a specified value, and provide, based on identifying the physical depleted state, information on an exercise motion of the person corresponding to the second frame interval.

10 . The electronic apparatus of claim 1 , wherein
the processor is configured to: obtain an original moving image, and obtain, based on an exercise data base (DB) storing a plurality of moving images that comprise a plurality of exercise motions, the moving image by identifying a frame corresponding to an exercise motion of the person from among a plurality of frames comprising in the original moving image.

11 . A method of controlling an electronic apparatus, the method comprising:
obtaining a moving image; identifying a person and pose data of the person from a plurality of frames in the moving image; obtaining exercise pattern information corresponding to the plurality of frames using pose data of the identified person; recognizing an exercise motion of the identified person by inputting at least one frame of the moving image into at least one neural network model, and obtaining exercise feature information corresponding to the recognized exercise motion; identifying, based on the exercise feature information and the exercise pattern information, a first frame interval and a second frame interval different from the first frame interval from among the plurality of frames in the moving image; and providing information on an exercise motion corresponding to the second frame by comparing the first frame interval with the second frame interval.

12 . The method of claim 11 , wherein
the obtaining exercise feature information comprises: inputting at least one frame of the moving image into a first neural network model to identify whether an exercise motion of a person is a symmetrical motion, and identifying whether an exercise motion of the person is a symmetrical motion, and inputting at least one frame of the moving image into a second neural network model to identify whether an exercise motion of a person is a hold motion, and identifying whether an exercise motion of the person is a hold motion.

13 . The method of claim 12 , wherein:
the first neural network model is configured to be trained based on a learning moving image which captured an exercise motion of a person and information on whether the exercise motion of the person corresponding to the learning moving image is a symmetrical motion, and the second neural network model is configured to be trained based on a learning moving image which captured an exercise motion of a person and information on whether the exercise motion of the person corresponding to the learning

Assignees

• Samsung Electronics Co Ltd

Inventors

• Kim Hongyoon
• Kim Kiljong
• Kim Jongwon
• Song Seokwoo
• Cho Hyunkook

Classifications

• G06T2207/20084

  Artificial neural networks [ANN] · CPC title

• G06T2207/20081

  Training; Learning · CPC title

• G06T2207/10016

  Video; Image sequence · CPC title

• G06T7/70

  Determining position or orientation of objects or cameras (camera calibration G06T7/80) · CPC title

• G06N3/08

  Learning methods · CPC title