Electronic device and methods for out of field of view hand tracking

Patent metadata

Field	Value
Publication number	US-2025046122-A1
Application number	US-202418907042-A
Country	US
Kind code	A1
Filing date	Oct 4, 2024
Priority date	Aug 4, 2023
Publication date	Feb 6, 2025
Grant date	—

Abstract

There is provided a method and apparatus for performing gesture recognition in an electronic device, including: tracking a visible trajectory of a hand of a user from a plurality of frames captured by the electronic device, identifying, by the electronic device, a first frame where the hand of the user has gone out of a Field of View (FOV) of the electronic device, identifying the second frame where the hand of the user has come back into the FOV, predicting, using an Artificial Intelligence (AI) model, a trajectory of the hand of the user using one or more frames captured before the first frame, and one or more frames captured after the second frame, and recognizing at least one hand gesture performed during the visible trajectory of the hand of the user, and the predicted trajectory of the hand of the user.

First claim

What is claimed is:

1 . A method for performing gesture recognition in an electronic device, comprising:
tracking, by the electronic device, a visible trajectory of a hand of a user from a plurality of frames captured by the electronic device; identifying, by the electronic device, the first frame, where the hand of the user has gone out of a Field of View (FOV) of the electronic device, among the plurality of frames; identifying, by the electronic device, the second frame, where the hand of the user has come back into the FOV of the electronic device, among the plurality of frames; obtaining, by the electronic device, using an Artificial Intelligence (AI) model, a trajectory of the hand of the user using the one or more frames captured before the first frame among the plurality of frames and the one or more frames captured after the second frame among the plurality of frames; and recognizing, by the electronic device, at least one hand gesture performed during the visible trajectory of the hand of the user, and the obtained trajectory of the hand of the user.

2 . The method as claimed in claim 1 , further comprising:
selecting, by the electronic device, a frame with the presence of the hand of the user from the visible trajectory, among the plurality of frames; and generating, by the electronic device, references of one or more hand landmarks from the frame with the presence of the hand of the user.

3 . The method as claimed in claim 2 , further comprising:
estimating, by the electronic device, a location of the one or more hand landmarks of the one or more frames captured before the first frame, based on the generated references of the one or more hand landmarks; calculating, by the electronic device, one or more kinetic parameters of each hand landmark, using the estimated location of the one or more hand landmarks of consecutive frames, wherein the consecutive frames comprise the one or more frames captured before the first frame; and obtaining, by the electronic device, a first trajectory of the hand of the user corresponding to the one or more frames captured before the first frame, using the calculated one or more kinetic parameters of each of the one or more hand landmarks.

4 . The method as claimed in claim 2 , further comprising:
reversing, by the electronic device, order of the plurality of frames captured by the electronic device; estimating, by the electronic device, a location of the one or more hand landmarks of the one or more frames captured after the second frame, based on the generated references of the one or more hand landmarks; calculating, by the electronic device, one or more kinetic parameters of each hand landmark, using the estimated location of the one or more hand landmarks of consecutive frames, wherein the consecutive frames comprise the one or more frames captured after the second frame; and obtaining, by the electronic device, a second trajectory of the hand of the user using the one or more frames captured after the second frame, using the calculated one or more kinetic parameters of each of the one or more hand landmarks.

5 . The method as claimed in claim 4 , wherein the one or more kinetic parameters of the hand of the user comprise at least one of a velocity and an acceleration of each of the one or more hand landmarks.

6 . The method as claimed in claim 4 , wherein the method comprises:
verifying, by the electronic device, if the hand of the user is in the FOV of the electronic device, after calculating one or more kinetic parameters of each of the one or more hand landmarks; and calculating, by the electronic device, a velocity and a position of the one or more hand landmarks in a next frame, after the one or more frames captured after the second frame, using the calculated one or more kinetic parameters of each of the one or more hand landmarks, if the hand of the user is not in the FOV of the electronic device.

7 . The method as claimed in claim 6 , further comprising:
verifying, by the electronic device, at least one parameter of the hand of the user, after calculating the velocity and the position of the one or more hand landmarks in the next frame, wherein the at least one parameter comprises at least one of whether a velocity goes to zero, a hand position is beyond a threshold, and the one or more hand landmarks no longer conform to predetermined bio-mechanical constraints of a human hand; repeating, by the electronic device, a verification of the hand of the user in the FOV of the electronic device, if the at least one parameter of the hand of the user is not satisfied; and repeating, by the electronic device, estimation of the location of the one or more hand landmarks from a previous frame, previous to the next frame, if the hand of the user is stationary and the at least one parameter of the hand of the user is satisfied.

8 . The method as claimed in claim 3 , further comprising:
checking, by the electronic device, closeness of the one or more hand landmarks for each frame, of the plurality of frames, from the first trajectory and the second trajectory of the hand of the user; obtaining, by the electronic device, a spatio-temporal convergence at a frame, of the plurality of frames, where a distance between two extrapolated hand landmarks is below a certain threshold, wherein a trajectory until the frame of the spatio-temporal convergence is considered as the first trajectory, wherein a trajectory after the frame of the spatio-temporal convergence is considered as the second trajectory; estimating, by the electronic device, a hand pose by encoding the two extrapolated hand landmarks of the spatio-temporal convergence; and recognizing, by the electronic device, the at least one hand gesture, based on a sequence of hand pose information.

9 . An electronic device, comprising:
a memory storing at least one instruction; and, at least one processor configured to execute the at least one instruction stored in the memory; wherein the at least one processor is configured to execute the at least one instruction to: track a visible trajectory of a hand of the user from a plurality of frames captured by the electronic device, wherein the plurality of frames comprise a first frame, a second frame, one or more frames captured before the first frame, and one or more frames captured after the second frame; identify the first frame, where the hand of the user has gone out of a Field of View (FOV) of the electronic device, among the plurality of frames; identify the second frame, where the hand of the user has come back into the FOV of the electronic device, among the plurality of frames; obtain, using an Artificial Intelligence (AI) model, a trajectory of the hand of the user using the one or more frames captured before the first frame among the plurality of frames, and the one or more frames captured after the second frame among the plurality of frames; and recognize at least one hand gesture performed during the visible trajectory of the hand of the user, and the obtained trajectory of the hand of the user.

10 . The electronic device as claimed in claim 9 , wherein the at least one processor is configured to execute the at least one instruction to:
select a frame, among the plurality of frames, with the presence of the hand of the user from the visible trajectory; and generate references of one or more hand landmarks from the frame with the presence of the hand of the user.

11 . The electronic device as claimed in claim 10 , wherein the at least one processor is configured to execute the at least one instruction to:
estimate a location of the one or more hand landmarks of the one or more frames captu

Assignees

• Samsung Electronics Co Ltd

Inventors

• S R Vishakha
• Prasad Bindigan Hariprasanna Pawan
• K S Green Rosh
• Jayaprakash Akula
• Shankar Meghana
• Kukreja Prateek
• Thakur Shubham Mallik
• Parmar Sagar
• Jha Mukesh
• Choi Sungsoo
• Woo Hyuntaek
• Lee Chaeeun
• Kim Beomsu

Classifications

• G06V40/28 Primary

  Recognition of hand or arm movements, e.g. recognition of deaf sign language (static hand signs G06V40/113) · CPC title

• A63F13/213

  comprising photodetecting means, e.g. cameras, photodiodes or infrared cells (A63F13/219 takes precedence) · CPC title

• G06V10/7715

  Feature extraction, e.g. by transforming the feature space, e.g. multi-dimensional scaling [MDS]; Mappings, e.g. subspace methods · CPC title

• A63F13/428

  involving motion or position input signals, e.g. signals representing the rotation of an input controller or a player's arm motions sensed by accelerometers or gyroscopes · CPC title

• G06V10/751 Primary

  Comparing pixel values or logical combinations thereof, or feature values having positional relevance, e.g. template matching · CPC title