Method for predicting intention of user and apparatus for performing same

Patent metadata

Field	Value
Publication number	US-2021256250-A1
Application number	US-202117246299-A
Country	US
Kind code	A1
Filing date	Apr 30, 2021
Priority date	Nov 2, 2018
Publication date	Aug 19, 2021
Grant date	—

Abstract

A method for predicting the intention of a user through an image acquired by capturing the user includes: receiving an image acquired by capturing a user; and predicting the intention of the user for the next motion by using spatial information and temporal information about the user and a target object included in the image.

First claim

1 . A method for predicting an intention of a user through an image acquired by capturing the user, the method comprising:
receiving an image acquired by capturing a user; and predicting an intention of the user for a next motion by using spatial information and temporal information about the user and a target object included in the image.

2 . The method of claim 1 , wherein the spatial information comprises a pose of a body part of the user and an interaction between the body part of the user and the target object.

3 . The method of claim 2 , wherein the interaction comprises at least one of a distance between the body part and the target object, and a position and direction of the body part based on the target object.

4 . The method of claim 2 , wherein the temporal information comprises changes in the pose of the body part of the user and the interaction over time.

5 . The method of claim 4 , wherein the temporal information comprises at least one of a trajectory along which the body part moves and a speed at which the body part moves toward the target object.

6 . The method of claim 1 , wherein predicting the intention of the user comprises:
applying the image to a deep learning network as an input; extracting features of spatial information for each of a plurality of frames constituting the image through a convolution neural network (CNN) included in the deep learning network; extracting features of temporal information, included in consecutive frames, from the extracted features of the spatial information through a recurrent neural network (RNN) included in the deep learning network; and outputting an intention for a next motion as a result value based on the extracted features of the spatial information and the temporal information.

7 . The method of claim 6 , further comprising applying a driving signal to a device for assisting the user in performing motions according to the predicted intention.

8 . The method of claim 7 , wherein applying the driving signal comprises:
selecting a result value occupying a predetermined percentage or more among a plurality of result values output from the deep learning network for a preset predetermined period of time; and applying a driving signal corresponding to the selected result value.

9 . A computer-readable storage medium having stored thereon a program that performs the method set forth in claim 1 .

10 . An apparatus for prediction an intention, the apparatus comprising:
an input/output unit configured to receive an image acquired by capturing a user from an outside and to output an intention of the user for a next motion predicted by analyzing the image; a storage unit configured to store a program for predicting an intention of the user for a next motion by analyzing the image; and a control unit including at least one processor; wherein the control unit predicts an intention of the user for a next motion using spatial information and temporal information about the user and a target object, included in the image, by executing the program.

11 . The apparatus of claim 10 , wherein the spatial information comprises a pose of a body part of the user and an interaction between the body part of the user and the target object.

12 . The apparatus of claim 11 , wherein the spatial information further comprises at least one of a size, shape, texture, stiffness, and color of the target object.

13 . The apparatus of claim 11 , wherein the interaction comprises at least one of a distance between the body part and the target object, and a position and direction of the body part based on the target object.

14 . The apparatus of claim 11 , wherein the temporal information comprises changes in the pose of the body part of the user and the interaction over time.

15 . The apparatus of claim 14 , wherein the temporal information comprises at least one of a trajectory along which the body part moves and a speed at which the body part moves toward the target object.

16 . The apparatus of claim 10 , wherein a deep learning network that is implemented by the control unit by executing the program and receives the image as an input comprises:
a spatial information extraction unit configured to extract features of spatial information for each of a plurality of frames constituting the image; a temporal information extraction unit configured to extract features of temporal information, included in consecutive frames, from the extracted features of the spatial information; and an intention output unit configured to output the intention for the next motion as a result value based on outputs of the spatial information extraction unit and the temporal information extraction unit.

17 . The apparatus of claim 16 , wherein the control unit applies a driving signal to a device for assisting the user in performing motions through the input/output unit according to the predicted intention.

18 . The apparatus of claim 17 , wherein the control unit, when applying the driving signal, selects a result value occupying a predetermined percentage or more among a plurality of result values output from the deep learning network for a preset predetermined period of time, and applies a driving signal corresponding to the selected result value.

19 . The apparatus of claim 17 , wherein:
motions that can be performed by the user are classified into at least two types; and the control unit, when applying the driving signal, checks a type of motion currently performed by the user, selects only an intention for a motion different from the identified type of motion from intentions output as result values from the deep learning network for a predetermined period of time, and applies a driving signal corresponding to the selected intention.

20 . The apparatus of claim 10 , wherein the image acquired by capturing the user is an image that is captured from a first-person point of view of the user such that at least a body part of the user appears in the image.

Assignees

• Seoul Nat Univ R&Db Foundation
• Korea Advanced Inst Sci & Tech

Inventors

• Cho Kyu-Jin
• Jo Sungho
• Kang Byunghyun
• Kim Daekyum
• Choi Hyungmin
• Kim Kyu Bum

Classifications

• A61B5/0077 Primary

  Devices for viewing the surface of the body, e.g. camera, magnifying lens · CPC title

• G06V40/28

  Recognition of hand or arm movements, e.g. recognition of deaf sign language (static hand signs G06V40/113) · CPC title

• G06V40/20

  Movements or behaviour, e.g. gesture recognition (recognition of facial expressions G06V40/16) · CPC title

• G06N3/045

  Combinations of networks · CPC title

• G06N3/044

  Recurrent networks, e.g. Hopfield networks · CPC title