Method and system for obtaining joint positions, and method and system for motion capture

Patent metadata

Field	Value
Publication number	US-11967101-B2
Application number	US-201917274744-A
Country	US
Kind code	B2
Filing date	Aug 29, 2019
Priority date	Sep 10, 2018
Publication date	Apr 23, 2024
Grant date	Apr 23, 2024

Abstract

The present invention provides a motion capture with a high accuracy which can replace an optical motion capture technology, without attaching optical markers and sensors to a subject. A subject with an articulated structure has a plurality of feature points in the body of the subject including a plurality of joints wherein a distance between adjacent feature points is obtained as a constant. A spatial distribution of a likelihood of a position of each feature point is obtained based on a single input image or a plurality of input images taken at the same time. One or a plurality of position candidates corresponding to each feature point are obtained based on the spatial distribution of the likelihood of the position of each feature point. Each join angle is obtained by performing an optimization calculation based on inverse kinematics using the candidates and the articulated structure. Positions of the feature points including the joints are obtained by performing a forward kinematics calculation using the joint angles.

First claim

The invention claimed is:

1. A method of obtaining joint positions of a subject with an articulated structure, said subject having a plurality of feature points in the body of said subject including a plurality of joints wherein a distance between adjacent feature points is obtained as a constant,
obtaining a spatial distribution of a likelihood of a position of each feature point based on a single input image or a plurality of input images taken at the same time;
obtaining one or a plurality of position candidates corresponding to each feature point based on the spatial distribution of said likelihood of the position of each feature point;
obtaining each joint angle by performing an optimization calculation based on inverse kinematics using said candidates and said articulated structure; and
obtaining positions of the feature points including the joints by performing a forward kinematics calculation using said joint angles.

2. The method according to claim 1 , wherein said obtaining position candidates for each feature point comprises obtaining one or a plurality of position candidates for each feature point by setting a a search range which is determined based on one or more positions of each feature point obtained in one or more previous frames.

3. The method according to claim 2 , wherein said search range is determined using at least a position of each feature point obtained in an immediate previous frame.

4. The method according to claim 2 , wherein said search range is a set of a predetermined number of points three-dimensionally distributed at a predetermined interval around a position which is determined based on one or more positions of each feature point obtained in one or more previous frames.

5. The method according to claim 1 , wherein the spatial distribution of the likelihood is used in the optimization calculation based on the inverse kinematics.

6. The method according to claim 1 further comprising:
smoothing the positions of feature points in a temporal direction using a plurality of positions of the feature points obtained at a plurality of the other frames; and
obtaining each joint angle of the subject by performing the optimization calculation based on the inverse kinematics using said smoothed positions of feature points and the articulated structure of the subject.

7. The method according to claim 6 , wherein the positions of the feature points including the joints are obtained by performing the forward kinematics calculation using said joint angles.

8. The method according to claim 1 , said method further comprising a preprocessing step based on information obtained at a frame prior to the input image,
said preprocessing step comprising at least one of:
(a) rotation of the input image;
(b) trimming and/or reducing of the input image;
(c) masking of the input image;
(d) selection of the input image by selecting a camera; and
(e) stitching of the input image.

9. The method according to claim 8 , wherein said preprocessing step comprises the rotation of the input image, and the preprocessing step comprising:
obtaining an inclination to an upright pose of the body of the subject at the frame prior to the input image;
rotating the input image in accordance with the inclination to obtain an upright pose or a nearly upright pose;
obtaining the spatial distribution of the likelihood based on the rotated image; and
rotating the rotated image with the spatial distribution of the likelihood back to the original input image.

10. The method according to claim 1 , wherein time series data of joint angles and joint positions is obtained by obtaining the joint angles and joint positions in each frame to capture the motion of the subject.

11. A system for obtaining joint positions of a subject comprising a memory and a processor,
said memory configured to store a file defining an articulated structure comprising a plurality of feature point in the body of the subject including a plurality of joints and a set of constants each defining a distance between adjacent feature points,
said processor configured to:
obtain a spatial distribution of a likelihood of a position of each feature point based on a single input image or a plurality of input images taken at the same time;
obtain one or a plurality of position candidates corresponding to each feature point based on the spatial distribution of said likelihood of the position of each feature point;
obtain each joint angle by performing an optimization calculation based on inverse kinematics using said candidates and said articulated structure; and
obtain positions of the feature points including the joints by performing a forward kinematics calculation using said joint angles.

12. The system according to claim 11 , wherein said memory is configured to store time series data of the obtained positions of the joints; and said processor is configured to obtain one or a plurality of position candidates for each feature point by setting a search range which is determined based on one or more positions of each feature point obtained in one or more previous frames.

13. The system according to claim 12 , wherein said processer is configured to determine the search range by using at least a position of each feature point obtained in an immediate previous frame.

14. The system according to claim 12 , wherein the search range is a set of a predetermined number of points three-dimensionally distributed at a predetermined interval around a position which is determined based on one or more positions of each feature point obtained in one or more previous frames.

15. The system according to claim 11 , wherein the spatial distribution of the likelihood is used in the optimization calculation based on the inverse kinematics.

16. The system according to claim 11 , wherein said processor is configured to:
smooth the positions of feature points in a temporal direction using a plurality of positions of the feature points obtained at a plurality of the other frames; and
obtain each joint angle of the subject by performing the optimization calculation based on the inverse kinematics using said smoothed positions of the feature points and the articulated structure of the subject.

17. The system according to claim 16 wherein the position of the feature points including the joints is obtained by performing the forward kinematics calculation using said joint angles.

18. The system according to claim 11 , said processor is configured to perform a preprocessing based on information obtained at a frame prior to the input image, and
said preprocessing comprise at least one of:
(a) rotation of the input image;
(b) trimming and/or reducing of the input image;
(c) masking of the input image;
(d) selection of the input image by selecting a camera; and
(e) stitching of the input image.

19. The system according to claim 18 , wherein said preprocessing comprises the rotation of the input image, and the processor is configured to:
obtain an inclination to an upright pose of the body of the subject at the frame prior to the input image;
rotate the input image in accordance with the inclination to obtain an upright pose or a nearly upright pose;
obtain the spatial distribution of the likelihood based on the rotated image; and
rotate the rotated image with the spatial distribution of the likelihood back to the original input image.

20. The system according to claim 11 further comprising one or a plurality of cameras capturing the input images wherein said

Assignees

• Univ Tokyo

Inventors

• Nakamura Yoshihiko
• Takano Wataru
• Ikegami Yosuke
• Ohashi Takuya
• Yamamoto Kazuki
• Takemoto Kentaro

Classifications

• G06T7/73 Primary

  using feature-based methods · CPC title

• G06V10/242

  by image rotation, e.g. by 90 degrees · CPC title

• G06V40/103

  Static body considered as a whole, e.g. static pedestrian or occupant recognition · CPC title

• G06T2207/30196

  Human being; Person · CPC title

• G06V20/64 Primary

  Three-dimensional [3D] objects · CPC title