System and method for tracking body joints

Patent metadata

Field	Value
Publication number	US-10980447-B2
Application number	US-201815908731-A
Country	US
Kind code	B2
Filing date	Feb 28, 2018
Priority date	Jul 7, 2017
Publication date	Apr 20, 2021
Grant date	Apr 20, 2021

Abstract

Body joint tracking is applied in various industries and medical field. In body joint tracking, marker less devices plays an important role. However, the marker less devices are facing some challenges in providing optimal tracking due to occlusion, ambiguity, lighting conditions, dynamic objects etc. System and method of the present disclosure provides an optimized body joint tracking. Here, motion data pertaining to a first set of motion frames from a motion sensor are received. Further, the motion data are processed to obtain a plurality of 3 dimensional cylindrical models. Here, every cylindrical model among the plurality of 3 dimensional cylindrical model represents a body segment. The coefficients associated with the plurality of 3 dimensional cylindrical models are initialized to obtain a set of initialized cylindrical models. A set of dynamic coefficients associated with the initialized cylindrical models are utilized to track joint motion trajectories of a set of subsequent frames.

First claim

What is claimed is:

1. A processor implemented method for tracking body joints, the method comprising:
receiving, by one or more hardware processors, motion data pertaining to a first set of motion frames from a motion sensor device, wherein the motion data comprises an initial joint motion data and a plurality of depth frames, wherein the initial joint motion data comprises locations in 3 dimensional world coordinates of a plurality of body joints associated with the plurality of depth frames and each depth frame among the plurality of depth frames comprises a set of 2D image coordinates and a depth value;
initializing, by the one or more hardware processors, a plurality of cylindrical models for each motion frame among the first set of motion frames by processing the initial joint motion data and the plurality of depth frames, wherein the plurality of cylindrical models represents a plurality of body segments, connecting adjacent body joints among the plurality of body joints, and wherein initializing the plurality of cylindrical models for each motion frame among the first set of motion frames comprises:
obtaining a plurality of 3 dimensional point clouds corresponding to the plurality of depth frames by mapping every 2 dimensional image coordinates to the 3 dimensional world coordinates, based on a perspective projection;
segmenting the plurality of 3 dimensional point clouds based on a segmentation threshold to obtain a first set of segmented 3 dimensional point clouds, wherein the first set of segmented 3 dimensional point clouds corresponds to the plurality of body segments, wherein the segmentation threshold is obtained based on an Euclidean distance computed for 3 dimensional world coordinates corresponding to the plurality of body joints of the initial joint motion data;
smoothening and reconstructing the first set of segmented 3 dimensional point clouds corresponding to the plurality of body segments prior to model fitting to reduce one or more model outliers, by using a Moving Least Squares (MLS) mechanism; and
fitting the plurality of cylindrical models to the set of smoothened point clouds corresponding to the plurality of body segments associated with each motion frame among the first set of motion frames, wherein a set of cylinder coefficients of each cylindrical model among the plurality of cylindrical models includes radius, length, the one or more direction angles and the one or more base coordinates; and

tracking, by the one or more hardware processors, the plurality of initialized cylindrical models to obtain one or more an optimized motion trajectories of the plurality of body joints in a plurality of motion frames received in succession to the first set of motion frames based on one or more direction angles and one or more base coordinates associated with each of the plurality of cylindrical models, wherein the tracking is performed by utilizing a particle filter mechanism, and wherein tracking the plurality of initialized cylindrical models to obtain an optimized motion trajectories comprises:
receiving the plurality of initialized cylindrical models and a second set of segmented point cloud data corresponding to the plurality of motion frames received in succession to the first set of motion frames from the motion sensor device, wherein the initialized plurality of cylindrical models are represented as a plurality of particles, and wherein each particle among the plurality of particles comprises a state represented by the one or more direction angles and the one or more base coordinates, and a weight;
propagating the plurality of particles to a new state and updating the state based on a state propagation rule by utilizing a state space model, wherein the state propagation rule is based on the plurality of initialized cylindrical models and variance of Gaussian noise with zero mean for the one or more direction angles and the one or more base coordinates;
updating the weight associated with the plurality of particles based on a variance of Gaussian likelihood function and a weight update rule by utilizing an observation model, wherein the weight update rule is based on the second set of segmented point cloud data and the plurality of initialized cylindrical models, and wherein the weight update rule considers a distance (D), which is a sum of all distances (d) of the second set of segmented point cloud data from a cylindrical model represented by a particle for which the weight is being updated; and
tracking the plurality of initialized cylindrical models based on the state and the weight of the plurality of particles associated with the plurality of initialized cylindrical models to obtain the optimized motion trajectories of the plurality of body joints, wherein a cylindrical model among the plurality of initialized cylindrical model having maximum weight is selected as an optimal cylindrical model.

2. The processor implemented method of claim 1 , wherein initializing the plurality of cylindrical models for each motion frame among the first set of motion frames further comprises:
initializing the radius of the plurality of cylindrical models, a mean radius value obtained from a cylinder radius corresponding to the plurality of motion frames;
initializing the one or more direction angles of the plurality of cylindrical models, a mean direction value obtained from a set of cylinder directions corresponding to from the plurality of motion frames;
initializing the one or more base coordinates of the plurality of cylindrical models, an average base value obtained by projecting a plurality of joint center coordinates in the initial joint motion data to an axis of a corresponding cylindrical model from the plurality of motion frames; and
initializing the length of the plurality of cylindrical models, a mean length value obtained from a distance between adjacent body joints in the initial joint motion data stream for plurality of motion frames.

3. A body joint tracking system, the system comprising:
a motion sensor device;
one or more memories comprising programmed instructions; and
one or more hardware processors operatively coupled to the one or more memories, wherein the one or more hardware processors are configured to execute the programmed instructions stored in the one or more memories to:
receive, motion data pertaining to a first set of motion frames from the motion sensor device, wherein the motion data comprises an initial joint motion data and a plurality of depth frames, wherein the initial joint motion data comprises locations in 3 dimensional world coordinates of a plurality of body joints associated with the plurality of depth frames and each depth frame among the plurality of depth frames comprises a set of 2D image coordinates and a depth value;
initialize, a plurality of cylindrical models for each motion frame among the first set of motion frames by processing the initial joint motion data and the plurality of depth frames, wherein the plurality of cylindrical models represents a plurality of body segments, connecting adjacent body joints among the plurality of body joints, and wherein initializing the plurality of cylindrical models for each motion frame among the first set of motion frames comprises:
obtaining a plurality of 3 dimensional point clouds corresponding to the plurality of depth frames by mapping every 2 dimensional image coordinates to the 3 dimensional world coordinates, based on a perspective projection;

segmenting the plurality of 3 dimensional point clouds based on a segmentation threshold to obtain a first set of segmented 3 dimensional point clouds, wherein the first set of segmented 3 dimensional point clouds corresponds to the plurality of body segments, wherein the segmentation threshold is obtained based on an Euclidean distance computed for 3 dimensional world coordinates correspondin

Assignees

• Tata Consultancy Services Ltd
• Tata Consultancy Ltd Services

Inventors

• Sinha Sanjana
• Bhowmick Brojeshwar
• Sinha Aniruddha
• Das Abhijit

Classifications

• G06T7/251 Primary

  involving models · CPC title

• G06V40/20

  Movements or behaviour, e.g. gesture recognition (recognition of facial expressions G06V40/16) · CPC title

• G06T2207/30241

  Trajectory · CPC title

• G06T7/277

  involving stochastic approaches, e.g. using Kalman filters · CPC title

• G06T2207/30196

  Human being; Person · CPC title