Systems and methods for three-dimensional (3D) reconstruction of human gestures from radar based measurements

Patent metadata

Field	Value
Publication number	US-11036303-B2
Application number	US-202016825140-A
Country	US
Kind code	B2
Filing date	Mar 20, 2020
Priority date	Mar 29, 2019
Publication date	Jun 15, 2021
Grant date	Jun 15, 2021

Abstract

This disclosure relates generally to radar based human activity detection, and, more particularly to, systems and methods from radar based human activity detection and three-dimensional (3D) reconstruction of human gestures using configurable panel radar system. Traditional systems and methods may not provide for a separate capturing of top and bottom parts of the human body. Embodiment of the present disclosure overcome the limitations faced by the traditional systems and methods by identifying a user that performed a gesture; detecting each gesture performed by the identified user; generating, by simulating a set of gesture labels, a sensor data and the generated metadata, a two-dimensional (2D) reference database of different speeds of the detected gestures; computing a displacement and a time of the detected gestures via a pattern matching technique; and reconstructing a video of the identified user performing the detected gestures in 3D.

First claim

What is claimed is:

1. A method for three-dimensional (3D) reconstruction of human gestures from radar based measurements, the method comprising:
acquiring, by one or more hardware processors, a time series data on radar measurements of gestures being performed by a plurality of users corresponding to a user database, wherein the time series data is acquired by implementing a configurable panel radar system ( 301 );
performing, by implementing a machine learning classification technique on the acquired time series data, a plurality of steps, wherein the plurality of steps comprise ( 302 ):
identifying a user amongst the plurality of users that performed a gesture;
classifying the identified user as an identified first user upon determining that the identified user corresponds to the user database;
classifying the identified user as an identified second user upon determining that the identified user does not corresponds to the user database;
generating a metadata corresponding to the identified second user; and
detecting the gestures performed by the identified first user or the identified second user, wherein the detected gestures comprise a corresponding set of gesture labels;

generating, by simulating the set of gesture labels, a sensor data and the generated metadata, a two-dimensional (2D) reference database of different speeds of the detected gestures, wherein the sensor data corresponds to the identified first user or the identified second user ( 303 );
computing, using the 2D reference database, a displacement and a time of the detected gestures by implementing a pattern matching technique; wherein the displacement corresponds to span of limbs of the detected gesture, and wherein the time is time taken to perform the detected gesture ( 304 ); and
reconstructing, using the computed displacement and time of the detected gestures, a video of the identified first user or of the identified second user performing the detected gestures in 3D via the configurable panel radar system ( 305 ).

2. The method as claimed in claim 1 , wherein the pattern matching technique comprises performing a comparison of a spectrogram of a buffered data with a spectrogram of the 2D reference database to compute the displacement speed and the time speed of the detected gestures, and wherein the buffered data corresponds to the time series data.

3. The method as claimed in claim 1 , wherein the step of simulating comprises a first modelling of trajectory of different joints for human gestures based upon the sensor data of the identified first user or the identified second user.

4. The method as claimed in claim 1 , wherein the step of simulating further comprises a second modelling of joints and a modelling of segments between the joints of the identified first user or the identified second user as ellipsoids.

5. The method as claimed in claim 4 , wherein the step of second modelling comprises generating, based upon the ellipsoids, a plurality of radar micro doppler signatures for different gestures of the identified first user or the identified second user.

6. The method as claimed in claim 5 , wherein the step of generating the plurality of radar micro doppler signatures is preceded by computing a Radar Cross Section (RCS) and a distance information of each ellipsoid for reconstructing the 3D video of the identified first user or of the identified second user.

7. The method as claimed in claim 1 , wherein the configurable panel radar system facilitates analyzing, based upon the detected gesture, the top and the bottom parts of the body of the identified first user or of the identified second user separately.

8. The method as claimed in claim 7 , wherein the step of analyzing is executed by a vertical placement of a plurality of radars at a predefined distance from each other for illuminating the top and the bottom parts of the body of the identified first user or of the identified second user separately, and wherein the plurality of radars correspond to the configurable panel radar system.

9. A system ( 100 ) for three-dimensional (3D) reconstruction of human gestures from radar based measurements, the system ( 100 ) comprising:
a memory ( 102 ) storing instructions;
one or more communication interfaces ( 106 ); and
one or more hardware processors ( 104 ) coupled to the memory ( 102 ) via the one or more communication interfaces ( 106 ), wherein the one or more hardware processors ( 104 ) are configured by the instructions to:
acquire a time series data on radar measurements of gestures being performed by a plurality of users corresponding to a user database, wherein the time series data is acquired by implementing a configurable panel radar system;
perform, by implementing a machine learning classification technique on the acquired time series data, a plurality of steps, wherein the plurality of steps comprise:
identify a user amongst the plurality of users that performed a gesture;
classifying the identified user as an identified first user upon determining that the identified user corresponds to the user database;
classifying the identified user as an identified second user upon determining that the identified user does not corresponds to the user database;
generate a metadata corresponding to the identified second user; and
detect the gestures performed by the identified first user or the identified second user, wherein the detected gestures comprise a corresponding set of gesture labels;

generate, by simulating the set of gesture labels, a sensor data and the generated metadata, a two-dimensional (2D) reference database of different speeds of the detected gestures, wherein the sensor data corresponds to the identified first user or the identified second user;
compute, using the 2D reference database, a displacement and a time of the detected gestures by implementing a pattern matching technique; wherein the displacement corresponds to span of limbs of the detected gesture, and wherein the time is time taken to perform the detected gesture; and
reconstruct, using the computed displacement and time of the detected gestures, a video of the identified first user or of the identified second user performing the detected gestures in 3D via the configurable panel radar system.

10. The system ( 100 ) as claimed in claim 9 , wherein the one or more hardware processors ( 104 ) are configured to implement the pattern matching technique by performing a comparison of a spectrogram of a buffered data with a spectrogram of the 2D reference database to compute the displacement speed and the time speed of the detected gestures, and wherein the buffered data corresponds to the time series data.

11. The system ( 100 ) as claimed in claim 9 , wherein the step of simulating comprises a first modelling of trajectory of different joints for human gestures based upon the sensor data of the identified first user or the identified second user.

12. The system ( 100 ) as claimed in claim 9 , step of simulating further comprises a second modelling of joints and a modelling of segments between the joints of the identified first user or the identified second user as ellipsoids.

13. The system ( 100 ) as claimed in claim 12 , wherein the one or more hardware processors ( 104 ) are configured to perform the second modelling by generating, based upon the ellipsoids, a plurality of radar micro doppler signatures for different gestures of the identified first user or the identified second user.

14. The system ( 100 ) as claimed in claim 13 , wherein the step of generating the plurality of radar micro doppler signatures is preceded by computing a Radar Cross Section (RC

Assignees

• Tata Consultancy Services Ltd
• Tata Consultancy Services Llc

Inventors

• Rani Smriti
• Gigie Andrew
• Chowdhury Arijit
• Chakravarty Tapas

Classifications

• G06F18/24

  Classification techniques · CPC title

• G06V40/20

  Movements or behaviour, e.g. gesture recognition (recognition of facial expressions G06V40/16) · CPC title

• G06V40/10

  Human or animal bodies, e.g. vehicle occupants or pedestrians; Body parts, e.g. hands · CPC title

• G06F3/017 Primary

  Gesture based interaction, e.g. based on a set of recognized hand gestures (interaction based on gestures traced on a digitiser G06F3/04883) · CPC title

• G01S13/88

  Radar or analogous systems specially adapted for specific applications (electromagnetic prospecting or detecting of objects, e.g. near-field detection, G01V3/00) · CPC title