Human posture determination method and mobile machine using the same

Patent metadata

Field	Value
Publication number	US-11837006-B2
Application number	US-202117364743-A
Country	US
Kind code	B2
Filing date	Jun 30, 2021
Priority date	Jun 30, 2021
Publication date	Dec 5, 2023
Grant date	Dec 5, 2023

Abstract

Human posture determination is disclosed. Human posture is determined by obtaining range image(s) through a range camera, detecting key points of an estimated skeleton of a human in color data of the range image(s) and calculating positions of the detected key points based on depth data of the range image(s), choosing a feature map from a set of predefined feature maps based on the detected key points among a set of predefined key points, obtaining two features of a body of the human corresponding to the chosen feature map based on the positions of the detected key points, and determining a posture of the human according to the two features in the chosen feature map.

First claim

What is claimed is:

1. A human posture determination method, comprising:
obtaining, through a range camera, one or more range images, wherein the one or more range images include color data and depth data;
detecting key points of an estimated skeleton of a human in the color data and calculating positions of the detected key points based on the depth data, wherein the estimated skeleton has a set of predefined key points;
choosing a feature map from a set of predefined feature maps based on the detected key points among the predefined key points;
obtaining two features of a body of the human corresponding to the chosen feature map based on the positions of the detected key points; and
determining a posture of the human according to the two features in the chosen feature map;
wherein when the detected key points comprise at least a shoulder key point, at least a hip key point, and at least a knee key point among the predefined key points, a second feature map with the two features of a body ratio and an upper body angle is chosen from the set of predefined feature maps, and the second feature map comprises: a first threshold curve for distinguishing a standing posture and a sitting posture, and a second threshold curve for distinguishing a lying posture, the standing posture, and the sitting posture; and
when the determined posture is not the lying posture and the detected key points comprise all the predefined key points, a first feature map with the two features of an internal angle and the body ratio is chosen from the set of predefined feature maps, and the first feature map comprises a threshold curve for distinguishing the standing posture and the sitting posture.

2. The method of claim 1 ,
wherein when the first feature map is chosen, the obtaining the two features of the body of the human corresponding to the chosen feature map based on the positions of the detected key points comprises:
in response to the detected key points including all the predefined key points, obtaining a detected internal angle of the body of the human and a detected body ratio of the body of the human based on the positions of the detected key points; and
the determining the posture of the human according to the two features in the chosen feature map comprises:
in response to the detected key points including all the predefined key points, determining the posture of the human based on a position of an intersection of the detected internal angle and the detected body ratio in the first feature map.

3. The method of claim 2 , wherein the obtaining the internal angle and the body ratio based on the positions of the key points comprises:
obtaining a middle position p h between the positions of two hip key points among the detected key points;
obtaining an upper body plane based on the middle position p h and the positions of two shoulder key points among the detected key points;
obtaining a lower body plane based on the middle position p h and the positions of two knee key points among the detected key points;
obtaining an angle between the upper body plane and the lower body plane to take as the internal angle;
obtaining a ratio between a lower body height h low and an upper body height h up to take as the body ratio, wherein h up =p s ·y−p h ·y and h low =p h ·y−p k ·y, p s is a middle position between the positions of the two shoulder key points, p s ·y is a y coordinate of p s , p h ·y is a y coordinate of p h , p k is a middle position between the positions of the two knee key points, and p k ·y is a y coordinate of p k ; and
the determining the posture of the human based on the position of the intersection of the internal angle and the body ratio in the first feature map comprises:
determining the posture of the human according to the position of the intersection of the internal angle and the body ratio in the first feature map with respect to the threshold curve for distinguishing the standing posture and the sitting posture.

4. The method of claim 1 ,
wherein when the second feature map is chosen, the obtaining the two features of the body of the human corresponding to the chosen feature map based on the positions of the detected key points comprises:
in response to the detected key points including at least a shoulder key point, at least a hip key point, and at least a knee key point among the predefined key points, obtaining a detected body ratio of the body of the human and a detected upper body angle of the body of the human based on the positions of the detected key points; and
the determining the posture of the human according to the two features in the chosen feature map comprises:
in response to the detected key points including at least a shoulder key point, at least a hip key point, and at least a knee key point among the predefined key points, determining the posture of the human based on a position of an intersection of the detected body ratio and the detected upper body angle in the second feature map.

5. The method of claim 4 , wherein the obtaining the body ratio and the upper body angle based on the positions of the detected key points comprises:
obtaining a middle position p h between the positions of two hip key points among the detected key points;
obtaining a ratio between a lower body height h low and an upper body height h up to take as the body ratio, wherein h up =p s ·y−p h ·y and h low =p h ·y p k ·y, p s is a middle position between the positions of two shoulder key points, p s ·y is a y coordinate of p s , p h ·y is a y coordinate of p h , p k is a middle position between the positions of two knee key points, and p k ·y is a y coordinate of p k ;
obtaining an angle between a vector {right arrow over (v)} hs between the middle position p h and the middle position p s and a unit vector {right arrow over (y)} in y-direction to take as the upper body angle; and
the determining the posture of the human based on the position of the intersection of the body ratio and the upper body angle in the second feature map comprises:
determining the posture of the human according to the position of the intersection of the body ratio and the upper body angle in the second feature map with respect to the first threshold curve for distinguishing the standing posture and the sitting posture and the second threshold curve for distinguishing the lying posture, the standing posture, and the sitting posture.

6. The method of claim 5 , further comprising:
in response to the detected key points including one shoulder key point, obtaining the position of another of the two shoulder key points based on the position of the detected shoulder key point;
in response to the detected key points including one hip key point, obtaining the position of another of the two hip key points based on the position of the detected hip key point; and
in response to the detected key points including one knee key point, obtaining the position of another of the two knee key points based on the position of the detected knee key point.

7. The method of claim 1 , further comprising:
in response to the detected key points including a plurality of head key points among the predefined key points, calculating a head height H head based on the head key points; and
determining the posture of the human by comparing the head height H head with a first head height threshold for distinguishing a standing posture and a sitting posture and a second head height threshold for distinguishing the sitting posture and a lying posture.

8. The method of claim 1 , wherein the predefined key points include two eye key points, a nose key point, two ear key points, a neck key point, two shoulder key points, two elbow key points, two hand key points, two hip key points

Assignees

• Ubtech North America Res And Development Center Corp
• Ubtech Robotics Corp Ltd

Inventors

• Dong Chuqiao
• Shao Dan
• Xiu Zhen
• Guo Dejun
• Tan Huan

Classifications

• G06V40/10 Primary

  Human or animal bodies, e.g. vehicle occupants or pedestrians; Body parts, e.g. hands · CPC title

• G06T7/74

  involving reference images or patches · CPC title

• G06V10/462

  Salient features, e.g. scale invariant feature transforms [SIFT] · CPC title

• G06T2207/10024

  Color image · CPC title

• G06T2207/10028

  Range image; Depth image; 3D point clouds · CPC title