System and method for generating acupuncture points on reconstructed 3D human body model for physical therapy

Patent metadata

Field	Value
Publication number	US-11475630-B2
Application number	US-201916560913-A
Country	US
Kind code	B2
Filing date	Sep 4, 2019
Priority date	Oct 17, 2018
Publication date	Oct 18, 2022
Grant date	Oct 18, 2022

Abstract

System and method for generating acupuncture points on reconstructed 3D human body mesh for physical treatment are disclosed. The computing device obtains a first two-dimensional image of a human subject that captures at least a predefined portion of the human subject; processes the first two-dimensional image of the first human subject using a trained human body recovery model to obtain a plurality of parameters representing a three-dimensional human body mesh with corresponding acupuncture points. The trained human body recovery model includes an iterative three-dimensional regression module that is supervised by a discriminator and that minimizes a combined loss below a preset threshold. The combined loss includes a discriminator error that provides a measure of whether the obtained three-dimensional human body mesh with corresponding acupuncture points correspond to real human shape, pose, and acupuncture points.

First claim

What is claimed is:

1. A method, comprising:
at a computing system having one or more processors and memory:
obtaining a first two-dimensional image of a human subject, the first two-dimensional image of the first human subject captures at least a predefined portion of the human subject;
processing the first two-dimensional image of the first human subject using a trained human body recovery model to obtain a plurality of parameters representing a three-dimensional human body mesh with corresponding acupuncture points, wherein:
the trained human body recovery model includes an iterative three-dimensional regression module that is supervised by a discriminator and that minimizes a combined loss below a preset threshold, the combined loss including a reprojection error of an encoded two-dimensional input image of a human body, a three-dimensional ground truth error, and a discriminator error,
the three-dimensional ground truth error includes respective errors related to a pose estimation, a shape estimation, and an acupuncture points estimation relative to annotated three-dimensional human bodies, and
the discriminator error provides a measure of whether the obtained three-dimensional human body mesh with corresponding acupuncture points corresponds to real human shape, pose, and acupuncture points; and
generating treatment data corresponding to the first human subject in accordance with the obtained three-dimensional human body mesh with corresponding acupuncture points.

2. The method of claim 1 , further comprising training the human body recovery model using annotated 2D images, annotated 3D human body meshes, and 3D datasets including annotated acupuncture points marked on respective 3D human body meshes.

3. The method of claim 1 , wherein the human body recovery model comprises a plurality of human body recovery sub-models corresponding to a plurality of portions of a human body respectively.

4. The method of claim 3 , further comprising training a respective human body recovery sub-model using annotation data of the corresponding portion on the three-dimensional human body corresponding to pose, shape, and a subset of acupuncture points associated with the corresponding portion.

5. The method of claim 3 , further comprising reconstructing a portion of a human body with corresponding acupuncture points, comprising:
dividing a reconstructed 3D full human body mesh into a plurality of reconstructed portions; and
identifying a reconstructed portion corresponding to the received 2D image of the portion of the human subject.

6. The method of claim 1 , wherein the first two-dimensional image of the first human subject captures a first portion of the first human subject, and wherein the two-dimensional image of the first human subject is processed using a first human body recovery sub-model corresponding to the first portion of the first human subject.

7. The method of claim 1 , further comprising:
obtaining a second two-dimensional image that captures a second portion of the of the human subject that is distinct from the first portion;
processing the second two-dimensional image of the second portion of the first human subject using a second human body recovery sub-model to obtain a second set of parameters representing a second three-dimensional human body mesh corresponding to the second portion of the human subject with a second set of acupuncture points associated with the second portion of the first human body; and
updating the treatment data in accordance with the second three-dimensional human body mesh corresponding to the second portion with a corresponding second set of acupuncture points.

8. A computing device, comprising:
one or more processors; and
memory storing instructions, wherein the instructions, when executed by the one or more processors, cause the processors to perform operations comprising:
obtaining a first two-dimensional image of a human subject, the first two-dimensional image of the first human subject captures at least a predefined portion of the human subject;
processing the first two-dimensional image of the first human subject using a trained human body recovery model to obtain a plurality of parameters representing a three-dimensional human body mesh with corresponding acupuncture points, wherein:
the trained human body recovery model includes an iterative three-dimensional regression module that is supervised by a discriminator and that minimizes a combined loss below a preset threshold, the combined loss including a reprojection error of an encoded two-dimensional input image of a human body, a three-dimensional ground truth error, and a discriminator error,
the three-dimensional ground truth error includes respective errors related to a pose estimation, a shape estimation, and an acupuncture points estimation relative to annotated three-dimensional human bodies, and
the discriminator error provides a measure of whether the obtained three-dimensional human body mesh with corresponding acupuncture points corresponds to real human shape, pose, and acupuncture points; and
generating treatment data corresponding to the first human subject in accordance with the obtained three-dimensional human body mesh with corresponding acupuncture points.

9. The computing device of claim 8 , wherein the operations include training the human body recovery model using annotated 2D images, annotated 3D human body meshes, and 3D datasets including annotated acupuncture points marked on respective 3D human body meshes.

10. The computing device of claim 8 , wherein the human body recovery model comprises a plurality of human body recovery sub-models corresponding to a plurality of portions of a human body respectively.

11. The computing device of claim 10 , wherein the operations include training a respective human body recovery sub-model using annotation data of the corresponding portion on the three-dimensional human body corresponding to pose, shape, and a subset of acupuncture points associated with the corresponding portion.

12. The computing device of claim 10 , wherein the operations further include reconstructing a portion of a human body with corresponding acupuncture points, comprising:
dividing a reconstructed 3D full human body mesh into a plurality of reconstructed portions; and
identifying a reconstructed portion corresponding to the received 2D image of the portion of the human subject.

13. The computing device of claim 8 , wherein the first two-dimensional image of the first human subject captures a first portion of the first human subject, and wherein the two-dimensional image of the first human subject is processed using a first human body recovery sub-model corresponding to the first portion of the first human subject.

14. The computing device of claim 8 , wherein the operations further include:
obtaining a second two-dimensional image that captures a second portion of the human subject that is distinct from the first portion;
processing the second two-dimensional image of the second portion of the first human subject using a second human body recovery sub-model to obtain a second set of parameters representing a second three-dimensional human body mesh corresponding to the second portion of the human subject with a second set of acupuncture points associated with the second portion of the first human body; and
updating the treatment data in accordance with the second three-dimensional human body mesh corresponding to the second portion with a corresponding second set of acupuncture points.

15. A non-transitory computer-readable storage medium stores instructions

Assignees

• Midea Group Co Ltd

Inventors

• Tian Yuan
• Mani Suresh

Classifications

• A61B5/1116

  Determining posture transitions · CPC title

• G06T7/75 Primary

  involving models · CPC title

• A61H2230/085

  used as a control parameter for the apparatus · CPC title

• G06T19/00

  Manipulating three-dimensional [3D] models or images for computer graphics · CPC title

• A61H39/02

  Devices for locating such points · CPC title