Systems and methods for generating three dimensional skeleton representations

Patent metadata

Field	Value
Publication number	US-10607079-B2
Application number	US-201815868374-A
Country	US
Kind code	B2
Filing date	Jan 11, 2018
Priority date	Sep 26, 2017
Publication date	Mar 31, 2020
Grant date	Mar 31, 2020

Abstract

Systems, robots, and methods for generating three-dimensional skeleton representations of people are disclosed. A method includes generating, from a two-dimensional image, a two-dimensional skeleton representation of a person present in the two-dimensional image. The two-dimensional skeleton representation includes a plurality of joints and a plurality of links between individual joints of the plurality of joints. The method further includes positioning a cone around one or more links of the plurality of links, and identifying points of a depth cloud that intersect with the cone positioned around the one or more links of the two-dimensional skeleton. The points of the depth cloud are generated by a depth sensor and each point provides depth information. The method also includes projecting the two-dimensional skeleton representation into three-dimensional space using the depth information of the points that intersect with the cone, thereby generating the three-dimensional skeleton representation of the person.

First claim

The invention claimed is:

1. A method of generating a three-dimensional skeleton representation of a person and recording gait information, the method comprising:
generating, from a two-dimensional image, a two-dimensional skeleton representation of a person present in the two-dimensional image, wherein the two-dimensional skeleton representation comprises a plurality of joints and a plurality of links between individual joints of the plurality of joints;
positioning one or more ring structures around one or more links of the plurality of links such that the one or more links pass through a center of the one or more ring structures;
identifying points of a depth cloud that intersect with the one or more ring structures positioned around the one or more links of the plurality of links, wherein the points of the depth cloud are generated by a depth sensor and each point provides depth information;
projecting the two-dimensional skeleton representation into three-dimensional space using the depth information of the points of the depth cloud that intersect with the one or more ring structures positioned around the one or more links of the plurality of links, thereby generating the three-dimensional skeleton representation of the person; and
recording gait information from the three-dimensional skeleton representation.

2. The method of claim 1 , wherein projecting the two-dimensional skeleton representation into three-dimensional space comprises averaging the depth information of all of the points of the depth cloud that intersect with the one or more ring structures positioned around the one or more links of the plurality of links.

3. The method of claim 1 , further comprising authenticating a user based at least in part on the three-dimensional skeleton representation.

4. The method of claim 3 , wherein authenticating the user comprises:
obtaining a pose of the user from the three-dimensional skeleton representation;
accessing a memory comprising a plurality of stored poses for a plurality of registered users; and
matching the pose of the user with one of the plurality of stored poses.

5. The method of claim 1 , further comprising obtaining image data from one or more sensors, wherein the image data comprises the two-dimensional image.

6. The method of claim 1 , wherein the one or more ring structures are shaped and sized to correspond to one or more body parts of the person.

7. The method of claim 1 , further comprising authenticating a user based at least in part on the gait information.

8. The method of claim 7 , wherein authenticating the user comprises:
accessing a memory comprising a plurality of stored gaits for a plurality of registered users; and
matching the gait information with one of the plurality of stored gaits.

9. A robot comprising:
a processor and a non-transitory memory device storing machine-readable instructions that, when executed by the processor, cause the processor to:
generate, from a two-dimensional image, a two-dimensional skeleton representation of a person present in the two-dimensional image, wherein the two-dimensional skeleton representation comprises a plurality of joints and a plurality of links between individual joints of the plurality of joints;
position one or more ring structures around one or more links of the plurality of links such that the one or more links pass through a center of the one or more ring structures;
identify points of a depth cloud that intersect with the one or more ring structures positioned around the one or more links of the plurality of links, wherein the points of the depth cloud are generated by a depth sensor and each point provides depth information;
project the two-dimensional skeleton representation into three-dimensional space using the depth information of the points of the depth cloud that intersect with the one or more ring structures positioned around the one or more links of the plurality of links, thereby generating a three-dimensional skeleton representation of the person; and
record gait information from the three-dimensional skeleton representation.

10. The robot of claim 9 , wherein projecting the two-dimensional skeleton representation into three-dimensional space comprises averaging the depth information of all of the points of the depth cloud that intersect with the one or more ring structures positioned around the one or more links.

11. The robot of claim 9 , wherein the machine-readable instructions further cause the processor to authenticate a user based at least in part on the three-dimensional skeleton representation.

12. The robot of claim 11 , wherein the machine-readable instructions that cause the processor to authenticate the user further cause the processor to:
obtain a pose of the user from the three-dimensional skeleton representation;
access a memory comprising a plurality of stored poses for a plurality of registered users; and
match the pose of the user with one of the plurality of stored poses.

13. The robot of claim 9 , further comprising an arm and end effector positioned at an end of the arm, wherein the machine-readable instructions further cause the processor to:
determine an object held in a hand of the person, and
determine a location of the hand of the person in three-dimensional space from the three-dimensional skeleton representation of the person.

14. The robot of claim 13 , wherein the machine-readable instructions further cause the arm and end effector to move toward the hand of the person and grasp the object.

15. The robot of claim 13 , further comprising a plurality of sensors communicatively coupled to the processor, wherein the plurality of sensors provide the two-dimensional image.

16. A system comprising:
a processor; and
a non-transitory, processor-readable storage device, the non-transitory, processor-readable storage device comprising one or more machine-readable instructions thereon that, when executed by the processor, cause the processor to:
generate, from a two-dimensional image, a two-dimensional skeleton representation of a person present in the two-dimensional image, wherein the two-dimensional skeleton representation comprises a plurality of joints and a plurality of links between individual joints of the plurality of joints;
position one or more ring structures around one or more links of the plurality of links such that the one or more links pass through a center of the one or more ring structures;
identify points of a depth cloud that intersect with the one or more ring structures positioned around the one or more links of the plurality of links, wherein the points of the depth cloud are generated by a depth sensor and each point provides depth information;
project the two-dimensional skeleton representation into three-dimensional space using the depth information of the points of the depth cloud that intersect with the one or more ring structures positioned around the one or more links of the plurality of links, thereby generating a three-dimensional skeleton representation of the person; and
record gait information from the three-dimensional skeleton representation.

17. The system of claim 16 , wherein the one or more machine-readable instructions that cause the processor to project the two-dimensional skeleton representation into three-dimensional space further causes the processor to average the depth information of all of the points of the depth cloud that intersect with the one or more ring structures positioned around the one or more links of the plurality of links.

18. The system of cl

Assignees

• Toyota Res Inst Inc

Inventors

• Northcutt Brandon
• Stone Kevin
• Mushegian Konstantine
• Bouma Katarina
• Murase Kazuto
• Ochiai Akiyoshi

Classifications

• G06T15/205

  Image-based rendering · CPC title

• G06T2207/10024

  Color image · CPC title

• G06T2207/30196

  Human being; Person · CPC title

• G06T7/50 Primary

  Depth or shape recovery · CPC title

• G06T7/75

  involving models · CPC title