Techniques for inferring the configuration of a room from skeleton tracking

Patent metadata

Field	Value
Publication number	US-11587239-B2
Application number	US-201916354094-A
Country	US
Kind code	B2
Filing date	Mar 14, 2019
Priority date	Mar 14, 2019
Publication date	Feb 21, 2023
Grant date	Feb 21, 2023

Abstract

In various embodiments, a map inference application automatically maps a user space. A camera is positioned within the user space. In operation, the map inference application determines a path of a first moving object within the user space based on a tracking dataset generated from images captured by the camera. Subsequently, the map inference application infers a walking space within the user space based on the path. The map inference application then generates a model of at least a portion of the user space based on the walking space. One or more movements of a second object within the user space are based on the model. Advantageously, unlike prior art solutions, the map inference application enables a model of a user space to be automatically and efficiently generated based on images from a single stationary camera.

First claim

What is claimed is:

1. A computer-implemented method for mapping a user space, comprising:
determining a path of a first moving object within the user space based on one or more skeletons included in a skeleton sequence, wherein the skeleton sequence includes one or more two-dimensional (2D) coordinates for one or more joints, and wherein the skeleton sequence is generated from images that visually depict motion and are captured by a camera positioned within the user space;
inferring a walking space within the user space based on the path;
determining that at least one skeleton in the skeleton sequence is missing a first joint;
generating an obstacle within the user space based on the first joint; and
generating a model of at least a portion of the user space based on the walking space and the obstacle, wherein one or more movements of a second object within the user space are based on the model.

2. The computer-implemented method of claim 1 , wherein generating the model comprises:
inferring a location and a size of a third stationary object within the user space based on an occlusion associated with the first moving object;
generating a second obstacle based on the location and the size of the third stationary object; and
representing a spatial relationship between the walking space and the second obstacle in the model.

3. The computer-implemented method of claim 1 , wherein generating the model comprises:
inferring a wall based on the path; and
representing the walking space as a horizontal surface area and the wall as a vertical surface area in the model.

4. The computer-implemented method of claim 1 , wherein the model comprises a three-dimensional (3D) model of the at least a portion of the user space or a 2D model of at least a portion of the walking space.

5. The computer-implemented method of claim 1 , wherein the one or more skeletons represent the first moving object at different points in time.

6. The computer-implemented method of claim 1 , wherein the one or more joints comprise a physical joint in a human skeleton.

7. The computer-implemented method of claim 1 , wherein the camera comprises a video camera or a still camera.

8. The computer-implemented method of claim 1 , further comprising performing one or more tracking operations on the images captured by the camera to generate the skeleton sequence.

9. The computer-implemented method of claim 1 , wherein the first moving object comprises a person, and further comprising determining the path based on one or more sets of locations for one or more facial features associated with the person.

10. The computer-implemented method of claim 1 , wherein the second object comprises a person or an inanimate object.

11. One or more non-transitory computer readable media including instructions that, when executed by one or more processors, cause the one or more processors to map a user space by performing the steps of:
determining a path of a first moving object within the user space based on one or more skeletons included in a skeleton sequence, wherein the skeleton sequence includes one or more two-dimensional (2D) coordinates for one or more joints, and wherein the skeleton sequence is generated from images that visually depict motion and are captured by a camera positioned within the user space;
inferring a walking space within the user space based on the path;
determining that at least one skeleton in the skeleton sequence is missing a first joint;
generating an obstacle within the user space based on the first joint; and
generating a model of at least a portion of the user space based on the walking space and the obstacle, wherein one or more movements of a second object within the user space are based on the model.

12. The one or more non-transitory computer readable media of claim 11 , wherein generating the model comprises:
detecting an occlusion of the first moving object based on one or more missing joints associated with a partial skeleton included in the skeleton sequence;
generating a second obstacle based on the occlusion; and
representing a spatial relationship between the walking space and the second obstacle in the model.

13. The one or more non-transitory computer readable media of claim 11 , wherein generating the model comprises:
inferring a wall based on the path; and
representing the walking space as a horizontal surface area and the wall as a vertical surface area in the model.

14. The one or more non-transitory computer readable media of claim 11 , wherein the model comprises a 3D model of the at least a portion of the user space or a 2D model of at least a portion of the walking space.

15. The one or more non-transitory computer readable media of claim 11 , wherein the skeleton sequence includes skeleton objects that represent the first moving object at different points in time.

16. The one or more non-transitory computer readable media of claim 15 , wherein each skeleton object included in the skeleton sequence comprises a plurality of positions for a plurality of joints.

17. The one or more non-transitory computer readable media of claim 11 , wherein the camera is at a fixed position and a fixed orientation within the user space.

18. The one or more non-transitory computer readable media of claim 11 , further comprising performing one or more tracking operations on the images captured by the camera to generate the skeleton sequence.

19. The one or more non-transitory computer readable media of claim 11 , further comprising performing one or more facial recognition operations on the images captured by the camera to generate one or more facial features associated with the first moving object.

20. A system, comprising:
one or more memories storing instructions; and
one or more processors that are coupled to the one or more memories and,
when executing the instructions, are configured to:
determine a path of a first moving object within a user space based on one or more skeletons included in a skeleton sequence, wherein the skeleton sequence includes one or more two-dimensional (2D) coordinates for one or more joints, and wherein the skeleton sequence is generated from images that visually depict motion and are captured by a camera positioned within the user space;
infer a walking space within the user space based on the path;
determine that at least one skeleton in the skeleton sequence is missing a first joint;
generate an obstacle within the user space based on the first joint; and
generate a model of at least a portion of the user space based on the walking space, wherein one or more movements of a second object within the user space are based on the model.

21. The computer-implemented method of claim 1 , wherein determining the path of the first moving object within the user space is further based on a trajectory of the first moving object and a walking plane associated with the trajectory of the first moving object.

22. The computer-implemented method of claim 1 , wherein the first joint is included in the one or more joints.

23. The computer-implemented method of claim 1 , wherein determining that at least one skeleton in the skeleton sequence is missing the first joint comprises determining that a first skeleton in the skeleton sequence is a complete skeleton and a second skeleton in the skeleton sequence is a partial skeleton, and wherein the first joint is included in the first skeleton but is not included in the second skeleton.

Assignees

• Disney Entpr Inc

Inventors

• Baumbach Elliott
• Goslin Michael

Classifications

• G06T2207/30241

  Trajectory · CPC title

• G06T15/00

  Three-dimensional [3D] image rendering · CPC title

• G06V20/10

  Terrestrial scenes (scenes under surveillance with static cameras G06V20/52; scenes perceived from the exterior of a vehicle G06V20/56; scenes perceived from the interior of a vehicle G06V20/59) · CPC title

• G06T2207/10016

  Video; Image sequence · CPC title

• G06T7/246 Primary

  using feature-based methods, e.g. the tracking of corners or segments · CPC title