Subject identification and tracking using image recognition

Patent metadata

Field	Value
Publication number	US-10055853-B1
Application number	US-201715847796-A
Country	US
Kind code	B1
Filing date	Dec 19, 2017
Priority date	Aug 7, 2017
Publication date	Aug 21, 2018
Grant date	Aug 21, 2018

Abstract

Systems and techniques are provided for tracking multi joint subjects in real space having a plurality of cameras. The field of view of each camera overlaps with at least one other camera. The plurality of cameras produce respective sequences of images of corresponding fields of view in the real space. A processing system is coupled to the plurality of cameras. In one embodiment, the processing system comprises image recognition engines receiving sequence of images from the plurality of cameras and generating corresponding arrays of joint data structures. A tracking engine is configured to receive the arrays of joint data structures and generate candidate joints having coordinates in the real space. The processing system includes the logic to identify sets of candidate joints having coordinates in the real space as multi-joint subjects in the real space.

First claim

What is claimed is:

1. A system for tracking multi joint subjects in an area of real space, comprising:
a plurality of cameras, cameras in the plurality of cameras producing respective sequences of images of corresponding fields of view in the real space, the field of view of each camera overlapping with the field of view of at least one other camera in the plurality of cameras;
a processing system coupled to the plurality of cameras, the processing system including:
image recognition engines, receiving the sequences of images from the plurality of cameras, which process images to generate corresponding arrays of joint data structures, the arrays of joint data structures corresponding to particular images classifying elements of the particular images by joint type, time of the particular image, and coordinates of the element in the particular image;
a tracking engine configured to receive the arrays of joint data structures corresponding to images in sequences of images from cameras having overlapping fields of view, and translate the coordinates of the elements in the arrays of joint data structures corresponding to images in different sequences into candidate joints having coordinates in real space; and
logic to identify sets of candidate joints having coordinates in real space as multi-joint subjects in the real space.

2. The system of claim 1 , wherein the image recognition engines comprise convolutional neural networks.

3. The system of claim 1 , wherein image recognition engines process images to generate confidence arrays for elements of the image, where a confidence array for a particular element of an image includes confidence values for a plurality of joint types for the particular element, and to select a joint type for the joint data structure of the particular element based on the confidence array.

4. The system of claim 1 , wherein the logic to identify sets of candidate joints comprises heuristic functions based on physical relationships among joints of subjects in real space to identify sets of candidate joints as multi-joint subjects.

5. The system of claim 4 , including logic to store the sets of joints identified as multi-joint subjects, and wherein the logic to identify sets of candidate joints includes logic to determine whether a candidate joint identified in images taken at a particular time corresponds with a member of one of the sets of candidate joints identified as multi-joint subjects in preceding images.

6. The system of claim 1 , wherein cameras in the plurality of cameras are configured to generate synchronized sequences of images.

7. The system of claim 1 , wherein the plurality of cameras comprise cameras disposed over and having fields of view encompassing respective parts of the area in real space, and the coordinates in real space of members of a set of candidate joints identified as a multi-joint subject identify locations in the area of the multi-joint subject.

8. The system of claim 1 , including logic to track locations of a plurality of multi-joint subjects in the area of real space.

9. The system of claim 8 , including logic to determine when multi joint subjects in the plurality of multi-joint subjects leave the area of real space.

10. The system of claim 1 , including logic to track locations in the area of real space of multiple candidate joints that are members of a set of candidate joints identified as a particular multi joint subject.

11. A method for tracking multi-joint subjects in an area of real space, comprising:
using a plurality of cameras to produce respective sequences of images of corresponding fields of view in the real space, the field of view of each camera overlapping with the field of view of at least one other camera in the plurality of cameras;
processing images in the sequences of images to generate corresponding arrays of joint data structures, the arrays of joint data structures corresponding to particular images classifying elements of the particular images by joint type, time of the particular image, and coordinates of the element in the particular image;
translating the coordinates of the elements in the arrays of joint data structures corresponding to images in different sequences into candidate joints having coordinates in the real space; and
identifying sets of candidate joints having coordinates in real space as multi-joint subjects in the real space.

12. The method of claim 11 , wherein said processing images includes using convolutional neural networks.

13. The method of claim 11 , wherein said processing images includes generating confidence arrays for elements of the image, where a confidence array for a particular element of an image includes confidence values for a plurality of joint types for the particular element, and selecting a joint type for the joint data structure of the particular element based on the confidence array.

14. The method of claim 11 , wherein identifying sets of candidate joints comprises applying heuristic functions based on physical relationships among joints of subjects in real space to identify sets of candidate joints as multi-joint subjects.

15. The method of claim 14 , including storing the sets of joints identified as multi-joint subjects, and wherein the identifying sets of candidate joints includes determining whether a candidate joint identified in images taken at a particular time corresponds with a member of one of the sets of candidate joints identified as a multi-joint subject in a preceding image.

16. The method of claim 11 , wherein the sequences of images are synchronized.

17. The method of claim 11 , wherein the plurality of cameras comprise cameras disposed over and having fields of view encompassing respective parts of the area in real space, and the coordinates in real space of members of a set of candidate joints identified as a multi-joint subject identify locations in the area of the multi-joint subject.

18. The method of claim 11 , including tracking locations of a plurality of multi-joint subjects in the area of real space.

19. The method of claim 18 , including determining when a multi-joint subject in the plurality of multi-joint subjects leaves the area of real space.

20. The method of claim 11 , including tracking locations in the area of real space of multiple candidate joints that are members of a set of candidate joints identified as a particular multi joint subject.

21. A computer program product, comprising:
a computer readable memory comprising a non-transitory data storage medium;
computer instructions stored in the memory executable by a computer to track multi-joint subjects in an area of real space by a process including:
using sequences of images from a plurality of cameras having corresponding fields of view in real space, the field of view of each camera overlapping with the field of view of at least one other camera in the plurality of cameras;
processing images in the sequences of images to generate corresponding arrays of joint data structures, the arrays of joint data structures corresponding to particular images classifying elements of the particular images by joint type, time of the particular image, and coordinates of the element in the particular image;
translating the coordinates of the elements in the arrays of joint data structures corresponding to images in different sequences into candidate joints having coordinates in the real space; and
identifying sets of candidate joints having coordinates in real space as multi-joint subjects in the

Assignees

• Standard Cognition Corp

Inventors

• Fisher Jordan
• Valdman David
• Novak John
• Ogle Brandon

Classifications

• H04N23/90

  Arrangement of cameras or camera modules, e.g. multiple cameras in TV studios or sports stadiums · CPC title

• H04N25/61

  the noise originating only from the lens unit, e.g. flare, shading, vignetting or "cos4" · CPC title

• G06V10/147

  Details of sensors, e.g. sensor lenses (fingerprint or palmprint sensors G06V40/13; vascular sensors G06V40/145; eye sensors G06V40/19) · CPC title

• G06V10/16

  using multiple overlapping images; Image stitching · CPC title

• G06V10/454

  Integrating the filters into a hierarchical structure, e.g. convolutional neural networks [CNN] · CPC title