ID association and indoor localization via passive phased-array and computer vision motion correlation

Patent metadata

Field	Value
Publication number	US-10915796-B2
Application number	US-201816175727-A
Country	US
Kind code	B2
Filing date	Oct 30, 2018
Priority date	Oct 30, 2018
Publication date	Feb 9, 2021
Grant date	Feb 9, 2021

Abstract

A system and method for combining computer vision information about human subjects within the field-of-view of a computer vision subsystem with RF Angle of Arrival (AoA) information from an RF receiver subsystem to locate, identify, and track individuals and their location. The RF receiver subsystem may receive RF signals emitted by one or more electronic devices (e.g., a mobile phone) carried, held, or otherwise associated with am individual. Further, gestures can be made with the device and they can be detected by the system.

First claim

I claim:

1. A system for identifying and tracking the location of human subjects, comprising:
an RF receiver subsystem that includes a phased array antenna of multiple antenna elements, wherein the RF receiver subsystem receives RF signals emitted from a plurality of electronic devices each being worn or carried by a human subject, the RF receiver subsystem detecting identifying information in the received RF signals and assigning an identifier (ID) to each of the electronic devices based on the identifying information, and the RF receiver subsystem determining the Angle of Arrival (AoA) of the received RF signals; and
a computer vision subsystem that includes at least one camera to generate image information based on a scene viewed by the camera, the computer vision subsystem processing the image information to locate one or more human subjects in the scene, and to determine the angular position of each of the one or more human subjects relative to the computer vision subsystem;
wherein the identifying information in the received RF signals and the AoA of the received RF signals are fused with the determined angular position of each of the one or more human subjects from the computer vision subsystem based on a probability, determined by comparing the AoA of the received signals to the angular position of each of the one or more human subjects, that one of the electronic devices and one of the one or more human subjects are in a matching location so that each ID assigned to the electronic devices is associated with a specific one of the one or more human subjects and the angular position of each of the one or more human subjects is known.

2. A system as defined in claim 1 , wherein the camera generates three-dimensional image information.

3. A system as defined in claim 2 , wherein the three-dimensional image information includes video image information.

4. A system as defined in claim 1 , wherein the image information includes video image information.

5. A system as defined in claim 1 , wherein the identifying of one or more human subjects includes skeleton tracking.

6. A system as defined in claim 1 , wherein the fusing occurs by correlating AoA and CV with machine learning support vector machines.

7. A system as defined in claim 1 , wherein fusing of the AoA of the received signals and the determined angular position of each of the one or more human subjects to associate each of the IDs with a specific one of the one or more human subjects is performed using a static nearest neighbors algorithm.

8. A system as defined in claim 1 , wherein fusing of the AoA of the received signals and the determined angular position of each of the one or more human subjects to associate each of the IDs with a specific one of the one or more human subjects is performed using path correlation.

9. A system as defined in claim 1 , wherein the computer vision subsystem includes a second camera to allow the system to re-identify the one or more human subjects in a second scene that does not overlap with the first scene.

10. A system as defined in claim 1 , wherein the computer vision subsystem includes a second camera to allow the system to re-identify the one or more human subjects in a second scene that does overlap with the first scene.

11. A system as defined in claim 1 , wherein the computer vision subsystem includes a second camera to allow the system to re-identify the one or more human subjects and wherein the re-identification is performed in part with AoA of received RF signals.

12. A system as defined in claim 1 , wherein the determining of angular position of each human subject relative to the computer vision subsystem further includes determining the distance of each human subject from the computer vision subsystem so as to determine the three-dimensional location of each subject relative to the computer vision subsystem.

13. A system as defined in claim 1 , wherein depth information about each human subject is obtained by using RSSI.

14. A system as defined in claim 1 , wherein the RF signals received by the at least one electronic device include BLE signals.

15. A system as defined in claim 1 , wherein IMU data about movement of the at least one electronic device is provided in the RF signals received therefrom and used in the fusing and identifying.

16. A method for identifying and tracking the location of human subjects, comprising:
RF signals from electronic devices associated with one or more individuals are received;
the angles of arrival and the relative signal strengths of the RF signals are determined;
assigning identification information to each of the electronic devices;
images are obtained of a scene and the presence and location of one or more humans are detected;
processing the images to determine an angular position for each of the one or more humans in the scene and without identifying any of the one or more humans by their name; and
the identification information associated with the electronic devices is then associated with the one or more humans detected in the images after pairing one of the angles of arrival with one of the angular positions of the one or more humans based on a probability, determined by comparing the angles of arrival of the received signals to the angular position of the one or more human, that one of the electronic devices and one of the one or more humans are in a matching location so the identification information assigned to each of the electronic devices is associated with a specific one of the one or more humans and the angular position of the one or more humans is known.

17. A system for identifying and tracking the location of human subjects, comprising:
an RF receiver subsystem that includes a phased array antenna of multiple antenna elements, wherein the RF receiver subsystem receives RF signals emitted from a plurality of electronic devices each being worn or carried by a human subject, the RF receiver subsystem detecting identifying information in the received RF signals, and the RF receiver subsystem determining the Angle of Arrival (AoA) of the received RF signals and assigning an ID to each of the electronic devices based on the detected identifying information; and
a computer vision subsystem that includes at least one camera to generate image information based on a scene viewed by the camera, the computer vision subsystem processing the image information to locate one or more human subjects in the scene and the computer vision subsystem processing the image information to generate the angular position of each of the one or more human subjects relative to the computer vision subsystem;
wherein the AoA of the received RF signals is processed with the generated angular position of each of the one or more human to associate each of the IDs with one of the one or more human subjects;
wherein the processing of the AoA of the received RF signals and the generated angular positions is performed using at least one of a static nearest neighbors algorithm and path correlation;
wherein the identifying information in the received RF signals and the AoA of the received RF signals are associated with the determined angular position of each of the one or more human subjects from the computer vision subsystem based on a probability, determined by comparing the AoA of the received RF signals to the angular position of each of the one or more human subjects, that one of the electronic devices and one of the one or more human subjects are in a matching location; and
wherein the system further determines the distance of each human subjects from th

Assignees

• Disney Entpr Inc

Inventors

• Yang Chouchang
• Sample Alanson

Classifications

• G06V10/80

  Fusion, i.e. combining data from various sources at the sensor level, preprocessing level, feature extraction level or classification level (multimodal speaker identification or verification G10L17/10) · CPC title

• G06V20/10

  Terrestrial scenes (scenes under surveillance with static cameras G06V20/52; scenes perceived from the exterior of a vehicle G06V20/56; scenes perceived from the interior of a vehicle G06V20/59) · CPC title

• G06F18/25

  Fusion techniques · CPC title

• H04W4/029 Primary

  Location-based management or tracking services · CPC title

• G06F18/21

  Design or setup of recognition systems or techniques; Extraction of features in feature space; Blind source separation · CPC title