Augmented reality system for guards of controlled environment residents

Patent metadata

Field	Value
Publication number	US-10810862-B2
Application number	US-201916557882-A
Country	US
Kind code	B2
Filing date	Aug 30, 2019
Priority date	Oct 31, 2017
Publication date	Oct 20, 2020
Grant date	Oct 20, 2020

Abstract

The present disclosure provides details of a system and method for a communication device for guards in a controlled environment. The communication device is established based on a wireless infrastructure in the controlled environment, receiving wireless positioning signals to calculate and determine the real-time location of personnel carrying the device. The indoor wireless positioning can be combined with other available positioning methods to provide highly accurate positioning information of the personnel. The communication device detects and records activities of the personnel during a period of time, and is controlled by the control center. The status of personnel is monitored by control center. Meanwhile, the communication device has AR functions that allow physical objects and augmented reality element to be displayed to the personnel at the same time. The personnel can thus have improved awareness of the surrounding environment and can respond to various potential dangers more effectively.

First claim

What is claimed is:

1. A multi-functional device for use in a controlled environment facility, comprising:
a transceiver configured to communicate with a communication network;
a communication interface connected with a database that stores positioning data for determining a location of a user, monitoring data for determining a real-time status associated with the user, and identification data for identifying objects in a physical environment of the user, and
a processor configured to:
activate a camera in response to an activation signal received from the communication network;
capture, using the camera, object images of objects in the physical environment based on the received activation signal, at least one of the objects being an inmate of the controlled environment facility;
receive a positioning signal reflecting the location of the user and surrounding data reflecting the physical environment from the communication network;
determine the location of the user based on the positioning signal and the positioning data;
recognize the captured objects in the physical environment at the location of the user based on object coordinates included in the surrounding data and the identification data;
determine whether the real-time status of the user is an abnormal status based on at least one of the recognized objects in the physical environment at the location of the user;
in response to the real-time status being an abnormal status, perform an action based on at least one of the objects in the physical environment at the location of the user, or the location of the user; and
determine, based on the positioning signal and the captured object images, whether the inmate is located in a forbidden location.

2. The multi-functional device according to claim 1 , wherein the processor is further configured to: receive a piece of status data; and determine whether the real-time status of the user is an abnormal status based on at least one of the positioning data, the monitoring data, or the piece of status data.

3. The multi-functional device according to claim 1 , wherein the real-time status comprises at least one of a path of the user, the location of the user, a moving speed of the user, a dwelling time of the user, a heart rate of the user, a body motion of the user, a video recording of the user, an audio recording of the user, a message by the user, a response time of the user, a presence of the inmate, an identity of the inmate, a location of the inmate, or a moving speed of the inmate.

4. The multi-functional device according to claim 3 , wherein the real-time status comprises at least one of a frequency of a wireless communication signal or a location of the wireless communication signal.

5. The multi-functional device according to claim 4 , wherein the abnormal status comprises at least one of the path of the user being deviated from a designed path, the location of the user being deviated from a designed location, the moving speed of the user being overly slow, the moving speed of the user being overly fast, the dwelling time of the user being overly long, a sudden change of the heart rate of the user, a sudden change of the body motion of the user, abnormal content in the video recording of the user, abnormal content in the audio recording of the user, abnormal content in the message by the user, an overly long response time of the user, an abnormal presence of the inmate, an identity mismatch of the inmate, an abnormal location of the inmate, an overly fast moving speed of the inmate, an unknown wireless communication signal, a forbidden wireless communication signal, an object at an abnormal condition, or an object at a forbidden location.

6. The multi-functional device according to claim 5 , wherein the action comprises at least one of sending an alert to the user, enabling video recording, enabling image recording, enabling video streaming, sharing the location of the user, sending an alert to other users, sharing the location of the inmate with the other users, or sharing the location of the wireless communication signal with other users.

7. The multi-functional device according to claim 1 , wherein the database further stores profile data for describing the objects in the physical environment of the user, and the processor is further configured to generate an augmented reality (AR) element based on the profile data and the objects in the physical environment.

8. The multi-functional device according to claim 7 , wherein the AR element includes at least one of identities of the objects, profile of the objects, status of the objects, or the real-time status of the user.

9. The multi-functional device according to claim 1 , wherein the physical environment includes at least one of a front environment, a rear environment, or side environments of the user.

10. The multi-functional device according to claim 1 , wherein a plurality of positioning signals comprises at least one of an indoor positioning signal or an outdoor positioning signal.

11. The multi-functional device according to claim 10 , wherein the indoor positioning signal includes at least one of a radio frequency signal, a Bluetooth signal, a WiFi signal, or an ultrasonic signal.

12. The multi-functional device according to claim 1 , wherein the positioning data comprises reference positioning data for determining the location of the user, and the monitoring data comprises criteria for determining the real-time status of the user.

13. A multi-functional device for positioning and monitoring a first user for use in a controlled environment facility, the multi-functional device comprising:
a transceiver configured to communicate with a communication network;
a communication interface connected with a database that stores positioning data for determining a location of the first user, monitoring data for determining a real-time status associated with the first user, and identification data for identifying objects in a physical environment of the first user, and
a processor configured to:
activate a camera in response to an activation signal received from the communication network;
capture, using the camera, object images of objects in the physical environment based on the received activation signal, at least one of the objects being an inmate of the controlled environment facility;
receive a positioning signal reflecting the location of the first user and surrounding data reflecting the physical environment from the communication network;
determine the location of the first user based on the positioning signal and the positioning data;
recognize the captured objects in the physical environment at the location of the first user based on object coordinates included in the surrounding data and profile data;
determine whether the real-time status of the first user is an abnormal status based on at least one of the recognized objects in the physical environment at the location of the first user;
receive an interrupting command from a second user;
in response to the real-time status being an abnormal status, perform an action based on at least one of the objects in the physical environment at the location of the first user, the location of the first user, or the interrupting command from the second user; and
determine, based on the positioning signal and the object images, whether the inmate is in a forbidden location.

14. The multi-functional device according to claim 13 , wherein the processor is further configured to: receive a piece of status data; and determine whether the real-time status of the first user is an abnormal status based on at least one of the p

Assignees

• Global Telstarlink Corp

Inventors

• Hodge Stephen L

Classifications

• G06Q10/063114

  Status monitoring or status determination for a person or group · CPC title

• G06Q10/0639

  Performance analysis of employees; Performance analysis of enterprise or organisation operations · CPC title

• H04W4/021 Primary

  Services related to particular areas, e.g. point of interest [POI] services, venue services or geofences · CPC title

• G06V20/52

  Surveillance or monitoring of activities, e.g. for recognising suspicious objects (recognising microscopic objects G06V20/69) · CPC title

• G06V20/20

  in augmented reality scenes · CPC title