Augmented reality of a building

Patent metadata

Field	Value
Publication number	US-11875665-B2
Application number	US-202217970128-A
Country	US
Kind code	B2
Filing date	Oct 20, 2022
Priority date	Jun 27, 2017
Publication date	Jan 16, 2024
Grant date	Jan 16, 2024

Abstract

The present disclosure relates to an augmented reality system including one or more of a building, sensor(s), a gateway, a server, and a monitoring computer. The augmented reality system may further includes drones that capture video or other images of the building. The monitoring computer gathers information from the sensor(s) and/or drones and facilitates the creation of an augmented view of the building to provide intelligence to emergency responders.

First claim

What is claimed is:

1. A method of monitoring a structure with an augmented reality system, the method comprising:
receiving, at a self-propelled monitoring device, instructions to monitor an assigned area associated with a monitored structure;
in response to the instructions to monitor the assigned area, the self-propelled monitoring device traveling to the assigned area;
receiving, by the self-propelled monitoring device, sensor data from one or more building sensors associated with the assigned area;
determining one or more vantage points from which to monitor the assigned area, the determination based on the received sensor data;
capturing, by the self-propelled monitoring device, image data of the assigned area from at least one of the vantage points; and
creating, by the self-propelled monitoring device, an augmented view of the monitored structure, wherein the augmented view incorporates, occupancy data, the image data of the assigned area, a building layout, and the sensor data from the one or more building sensors.

2. The method of claim 1 , wherein the self-propelled monitoring device comprises an aerial drone.

3. The method of claim 1 , wherein determining the one or more vantage points is based on a safety of the vantage point, an availability of sensor data from the vantage point, and an availability of image data from the vantage point; and
wherein the image data of the monitored structure comprises at least one of real-time video and one or more still images.

4. The method of claim 1 , wherein creating the augmented view of the monitored structure comprises selectively overlaying at least a portion of the sensor data on the image data of the monitored structure.

5. The method of claim 1 , wherein the sensor data comprises broadcast information indicative of a sensed condition, an identity code associated with each of the respective building sensors, and location information for each of the respective building sensors.

6. The method of claim 1 , further comprising transmitting the augmented view to a monitoring computer for display.

7. An augmented reality system, the system comprising:
a monitoring computer configured to receive information regarding a monitored structure, the information comprising a building layout; and
a self-propelled monitoring device communicatively coupled to the monitoring computer, wherein the movable monitoring device is configured to:
receive, from the monitoring computer, instructions to monitor an assigned area associated with the monitored structure;
in response to the instructions, travel to the assigned area;
receive sensor data from one or more of the building sensors associated with the monitored structure;
determine one or more vantage points from which to monitor the assigned area based on received sensor date from at least one of the building sensors;
travel to at least one of the vantage points; and
capture image data of the assigned area associated with the monitored structure;

wherein the self-propelled monitoring device is further configured to create an augmented view of the monitored structure, wherein the augmented view selectively incorporates the building layout, the image data of the assigned area, and the sensor data from the one or more building sensors, the augmented view further incorporating an indication of whether the monitored structure is occupied.

8. The system of claim 7 , wherein the monitoring computer is
communicatively coupled to a server, and wherein the monitoring computer is configured to receive layout information for the monitored structure from the server.

9. The system of claim 8 , wherein one of the monitoring computer and the self-propelled monitoring device is configured to determine the instructions to travel to the vantage point based on the layout information received from the server.

10. The system of claim 7 , further comprising the one or more building sensors, wherein the one or more building sensors are configured to broadcast the sensor data as a signal at regular intervals, and wherein the signal comprises identification information for each of the one or more building sensors and data indicative a sensed condition.

11. The system of claim 7 , wherein the self-propelled monitoring device comprises an aerial drone.

12. The system of claim 7 , wherein the sensed environment associated with the structure comprises at least one of a safety of the vantage point, an availability of the sensor data from the vantage point, and an availability of the image data from the vantage point; and wherein the image data of the monitored structure comprises real-time video.

13. The system of claim 7 , wherein the self-propelled monitoring device is further configured to at least one of:
(a) determine the sensed environment based on the received sensor data from the one or more building sensors; or
(b) monitor at least one of the structure and surroundings of the structure to determine the sensed environment.

14. The system of claim 13 , wherein the self-propelled monitoring device is further configured to:
receive building data representing building shape, height, and structures or topology around the building; and
determine the one or more vantage points based on the received sensor and building data.

15. A method of monitoring a structure with an augmented reality system, the method comprising:
transmitting, by a monitoring computer to a self-propelled monitoring device, instructions to monitor an assigned area associated with a monitored structure;
traveling, by the self-propelled monitoring device, to the assigned area;
receiving, by the self-propelled monitoring device, sensor data from one or more building sensors associated with the assigned area;
receiving, by the monitoring computer from the self-propelled monitoring device, the sensor data from one or more building sensors associated with the assigned area;
determining one or more a vantage points from which to monitor the assigned area based on the sensor data from one or more building sensors associated with the assigned area;
receiving, by the monitoring computer from the self-propelled monitoring device, image data of the assigned area;
creating, by the monitoring computer, an augmented view of the monitored structure, wherein the augmented view selectively incorporates the image data of the assigned area, a floorplan of the structure, and the sensor data from the one or more of the building sensors; and
determining the severity of a hazardous condition using sensor data and image data and detecting a rate of spread of the hazardous condition.

16. The method of claim 15 , wherein the sensor data comprises broadcast information indicative of a sensed condition, an identity code associated with each of the respective building sensors, and location information for each of the respective building sensors.

17. The method of claim 15 , further comprising:
receiving, by the monitoring computer, information regarding a layout of the monitored structure; and
determining the instructions to travel to the destination based on the information regarding the layout of the monitoring structure.

18. The method of claim 15 , further comprising:
displaying, by the monitoring computer, a list of the one or more building sensors;
receiving, at the monitoring computer, a selection of a specific location within the monitored structure from which to receive additional data; and
presenting, by the monitoring computer, additional information regarding the specific location in response to the sel

Assignees

• Oneevent Tech Inc

Inventors

• Wedig Kurt Joseph
• Parent Daniel Ralph

Classifications

• G08B27/001 Primary

  Signalling to an emergency team, e.g. firemen · CPC title

• G08B17/00

  Fire alarms; Alarms responsive to explosion · CPC title

• H04N7/181 Primary

  for receiving images from a plurality of remote sources · CPC title

• H04N7/185

  from a mobile camera, e.g. for remote control · CPC title

• G08B13/1965

  the vehicle being an aircraft · CPC title