Mapping sensor data using a mixed-reality cloud

Patent metadata

Field	Value
Publication number	US-2022058884-A1
Application number	US-202117517951-A
Country	US
Kind code	A1
Filing date	Nov 3, 2021
Priority date	Jul 22, 2019
Publication date	Feb 24, 2022
Grant date	—

Abstract

Improved techniques for re-localizing Internet-of-Things (IOT) devices are disclosed herein. Sensor data digitally representing one or more condition(s) monitored by an IOT device is received. In response, a sensor readings map is accessed, where this map is associated with the IOT device. The map also digitally represents the IOT device's environment and includes data representative of a location of the IOT device within the environment. The map also includes data representative of the conditions monitored by the IOT device. Additionally, the map is updated by attaching the sensor data to the map. In some cases, a coverage map can also be computed. Both the sensors readings map and the coverage map can be automatically updated in response to the IOT device being re-localized.

First claim

What is claimed is:

1 . A server computer system configured to execute a mixed-reality (MR) service, said server computer system comprising:
one or more processor(s); and one or more computer-readable hardware storage device(s) that store instructions that are executable by the one or more processor(s) to cause the server computer system to at least:
receive sensor data of an internet of things (IOT) device;
access a sensor readings map associated with the IOT device, the sensor readings map includes the sensor data; and
generate a combined map that includes a first visualization reflecting a coverage map and a second visualization reflecting the sensor readings map, wherein:
the coverage map reflects an operational coverage area of the IOT device,
the operational coverage area starts at the IOT device and spans a determined distance away from the IOT device in the environment, and
the second visualization is structured to overlap at least a part of the first visualization in the combined map.

2 . The server computer system of claim 1 , the MR service also being configured to transmit at least some of the sensor data to a mixed-reality (MR) device that is either (i) physically operating within a same environment as where the IOT device is operating or (ii) displaying a visualization of the environment.

3 . The server computer system of claim 1 , the sensor data including measurement data obtained from a sensor of the IOT device such that the sensor readings map is updated to include the measurement data and such that the server computer system includes updated information regarding sensed operational conditions that are sensed by the IOT device.

4 . The server computer system of claim 1 , the sensor data including image data captured by a camera of the IOT device.

5 . The server computer system of claim 4 , the MR service being further configured to:
in response to receiving the image data from the IOT device, determine re-localization position data for the IOT device.

6 . The server computer system of claim 5 , the MR service being further configured to:
in response to determining the re-localization position data for the IOT device, update the coverage map associated with the IOT device.

7 . The server computer system of claim 1 , the sensor data including inertial measurement unit (IMU) data that is representative of a movement of the IOT device within the environment.

8 . The server computer system of claim 1 , the sensor data including environmental data that is representative of an environment in which the IOT device is operating.

9 . The server computer system of claim 1 , the sensor data including image data that digitally represents a location of the IOT device within an environment, and the MR service being further configured to:
use the image data to determine whether the location of the IOT device has changed by comparing the image data against location data of the IOT device included within a digital representation of the environment.

10 . The server computer system of claim 9 , the digital representation of the environment digitally representing semantically segmented objects located within the environment.

11 . A method for executing a mixed-reality (MR) service, the method comprising:
receiving sensor data of an internet of things (IOT) device; accessing a sensor readings map associated with the IOT device, the sensor readings map includes the sensor data; and generating a combined map that includes a first visualization reflecting a coverage map and a second visualization reflecting the sensor readings map, wherein:
the coverage map reflects an operational coverage area of the IOT device,
the operational coverage area starts at the IOT device and spans a determined distance away from the IOT device in the environment, and
the second visualization is structured to overlap at least a part of the first visualization in the combined map.

12 . The method of claim 11 , wherein the MR service is configured to transmit at least some of the sensor data to a remote mixed-reality device.

13 . The method of claim 11 , wherein the sensor data includes one or more of measurement data, image data, or inertial measurement unit (IMU) data.

14 . The method of claim 11 , wherein the sensor data includes one or more of environmental data or image data.

15 . The method of claim 11 , wherein the method further includes determining a layout of an environment in which the IOT device is operating.

16 . The method of claim 11 , wherein the operational coverage area includes a buffer.

17 . The method of claim 11 , wherein the MR service relocalizes a position of the IOT device.

18 . A server computer system comprising:
one or more processor(s); and one or more computer-readable hardware storage device(s) having stored thereon computer-executable instructions that are executable by the one or more processor(s) to cause the server computer system to execute a mixed-reality (MR) service configured to perform at least the following:
receive sensor data of an internet of things (IOT) device;
access a sensor readings map associated with the IOT device, the sensor readings map includes the sensor data;
determine a layout of an environment in which the IOT device is operating; and
generate a combined map that includes a first visualization reflecting a coverage map and a second visualization reflecting the sensor readings map, wherein:
the coverage map reflects an operational coverage area of the IOT device, the operational coverage area being determined based on the layout of the environment,
the operational coverage area starts at the IOT device and spans a determined distance away from the IOT device in the environment, and
the second visualization is structured to overlap at least a part of the first visualization in the combined map.

19 . The server computer system of claim 18 , the sensor data including one or more of:
image data corresponding to one or more image(s) of the environment; or signal strength data between the IOT device and another device located within the environment.

20 . The server computer system of claim 18 , wherein the operational coverage map includes a buffer.

Assignees

• Microsoft Technology Licensing Llc

Inventors

• Bleyer Michael
• Pekelny Yuri
• Price Raymond Kirk

Classifications

• H04L67/131

  Protocols for games, networked simulations or virtual reality · CPC title

• H04L67/52

  specially adapted for the location of the user terminal · CPC title

• H04W4/70

  Services for machine-to-machine communication [M2M] or machine type communication [MTC] · CPC title

• H04W4/029

  Location-based management or tracking services · CPC title

• G06V20/20

  in augmented reality scenes · CPC title