Mapping sensor data using a mixed-reality cloud

Patent metadata

Field	Value
Publication number	US-2021027538-A1
Application number	US-201916517976-A
Country	US
Kind code	A1
Filing date	Jul 22, 2019
Priority date	Jul 22, 2019
Publication date	Jan 28, 2021
Grant date	—

Abstract

Improved techniques for re-localizing Internet-of-Things (IOT) devices are disclosed herein. Sensor data digitally representing one or more condition(s) monitored by an IOT device is received. In response, a sensor readings map is accessed, where this map is associated with the IOT device. The map also digitally represents the IOT device's environment and includes data representative of a location of the IOT device within the environment. The map also includes data representative of the conditions monitored by the IOT device. Additionally, the map is updated by attaching the sensor data to the map. In some cases, a coverage map can also be computed. Both the sensors readings map and the coverage map can be automatically updated in response to the TOT device being re-localized.

First claim

1 . A server computer system comprising:
one or more processor(s); and one or more computer-readable hardware storage device(s) having stored thereon computer-executable instructions that are executable by the one or more processor(s) to cause the server computer system to execute a mixed-reality (MR) service configured to perform at least the following:
receive, from an internet-of-things (IOT) device operating in an environment, sensor data representative of one or more condition(s) monitored by the IOT device;
in response to receiving the sensor data, access a sensor readings map associated with the IOT device, the sensor readings map digitally representing the environment and including data representative of a location of the IOT device within the environment, the sensor readings map also including data representative of the one or more condition(s) monitored by the IOT device;
update the sensor readings map by attaching the sensor data to the sensor readings map;
determine a layout of the environment in which the IOT device is operating, the layout also reflecting the location of the IOT device within the environment; and
generate a combined map that includes a first visualization reflecting a coverage map and a second visualization reflecting the sensor readings map, the coverage map reflecting an operational range of the IOT device based on the layout, the operational range starting at the IOT device and spanning a determined distance away from the IOT device in the environment, the second visualization being structured to overlap at least a part of the first visualization in the combined map.

2 . The server computer system of claim 1 , the MR service also being configured to transmit at least some of the sensor data to a mixed-reality (MR) device that is either (i) physically operating within the environment or (ii) displaying a visualization of the environment.

3 . The server computer system of claim 1 , the sensor data also including measurement data obtained from a sensor of the IOT device such that the sensor readings map is also updated to include the measurement data and such that the server computer system includes updated information regarding sensed operational conditions of the IOT device.

4 . The server computer system of claim 1 , the sensor data also including image data captured by a camera of the IOT device.

5 . The server computer system of claim 4 , the MR service being further configured to:
in response to receiving the image data from the IOT device, determine re-localization position data for the IOT device.

6 . The server computer system of claim 5 , the MR service being further configured to:
in response to determining the re-localization position data for the IOT device, update the coverage map associated with the IOT device.

7 . The server computer system of claim 1 , the sensor data also including inertial measurement unit (IMU) data that is representative of a movement of the IOT device within the environment.

8 . The server computer system of claim 1 , the sensor data also including environmental data that is representative of the environment.

9 . The server computer system of claim 1 , the sensor data also including image data that digitally represents the location of the IOT device within the environment, and the MR service being further configured to:
use the image data to determine whether the location of the IOT device has changed by comparing the image data against location data of the IOT device included within a digital representation of the environment.

10 . The server computer system of claim 9 , the digital representation of the environment digitally representing semantically segmented objects located within the environment, and wherein the MR service is further configured to:
determine a relative proximity between the IOT device and one of the semantically segmented objects; determine a probability that the relative proximity of the IOT device to the one of the semantically segmented objects will negatively impact a performance of the IOT device; and in response to determining that the probability exceeds a threshold limit indicating that the performance of the IOT device is below a performance threshold as a result of the relative proximity, send an alert to indicate that the IOT device should be moved to a new location.

11 . A head-mounted device (HMD) comprising:
a wearable display, the HMD being configured to display virtual images on the wearable display and being configured to update the virtual images in response to unanticipated external stimuli; one or more processor(s); and one or more computer-readable hardware storage device(s) having stored thereon computer-executable instructions that are executable by the one or more processor(s) to cause the HMD to perform at least the following while either (i) physically operating within a same environment as an internet-of-things (IOT) device or (ii) displaying a visualization of the environment in which the IOT device is located:
receive sensor data that was generated by the IOT device, the sensor data digitally representing one or more condition(s) monitored by the IOT device while the IOT device operates in the environment;
access a digital representation of the environment;
associate the sensor data with the digital representation of the environment, the sensor data being associated with a specific area in the environment, associating the sensor data with the digital representation includes associating the sensor data with a portion of the digital representation corresponding to the specific area;
in response to (i) determining the HMD is physically proximate to the specific area or (ii) determining a scene rendered by the HMD is visualizing the specific area, render a virtual image on the wearable display, the virtual image being representative of the sensor data such that the HMD displays a visualization corresponding to the one or more condition(s) monitored by the IOT device;
access a combined map that includes a first visualization reflecting a coverage map and a second visualization reflecting a sensor readings map, the coverage map reflecting an operational range of the IOT device based on an identified layout of the environment, the operational range starting at the IOT device and spanning a determined distance away from the IO device in the environment, the sensor readings map comprising the sensor data, the second visualization being structured to overlap at least a part of the first visualization in the combined map; and
display the combined map on the wearable display.

12 . The HMD of claim 11 , the sensor data being received either (i) from a mixed-reality (MR) service operating in a cloud environment, the MR service receiving the sensor data prior to transmitting the sensor data to the HMD or (ii) from the IOT, which bypassed the MR service.

13 . The HMD of claim 11 , the sensor data being received in response to a change in location by the IOT device.

14 . The HMD of claim 11 , the HMD rendering a second virtual image representative of the operational range of the IOT device.

15 . The HMD of claim 11 , the HMD scanning the IOT device to identify the IOT device within the environment.

16 . The HMD of claim 15 , wherein scanning the IOT device to identify the IOT device includes one or more of the following:
scanning an identifier affixed to the IOT device, the identifier then being used to identify the IOT device; or scanning the IOT device and then semantically segmenting the IOT device to identify the IOT device.

17

Assignees

• Microsoft Technology Licensing Llc

Inventors

• Bleyer Michael
• Pekelny Yuri
• Price Raymond Kirk

Classifications

• H04L67/52

  specially adapted for the location of the user terminal · CPC title

• H04L67/131

  Protocols for games, networked simulations or virtual reality · CPC title

• G06V2201/10

  Recognition assisted with metadata · CPC title

• G06V20/20

  in augmented reality scenes · CPC title

• H04W4/029

  Location-based management or tracking services · CPC title