Mapping sensor data using a mixed-reality cloud

Patent metadata

Field	Value
Publication number	US-11961192-B2
Application number	US-202117517951-A
Country	US
Kind code	B2
Filing date	Nov 3, 2021
Priority date	Jul 22, 2019
Publication date	Apr 16, 2024
Grant date	Apr 16, 2024

Abstract

Improved techniques for re-localizing Internet-of-Things (IOT) devices are disclosed herein. Sensor data digitally representing one or more condition(s) monitored by an IOT device is received. In response, a sensor readings map is accessed, where this map is associated with the IOT device. The map also digitally represents the IOT device's environment and includes data representative of a location of the IOT device within the environment. The map also includes data representative of the conditions monitored by the IOT device. Additionally, the map is updated by attaching the sensor data to the map. In some cases, a coverage map can also be computed. Both the sensors readings map and the coverage map can be automatically updated in response to the IOT device being re-localized.

First claim

What is claimed is:

1. A server computer system that executes a mixed-reality (MR) service, said server computer system comprising:
at least one processor; and
at least one hardware storage device that stores instructions that are executable by the at least one processor to cause the server computer system to:
receive sensor data of an internet of things (IOT) device located in an environment;
determine whether the sensor data is fresh or stale, the sensor data being fresh during a period of time whose duration is dependent on a number of moveable objects identified within the environment, a relatively higher number of moveable objects results in the duration being relatively shorter while a relatively lower number of moveable objects results in the duration being relatively longer;
display a notification regarding whether the sensor data is fresh or stale;
access a sensor readings map associated with the IOT device, the sensor readings map includes the sensor data;
generate a combined map that includes a first visualization reflecting a coverage map and a second visualization reflecting the sensor readings map, wherein:
the coverage map reflects an operational coverage area of the IOT device,
the operational coverage area starts at the IOT device and extends a determined distance away from the IOT device in the environment, and
the second visualization overlaps at least a part of the first visualization in the combined map;

identify an impedance that blocks the IOT device from sensing an area of the environment, the identified impedance being located between the IOT device and the area, the impedance being identified as one that blocks the IOT device from acquiring particular sensor data for that area; and
update, based on the identified impedance, the operational coverage area of the IOT device using a machine learning (ML) algorithm, said updating includes using the ML algorithm to modify the operational coverage area to reflect that the IOT device is blocked from acquiring the particular sensor data for the area.

2. The server computer system of claim 1 , the MR service also being configured to transmit at least some of the sensor data to a mixed-reality (MR) device that is either (i) physically operating within the same environment as where the IOT device is operating or (ii) displaying a visualization of the environment.

3. The server computer system of claim 1 , the sensor data including measurement data obtained from a sensor of the IOT device such that the sensor readings map is updated to include the measurement data and such that the server computer system includes updated information regarding sensed operational conditions that are sensed by the IOT device.

4. The server computer system of claim 1 , the sensor data including image data captured by a camera of the IOT device.

5. The server computer system of claim 4 , the MR service being further configured to:
in response to receiving the image data from the IOT device, determine relocalization position data for the IOT device.

6. The server computer system of claim 5 , the MR service being further configured to:
in response to determining the re-localization position data for the IOT device, update the coverage map associated with the IOT device.

7. The server computer system of claim 1 , the sensor data including inertial measurement unit (IMU) data that is representative of a movement of the IOT device within the environment.

8. The server computer system of claim 1 , the sensor data including environmental data that is representative of an environment in which the IOT device is operating.

9. The server computer system of claim 1 , the sensor data including image data that digitally represents a location of the IOT device within an environment, and the MR service being further configured to:
use the image data to determine whether the location of the IOT device has changed by comparing the image data against location data of the IOT device included within a digital representation of the environment.

10. The server computer system of claim 9 , the digital representation of the environment digitally representing semantically segmented objects located within the environment.

11. A method for executing a mixed-reality (MR) service, the method comprising:
receiving sensor data of an internet of things (IOT) device located in an environment;
determining whether the sensor data is fresh or stale, the sensor data being fresh during a period of time whose duration is dependent on a number of moveable objects identified within the environment, a relatively higher number of moveable objects results in the duration being relatively shorter while a relatively lower number of moveable objects results in the duration being relatively longer;
displaying a notification regarding whether the sensor data is fresh or stale;
accessing a sensor readings map associated with the IOT device, the sensor readings map includes the sensor data;
generating a combined map that includes a first visualization reflecting a coverage map and a second visualization reflecting the sensor readings map, wherein:
the coverage map reflects an operational coverage area of the IOT device,
the operational coverage area starts at the IOT device and spans a determined distance away from the IOT device in the environment, and
the second visualization is structured to overlap at least a part of the first visualization in the combined map;

identifying an impedance that blocks the IOT device from sensing an area of the environment, the identified impedance being located between the IOT device and the area, the impedance being identified as one that blocks the IOT device from acquiring particular sensor data for that area; and
updating, based on the identified impedance, the operational coverage area of the IOT device using a machine learning (ML) algorithm, said updating includes using the ML algorithm to modify the operational coverage area to reflect that the IOT device is blocked from acquiring the particular sensor data for the area.

12. The method of claim 11 , wherein the MR service is configured to transmit at least some of the sensor data to a remote mixed-reality device.

13. The method of claim 11 , wherein the sensor data includes one or more of measurement data, image data, or inertial measurement unit (IMU) data.

14. The method of claim 11 , wherein the sensor data includes one or more of environmental data or image data.

15. The method of claim 11 , wherein the method further includes determining a layout of the environment in which the IOT device is operating.

16. The method of claim 11 , wherein the operational coverage area includes a buffer.

17. The method of claim 11 , wherein the MR service relocalizes a position of the IOT device.

18. A server computer system comprising:
one or more processor(s); and
one or more computer-readable hardware storage device(s) having stored thereon computer-executable instructions that are executable by the one or more processor(s) to cause the server computer system to execute a mixed-reality (MR) service configured to perform at least the following:
receive sensor data of an internet of things (IOT) device located in an environment;
determine whether the sensor data is fresh or stale, the sensor data being fresh during a period of time whose duration is dependent on a number of moveable objects identified within the environment, a relatively higher number of moveable objects results in the duration being relatively shorter while a relatively lower number of

Assignees

• Microsoft Technology Licensing Llc

Inventors

• Bleyer Michael
• Pekelny Yuri
• Price Raymond Kirk

Classifications

• G06T19/006 Primary

  Mixed reality (object pose determination, tracking or camera calibration for mixed reality G06T7/00) · CPC title

• G02B27/0101

  characterised by optical features (G02B27/0172 takes precedence) · CPC title

• G06F16/5866

  using information manually generated, e.g. tags, keywords, comments, manually generated location and time information · CPC title

• G06T7/70

  Determining position or orientation of objects or cameras (camera calibration G06T7/80) · CPC title

• G06V20/20

  in augmented reality scenes · CPC title