System and method for spatially mapping smart objects within augmented reality scenes

Patent metadata

Field	Value
Publication number	US-2021365681-A1
Application number	US-201916977616-A
Country	US
Kind code	A1
Filing date	Feb 27, 2019
Priority date	Mar 2, 2018
Publication date	Nov 25, 2021
Grant date	—

Abstract

A method and system for localizing a plurality of stationary devices, such as Internet of Things (IoT devices), arranged in an environment is disclosed. A mobile device is configured to survey an environment to generate a three-dimensional map of the environment using simultaneous localization and mapping (SLAM) techniques. The mobile device and the stationary devices are equipped with wireless transceivers, such as Ultra-wideband radios, for measuring distances between the devices using wireless ranging techniques. Based on the measured distances, the mobile device is configured to determine locations of the stationary devices in a reference frame of the three-dimensional map. In some embodiments, the determined locations can be used to enable a variety of spatially aware augmented reality features and interactions between the mobile device and the stationary device.

First claim

What is claimed is:

1 . A method for localizing a plurality of stationary devices in an environment with a mobile device, the method comprising:
capturing a first plurality of images of the environment with a camera of the mobile device, the first plurality of images being captured as the mobile device is moved throughout the environment; generating, with a processor of the mobile device, a three-dimensional map of the environment based on the first plurality of images of the environment; measuring a first plurality of distance measurements between the mobile device and each individual stationary device in the plurality of stationary devices, the first plurality of distance measurements being measured by operating a first wireless transceiver of the mobile device to exchange messages with respective second wireless transceivers of each of the plurality of stationary devices; receiving, from the plurality of stationary devices, a second plurality of distance measurements between each stationary device in the plurality of stationary devices and each other stationary device in the plurality of stationary devices, the second plurality of distance measurements being measured by operating the second wireless transceiver of each stationary device in the plurality of stationary devices to exchange messages with the second wireless transceivers of each other stationary device in the plurality of stationary devices; and determining, with the processor, locations of the plurality of stationary devices in a reference frame of the three-dimensional map of the environment based on the first plurality of distance measurements and the second plurality of distance measurements.

2 . The method according to claim 1 , the measuring the first plurality of distance measurements further comprising:
measuring the first plurality of distance measurements while the mobile device is moved along a path through the environment, the path comprising at least four distinct measurement locations from which at least four respective subsets of the distance measurements in the first plurality of distance measurements are measured, each respective subset of the first plurality of distance measurements including distance measurements between the mobile device and each individual stationary device in the plurality of stationary devices while the mobile device is at a respective one of the at least four distinct measurement locations along the path.

3 . The method according to claim 2 further comprising:
determining, with the processor, the at least four distinct measurement locations in the reference frame of the three-dimensional map of the environment based on images of the environment captured by the camera at the at least four distinct measurement locations.

4 . The method according to claim 3 , the determining the locations of the plurality of stationary devices further comprising:
determining the locations of the plurality of stationary devices based on the first plurality of distance measurements, the second plurality of distance measurements, and the at least four distinct measurement locations in the reference frame of the three-dimensional map.

5 . The method according to claim 4 , the determining the locations of the plurality of stationary devices further comprising:
estimating the locations of the plurality of stationary devices by minimizing differences between (i) the first plurality of distance measurements and the second plurality of distance measurements and (ii) corresponding distances calculated from the estimated locations of the plurality of stationary devices.

6 . The method according to claim 5 , the minimizing differences further comprising:
calculating a weighted sum of differences between (i) the first plurality of distance measurements and the second plurality of distance measurements and (ii) corresponding distances calculated from the estimated locations of the plurality of stationary devices.

7 . The method according to claim 6 , wherein differences between (i) the first plurality of distance measurements and (ii) corresponding distances calculated from the estimated locations of the plurality of stationary devices are weighted more heavily in the weighted sum than differences between (i) the second plurality of distance measurements and (ii) corresponding distances calculated from the estimated locations of the plurality of stationary devices.

8 . The method according to claim 5 , the minimizing differences further comprising:
iteratively calculating values for the locations of the plurality of stationary devices using a majorizing function until the calculated values of the locations of the plurality of stationary devices satisfy a convergence criterion.

9 . The method according to claim 8 , the iteratively calculating values for the locations of the plurality of stationary devices further comprising:
calculating the majorizing function based on the first plurality of distance measurements, the second plurality of distance measurements, and the locations of each of the at least four distinct measurement locations.

10 . The method according to claim 1 , wherein the first wireless transceiver of the mobile device is an ultra-wideband transceiver and the second wireless transceivers of each of the plurality of stationary devices are ultra-wideband transceivers.

11 . The method according to claim 1 , wherein at least of the stationary devices has an actuator and is configured to receive control messages from the mobile device and operate the actuator based on the control message.

12 . The method according to claim 1 , the receiving of the second plurality of distance measurements further comprising:
receiving the second plurality of distance measurements from the plurality of stationary devices with a third transceiver of the mobile device, which is different from the first transceiver of the mobile device.

13 . The method according to claim 12 , wherein the third transceiver is a Wi-Fi transceiver.

14 . The method according to claim 1 , the measuring the first plurality of distance measurements further comprising:
measuring times of flight of messages transmitted between the mobile device and individual stationary devices in the plurality of stationary devices.

15 . The method according to claim 1 further comprising:
displaying, on a display screen of the mobile device, a graphical user interface having graphical elements that are rendered depending on the locations of the plurality of stationary devices.

16 . The method according to claim 15 , the displaying further comprising:
capturing, with the camera, a second plurality of images of the environment; and displaying the graphical user interface having the second plurality of images of the environment displayed in real-time with the graphical elements superimposed on the second plurality of images.

17 . The method according to claim 15 further comprising:
receiving a status parameter from at least one stationary device in the plurality of stationary devices,
wherein the graphical elements include at least one graphical element associated with the at least one stationary device and which is rendered depending on the status parameter.

18 . The method according to claim 15 further comprising:
determining, with the processor, at least one of (i) a distance of the mobile device from at least one stationary device in the plurality of stationary devices and (ii) an orientation of the mobile device with respect to at least one stationary device in the plurali

Assignees

• Purdue Research Foundation

Inventors

• Huo Ke
• Ramani Karthik

Classifications

• G06F3/0346 Primary

  with detection of the device orientation or free movement in a three-dimensional [3D] space, e.g. 3D mice, 6-DOF [six degrees of freedom] pointers using gyroscopes, accelerometers or tilt-sensors · CPC title

• G06V20/20 Primary

  in augmented reality scenes · CPC title

• G06V20/64

  Three-dimensional [3D] objects · CPC title

• H04W88/02

  Terminal devices · CPC title

• G06T2200/24

  involving graphical user interfaces [GUIs] · CPC title