Cross reality system with wireless fingerprints

Patent metadata

Field	Value
Publication number	US-2025063390-A1
Application number	US-202418936094-A
Country	US
Kind code	A1
Filing date	Nov 4, 2024
Priority date	Oct 15, 2019
Publication date	Feb 20, 2025
Grant date	—

Abstract

A cross reality system enables any of multiple devices to efficiently and accurately access previously stored maps and render virtual content specified in relation to those maps. Both stored maps and tracking maps used by portable devices may have wireless fingerprints associated with them. The portable devices may maintain wireless fingerprints based on wireless scans performed repetitively, based on one or more trigger conditions, as the devices move around the physical world. The wireless information obtained from these scans may be used to create or update wireless fingerprints associated with locations in a tracking map on the devices. One or more of these wireless fingerprints may be used when a previously stored map is to be selected based on its coverage of an area in which the portable device is operating. Maintaining wireless fingerprints in this way provides a reliable and low latency mechanism for performing map-related operations.

First claim

1 . A portable device configured to operate within a three-dimensional (3D) environment and display virtual content of a cross reality system, the 3D environment comprising a plurality of locations associated with a plurality of features extracted from image data and a wireless fingerprint linked with a location of the plurality of locations, the wireless fingerprint comprising first network access point information obtained from one or more network access points at a first time, the portable device comprising:
at least one processor; an image sensor; wireless hardware; and computer executable instructions configured to, when executed by the at least one processor, perform a method comprising:
obtaining second network access point information from the one or more network access points at a second time after the first time using the wireless hardware;
extracting one or more features from image data obtained by the image sensor;
identifying the location from among the plurality of locations of the 3D environment based on correspondence between the one or more features extracted from the image data obtained by the image sensor and features of the plurality of features of the 3D environment; and
based on selecting the location, updating the wireless fingerprint linked with the location of the 3D environment by merging the second network access point information obtained at the second time with the first network access point information obtained at the first time.

2 . The portable device of claim 1 , wherein the method further comprises:
for the one or more features of the plurality of features extracted from the image data obtained by the image sensor, sending to a localization service at least one feature of the one or more features extracted from the image data obtained by the image sensor and the updated wireless fingerprint linked with the location of the 3D environment; and based on sending to the localization service the at least one feature and the updated wireless fingerprint linked with the location, receiving from the localization service a coordinate transformation between a first coordinate frame of the 3D environment and a coordinate frame of a stored map.

3 . The portable device of claim 1 , wherein obtaining network access point information comprises triggering a scan for network access points.

4 . The portable device of claim 1 , wherein obtaining network access point information comprises receiving the network access point information pushed from wireless hardware components after a scan for network access points.

5 . The portable device of claim 1 , wherein obtaining the second network access point information comprises obtaining access point identifiers of the one or more network access points.

6 . The portable device of claim 5 , wherein obtaining the second network access point information further comprises obtaining one or more signal strength indicator values for access points identified by the access point identifiers.

7 . The portable device of claim 6 , wherein the access point identifiers are Basic Service Set Identifiers (BSSIDs).

8 . The portable device of claim 1 , wherein obtaining network access point information further comprises filtering, clustering, and/or normalizing the network access point information.

9 . The portable device of claim 1 , wherein:
each location of the plurality of locations of the 3D environment comprises information configured to orient the portable device relative to the 3D environment.

10 . A non-transitory computer-readable medium storing computer executable instructions configured to, when executed by at least one processor, perform a method for operating a portable device comprising an image sensor and wireless hardware and configured to operate within a three-dimensional (3D) environment and display virtual content of a cross reality system, the 3D environment comprising a plurality of locations associated with a plurality of features extracted from image data and a wireless fingerprint linked with a location of the plurality of locations, the wireless fingerprint comprising first network access point information obtained from one or more network access points at a first time, the method comprising:
obtaining second network access point information from the one or more network access points at a second time different than the first time using the wireless hardware; extracting one or more features from image data obtained by the image sensor; identifying the location from among the plurality of locations of the 3D environment based on correspondence between the one or more features extracted from the image data obtained by the image sensor and features of the plurality of features of the 3D environment; and based on selecting the location, updating the wireless fingerprint linked with the location of the 3D environment by merging the second network access point information obtained at the second time with the first network access point information obtained at the first time.

11 . The non-transitory computer-readable medium of claim 10 , wherein the method further comprises:
for the one or more features of the plurality of features extracted from the image data obtained by the image sensor, sending to a localization service at least one feature of the one or more features extracted from the image data obtained by the image sensor and the updated wireless fingerprint linked with the location of the 3D environment; and based on sending to the localization service the at least one feature and the updated wireless fingerprint linked with the location, receiving from the localization service a coordinate transformation between a first coordinate frame of the 3D environment and a coordinate frame of a stored map.

12 . The non-transitory computer-readable medium of claim 10 , wherein obtaining network access point information comprises triggering a scan for network access points;
wherein the scan for network access points is triggered when an amount of time passed since a previous scan of network access points was performed exceeds a threshold.

13 . The non-transitory computer-readable medium of claim 10 , wherein obtaining the second network access point information comprises obtaining access point identifiers of the one or more network access points.

14 . The non-transitory computer-readable medium of claim 13 , wherein obtaining the second network access point information further comprises obtaining one or more signal strength indicator values for access points identified by the access point identifiers.

15 . The non-transitory computer-readable medium of claim 14 , wherein updating the wireless fingerprint linked with the location comprises storing the one or more signal strength indicator values in association with the access point identifiers.

16 . The non-transitory computer-readable medium of claim 15 , wherein updating the wireless fingerprint linked with the location comprises identifying a subset of the access point identifiers to be excluded.

17 . The non-transitory computer-readable medium of claim 16 , wherein the subset of the access point identifiers to be excluded is based at least on the one or more signal strength indicator values.

18 . The non-transitory computer-readable medium of claim 15 , wherein storing the one or more signal strength indicator values comprises storing an average of multiple signal strength indicator values in association with the access point identifiers.

19 . A portable device configured to operate within a th

Assignees

• Magic Leap Inc

Inventors

• Shveki Gilboa
• Weisbih Ben
• Kapota Ofer
• Torres Rafael Domingos
• Olshansky Daniel
• Holder Joel David

Classifications

• H04W4/02

  Services making use of location information · CPC title

• G01S5/0252

  Radio frequency fingerprinting · CPC title

• H04W4/029 Primary

  Location-based management or tracking services · CPC title

• G06T19/006

  Mixed reality (object pose determination, tracking or camera calibration for mixed reality G06T7/00) · CPC title

• H04W48/16

  Discovering, processing access restriction or access information · CPC title