Cross reality system with wireless fingerprints

Patent metadata

Field	Value
Publication number	US-12170910-B2
Application number	US-202318179275-A
Country	US
Kind code	B2
Filing date	Mar 6, 2023
Priority date	Oct 15, 2019
Publication date	Dec 17, 2024
Grant date	Dec 17, 2024

Abstract

A cross reality system enables any of multiple devices to efficiently and accurately access previously stored maps and render virtual content specified in relation to those maps. Both stored maps and tracking maps used by portable devices may have wireless fingerprints associated with them. The portable devices may maintain wireless fingerprints based on wireless scans performed repetitively, based on one or more trigger conditions, as the devices move around the physical world. The wireless information obtained from these scans may be used to create or update wireless fingerprints associated with locations in a tracking map on the devices. One or more of these wireless fingerprints may be used when a previously stored map is to be selected based on its coverage of an area in which the portable device is operating. Maintaining wireless fingerprints in this way provides a reliable and low latency mechanism for performing map-related operations.

First claim

What is claimed is:

1. A portable device configured to operate within a three-dimensional (3D) environment and display virtual content of a cross reality system, the portable device comprising:
at least one processor;
an image sensor configured to output image data; and
computer executable instructions configured to, when executed by the at least one processor, perform a method comprising:
extracting a plurality of features from the image data;
forming a map of the 3D environment based on the image data produced by the image sensor as the portable device moves in the 3D environment, the map of the 3D environment
comprising a first coordinate frame and a plurality of persistent poses associated with feature descriptors of features of the plurality of features extracted from the image data and associated with locations within the map of the 3D environment;
maintaining a wireless fingerprint associated with the map of the 3D environment by repeatedly:
obtaining network access point information from network access points sending wireless signals received by the portable device at a location within the 3D environment;
selecting a persistent pose of the plurality of persistent poses at a location within the map of the 3D environment based on correspondence between features of the plurality of features and features associated with the persistent pose; and
based on selecting the persistent pose, updating a wireless fingerprint stored in association with the location within the map of the 3D environment by merging network access point information obtained at a second time with network access point information obtained at a first time and linked with the selected pose to produce a merged wireless fingerprint linked with the selected persistent pose;
for a set of features of the plurality of features associated with a position of the portable device, determining the location within the map associated with the features and sending to a localization service the merged wireless fingerprint stored in association with the determined location within the map and a plurality of feature descriptors associated with features of the set of features; and
based on sending the merged wireless fingerprint associated with the determined location and the plurality of feature descriptors to the localization service, receiving, from the localization service, a coordinate transformation between the first coordinate frame of the map of the 3D environment and a coordinate frame of a stored map.

2. The portable device of claim 1 , wherein updating the wireless fingerprint stored in association with the location of the selected persistent pose comprises combining the obtained network access point information with previously obtained network access point information associated with the location.

3. The portable device of claim 1 , wherein obtaining network access point information comprises triggering a scan for network access points.

4. The portable device of claim 1 , wherein obtaining network access point information comprises receiving the network access point information pushed from wireless hardware components after a scan for network access points.

5. The portable device of claim 1 , wherein obtaining network access point information comprises obtaining access point identifiers of the network access points.

6. The portable device of claim 5 , wherein obtaining network access point information further comprises obtaining one or more signal strength indicator values for access points identified by the access point identifiers.

7. The portable device of claim 6 , wherein the access point identifiers are Basic Service Set Identifiers (BSSIDs).

8. The portable device of claim 1 , wherein obtaining network access point information further comprises filtering, clustering, and/or normalizing the network access point information.

9. The portable device of claim 1 , wherein:
each persistent pose of the plurality of persistent poses comprises information configured to orient the portable device relative to the 3D environment.

10. A non-transitory computer-readable medium storing computer executable instructions configured to, when executed by at least one processor, perform a method for operating a portable device configured to operate within a three-dimensional (3D) environment and display virtual content of a cross reality system, the method comprising:
extracting a plurality of features from image data output by an image sensor;
forming a map of the 3D environment based on the image data generated as the portable device moves in the 3D environment, the map of the 3D environment comprising a first coordinate frame and feature descriptors of features of the plurality of features extracted from the image data and associated with locations within the map of the 3D environment;
maintaining a wireless fingerprint associated with the map of the 3D environment by repeatedly:
obtaining network access point information from network access points sending wireless signals received by the portable device at a location within the 3D environment;
selecting a location within the map of the 3D environment based on correspondence between features of the plurality of features and features within the map of the 3D environment; and
based on selecting the location, updating a wireless fingerprint stored in association with the selected location within the map of the 3D environment based on the obtained network access point information by merging a plurality of wireless fingerprints each obtained at a respective time for the selected location to produce a merged wireless fingerprint for the selected location; and

for a set of features of the plurality of features associated with a position of the portable device, selecting the location within the map associated with the features and sending to a localization service the merged wireless fingerprint stored in association with the selected location within the map and a plurality of feature descriptors associated with features of the set of features; and
based on sending the merged wireless fingerprint associated with the selected location and the plurality of feature descriptors to the localization service, receiving, from the localization service, a coordinate transformation between the first coordinate frame of the map of the 3D environment and a coordinate frame of a stored map.

11. The non-transitory computer-readable medium of claim 10 , wherein merging the plurality of wireless fingerprints for the selected location comprises combining the obtained network access point information with previously obtained network access point information associated with the selected location.

12. The non-transitory computer-readable medium of claim 10 , wherein obtaining network access point information comprises triggering a scan for network access points;
wherein the scan for network access points is triggered when an amount of time passed since a previous scan of network access points was performed exceeds a threshold.

13. The non-transitory computer-readable medium of claim 10 , wherein obtaining network access point information comprises obtaining access point identifiers of the network access points.

14. The non-transitory computer-readable medium of claim 13 , wherein obtaining network access point information further comprises obtaining one or more signal strength indicator values for access points identified by the access point identifiers.

15. The non-transitory computer-readable medium of claim 14 , wherein updating the wireless fingerprint stored in association with the selected location within the map comprise

Assignees

• Magic Leap Inc

Inventors

• Shveki Gilboa
• Weisbih Ben
• Kapota Ofer
• Torres Rafael Domingos
• Olshansky Daniel
• Holder Joel David

Classifications

• H04W4/02

  Services making use of location information · CPC title

• G01S5/0252

  Radio frequency fingerprinting · CPC title

• H04W4/029 Primary

  Location-based management or tracking services · CPC title

• G06T19/006

  Mixed reality (object pose determination, tracking or camera calibration for mixed reality G06T7/00) · CPC title

• H04W48/16

  Discovering, processing access restriction or access information · CPC title