Network-based spatial computing for extended reality (XR) applications

Patent metadata

Field	Value
Publication number	US-11748955-B2
Application number	US-202117495329-A
Country	US
Kind code	B2
Filing date	Oct 6, 2021
Priority date	Oct 6, 2020
Publication date	Sep 5, 2023
Grant date	Sep 5, 2023

Abstract

Feature information is received from a client device and describes extended reality scene(s) in an extended reality (ER) environment at the client device. An ER description is formed in an ER description format corresponding to the feature information and is stored. Some of the stored ER description format is provided in a representational format upon request of the client device or other client devices viewing the one or more ER scenes and assisting the positioning of corresponding client devices in the ER environment. A client device captures environmental visual data and generates feature information, describing ER scene(s) in an extended reality environment at the client device, from the environmental visual data, and sends the generated feature information toward a server. A client device can localize itself in a 3D environment of an ER description, and generate ER anchors and objects, link them, and send them to a server.

First claim

What is claimed is:

1. A method, comprising:
receiving, from a client device in a wireless network, feature information describing at least part of one or more extended reality scenes in an extended reality environment at the client device, the receiving determining information indicating whether the feature information comprises one or both of visual features or non-visual features, wherein the feature information is defined using a structure providing an indication of feature type that indicates at least visual features or non-visual features, the non-visual features comprising radio frequency features, and where indications of both visual and non-visual features use a same structure;
forming an extended reality description in an extended reality description format corresponding to the feature information;
storing the extended reality description format in a database; and
providing at least a portion of the stored extended reality description format in a representational format upon request of the client device or other client devices viewing the one or more extended reality scenes and assisting positioning of corresponding client devices in the extended reality environment.

2. The method of claim 1 , wherein:
the feature information comprises a plurality of environment features from the one or more extended reality scenes in the extended reality environment at the client device; and
the forming an extended reality description comprises generating an extended reality description, corresponding to the plurality of environment features, in the extended reality description format.

3. The method of claim 1 , wherein:
the received feature information comprises an extended reality description received from the client device and corresponding to environment features from the one or more extended reality scenes; and
the forming an extended reality description in an extended reality description format comprises fusing the received extended reality description from the client device with the stored extended reality description format in the database.

4. An apparatus, comprising:
one or more processors; and
one or more memories storing instructions that, when executed by the one or more processors, cause the apparatus at least to perform:
receiving, from a client device in a wireless network, feature information describing at least part of one or more extended reality scenes in an extended reality environment at the client device, the receiving determining information indicating whether the feature information comprises one or both of visual features or non-visual features, wherein the feature information is defined using a structure providing an indication of feature type that indicates at least visual features or non-visual features, the non-visual features comprising radio frequency features and where indications of both visual and non-visual features use a same structure;
forming an extended reality description in an extended reality description format corresponding to the feature information;
storing the extended reality description format in a database; and
providing at least a portion of the stored extended reality description format in a representational format upon request of the client device or other client devices viewing the one or more extended reality scenes and assisting positioning of corresponding client devices in the extended reality environment.

5. The apparatus of claim 4 , wherein:
the feature information comprises a plurality of environment features from the one or more extended reality scenes in the extended reality environment at the client device; and
the forming an extended reality description comprises generating an extended reality description, corresponding to the plurality of environment features, in the extended reality description format.

6. The apparatus of claim 5 , wherein the plurality of environment features comprise environment features that have been generated by the client device for one or more real objects in the one or more extended reality scenes.

7. The apparatus of claim 5 , wherein storing the extended reality description format comprises storing the extended reality description format using a hierarchical data structure that represents spatial relationships of the plurality of environment features with respect to a three-dimensional representation of the one or more extended reality scenes.

8. The apparatus of claim 7 , wherein the data structure is stored using one or both of a persistent format or a payload format.

9. The apparatus of claim 7 , wherein the data structure has capabilities of multi-scaling to compute an absolute scale with respect to a physical environment, or supporting multiple client devices, or both the multi-scaling and supporting of the multiple client devices.

10. The apparatus of claim 5 , wherein generating the extended reality description comprises generating an extended reality description corresponding to the environment features that can be used for device localization and environment reconstruction technologies using both vision-based and radio frequency-based technologies.

11. The apparatus of claim 4 , wherein:
the received feature information comprises an extended reality description received from the client device and corresponding to environment features from the one or more extended reality scenes; and
the forming an extended reality description in an extended reality description format comprises fusing the received extended reality description from the client device with the stored extended reality description format in the database.

12. The apparatus of claim 11 , wherein fusing the extended reality description from the client device with the extended reality description format further comprises one or more of the following operations:
converting the received extended reality description to a same or a common coordinate system as used by previously stored extended reality description format;
converting different extended reality information sources to a generic standardized format;
scaling information in the received extended reality description up or down to a same global size to be used for the extended reality description format; or
updating existing extended reality description in the database with newly received extended reality data elements from the client device.

13. The apparatus of claim 4 , wherein:
the feature information comprises a representation of visual data corresponding to environment features from the one or more extended reality scenes; and
the forming an extended reality description in an extended reality description format comprises forming a received extended reality description from the received representation of visual data.

14. The apparatus of claim 13 , wherein forming a received extended reality description further comprises fusing the formed extended reality description with previously stored extended reality description.

15. The apparatus of claim 4 , comprising a server, wherein the feature information is received by the server or sent by the client device without corresponding original image or video information.

16. The apparatus of claim 4 , wherein the representational format comprises graphics library transmission format.

17. The apparatus of claim 4 , wherein:
the one or more memories further store instructions that, when executed by the one or more processors, cause the apparatus at least to perform: receiving from the client device one or more extended reality anchors and one or more extended reality objects that place the one or m

Assignees

• Nokia Technologies Oy

Inventors

• You Yu
• Fasogbon Peter
• Aksu Emre
• Curcio Igor Danilo Diego
• Ahsan Saba
• Mattila Ville-Veikko

Classifications

• G06T19/006 Primary

  Mixed reality (object pose determination, tracking or camera calibration for mixed reality G06T7/00) · CPC title

• G06F16/487

  using geographical or spatial information, e.g. location · CPC title

• H04L67/131

  Protocols for games, networked simulations or virtual reality · CPC title

• G06T19/003 Primary

  Navigation within 3D models or images · CPC title

• G06T2219/024

  Multi-user, collaborative environment · CPC title