Pose validity for XR based services

Patent metadata

Field	Value
Publication number	US-11989343-B2
Application number	US-202318093018-A
Country	US
Kind code	B2
Filing date	Jan 4, 2023
Priority date	Jan 5, 2022
Publication date	May 21, 2024
Grant date	May 21, 2024

Abstract

An apparatus comprising: at least one processor; and at least one non-transitory memory storing instructions that, when executed by the at least one processor, cause the apparatus at least to: transmit a pose and interactivity stream to a server, the pose and interactivity stream comprising data based on information obtained from at least one sensor; determine a validity of the pose and interactivity stream, the validity comprising at least one criterion for which the data within the pose and interactivity stream is considered valid; signal at least one parameter that defines the validity of the pose and interactivity stream; and signal a current state of the validity of the pose and interactivity stream.

First claim

What is claimed is:

1. An apparatus comprising:
at least one processor; and
at least one non-transitory memory storing instructions that, when executed by the at least one processor, cause the apparatus at least to:
transmit, from a client device to a server, an estimate of pose data, the pose data indicating at least one of a position, translation, or orientation of the client device relative to a coordinate system, wherein the client device comprises the apparatus;
determine, with the client device, a confidence score that indicates a probability of the estimate of the pose data to be correct; and
transmit, from the client device to the server, the confidence score that indicates the probability of the estimate of the pose data to be correct;
wherein the confidence score is configured to be used with the server to determine a re-estimate of the pose data, in response to the estimate of the pose data transmitted from the client device to the server being inadequate based on the confidence score.

2. The apparatus of claim 1 , wherein the instructions, when executed by the at least one processor, cause the apparatus at least to:
signal, to the server, a threshold for assistance when the confidence score or a validity value is low; and
receive assistance from the server when the confidence score or the validity value is lower than the threshold, the assistance comprising a correction of the estimate of the pose data.

3. The apparatus of claim 1 , wherein the instructions, when executed by the at least one processor, cause the apparatus at least to:
receive from the server another confidence score that indicates a confidence of the re-estimate of the pose data.

4. The apparatus of claim 3 , wherein the confidence score is determined based on at least one of: a neural network trained on motion capture recordings of a plurality of users, or a type of the apparatus.

5. The apparatus of claim 1 , wherein the instructions, when executed by the at least one processor, cause the apparatus at least to:
signal at least one parameter that defines a validity of the pose data; and
signal a current state of the validity of the pose data;
wherein the at least one parameter that defines the validity of the pose data is signaled within a payload as part of a real time transport protocol delivery session, and the current state of the validity of the pose data is signaled within the payload as part of the real time transport protocol delivery session.

6. The apparatus of claim 1 , wherein the instructions, when executed by the at least one processor, cause the apparatus at least to:
determine a plurality of regions for rendering at the client device dependent on a viewport of a user of the client device, based on the confidence score;
wherein the plurality of regions comprises a first region to be delivered at a first quality, a second region to be delivered at a second quality, and a third region at least partially around the first region to be delivered at a third quality;
wherein the first quality is higher than the second quality, and the third quality is equivalent to or lower than the first quality and higher than the second quality.

7. The apparatus of claim 5 , wherein the at least one parameter that defines the validity of the pose data is based on at least one of: biometric data, a device operation metric, environmental data, or a dependency on a device different from the apparatus.

8. The apparatus of claim 1 , wherein the instructions, when executed by the at least one processor, cause the apparatus at least to:
stop transmission of the pose data, in response to the confidence score falling below a value, the confidence score indicating a confidence of the pose data.

9. The apparatus of claim 1 , wherein the instructions, when executed by the at least one processor, cause the apparatus at least to:
receive a heartbeat signal from the server to determine whether the pose data is still valid; and
transmit, to the server, a response to the heartbeat signal to indicate the validity of the pose data.

10. An apparatus comprising:
at least one processor; and
at least one non-transitory memory storing instructions that, when executed by the at least one processor, cause the apparatus at least to:
receive, with the apparatus from a client device, an estimate of pose data, the pose data indicating at least one of a position, translation, or orientation of the client device relative to a coordinate system, wherein a server comprises the apparatus; stream from a client device, the pose;
receive, with the server from the client device, a confidence score that indicates a probability of the estimate of the pose data received from the client device to be correct;
determine, based on the confidence score received from the client device, whether the estimate of the pose data received from the client device is inadequate; and
determine a re-estimate of the pose data, in response to determining that the estimate of the pose data received from the client device is inadequate.

11. The apparatus of claim 10 , wherein the instructions, when executed by the at least one processor, cause the apparatus at least to:
receive, from the client device, signaling of a threshold for assistance when the confidence score or a validity value is low; and
transmit assistance to the client device when the confidence score or the validity value is lower than the threshold, the assistance comprising a correction of the estimate of the pose data.

12. The apparatus of claim 10 , wherein the instructions, when executed by the at least one processor, cause the apparatus at least to:
determine another confidence score that indicates a confidence of the re-estimate of the pose data, and transmit the another confidence score to the client device or to another client device, or
receive from the another client device another confidence score that indicates a probability of an estimate of pose data received from the another client device.

13. The apparatus of claim 12 , wherein the confidence score is based on at least one of: a neural network trained on motion capture recordings of a plurality of users, or a type of the client device.

14. The apparatus of claim 10 , wherein the instructions, when executed by the at least one processor, cause the apparatus at least to:
receive signaling of at least one parameter that defines a validity of the pose data; and
receive signaling of a current state of the validity of the pose data;
wherein the at least one parameter that defines the validity of the pose data is signaled within a payload as part of a real time transport protocol delivery session, and the current state of the validity of the pose data is signaled within the payload as part of the real time transport protocol delivery session.

15. The apparatus of claim 10 , wherein the instructions, when executed by the at least one processor, cause the apparatus at least to:
determine, with the server, a plurality of regions for rendering at the client device dependent on a viewport of a user of the client device, based on the confidence score;
wherein the plurality of regions comprises a first region to be delivered at a first quality, a second region to be delivered at a second quality, and a third region at least partially around the first region to be delivered at a third quality;
wherein the first quality is higher than the second quality, and the third quality is equivalent to or lower than the first quality and higher than the second quality.

16. The apparatus of claim 14 , wherein the at least one paramet

Assignees

• Nokia Technologies Oy

Inventors

• Ahsan Saba
• Mate Sujeet Shyamsundar
• You Yu
• Fasogbon Peter Oluwanisola
• Curcio Igor Danilo Diego

Classifications

• G06F3/012 Primary

  Head tracking input arrangements · CPC title

• G06F3/011 Primary

  Arrangements for interaction with the human body, e.g. for user immersion in virtual reality (blind teaching G09B21/00) · CPC title

• G06F3/0346

  with detection of the device orientation or free movement in a three-dimensional [3D] space, e.g. 3D mice, 6-DOF [six degrees of freedom] pointers using gyroscopes, accelerometers or tilt-sensors · CPC title

• H04L67/131

  Protocols for games, networked simulations or virtual reality · CPC title