Creating cloud-hosted, streamed augmented reality experiences with low perceived latency

Patent metadata

Field	Value
Publication number	US-12272013-B2
Application number	US-202318395986-A
Country	US
Kind code	B2
Filing date	Dec 26, 2023
Priority date	Mar 31, 2020
Publication date	Apr 8, 2025
Grant date	Apr 8, 2025

Abstract

A technology that streams graphical components and rendering instructions to a client device, for the client device to perform the final rendering and overlaying of that content onto the client's video stream based on the client's most recent tracking of the device's position and orientation. A client device sends a request for augmented reality drawing data to a network device. In response, the network device generates augmented reality drawing data, which can be augmented reality change data based on the augmented reality information and previous client render state information, and sends the augmented reality drawing data to the client device. The client device receives the augmented reality drawing data and renders a visible representation of an augmented reality scene comprising overlaying augmented reality graphics over a current video scene obtained from a camera of the client device.

First claim

What is claimed is:

1. A client device, comprising:
a processor; and
a memory that stores executable instructions that, when executed by the processor, facilitate performance of operations, the operations comprising:
maintaining a high framerate at a display of the client device comprising:
initializing rendering of a plurality of augmented reality scenes at least partially in parallel with one another,
wherein the plurality of augmented reality scenes is based at least on initial augmented reality information, and
wherein the initializing is performed prior to a rendering of any of the plurality of augmented reality scenes on the display; and
configuring the plurality of augmented reality scenes to be rendered in succession relative to one another.

2. The client device of claim 1 , wherein the operations further comprise:
obtaining live video data from a camera; and
generating, based on the live video data, the initial augmented reality information.

3. The client device of claim 1 , wherein the operations further comprise:
rendering, on the display, the plurality of augmented reality scenes in succession to one another.

4. The client device of claim 3 , wherein the operations further comprise:
prior to the rendering on the display, obtaining, from a network device, augmented reality drawing data based on the initial augmented reality information,
wherein the plurality of augmented scenes rendered on the display are based on the augmented reality drawing data.

5. The client device of claim 4 , wherein the obtaining the augmented reality data comprises obtaining at least one of one or more render buffers or one or more draw commands.

6. The client device of claim 1 , wherein the operations further comprise:
detecting a changing of an orientation of a camera of the client device; and
in response thereto, executing the initializing the rendering of at least one of the plurality of augmented reality scenes.

7. The client device of claim 1 , wherein the initial augmented reality information comprises at least one of a set of one or more planes that represent respective one or more flat surfaces detected in a real-world environment, a three-dimensional mesh of a real-world environment, or lighting information.

8. A method, comprising:
making available to a network device, by a client device, at least partially in parallel with one another, a plurality of sets of augmented reality information comprised by the client device,
wherein the plurality of sets of augmented reality information correspond to a plurality of augmented reality scenes to be generated by the client device and to be subsequently displayed in succession on a display of the client device; and
obtaining from the network device, by the client device, only after having made the plurality of sets of augmented reality information available to the network device, augmented reality drawing data generated by the network device and based on the plurality of sets of augmented reality information.

9. The method of claim 8 , further comprising:
rendering, by the client device, on the display of the client device, the plurality of augmented reality scenes in succession to one another.

10. The method of claim 9 , wherein the plurality of augmented reality scenes rendered on the display are based on the plurality of sets of augmented reality information and on the augmented reality drawing data having been obtained from the network device.

11. The method of claim 8 , wherein the obtaining the augmented reality drawing data comprises obtaining one or more render buffers or one or more draw commands.

12. The method of claim 8 , wherein the obtaining the augmented reality drawing data comprises obtaining uncompressed data and compressed data.

13. The method of claim 8 , further comprising:
detecting, by the client device, a changing of a position of the client device; and
in response to the detecting, generating, by the client device, at least one of the sets of the plurality of sets of augmented reality information.

14. The method of claim 8 , further comprising:
capturing, by a camera of the client device, live video; and
generating, by the client device, the plurality of sets of augmented reality information based on the live video.

15. A non-transitory machine-readable medium, comprising executable instructions that, when executed by a processor of a client device, facilitate performance of operations, the operations comprising:
making available to a network device, at least partially in parallel with one another, a plurality of sets of augmented reality information comprised by the client device;
based on the plurality of sets of augmented reality information, initializing rendering of a plurality of augmented reality scenes at least partially in parallel with one another; and
obtaining, from the network device, only after having made the plurality of sets of augmented reality information available to the network device and only after having initialized the rendering of the plurality of augmented reality scenes based on the plurality of sets of augmented reality information, augmented reality drawing data generated by the network device and based on the plurality of sets of augmented reality information.

16. The non-transitory machine-readable medium of claim 15 , wherein the operations further comprise:
rendering, on a display of the client device, the plurality of augmented reality scenes in succession to one another.

17. The non-transitory machine-readable medium of claim 15 , wherein the initializing is performed prior to a rendering of any of the plurality of augmented reality scenes on the display.

18. The non-transitory machine-readable medium of claim 15 , wherein the operations further comprise:
in response to a detection of a change of a position of the client device, generating at least one of the sets of the plurality of sets of augmented reality information.

19. The non-transitory machine-readable medium of claim 15 , wherein the obtaining the augmented reality drawing data comprises obtaining one or more draw commands comprised by uncompressed data and compressed data.

20. The non-transitory machine-readable medium of claim 15 , wherein the operations further comprise:
acquiring video stream data from a camera of the client device; and
overlaying, on a display of the client device, based on the augmented reality drawing data that was obtained, augmented reality graphics on video stream data to provide the plurality of augmented reality scenes.

Assignees

• Home Box Office Inc

Inventors

• Parr Richard

Classifications

• G06T15/005

  General purpose rendering architectures · CPC title

• G06T15/50

  Lighting effects · CPC title

• H04L67/131

  Protocols for games, networked simulations or virtual reality · CPC title

• G06T19/006 Primary

  Mixed reality (object pose determination, tracking or camera calibration for mixed reality G06T7/00) · CPC title

• G06F9/452

  Remote windowing, e.g. X-Window System, desktop virtualisation (protocols for virtual reality H04L67/131) · CPC title