Hybrid rendering

Patent metadata

Field	Value
Publication number	US-11170579-B2
Application number	US-201916379693-A
Country	US
Kind code	B2
Filing date	Apr 9, 2019
Priority date	Apr 9, 2019
Publication date	Nov 9, 2021
Grant date	Nov 9, 2021

Abstract

One disclosed example provides a computing device comprising a processing device and a storage device storing instructions executable by the processing device to execute in a first local process an application that outputs digital content for rendering and display. During execution of the application, the instructions are executable to provide, to a second local or remote process, object information regarding an object to be rendered by the second local or remote process, receive, from the second local or remote process, a rendering of the object, output the rendering of the object to display the object, receive a manipulation made to the object, provide, to the second local or remote process, updated object information based on the manipulation made to the object, receive, from the second local or remote process, an updated rendering of the object, and output the updated rendering of the object to display the object.

First claim

The invention claimed is:

1. A computing device, comprising:
a processing device; and
a storage device storing instructions executable by the processing device to:
execute in a first local process an application that outputs digital content comprising a first object for rendering and display of the first object, the rendering comprising depth data for the first object, the application comprising conditional logic configured to determine whether to render an object remotely or locally based at least on a threshold network latency;
during execution of the application, based on the conditional logic, provide, to a second local or remote computing system process, object information regarding a second object to be rendered by the second local or remote computing system process;
receive, from the second local or remote computing system process, a rendering of the second object and depth data for the second object, wherein the rendering of the second object received from the second local or remote computing system process comprises a first color buffer;

perform a reprojection of the first color buffer;
composite the first object as rendered and the second object as rendered for display as a composite frame by performing depth-correct blending between the first object and the second object based on the depth data for the first object and the depth data for the second object;
perform a reprojection of a color buffer of the composite frame;
output, to a display, the composite frame;
receive a manipulation made to the second object;
provide, to the second local or remote computing system process, updated object information based on the manipulation made to the second object;
receive, from the second local or remote computing system process, an updated rendering of the second object; and
output, to the display, the updated rendering of the second object to display the second object.

2. The computing device of claim 1 , wherein the rendering of the first object received from the second local or remote process comprises first depth data, and wherein the rendering of the second object comprises second depth data, and wherein the instructions are executable to composite the rendering of the first object and the rendering of the second object by performing a z-test on the first depth data and the second depth data and compositing the first object and the second object based on the z-test.

3. The computing device of claim 1 , wherein the instructions are executable to provide the object information by providing a three-dimensional model comprising a geometry of the second object.

4. The computing device of claim 1 , wherein the instructions are executable to provide the object information by providing a request to update a data structure for the digital rendered content.

5. The computing device of claim 1 , wherein the rendering of one or more of the first object and the second object comprises video data.

6. The computing device of claim 5 , wherein the video data further comprises a representation of depth.

7. The computing device of claim 1 , wherein the instructions are executable to receive the manipulation made to the second object by receiving sensor data indicating a change in head pose and/or eye gaze direction, and/or by receiving a user input to manipulate the object.

8. A method enacted on a computing device, the method comprising:
executing an application that outputs digital content for rendering and display, the application comprising conditional logic configured to determine whether to render an object remotely or locally based at least on a threshold network latency;
during execution of the application:
based on the conditional logic, identifying a first object of the digital content to be rendered remotely, and also identifying a second object of the digital content to be rendered locally;

providing, to a remote computing system, object information regarding the first object to be rendered by the remote computing system;
receiving, from the remote computing system, a rendering of the first object and depth data for the first object, wherein the rendering of the first object comprises a first color buffer;
performing a reprojection of the first color buffer;

locally rendering the second object to generate a rendering of the second object;
compositing the rendering of the first object and the rendering of the second object for display as a composite frame by performing depth-correct blending between the first object and the second object based on the depth data for the first object and depth data for the second object;

perform a reprojection of a color buffer of the composite frame; and
output the composite frame for display.

9. The method of claim 8 , wherein performing the reprojection of the first color buffer comprises performing the reprojection based on a head pose of a user.

10. The method of claim 8 , wherein the rendering of the first object comprises first depth data, and wherein the rendering of the second object comprises second depth data, the method further comprising:
performing a depth test on the first depth data and the second depth data; and
compositing the rendering of the first object and the rendering of the second object based on the depth test.

Assignees

• Microsoft Technology Licensing Llc

Inventors

• Frommhold Dag Birger
• Lyons Jonathan Michael
• Thaut Benjamin Markus
• Michail Ashraf Ayman

Classifications

• G06T2219/2012

  Colour editing, changing, or manipulating; Use of colour codes · CPC title

• G06T19/20 Primary

  Editing of three-dimensional [3D] images, e.g. changing shapes or colours, aligning objects or positioning parts · CPC title

• G06T17/10

  Constructive solid geometry [CSG] using solid primitives, e.g. cylinders, cubes · CPC title

• G06T15/405

  using Z-buffer · CPC title

• G06F9/452 Primary

  Remote windowing, e.g. X-Window System, desktop virtualisation (protocols for virtual reality H04L67/131) · CPC title