Techniques for extrapolating image frames

Patent metadata

Field	Value
Publication number	US-10708597-B2
Application number	US-201815886448-A
Country	US
Kind code	B2
Filing date	Feb 1, 2018
Priority date	Feb 1, 2018
Publication date	Jul 7, 2020
Grant date	Jul 7, 2020

Abstract

Examples described herein generally relate to performing frame extrapolation in image frame rendering. A vertex mesh as a set of vertices is generated, and each vertex is mapped to a screen space position for defining a texture. One or more motion vectors for one or more regions in a first image frame of a stream of image frames can be determined. The screen space positions associated with at least a portion of the set of vertices within the texture can be modified based at least in part on the one or more motion vectors. A graphics processing unit (GPU) can render the first image frame into the texture. The extrapolated image frame is displayed after the first image frame and before a next image frame in the stream of image frames.

First claim

What is claimed is:

1. A method for performing frame extrapolation in image frame rendering, comprising:
generating a vertex mesh as a set of vertices;
mapping each vertex in the set of vertices to a screen space position for defining a texture;
determining one or more motion vectors for one or more regions in a first image frame of a stream of image frames, wherein the one or more motion vectors represent motion of the one or more regions from a previous image frame in the stream of image frames;
dividing the first image frame into areas to be modified based on the motion vectors;
modifying screen space positions associated with at least a portion of the set of vertices within the areas to be modified based at least in part on the one or more motion vectors;
rendering, by a graphics processing unit (GPU), the first image frame into the texture, such that a pixel position of one or more pixels in the first image frame are moved based at least in part on the screen space positions of at least the portion of the set of vertices within the texture as modified, to generate an extrapolated image frame, wherein movement of corresponding pixels of the first image frame are determined by properties of the modified screen space positions based on functionality of the GPU; and
displaying the extrapolated image frame after the first image frame and before a next image frame in the stream of image frames.

2. The method of claim 1 , further comprising:
determining depth information for one or more objects in the first image frame; and
associating the depth information with at least a portion of the screen space positions that are determined to correspond to a position of the one or more objects in the first image frame.

3. The method of claim 2 , wherein rendering the first image frame into the texture is further based at least in part on the depth information.

4. The method of claim 3 , wherein rendering the first image frame based at least in part on the depth information comprises occluding, based on the depth information, one or more pixels of the one or more objects in generating the extrapolated image frame.

5. The method of claim 1 , wherein rendering the first image frame into the texture results in one or more original pixels, corresponding to at least one of the one or more regions in the first image frame, inheriting properties of the one or more original pixels in generating the extrapolated image frame.

6. The method of claim 1 , wherein rendering the first image frame into the texture results in one or more original pixels, corresponding to at least one of the one or more regions in the first image frame, inheriting properties of one or more adjacent pixels that are adjacent to the original pixel in generating the extrapolated image frame.

7. The method of claim 1 , wherein determining the one or more motion vectors is based at least in part on determining an acceleration computed from motion of the one or more regions of the image frame between multiple previous image frames in the stream of image frames.

8. The method of claim 1 , wherein modifying screen space positions associated with at least a portion of the set of vertices within the texture comprises applying a shader to modify the screen space positions based at least in part on the one or more motion vectors.

9. A computing device for performing frame extrapolation in image frame rendering, comprising:
a memory storing one or more parameters or instructions for executing an operating system and one or more applications;
a display interface coupled with a display device for communicating signals to display images on the display device; and
at least one processor coupled to the memory and the display interface, wherein the at least one processor is configured to:
generate a vertex mesh as a set of vertices;
map each vertex in the set of vertices to a screen space position for defining a texture;
determine one or more motion vectors for one or more regions in a first image frame of a stream of image frames, wherein the one or more motion vectors represent motion of the one or more regions from a previous image frame in the stream of image frames;
divide the first image frame into areas to be modified based on the motion vectors;
modify screen space positions associated with at least a portion of the set of vertices within the areas to be modified based at least in part on the one or more motion vectors;
render, by a graphics processing unit (GPU), the first image frame into the texture, such that a pixel position of one or more pixels in the first image frame are moved based at least in part on the screen space positions of at least the portion of the set of vertices within the texture as modified, to generate an extrapolated image frame, wherein movement of corresponding pixels of the image first frame are determined by properties of the modified screen space positions based on functionality of the GPU; and
display the extrapolated image frame after the first image frame and before a next image frame in the stream of image frames.

10. The computing device of claim 9 , wherein the at least one processor is further configured to:
determine depth information for one or more objects in the first image frame; and
associate the depth information with at least a portion of the screen space positions that are determined to correspond to a position of the one or more objects in the first image frame.

11. The computing device of claim 10 , wherein the GPU renders the first image frame into the texture further based at least in part on the depth information.

12. The computing device of claim 11 , wherein the GPU renders the first image frame based at least in part on the depth information by occluding, based on the depth information, one or more pixels of the one or more objects in generating the extrapolated image frame.

13. The computing device of claim 9 , wherein rendering, by the GPU, the first image frame into the texture, results in one or more original pixels, corresponding to at least one of the one or more regions in the first image frame, inheriting properties of the one or more original pixels in generating the extrapolated image frame.

14. The computing device of claim 9 , wherein rendering, by the GPU, the first image frame into the texture results in one or more original pixels, corresponding to at least one of the one or more regions in the first image frame, inheriting properties of one or more adjacent pixels that are adjacent to the original pixel in generating the extrapolated image frame.

15. The computing device of claim 9 , wherein the at least one processor is configured to determine the one or more motion vectors based at least in part on determining an acceleration computed from motion of the one or more regions of the image frame between multiple previous image frames in the stream of image frames.

16. The computing device of claim 9 , wherein the at least one processor is configured to modify the screen space positions associated with at least a portion of the set of vertices within the texture based at least in part on applying a shader to modify the screen space positions based at least in part on the one or more motion vectors.

17. A non-transitory computer-readable medium, comprising code executable by one or more processors for performing frame extrapolation in image frame rendering, the code comprising code for:
generating a vertex mesh as a set of vertices;
mapping each vertex in the set of vertices to a screen space position for defining a texture;
determining one or more

Assignees

• Microsoft Technology Licensing Llc

Inventors

• Yeung Andrew Zicheng
• Boulton Michael George
• Michail Ashraf Ayman
• Bronder Matt
• Elliott Jack Andrew
• Sandy Matthew David

Classifications

• G06T13/80 Primary

  Two-dimensional [2D] animation, e.g. using sprites · CPC title

• G06T11/00

  Two-dimensional [2D] image generation · CPC title

• G06T17/20

  Finite element generation, e.g. wire-frame surface description, {tesselation} · CPC title

• G06T15/04

  Texture mapping · CPC title

• H04N19/139 Primary

  Analysis of motion vectors, e.g. their magnitude, direction, variance or reliability · CPC title