Frame extrapolation via motion vectors

Patent metadata

Field	Value
Publication number	US-2020137409-A1
Application number	US-201816171969-A
Country	US
Kind code	A1
Filing date	Oct 26, 2018
Priority date	Oct 26, 2018
Publication date	Apr 30, 2020
Grant date	—

Abstract

Examples are disclosed that relate to producing an extrapolated frame based on motion vectors. One example provides a computing device comprising a logic machine and a storage machine comprising instructions executable by the logic machine to, for each block of one or more blocks of pixels in rendered image data, generate a motion vector indicating motion between a current frame and a prior frame, and for each block of the one or blocks, extrapolate a predicted block of pixels from the current frame based on the motion vector and one or more prior motion vectors for the block, the one or more prior motion vectors determined via one or more corresponding frames preceding the prior frame. The instructions are further executable to produce an extrapolated frame comprising the predicted block of pixels for each block of the one or more blocks, and display the extrapolated frame.

First claim

1 . A computing device, comprising:
a logic machine; and a storage machine comprising instructions executable by the logic machine to
for each block of one or more blocks of pixels in rendered image data, generate a motion vector indicating motion between a current frame and a prior frame;
for each block of the one or blocks, extrapolate a predicted block of pixels from the current frame based on the motion vector and one or more prior motion vectors for the block, the one or more prior motion vectors determined via one or more corresponding frames preceding the prior frame;
produce an extrapolated frame comprising the predicted block of pixels for each block of the one or more blocks; and
display the extrapolated frame.

2 . The computing device of claim 1 , wherein the instructions are executed in response to detecting that the current frame does not meet a framerate condition.

3 . The computing device of claim 1 , wherein the predicted block of pixels is extrapolated based upon a magnitude and a direction of the motion vector and respective magnitudes and respective directions of the one or more prior motion vectors.

4 . The computing device of claim 1 , wherein a respective contribution of each of the one or more prior motion vectors to the predicted block of pixels decays as a number of frames separating the current frame and the corresponding frame increases.

5 . The computing device of claim 1 , wherein the motion vector is weighted with a weight, the weight determined based upon a spatial correspondence between the motion vector for the block and one or more current motion vectors in spatially proximate blocks.

6 . The computing device of claim 5 , wherein the spatial correspondence is determined via a kernel, and wherein the kernel is configured to output a respective weight for each of the one or more current motion vectors.

7 . The computing device of claim 1 , wherein the instructions executable to generate the motion vector for each block of the one or more blocks are executed on a video encoder.

8 . The computing device of claim 1 , wherein the motion vector is generated via an application that generates the rendered image data.

9 . The computing device of claim 1 , further comprising instructions executable to determine whether the motion vector for the block meets a discard condition, and not use the motion vector for generating the predicted block of pixels.

10 . At a computing device, a method, comprising:
for each block of one or more blocks of pixels in rendered image data, generating a motion vector indicating motion between a current frame and a prior frame; for each block of the one or blocks, extrapolating a predicted block of pixels from the current frame based on the motion vector and one or more prior motion vectors for the block, the one or more prior motion vectors determined via one or more corresponding frames preceding the prior frame; producing an extrapolated frame comprising the predicted block of pixels for each block of the one or more blocks; and displaying the extrapolated frame.

11 . The method of claim 10 , wherein the method is executed in response to detecting that the current frame does not meet a framerate condition.

12 . The method of claim 10 , wherein the predicted block of pixels is extrapolated based upon a magnitude and a direction of the motion vector and respective magnitudes and respective directions of the one or more prior motion vectors.

13 . The method of claim 10 , wherein a respective contribution of each of the one or more prior motion vectors to the predicted block of pixels decays as a number of frames separating the current frame and the corresponding frame increases.

14 . The method of claim 10 , wherein the motion vector is weighted with a weight, the weight determined based upon a spatial correspondence between the motion vector for the block and one or more current motion vectors in spatially proximate blocks.

15 . The method of claim 14 , wherein the spatial correspondence is determined via a kernel, and wherein the kernel is configured to output a respective weight for each of the one or more current motion vectors.

16 . The method of claim 10 , wherein the motion vector generated for each block of the one or more blocks is generated via a video encoder.

17 . The method of claim 10 , wherein the motion vector is generated via an application that generates the rendered image data.

18 . The method of claim 10 , further comprising determining whether the motion vector for the block meets a discard condition, and not using the motion vector for extrapolating the predicted block of pixels for the block if the discard condition is met.

19 . A computing device, comprising:
a logic machine; and a storage machine comprising instructions executable by the logic machine to
for each block of one or more blocks of pixels in rendered image data, generate a motion vector indicating motion between a current frame and a prior frame;
for each block of the one or blocks, extrapolate a predicted block of pixels from the current frame based upon a spatial correspondence between the motion vector and one or more current motion vectors in spatially proximate blocks, and also based upon a temporal correspondence between the motion vector and one or more prior motion vectors for the block, the one or more prior motion vectors determined via one or more corresponding frames preceding the prior frame;
produce an extrapolated frame comprising the predicted block of pixels for each block of the one or more blocks; and
display the extrapolated frame.

20 . The computing device of claim 19 , wherein a respective contribution of each of the one or more prior motion vectors to the predicted block of pixels decays as a number of frames separating the current frame and the corresponding frame increases.

Assignees

• Microsoft Technology Licensing Llc

Inventors

• Michail Ashraf Ayman
• Boulton Michael George

Classifications

• H04N19/587 Primary

  involving temporal sub-sampling or interpolation, e.g. decimation or subsequent interpolation of pictures in a video sequence · CPC title

• H04N19/521

  for estimating the reliability of the determined motion vectors or motion vector field, e.g. for smoothing the motion vector field or for correcting motion vectors · CPC title

• H04N19/139

  Analysis of motion vectors, e.g. their magnitude, direction, variance or reliability · CPC title

• H04N19/59

  involving spatial sub-sampling or interpolation, e.g. alteration of picture size or resolution · CPC title

• H04N19/176

  the region being a block, e.g. a macroblock · CPC title