Computing images of dynamic scenes

Patent metadata

Field	Value
Publication number	US-2021390761-A1
Application number	US-202016927928-A
Country	US
Kind code	A1
Filing date	Jul 13, 2020
Priority date	Jun 15, 2020
Publication date	Dec 16, 2021
Grant date	—

Abstract

Computing an output image of a dynamic scene. A value of E is selected which is a parameter describing desired dynamic content of the scene in the output image. Using selected intrinsic camera parameters and a selected viewpoint, for individual pixels of the output image to be generated, the method computes a ray that goes from a virtual camera through the pixel into the dynamic scene. For individual ones of the rays, sample at least one point along the ray. For individual ones of the sampled points, a viewing direction being a direction of the corresponding ray, and E, query a machine learning model to produce colour and opacity values at the sampled point with the dynamic content of the scene as specified by E. For individual ones of the rays, apply a volume rendering method to the colour and opacity values computed along that ray, to produce a pixel value of the output image.

First claim

What is claimed is:

1 . An apparatus comprising:
at least one processor; a memory storing instructions that, when executed by the at least one processor ( 714 ), perform a method for computing an output image of a dynamic scene comprising: selecting: a value of E which is a parameter describing desired dynamic content of the scene in the output image, intrinsic camera parameter values of a virtual camera associated with the output image, a viewpoint for the virtual camera; accessing a trained machine learning model which has been trained to produce colour and density values, given points in the dynamic 3D scene, a viewing direction and a value of E; using the intrinsic camera parameters and the viewpoint, for individual pixels of the output image to be generated, compute a ray that goes from the virtual camera through the pixel into the dynamic scene; for individual ones of the rays, sample at least one point along the ray; for individual ones of the sampled points, a viewing direction being a direction of the corresponding ray, and E, query the machine learning model to produce colour and opacity values at the sampled point with the dynamic content of the scene as specified by E; for individual ones of the rays, apply a volume rendering method to the colour and opacity values computed along that ray, to produce a pixel value of the output image.

2 . The apparatus of claim 1 wherein the instructions comprise one or more of: storing the output image, transmitting the output image to a computer game application, transmitting the output image to a telepresence application, inserting the output image into a virtual webcam stream.

3 . The apparatus of claim 1 wherein the machine learning model has been trained using a plurality of images of the dynamic scene from a plurality of different viewpoints over a period of time.

4 . The apparatus of claim 1 wherein the parameter E is a time signal.

5 . The apparatus of claim 1 wherein the parameter E comprises values of parameters of a 3D model of an object in the dynamic scene at a specified time.

6 . The apparatus of claim 5 wherein the parameters of the 3D model comprise position, orientation and shape parameters.

7 . The apparatus of claim 5 wherein the object is a face and the parameter E comprises values of expression parameters of a 3D model of the face in the dynamic scene at the specified time.

8 . The apparatus of claim 5 wherein the object is a person and the parameter E comprises values of joint position and joint orientation parameters of a 3D model of the person in the dynamic scene at the specified time.

9 . The apparatus of claim 1 wherein querying the machine learning model comprises inputting the selected value of E to the machine learning model together with the associated sampled point.

10 . The apparatus of claim 1 wherein querying the machine learning model comprises inputting the selected value of E to the machine learning model after encoding the selected value of E using a positional encoding.

11 . The apparatus of claim 1 wherein the machine learning model is a neural network and wherein querying the machine learning model comprises using the selected value of E to modify one or more activations of the neural network.

12 . The apparatus of claim 1 wherein the machine learning model is a neural network and wherein querying the machine learning model comprises using the selected value of E to modify one or more weights of the neural network.

13 . The apparatus of claim 1 wherein the machine learning model is a neural network and wherein querying the machine learning model comprises one or more of: inputting the selected value of E to the machine learning model together with the associated sampled point, using the selected value of E to modify one or more activations of the neural network, using the selected value of E to modify one or more weights of the neural network.

14 . The apparatus of claim 1 wherein sampling at least one point along the ray comprises taking into account bounds of the scene.

15 . The apparatus of claim 1 integral with a head mounted display.

16 . The apparatus of claim 1 wherein the instructions comprise training the machine learning model using training data comprising images of the dynamic scene from a plurality of viewpoints at a plurality of different times.

17 . A computer-implemented method for computing an output image of a dynamic scene comprising:
selecting: a value of E which is a parameter describing desired dynamic content of the scene in the output image, intrinsic camera parameter values of a virtual camera associated with the output image, a viewpoint for the virtual camera; accessing a trained machine learning model which has been trained to produce colour and density values, given points in the dynamic 3D scene, a viewing direction and a value of E; using the intrinsic camera parameters and the viewpoint, for individual pixels of the output image to be generated, compute a ray that goes from the virtual camera through the pixel into the dynamic scene; for individual ones of the rays, sample at least one point along the ray; for individual ones of the sampled points, a viewing direction being a direction of the corresponding ray, and E, query the machine learning model to produce colour and opacity values at the sampled point with the dynamic content of the scene as specified by E; for individual ones of the rays, apply a volume rendering method to the colour and opacity values computed along that ray, to produce a pixel value of the output image.

18 . The computer-implemented method of claim 17 comprising selecting the value of E according to one or more of: user input, captured sensor data, computer game state.

19 . The computer-implemented method of claim 17 wherein querying the machine learning model comprises one or more of: inputting the selected value of E to the machine learning model together with the associated sampled point, using the selected value of E to modify one or more activations of the neural network, using the selected value of E to modify one or more weights of the neural network.

20 . A computer-implemented method of training a machine learning model the method comprising:
accessing a plurality of training images of a dynamic scene, the training images having been captured from a plurality of different viewpoints and at a plurality of different times; for individual ones of the training images, specifying a viewing direction according to a known viewpoint of a capture device which captured the image; for individual ones of the training images, specifying a value of E using one or more of: a time when the image was captured, a value of parameters of a 3D model of an object in the scene at the time when the image was captured; for individual ones of the training images, extracting colour of points in the dynamic 3D scene; training the machine learning model using supervised learning given the training images such that the machine learning model produces colour and density values, given points in the dynamic 3D scene, a viewing direction and a value of E.

Assignees

• Microsoft Technology Licensing Llc

Inventors

• Kowalski Marek Adam
• Johnson Matthew Alastair
• Shotton Jamie Daniel Joseph

Classifications

• G06N3/048

  Activation functions · CPC title

• G06N3/09

  Supervised learning · CPC title

• G06N3/0499

  Feedforward networks · CPC title

• A63F13/355

  Performing operations on behalf of clients with restricted processing capabilities, e.g. servers transform changing game scene into an encoded video stream for transmitting to a mobile phone or a thin client · CPC title

• G06N20/20

  Ensemble learning · CPC title