Efficient image population from captured scene

Patent metadata

Field	Value
Publication number	US-11146777-B2
Application number	US-201916276831-A
Country	US
Kind code	B2
Filing date	Feb 15, 2019
Priority date	Feb 15, 2019
Publication date	Oct 12, 2021
Grant date	Oct 12, 2021

Abstract

A method for efficiently populating a display is provided. The method can include identifying a point at which a world ray intersects a capture surface defined by capture points of a scene, identifying a capture point closest to the identified point, generating a motion vector based on the motion vectors for each of two directly adjacent capture points, identifying a vector in the generated motion vector at a location at which the world ray intersects an image surface, and providing a pixel value from the image data of the capture point, the pixel value corresponding to a location in the image surface at which a vector of the generated motion vector points to the location at which the world ray intersects the image surface within a threshold distance or after a specified number of iterations.

First claim

What is claimed is:

1. A virtual reality (VR) system comprising:
a memory device including data regarding a scene that is captured from multiple capture points, the data indicating, for each capture point of the capture points of the scene, image data, two directly adjacent capture points to the capture point, offset vectors to the two directly adjacent capture points, and motion vectors for each of the two directly adjacent capture points, the capture points correspond to respective camera locations of an array of cameras that captured the scene; and
processing circuitry to perform operations for populating a VR display, the operations comprising:
identifying a first point at which a world ray intersects a capture surface defined by the capture points;
identifying a capture point of the capture points closest to the first point;
identifying a first vector based on the first point and the capture point;
identifying a first motion vector based on motion vectors from the two directly adjacent capture points
estimating a pixel location in the image data corresponding to the capture point based on a combination of the first vector and the first motion vector; and
obtaining a pixel value from the estimated pixel location in the image data to populate the VR display.

2. The VR system of claim 1 , wherein the memory further includes data identifying a cell of a partition of the capture surface and an associated capture point closest to all points in the cell, and wherein identifying the capture point closest to the first point includes identifying the cell to which the first point maps and the associated capture point.

3. The VR system of claim 2 wherein the cell is part of a Voronoi grid based on the capture points.

4. The VR system of claim 1 , wherein the motion vectors for the two directly adjacent capture points form an orthogonal basis.

5. The VR system of claim 1 , and further comprising:
determining a difference vector between the first point and the estimated pixel location;
updating the first vector by adding the difference vector from the first vector;
comparing the difference vector to a threshold magnitude; and
using the updated first vector as the estimated pixel location in response to the comparing to obtain the pixel value.

6. The VR system of claim 5 and further comprising repeating the elements of claim 5 while updating an iteration counter until the iteration counter reaches an iteration threshold.

7. A non-transitory machine-readable medium including instructions that, when executed by a machine, cause the machine to perform operations comprising:
identifying a first point at which a world ray intersects a capture surface defined by capture points;
identifying a capture point of the capture points closest to the first point;
identifying a first vector based on the first point and the capture point;
identifying a first motion vector based on the motion vectors from the two directly adjacent capture points;
estimating a pixel location in the image data corresponding to the capture point based on a combination of the first vector and the first motion vector; and
obtaining a pixel value from the estimated pixel location in the image data to populate the VR display.

8. The non-transitory machine-readable medium of claim 7 , wherein identifying the capture point closest to the first point includes identifying a cell of a partition of the capture surface to which the first point maps.

9. The non-transitory machine-readable medium of claim 7 , wherein the motion vectors for the two directly adjacent capture points form an orthogonal basis.

10. A method for populating a virtual reality (VR) headset display, the method comprising:
identifying a first point at which a world ray intersects a capture surface defined by capture points of a scene;
identifying a capture point of the capture points closest to the first point;
identifying a first vector based on the first point and the capture point;
identifying a first motion vector based on the motion vectors from the two directly adjacent capture points;
estimating a pixel location in the image data corresponding to the capture point based on a combination of the first vector and the first motion vector; and
obtaining a pixel value from the estimated pixel location in the image data to populate the VR display.

11. The method of claim 10 , wherein identifying the capture point closest to the first point includes identifying a cell of a partition of the capture surface to which the first point maps and an associated capture point of the cell.

Assignees

• Microsoft Technology Licensing Llc

Inventors

• Michail Ashraf A
• You Yang
• Boulton Michael G

Classifications

• G06T15/205 Primary

  Image-based rendering · CPC title

• H04N13/282 Primary

  for generating image signals corresponding to three or more geometrical viewpoints, e.g. multi-view systems · CPC title

• H04N2013/0085

  Motion estimation from stereoscopic image signals · CPC title

• G06F17/16

  Matrix or vector computation {, e.g. matrix-matrix or matrix-vector multiplication, matrix factorization (matrix transposition G06F7/78)} · CPC title

• H04N13/275

  from three-dimensional [3D] object models, e.g. computer-generated stereoscopic image signals · CPC title