Continuous depth-ordered image compositing

Patent metadata

Field	Value
Publication number	US-10681325-B2
Application number	US-201615156192-A
Country	US
Kind code	B2
Filing date	May 16, 2016
Priority date	May 16, 2016
Publication date	Jun 9, 2020
Grant date	Jun 9, 2020

Abstract

A system creates an output image of a scene using two-dimensional (2D) images of the scene. For a pixel in the output image, the system identifies, in the output image, 2D fragments that correspond to the pixel. The system converts the 2D fragments into three dimensional (3D) fragments, creates volume spans for the pixel based on the 3D fragments, determines a color of a volume span based on color contribution of respective one or more of the 3D fragments for the volume span, and determines a color of the pixel for the output image from determined colors of the volume spans.

First claim

What is claimed is:

1. A method comprising:
creating an output image of a scene for an output video stream by aligning a plurality of two-dimensional (2D) images of the scene that are captured by a plurality of cameras;
collecting, based on the plurality of two-dimensional (2D) images of the scene and using surface splatting point rendering, a plurality of 2D fragments for each of a plurality of pixels of the output image for the scene for the output video stream, wherein each 2D fragment contributes a color to a color for the pixel of the output image for the output video stream and is associated with a particular location on an axis of a third dimension;
converting, by a processing device, the plurality of 2D fragments into a plurality of three dimensional (3D) fragments by adding a pre-defined thickness along the axis of the third dimension to each 2D fragment at the particular location on the axis of the third dimension;
creating, by the processing device, a plurality of volume spans for the pixel based on locations of the plurality of 3D fragments along the axis of the third dimension;
determining a color of a volume span of the plurality of volume spans based on a color of one or more of the plurality of 3D fragments corresponding to the volume span; and
determining the color of the pixel for the output image for the output video stream from determined colors of the plurality of volume spans.

2. The method of claim 1 , wherein the pre-defined thickness is defined in a disparity space.

3. The method of claim 1 , wherein determining the color of the volume span comprises:
identifying the one or more of the plurality of 3D fragments that contribute color to the volume span; and
determining color contribution of the identified one or more of the plurality of 3D fragments.

4. The method of claim 3 , wherein determining the color contribution of the identified one or more of the plurality of 3D fragments comprises:
determining a length of the volume span;
determining a total thickness of individual 3D fragments of the identified one or more of the plurality of 3D fragments; and
dividing the length of the volume span by the total thickness.

5. The method of claim 1 , wherein creating the plurality of volume spans for the pixel based on locations of the plurality of 3D fragments along the axis of the third dimension comprises:
marking start events and end events for the 3D fragments;
ordering the start events and end events for the 3D fragments sequentially; and
defining start events and end events for the plurality of volume spans for the pixel based on a sequential ordering of the start events and end events for the 3D fragments.

6. A system comprising:
a memory; and
a processing device, coupled to the memory, to:
create an output image of a scene for an output video stream by aligning a plurality of two-dimensional (2D) images of the scene that are captured by a plurality of cameras;
collect, based on the plurality of two-dimensional (2D) images of the scene and using surface splatting point rendering, a plurality of 2D fragments for each of a plurality of pixels of the output image for the scene for the output video stream, wherein each 2D fragment contributes a color to a color for the pixel of the output image for the output video stream and is associated with a particular location on an axis of a third dimension;
convert the plurality of 2D fragments into a plurality of three dimensional (3D) fragments by adding a pre-defined thickness along the axis of the third dimension to each 2D fragment at the particular location on the axis of the third dimension;
create a plurality of volume spans for the pixel based on locations of the plurality of 3D fragments along the axis of the third dimension;
determine a color of a volume span of the plurality of volume spans based on a color of one or more of the plurality of 3D fragments corresponding to the volume span; and
determine the color of the pixel for the output image for the output video stream from determined colors of the plurality of volume spans.

7. The system of claim 6 , wherein the pre-defined thickness is defined in a disparity space.

8. The system of claim 6 , wherein to determine the color of the volume span, the processing device is to:
identify the one or more of the plurality of 3D fragments that contribute color to the volume span; and
determine color contribution of the identified one or more of the plurality of 3D fragments.

9. The system of claim 6 , wherein to create the plurality of volume spans for the pixel based on locations of the plurality of 3D fragments along the axis of the third dimension, the processing device is to:
mark start events and end events for the 3D fragments;
order the start events and end events for the 3D fragments sequentially; and
define start events and end events for the plurality of volume spans for the pixel based on a sequential ordering of the start events and end events for the 3D fragments.

10. A non-transitory computer readable medium having instructions stored thereon that, when executed by a processing device of a user device, cause the processing device to perform operations comprising:
creating an output image of a scene for an output video stream using a plurality of two-dimensional (2D) images of the scene that are captured by a plurality of cameras;
collecting, based on the plurality of two-dimensional (2D) images of the scene and using surface splatting point rendering, a plurality of 2D fragments for each of a plurality of pixels of the output image for the scene for the output video stream, wherein each 2D fragment contributes a color to a color for the pixel of the output image for the output video stream and is associated with a particular location on an axis of a third dimension;
converting, by the processing device, the plurality of 2D fragments into a plurality of three dimensional (3D) fragments by adding a pre-defined thickness along the axis of the third dimension to each 2D fragment at the particular location on the axis of the third dimension;
creating, by the processing device, a plurality of volume spans for the pixel based on locations of the plurality of 3D fragments along the axis of the third dimension;
determining a color of a volume span of the plurality of volume spans based on a color of one or more 3D fragments corresponding to the volume span; and
determining the color of the pixel for the output image for the output video stream from determined colors of the plurality of volume spans.

11. The non-transitory computer readable medium of claim 10 , wherein the pre-defined thickness is defined in a disparity space.

12. The non-transitory computer readable medium of claim 10 , wherein determining the color of the volume span comprises:
identifying the one or more of the plurality of 3D fragments that contribute color to the volume span; and
determining color contribution of the identified one or more of the plurality of 3D fragments.

13. The non-transitory computer readable medium of claim 12 , wherein determining the color contribution of the identified one or more of the plurality of 3D fragments comprises:
determining a length of the volume span;
determining a total thickness of individual 3D fragments of the identified one or more of the plurality of 3D fragments; and
dividing the length of the volume span by the total thickness.

14. The non-transitory computer readable medium of claim 10 , wherein creating the plurality of volume spans for the pixel based on locations of the plurality of 3D fragments along the axis of the third dimension comprises:

Assignees

• Google Inc
• Google Llc

Inventors

• Kontkanen Janne
• Snavely Noah

Classifications

• H04N13/257

  Colour aspects · CPC title

• H04N13/106 Primary

  Processing image signals (for multi-view video sequence encoding H04N19/597) · CPC title

• H04N13/204

  using stereoscopic image cameras (stereoscopic photography G03B35/00) · CPC title

• G06T15/40

  Hidden part removal · CPC title

• G06T15/503

  Blending, e.g. for anti-aliasing · CPC title