Method and apparatus for processing and distributing live virtual reality content

Patent metadata

Field	Value
Publication number	US-11282169-B2
Application number	US-201916958698-A
Country	US
Kind code	B2
Filing date	Jan 24, 2019
Priority date	Feb 7, 2018
Publication date	Mar 22, 2022
Grant date	Mar 22, 2022

Abstract

An apparatus, system, and method are described for providing real-time capture, processing, and distribution of panoramic virtual reality (VR) content. One embodiment of a graphics processor comprises a video interface to receive a plurality of images from a corresponding plurality of cameras; an image rectifier to perform a perspective re-projection of at least some of the images to a common image plane to generate a rectified plurality of images; a stitcher to analyze overlapping regions of adjacent images in the rectified images and to identify corresponding pixels in the overlapping regions and to stitch the adjacent images in accordance with the corresponding pixels to generate a panoramic image comprising a stitched combination of the rectified plurality of images; and a cylindrical projector to project the panoramic image onto a cylindrical surface to generate a final panoramic video image to be used to implement a VR environment on a VR apparatus.

First claim

What is claimed is:

1. A graphics processor comprising:
a video interface to receive a first plurality of images from a corresponding first plurality of cameras;
an image rectifier to perform a perspective re-projection of at least some of the first plurality of images to a common image plane to generate a rectified first plurality of images;
a stitcher to analyze overlapping regions of adjacent images in the rectified first plurality and to identify corresponding pixels in the overlapping regions and to stitch the adjacent images in accordance with the corresponding pixels to generate a panoramic image comprising a stitched combination of the rectified first plurality of images, wherein the stitcher is to analyze the overlapping regions by executing a sequence of Belief Propagation operations that comprise:
construction of an initial data cost volume using pixel data in the overlapping regions;
construction of a data cost pyramid based on the initial data cost volume comprising a series of smaller volumes;
iteration through the series of smaller volumes and the initial volume using Belief Propagation message passing to generate a final set of costs; and
construction of a stitch map from the final set of costs; and

a cylindrical projector to project the panoramic image onto a cylindrical surface to generate a final panoramic video image to be used to implement a virtual reality (VR) environment on a VR apparatus.

2. The graphics processor of claim 1 wherein the stitcher is to stitch the adjacent images in accordance with the final set of costs.

3. The graphics processor of claim 2 wherein for each pixel in the overlapping region the stitcher is to use the stitch map to determine which pixels to blend.

4. The graphics processor of claim 3 wherein the stitcher is to use a convex linear combination of pixels to blend from each image.

5. The graphics processor of claim 1 further comprising:
a lookahead buffer to store stitching parameters determined from one or more prior images, the stitcher to use at least a portion of the stitching parameters stored in the lookahead buffer to stitch the adjacent images in the rectified first plurality of images.

6. The graphics processor of claim 1 wherein the perspective re-projection comprises a homography transformation.

7. The graphics processor of claim 1 wherein the first plurality of images comprise left images to be displayed, following processing, in a left display of the VR apparatus.

8. The graphics processor of claim 7 wherein the video interface is to receive a right plurality of images from a corresponding second plurality of cameras,
wherein the image rectifier is to perform a perspective re-projection of at least some of the right plurality of images to a common image plane,
wherein the stitcher is to analyze overlapping regions of adjacent images in the right plurality and to identify corresponding pixels in the overlapping regions and to stitch the adjacent images in accordance with the corresponding pixels to generate a panoramic right eye image comprising a stitched combination of the right plurality of images; and
wherein the cylindrical projector is to project the panoramic right eye image onto a cylindrical surface to generate a final panoramic right eye image to be combined with a final panoramic left eye image to implement a virtual reality (VR) environment on a VR apparatus.

9. The graphics processor of claim 1 wherein the image rectifier, stitcher, and cylindrical projector comprise circuitry and/or executable software executed by an execution unit of the graphics processor.

10. A method comprising:
receiving a first plurality of images from a corresponding first plurality of cameras;
performing a perspective re-projection of at least some of the first plurality of images to a common image plane to generate a rectified first plurality of images;
analyzing overlapping regions of adjacent images in the rectified first plurality by performing a sequence of Belief Propagation operations and responsively identifying corresponding pixels in the overlapping regions, wherein the sequence of Belief Propagation operations comprise:
constructing an initial data cost volume using pixel data in the overlapping regions;
constructing a data cost pyramid based on the initial data cost volume comprising a series of smaller volumes;
iterating through the series of smaller volumes and the initial volume using Belief Propagation message passing to generate a final set of costs; and
constructing a stitch map from the final set of costs;

stitching the adjacent images in accordance with the corresponding pixels to generate a panoramic image comprising a stitched combination of the rectified first plurality of images; and
projecting the panoramic image onto a cylindrical surface to generate a final panoramic video image to be used to implement a virtual reality (VR) environment on a VR apparatus.

11. The method of claim 10 wherein stitching comprises stitching the adjacent images in accordance with the final set of costs.

12. The method of claim 11 wherein for each pixel in the overlapping region the stitch map is used to determine which pixels to blend.

13. The method of claim 12 wherein a convex linear combination of pixels is selected to blend from each image.

14. The method of claim 10 further comprising:
storing stitching parameters determined from one or more prior images, and
stitching the adjacent images in the rectified first plurality of images using at least a portion of the stitching parameters.

15. The method of claim 10 wherein the perspective re-projection comprises a homography transformation.

16. The method of claim 10 wherein the first plurality of images comprise left images to be displayed, following processing, in a left display of the VR apparatus.

17. The method of claim 16 wherein the video interface is to receive a right plurality of images from a corresponding second plurality of cameras, the method further comprising:
performing a perspective re-projection of at least some of the right plurality of images to a common image plane to generate rectified right plurality of images;
analyzing overlapping regions of adjacent images in the rectified right plurality and identifying corresponding pixels in the overlapping regions; and
stitching the adjacent images in accordance with the corresponding pixels to generate a panoramic right eye image comprising a stitched combination of the rectified right plurality of images; and
projecting the panoramic right eye image onto a cylindrical surface to generate a final panoramic right eye image to be combined with a final panoramic left eye image to implement a virtual reality (VR) environment on a VR apparatus.

18. A non-transitory machine-readable medium having program code stored thereon which, when executed by a machine, causes the machine to perform operations of:
receiving a first plurality of images from a corresponding first plurality of cameras;
performing a perspective re-projection of at least some of the first plurality of images to a common image plane to generate a rectified first plurality of images;
analyzing overlapping regions of adjacent images in the rectified first plurality by performing a sequence of Belief Propagation operations and responsively identifying corresponding pixels in the overlapping regions, wherein the sequence of Belief Propagation operations comprise:
constructing an initial data cost volume using pixel data in the overlapping regions;
constructing a data cost pyrami

Assignees

• Intel Corp

Inventors

• Cochran Wayne
• Yeung Fai
• Mathur Durga Raj
• Goncalves De Lima Gilson
• Shon Patrick Youngung
• Harrison John A
• Kim Ok Joon
• Gill Harleen
• Siehl Kyle
• Jayaram Uma
• Jayaram Sankar
• Sharma Archie
• Clingir Gockcen
• Baran Stanley
• Varerkar Mayuresh
• Das Barnan
• Biswal Narayan
• Shah Nilesh
• Kale Ritesh
• Weinstein Greg

Classifications

• H04N23/662

  by using camera arrangements where one camera controls another camera to affect the control of camera image capture, e.g. placing the camera in a desirable condition to capture a desired image · CPC title

• H04N23/698

  for achieving an enlarged field of view, e.g. panoramic image capture · CPC title

• H04N23/90

  Arrangement of cameras or camera modules, e.g. multiple cameras in TV studios or sports stadiums · CPC title

• H04N21/8133

  specifically related to the content, e.g. biography of the actors in a movie, detailed information about an article seen in a video programme · CPC title

• H04N21/234309

  by transcoding between formats or standards, e.g. from MPEG-2 to MPEG-4 or from Quicktime to Realvideo (conversion of standards in analog television systems H04N7/01) · CPC title