Transition between binocular and monocular views

Patent metadata

Field	Value
Publication number	US-9934615-B2
Application number	US-201615096162-A
Country	US
Kind code	B2
Filing date	Apr 11, 2016
Priority date	Apr 6, 2016
Publication date	Apr 3, 2018
Grant date	Apr 3, 2018

Abstract

An image processing system is designed to generate a canvas view that transitions between binocular views and monocular views. Initially, the image processing system receives top/bottom images and side images of a scene and calculates offsets to generate synthetic side images for left and right view of a user. To transition between binocular views and monocular views, the image processing system first warps top/bottom images onto corresponding synthetic side images to generate warped top/bottom images, which realizes the transition in terms of shape. The image processing system then morphs the warped top/bottom images onto the corresponding synthetic side images to generate blended images for left and right eye views with the blended images. The image processing system creates the canvas view which transitions between binocular views and monocular views in terms of image shape and color based on the blended images.

First claim

What is claimed is:

1. A method comprising:
receiving a top image of a scene;
receiving a first and second synthetic image of the scene, the first and second synthetic image separately corresponding to a left eye view and a right eye view of a user; and
for each of the first and second synthetic images:
identifying an overlapping portion of the top image of the scene with the corresponding synthetic image;
within the overlapping portion, determining an optical flow from the top image of the scene to the corresponding synthetic image;
generating a warped top image by blending the top image onto the corresponding synthetic image by applying the optical flow, wherein the optical flow is applied as a function of a vertical angle; and
generating a canvas view for a corresponding eye view of the user by combining the top image of the scene, the corresponding synthetic image, and the warped top image.

2. The method of claim 1 , wherein the function for applying optical flow applies no optical flow at a vertical angle where the overlapping portion begins adjacent to the top image, and applies a full optical flow at a pivotal row in the overlapping portion.

3. The method of claim 2 , further comprising,
for each of the first and second synthetic images: morphing the color of the top image with the corresponding synthetic image between the pivotal row and a side-only portion of the corresponding synthetic image where the overlapping portion is adjacent to the side-only portion.

4. The method of claim 3 , wherein the color morphing ranges from zero to complete color morphing of a side-only image color between the pivotal row and the overlapping portion adjacent to the side-only portion.

5. The method of claim 1 , wherein the left eye view and the right eye view are separated by a distance as a function of an interpupillary distance (IPD) between the left eye view and the right eye view of the user.

6. The method of claim 5 , wherein the IPD is a function of the vertical angle of the scene.

7. The method of claim 6 , wherein at a vertical angle of π/2, the IPD is a maximum IPD, and at a vertical angle of π, the IPD is zero.

8. A non-transitory computer-readable storage medium storing computer program instructions, the computer program instructions when executed by a computer processor causes the processor to perform steps of:
receiving a top image of a scene;
receiving a first and second synthetic image of the scene, the first and second synthetic image separately corresponding to a left eye view and a right eye view of a user; and
for each of the first and second synthetic images:
identifying an overlapping portion of the top image of the scene with the corresponding synthetic image;
within the overlapping portion, determining an optical flow from the top image of the scene to the corresponding synthetic image;
generating a warped top image by blending the top image onto the corresponding synthetic image by applying the optical flow, wherein the optical flow is applied as a function of a vertical angle; and
generating a canvas view for the corresponding eye view of the user by combining the top image of the scene, the corresponding synthetic image, and the warped top image.

9. The non-transitory computer-readable storage medium of claim 8 , wherein the function for applying optical flow applies no optical flow at a vertical angle where the overlapping portion begins adjacent to the top image, and applies a full optical flow at a pivotal row in the overlapping portion.

10. The non-transitory computer-readable storage medium of claim 9 , further comprising,
for each of the first and second synthetic images: morphing the color of the top image with the corresponding synthetic image between the pivotal row and a side-only portion of the corresponding synthetic image where the overlapping portion is adjacent to the side-only portion.

11. The non-transitory computer-readable storage medium of claim 10 , wherein the color morphing ranges from zero to complete color morphing of a side-only image color between the pivotal row and the overlapping portion adjacent to the side-only portion.

12. The non-transitory computer-readable storage medium of claim 8 , wherein the left eye view and the right eye view are separated by a distance as a function of an interpupillary distance (IPD) between the left eye view and the right eye view of the user.

13. The non-transitory computer-readable storage medium of claim 12 , wherein the IPD is a function of the vertical angle of the scene.

14. The non-transitory computer-readable storage medium of claim 13 , wherein at a vertical angle of π/2, the IPD is a maximum IPD, and at a vertical angle of π, the IPD is zero.

Assignees

• Facebook Inc

Inventors

• Cabral Brian Keith
• Briggs Forrest Samuel

Classifications

• G06T3/4038 Primary

  Image mosaicing, e.g. composing plane images from plane sub-images · CPC title

• G06T19/006 Primary

  Mixed reality (object pose determination, tracking or camera calibration for mixed reality G06T7/00) · CPC title

• H04N23/698

  for achieving an enlarged field of view, e.g. panoramic image capture · CPC title

• G06V10/147

  Details of sensors, e.g. sensor lenses (fingerprint or palmprint sensors G06V40/13; vascular sensors G06V40/145; eye sensors G06V40/19) · CPC title

• G06V10/16

  using multiple overlapping images; Image stitching · CPC title