Method and apparatus for obtaining binocular panoramic image, and storage medium

Patent metadata

Field	Value
Publication number	US-10855909-B2
Application number	US-201816041514-A
Country	US
Kind code	B2
Filing date	Jul 20, 2018
Priority date	Jul 29, 2016
Publication date	Dec 1, 2020
Grant date	Dec 1, 2020

Abstract

A method for obtaining a binocular panoramic image is performed at an electronic apparatus. The method includes: obtaining first and second panoramic images acquired by two panoramic cameras; obtaining at least one group of a first pixel located in the first panoramic image and a second pixel located in the second panoramic image; calculating a distance between the first pixel and the second pixel in each group, and obtaining depth information corresponding to the two panoramic image according to the distance between the first pixel and the second pixel; obtaining a second monocular panoramic image through mapping by using one of the two panoramic images as a first monocular panoramic image, in combination with the corresponding depth information and a preset pupil distance between a first eye and a second eye; and displaying the first monocular panoramic image and the second monocular panoramic image in corresponding display regions.

First claim

What is claimed is:

1. A method for obtaining a binocular panoramic image performed at an electronic apparatus having one or more processors and memory storing instructions to be executed by the one or more processors, comprising:
obtaining, by the electronic apparatus, a first panoramic image acquired by a first panoramic camera and a second panoramic image acquired by a second panoramic camera, wherein the first panoramic camera and the second panoramic camera are both horizontally disposed, the first panoramic image has multiple pixels, and the second panoramic image has multiple pixels;
obtaining, by the electronic apparatus, at least one group of a first pixel and a second pixel that match each other, wherein the first pixel is a pixel located in the first panoramic image, the second pixel is a pixel located in the second panoramic image, and for the first pixel and the second pixel that match each other, pixel values of the first pixel and adjacent pixels thereof and pixel values of the second pixel and adjacent pixels thereof meet a preset condition;
calculating, by the electronic apparatus, a distance between the matching first pixel and second pixel in the at least one group, wherein the distance between the first pixel and the second pixel is a distance from the second pixel to a pixel that is in the second panoramic image and that corresponds to a position of the first pixel;
obtaining, by the electronic apparatus, depth information of the first panoramic image and depth information of the second panoramic image according to the distance between the first pixel and the second pixel;
obtaining, by the electronic apparatus, a second monocular panoramic image through mapping by choosing one of the first panoramic image and the second panoramic image as a first monocular panoramic image, in combination with the depth information of a corresponding panoramic image used as the first monocular panoramic image, the first monocular panoramic image and a preset pupil distance between a first eye and a second eye; and
displaying, by the electronic apparatus, the first monocular panoramic image in a first display region corresponding to the first eye, and displaying the second monocular panoramic image in a second display region corresponding to the second eye.

2. The method according to claim 1 , further comprising:
performing, by the electronic apparatus, monocular calibration on the first panoramic camera, to determine a first internal parameter of the first panoramic camera and a first external parameter with respect to a real-world coordinate system;
performing, by the electronic apparatus, monocular calibration on the second panoramic camera, to determine a second internal parameter of the second panoramic camera and a second external parameter with respect to the real-world coordinate system;
performing, by the electronic apparatus, stereo calibration on the first panoramic camera and the second panoramic camera, wherein a relative position parameter between the first panoramic camera and the second panoramic camera is determined in the stereo calibration according to the first internal parameter, the first external parameter, the second internal parameter and the second external parameter; and
calculating, by the electronic apparatus, a rotation matrix and a translation matrix of the first panoramic camera and the second panoramic camera according to the parameters obtained through the monocular calibration and the stereo calibration.

3. The method according to claim 2 , wherein the obtaining a first panoramic image acquired by a first panoramic camera and a second panoramic image acquired by a second panoramic camera comprises:
capturing a first initial panoramic image by using the first panoramic camera, and capturing a second initial panoramic image by using the second panoramic camera at the same time;
calculating a first mapping table of the first initial panoramic image and a second mapping table of the second initial panoramic image according to the rotation matrix and the translation matrix, wherein the first mapping table stores a correspondence between coordinates of each pixel in the first initial panoramic image before and after transformation based on the rotation matrix and the translation matrix, and the second mapping table stores a correspondence between coordinates of each pixel in the second initial panoramic image before and after transformation based on the rotation matrix and the translation matrix; and
rectifying the first initial panoramic image according to the first mapping table, to obtain the first panoramic image, and rectifying the second initial panoramic image according to the second mapping table, to obtain the second panoramic image, wherein matching pixels in the first panoramic image obtained after the rectification and the second panoramic image obtained after the rectification are on the same horizontal axis.

4. The method according to claim 1 , wherein the obtaining at least one group of a first pixel and a second pixel that match each other comprises:
for any first pixel in the first panoramic image, selecting, in the second panoramic image, a pixel at a position corresponding to the first pixel, and determining pixels in a first predetermined region comprising the selected pixel as candidate pixels;
selecting, in the first panoramic image, a first region of a predetermined size by using the first pixel as a center;
for each candidate pixel in the second panoramic image, selecting a second region of a predetermined size by using the candidate pixel as a center;
for each second region, calculating pixel differences between pixels in the second region and pixels at corresponding positions in the first region, and a sum of absolute values of the pixel differences; and
determining a candidate pixel corresponding to a second region with a minimum sum of absolute values as the second pixel corresponding to the first pixel.

5. The method according to claim 4 , wherein after the determining a candidate pixel corresponding to a second region with a minimum sum of absolute values as the second pixel corresponding to the first pixel, the method further comprises:
for each determined group of the first pixel and the second pixel in the second panoramic image, determining, by the electronic apparatus, a pixel in the first panoramic image having a highest matching degree with the second pixel, and when the pixel in the first panoramic image having the highest matching degree with the second pixel is the first pixel, storing the first pixel and the second pixel as a group of first and second pixels that match each other.

6. The method according to claim 1 , wherein the obtaining at least one group of a first pixel and a second pixel that match each other comprises:
for any first pixel in the first panoramic image, selecting, in the second panoramic image, a pixel at a position corresponding to the first pixel, and determining pixels in a second predetermined region comprising the selected pixel as candidate pixels;
selecting, in the first panoramic image, a first line segment with a predetermined length in a predetermined direction by using the first pixel as a center;
for each candidate pixel in the second panoramic image, selecting a second line segment with the predetermined length in the predetermined direction by using the candidate pixel as a center;
for each second line segment, calculating pixel differences between pixels on the second line segment and pixels at corresponding positions on the first line segment, and a sum of absolute values of the pixel differences; and
determining a candidate pixel corresponding to a second line segment with a minimum sum of absolute values as the second pixel matching the first pixel.

Assignees

• Tencent Tech Shenzhen Co Ltd

Inventors

• Yao Li
• Wang Juhong
• Huang Tingting
• Zheng Xuan

Classifications

• H04N23/675

  comprising setting of focusing regions · CPC title

• H04N13/271 Primary

  wherein the generated image signals comprise depth maps or disparity maps · CPC title

• H04N23/698

  for achieving an enlarged field of view, e.g. panoramic image capture · CPC title

• G06F18/22

  Matching criteria, e.g. proximity measures · CPC title

• H04N13/239

  using two two-dimensional [2D] image sensors having a relative position equal to or related to the interocular distance (H04N13/243 takes precedence) · CPC title