Method and apparatus for obtaining binocular panoramic image, and storage medium

Patent metadata

Field	Value
Publication number	US-11632537-B2
Application number	US-202017100737-A
Country	US
Kind code	B2
Filing date	Nov 20, 2020
Priority date	Jul 29, 2016
Publication date	Apr 18, 2023
Grant date	Apr 18, 2023

Abstract

A method for obtaining a binocular panoramic image is performed at an electronic apparatus, including: obtaining first and second panoramic images acquired by two panoramic cameras; obtaining at least one group of a first pixel located in the first panoramic image and a second pixel located in the second panoramic image; calculating a distance between the first pixel and the second pixel in each group, and obtaining depth information corresponding to the two panoramic image according to the distance between the first pixel and the second pixel; converting the first panoramic image and the second panoramic image into a first monocular panoramic image and a second monocular panoramic image, respectively in accordance with the corresponding depth information and a preset pupil distance between a first eye and a second eye; and combining a display of the first monocular panoramic image and the second monocular panoramic image in corresponding display regions.

First claim

What is claimed is:

1. A method for obtaining a binocular panoramic image performed at an electronic apparatus having one or more processors and memory storing instructions to be executed by the one or more processors, comprising:
obtaining a first panoramic image captured by a first panoramic camera and a second panoramic image captured by a second panoramic camera;
identifying at least one group of matching pixels including a first pixel in the first panoramic image and a second pixel in the second panoramic image, wherein pixel values of the first pixel and the second pixel match each other according to a preset condition;
determining a distance between the first pixel and second pixel in the at least one group;
obtaining depth information of the first panoramic image and the second panoramic image according to the distance between the first pixel and the second pixel;
selecting one of the first panoramic image and the second panoramic image as a first monocular panoramic image;
determining a second monocular panoramic image in accordance with (i) the first monocular panoramic image, (ii) the depth information of the first panoramic image and the second panoramic image, and (iii) a preset pupil distance between a first eye and a second eye; and
combining a display of the first monocular panoramic image in a first display region corresponding to the first eye with a display of the second monocular panoramic image in a second display region corresponding to the second eye.

2. The method according to claim 1 , further comprising:
performing monocular calibration on the first panoramic camera, to determine a first internal parameter of the first panoramic camera and a first external parameter with respect to a real-world coordinate system;
performing monocular calibration on the second panoramic camera, to determine a second internal parameter of the second panoramic camera and a second external parameter with respect to the real-world coordinate system;
performing stereo calibration on the first panoramic camera and the second panoramic camera, wherein a relative position parameter between the first panoramic camera and the second panoramic camera is determined in the stereo calibration according to the first internal parameter, the first external parameter, the second internal parameter and the second external parameter; and
calculating a rotation matrix and a translation matrix of the first panoramic camera and the second panoramic camera according to the parameters obtained through the monocular calibration and the stereo calibration.

3. The method according to claim 1 , wherein the identifying at least one group of matching pixels including a first pixel in the first panoramic image and a second pixel in the second panoramic image comprises:
selecting, within the second panoramic image, a pixel at a position corresponding to the first pixel;
choosing pixels within a first predetermined region covering the selected pixel as candidate pixels;
selecting, within the first panoramic image, a first region of a predetermined size centering at the first pixel;
for each candidate pixel in the second panoramic image, selecting a second region of a predetermined size centering at the candidate pixel;
calculating a sum of absolute pixel value differences between pixels in the second region and pixels at corresponding positions in the first region; and
determining a candidate pixel associated with a minimum sum of absolute pixel value differences as the second pixel corresponding to the first pixel.

4. The method according to claim 3 , wherein after the determining a candidate pixel corresponding to a second region with a minimum sum of absolute values as the second pixel corresponding to the first pixel, the method further comprises:
for each determined group of the first pixel and the second pixel in the second panoramic image, determining, by the electronic apparatus, a pixel in the first panoramic image having a highest matching degree with the second pixel, and when the pixel in the first panoramic image having the highest matching degree with the second pixel is the first pixel, storing the first pixel and the second pixel as a group of first and second pixels that match each other.

5. The method according to claim 1 , wherein the identifying at least one group of matching pixels including a first pixel in the first panoramic image and a second pixel in the second panoramic image comprises:
selecting, within the second panoramic image, a pixel at a position corresponding to the first pixel;
choosing pixels within a second predetermined region covering the selected pixel as candidate pixels;
selecting, within the first panoramic image, a first line segment with a predetermined length in a predetermined direction centering at the first pixel;
for each candidate pixel in the second panoramic image, selecting a second line segment with the predetermined length in the predetermined direction centering at the candidate pixel;
calculating a sum of absolute pixel value differences between pixels on the second line segment and pixels at corresponding positions on the first line segment; and
determining a candidate pixel associated with a minimum sum of absolute pixel value differences as the second pixel matching the first pixel.

6. The method according to claim 1 , wherein the determining a distance between the first pixel and second pixel in the at least one group comprises:
generating a disparity map according to the distance between the first and second pixels in each group;
denoising the disparity map by using a filter, to obtain a denoised disparity map; and
calculating depth information of each group of matching pixels according to the denoised disparity map.

7. The method according to claim 1 , wherein the combining a display of the first monocular panoramic image in a first display region corresponding to the first eye with a display of the second monocular panoramic image in a second display region corresponding to the second eye comprises:
performing hole filling processing on the second monocular panoramic image for filling a hole that is generated due to image rotation or shielding during mapping; and
combining the display of the first monocular panoramic image in the first display region corresponding to the first eye with the display of the second monocular panoramic image in the second display region corresponding to the second eye after the hole filling processing in the display region corresponding to the second eye.

8. An electronic apparatus having one or more processors and memory coupled to the one or more processors and a plurality of instructions stored in the memory that, when executed by the one or more processors, cause the electronic apparatus to perform a plurality of operations including:
obtaining a first panoramic image captured by a first panoramic camera and a second panoramic image captured by a second panoramic camera;
identifying at least one group of matching pixels including a first pixel in the first panoramic image and a second pixel in the second panoramic image, wherein pixel values of the first pixel and the second pixel match each other according to a preset condition;
determining a distance between the first pixel and second pixel in the at least one group;
obtaining depth information of the first panoramic image and depth information of the second panoramic image according to the distance between the first pixel and the second pixel;
selecting one of the first panoramic image and the second panoramic image as a first monocular panoramic image; and
determining a second monocular panoramic image in accordance with (i) the first monocular panoramic image, (ii) the depth information of

Assignees

• Tencent Tech Shenzhen Co Ltd

Inventors

• Yao Li
• Wang Juhong
• Huang Tingting
• Zheng Xuan

Classifications

• H04N2013/0081

  Depth or disparity estimation from stereoscopic image signals · CPC title

• G06T7/80

  Analysis of captured images to determine intrinsic or extrinsic camera parameters, i.e. camera calibration · CPC title

• H04N23/675

  comprising setting of focusing regions · CPC title

• G06T2207/10012

  Stereo images · CPC title

• H04N13/271 Primary

  wherein the generated image signals comprise depth maps or disparity maps · CPC title