Method for generating and displaying panorama images based on rendering engine and a display apparatus

Patent metadata

Field	Value
Publication number	US-11589026-B2
Application number	US-201816626486-A
Country	US
Kind code	B2
Filing date	Nov 21, 2018
Priority date	Nov 21, 2018
Publication date	Feb 21, 2023
Grant date	Feb 21, 2023

Abstract

The present application discloses method for generating a panorama image based on a rendering engine associated with a display apparatus. The method includes determining a display model configured to be a polygon prism for a rendering engine corresponding to a panorama view of a scene. Additionally, the method includes using multiple sampling cameras associated with the rendering engine to capture multiple sample sub-images of the panorama view in respective directions towards multiple sub-planes of the polygon prism. The method further includes attaching the multiple sample sub-images to respective multiple sub-planes to form a constructed display model. Furthermore, the method includes rendering the constructed display model by the rendering engine to reconstruct a panorama image and displaying the panorama image on the display apparatus.

First claim

What is claimed is:

1. A method for generating a panorama image based on a rendering engine associated with a display apparatus comprising:
determining a display model configured to be a polygon prism for the rendering engine corresponding to a panorama view of a scene;
using multiple sampling cameras associated with the rendering engine to capture multiple sample sub-images of the panorama view in respective directions towards multiple sub-planes of the polygon prism;
attaching the multiple sample sub-images to respective multiple sub-planes to form a constructed display model;
rendering the constructed display model by the rendering engine to reconstruct a panorama image; and
displaying the panorama image on the display apparatus;
wherein the polygon prism is an equilateral N-side polygon prism; and
determining the display model comprises determining different image resolutions corresponding to respective multiple sub-planes of different planes of the equilateral N-side polygon prism and determining a width and a length as well as a width-to-length ratio of each of the multiple sub-planes based on a ratio of tan(u/2)/tan(v/2), where u is a horizontal field-of-view angle and v is a vertical field-of-view angle projected from at least one of the multiple sampling cameras to each of the multiple sub-planes.

2. The method of claim 1 , wherein N is an integer no smaller than 3.

3. The method of claim 2 , wherein determining the display model further comprises determining the multiple sub-planes divided respectively from each side or top or bottom plane of the equilateral N-side polygon prism.

4. The method of claim 1 , wherein the at least one of the multiple sampling cameras is located at a center of the display model.

5. The method of claim 1 , wherein using multiple sampling cameras associated with the rendering engine to capture multiple sample sub-images comprises separately sampling each of the multiple sample sub-images with an independently-defined image resolution depending on a scene in respective one direction toward one of multiple sub-planes projected from one of the multiple sampling cameras.

6. The method of claim 5 , wherein the independently-defined image resolution for a single sub-plane is configured to be several times greater than a maximum image resolution allowed for a single field-of-view image in a conventional model.

7. The method of claim 1 , wherein using multiple sampling cameras associated with the rendering engine to capture multiple sample sub-images comprises capturing at least two sample sub-images in projected squares respectively for top view toward a top plane and bottom view toward a bottom plane by slightly enlarging field-of-view angles to make the projected squares to be larger than the top plane or the bottom plane.

8. The method of claim 1 , wherein using multiple sampling cameras associated with the rendering engine to capture multiple sample sub-images comprises using two sets of sampling cameras to capture two sets of sample sub-images with parallax for the multiple sub-planes; and reconstructing a 3D panorama image based on the two sets of sample sub-images with parallax.

9. The method of claim 1 , wherein attaching the multiple sample sub-images to respective multiple sub-planes comprises performing UV mapping to add image textures of each sample sub-image to a corresponding sub-plane and generating the constructed display model for the rendering engine.

10. The method of claim 1 , wherein using multiple sampling cameras associated with the rendering engine to capture multiple sample sub-images further comprises sampling a series of sample sub-images time-sequentially with a sampling frequency for each of the multiple sub-planes of the display model; encoding all sample sub-images of a same sub-plane according to an order of being sampled to generate a sample sub-video; and attaching the sample sub-video to a corresponding sub-plane to form the constructed display model.

11. The method of claim 10 , wherein rendering the constructed display model comprises rendering multiple sample sub-images respectively for the multiple sub-planes that are sampled at a same time to generate one panorama image at the same time and further generating a panorama video by encoding a series of panorama images sequentially in time.

12. A method for generating a panorama image based on a rendering engine associated with a display apparatus comprising:
determining a display model configured to be a polygon prism for the rendering engine corresponding to a panorama view of a scene;
using multiple sampling cameras associated with the rendering engine to capture multiple sample sub-images of the panorama view in respective directions towards multiple sub-planes of the polygon prism;
attaching the multiple sample sub-images to respective multiple sub-planes to form a constructed display model;
rendering the constructed display model by the rendering engine to reconstruct a panorama image; and
displaying the panorama image on the display apparatus;
wherein rendering the constructed display model comprises rendering multiple sample sub-images respectively for the multiple sub-planes that are sampled at a same time to generate one panorama image at the same time and further generating a panorama video by encoding a series of panorama images sequentially in time; and
displaying the panorama image comprises displaying the panorama video on the display apparatus by at least displaying a sub-video separately for respective one sub-plane in a direction of a field-of-view.

13. An apparatus for generating a panorama image based on a rendering engine associated with a display apparatus, comprising:
a memory; and
one or more processors;
wherein the memory and the one or more processors are connected with each other; and
the memory stores computer-executable instructions for controlling the one or more processors to:
determine a display model configured to be a polygon prism for the rendering engine corresponding to a panorama view of a scene;
use multiple sampling cameras associated with the rendering engine to capture multiple sample sub-images of the panorama view in respective directions towards multiple sub-planes of the polygon prism;
attach the multiple sample sub-images to respective multiple sub-planes to form a constructed display model;
render the constructed display model by the rendering engine to reconstruct a panorama image; and
display the panorama image on the display apparatus;
wherein the polygon prism is an equilateral N-side polygon prism; and
the memory further stores computer-executable instructions for controlling the one or more processors to determine different image resolutions corresponding to respective multiple sub-planes of different planes of the equilateral N-side polygon prism and determining a width and a length as well as a width-to-length ratio of each of the multiple sub-planes based on a ratio of tan(u/2)/tan(v/2), where u is a horizontal field-of-view angle and v is a vertical field-of-view angle projected from at least one of the multiple sampling cameras to each of the multiple sub-planes.

14. The apparatus of claim 13 , wherein the one or more processors is controlled by a programmable instruction to create a display model configured as an equilateral polygon prism with each of side, top, and bottom planes being divided into one or more sub-planes, each sub-plane being independently defined with an image resolution.

15. The apparatus of claim 14 , wherein the one or more processors comprises a rendering engine having multiple sampling cameras config

Assignees

• Beijing Boe Optoelectronics Tech Co Ltd
• Boe Technology Group Co Ltd
• Boe Tech Groupco Ltd

Inventors

• Wang Xuefeng
• Sun Yukun
• Miao Jinghua
• Zhang Hao
• Chen Lili
• Li Xi
• Zhao Bin
• Suo Jianwen
• Peng Jinbao
• Li Wenyu
• Fan Qingwen
• Li Zhifu
• Lu Yuanjie

Classifications

• H04N13/117 Primary

  the virtual viewpoint locations being selected by the viewers or determined by viewer tracking · CPC title

• H04N13/243

  using three or more two-dimensional [2D] image sensors · CPC title

• H04N23/698 Primary

  for achieving an enlarged field of view, e.g. panoramic image capture · CPC title

• G06T15/04 Primary

  Texture mapping · CPC title

• H04N13/332

  Displays for viewing with the aid of special glasses or head-mounted displays [HMD] · CPC title