Method of real-time image processing based on rendering engine and a display apparatus

Patent metadata

Field	Value
Publication number	US-11263805-B2
Application number	US-201816483724-A
Country	US
Kind code	B2
Filing date	Nov 21, 2018
Priority date	Nov 21, 2018
Publication date	Mar 1, 2022
Grant date	Mar 1, 2022

Abstract

The present application discloses method for real-time processing image data based on a rendering engine. The method includes sampling a first set of data in a first pixel arrangement to be displayed as an image in a display screen with pixels in a second pixel arrangement. The method further includes creating a first model mapped with the first set of data to generate a second set of data and a second model mapped with the first set of data to generate a third set of data. Additionally, the method further includes rendering the second set of data using a first shader to output a fourth set of data and rendering the third set of data using a second shader to output a fifth set of data. Furthermore, the method includes superimposing the fourth set of data over the fifth set of data to display a sixth set of data.

First claim

What is claimed is:

1. A method for real-time processing image data based on a rendering engine comprising:
sampling a first set of data in a first pixel arrangement to be displayed as an image in a display screen with pixels in a second pixel arrangement;
mapping the first set of data to a first model for a rendering engine to generate a second set of data;
mapping the first set of data to a second model for the rendering engine to generate a third set of data;
loading the second set of data and the third set of data to the rendering engine;
processing the second set of data by the rendering engine using a first shader associated with the first pixel arrangement to output a fourth set of data;
processing the third set of data by the rendering engine using a second shader associated with the second pixel arrangement to output a fifth set of data;
superimposing the fourth set of data over the fifth set of data to obtain a sixth set of data; and
displaying the sixth set of data in the display screen with pixels in the second pixel arrangement.

2. The method of claim 1 , further comprising creating the first model and the second model for the rendering engine based on the display screen;
wherein creating the first model comprises generating a first number of rectangular stripe bodies each having a first width of a pixel and a first length of one row of second number of pixels and being separated from neighboring rectangular stripe body by one row of pixels, wherein the first number and the second number respectively represent half of a vertical display resolution and half of a lateral display resolution associated with the display screen; wherein the second model is characterized by a rectangular body corresponding to the display screen with a second width equal to the first length of one row of pixels and a second length equal to a sum of first widths of two times of the first number of rows of pixels.

3. The method of claim 1 , wherein sampling the first set of data comprises collecting the first set of data in real time using a sampling camera associated with the rendering engine.

4. The method of claim 3 , wherein the sampling camera comprises an orthogonal projection camera or a perspective projection camera.

5. The method of claim 2 , wherein mapping the first set of data to the first model to generate the second set of data comprises generating a first virtual image with a first edge aligned with a first one of the first number of rectangular stripe bodies of the first model by transforming the first set of data to the second set of data in terms of UV mapping; mapping the first set of data to the second model to generate the third set of data comprises generating a second virtual image attached to a single rectangular body of the second model by transforming the first set of data to the third set of data in terms of UV mapping.

6. The method of claim 5 , wherein loading the second set of data and the third set of data to the rendering engine comprises generating a first Filmbox (FBX) file bearing at least information of the second set of data associated with the first model and a second FBX file bearing at least information of the third set of data associated with the second model respectively loaded into the rendering engine.

7. The method of claim 6 , further comprising creating the first shader associated with the first pixel arrangement;
wherein creating the first shader comprises defining one or more computer-executable treatments on image data in the first pixel arrangement in which each row of pixels is arranged in real RGB subpixel order.

8. The method of claim 7 , wherein processing the second set of data comprises rendering the first virtual image by the rendering engine; and adjusting colors, textures, and mesh information on the second set of data associated with the first model by the first shader to obtain the fourth set of data in the first pixel arrangement.

9. The method of claim 6 , further comprising creating the second shader associated with the second pixel arrangement;
wherein creating the second shader comprises defining one or more computer-executable treatments on image data in the second pixel arrangement in which a number of each row of pixels is half of that in the first pixel arrangement, each odd numbered row of pixels is kept at real RGB subpixel order as in the first pixel arrangement, and each even numbered row of pixels is changed to BRG subpixel order and each subpixel of one color in the even numbered row is shifted to a position between two nearest subpixels of two different colors in the odd numbered row.

10. The method of claim 9 , wherein processing the third set of data comprises rendering the second virtual image by the rendering engine; and adjusting colors, textures, and mesh information on the third set of data associated with the second model by the second shader to obtain the fifth set of data in the second pixel arrangement.

11. The method of claim 1 , wherein superimposing the fourth set of data over the fifth set of data to obtain a sixth set of data comprises combining all fourth set of data in odd numbered rows of pixels and all fifth set of data in even numbered rows of pixels to effectively obtain the sixth set of data to generate a virtual scene in the rendering engine; and sampling the sixth set of data in the second pixel arrangement out of the virtual scene in real time using a sampling camera associated with the rendering engine.

12. The method of claim 1 , wherein displaying the sixth set of data in the display screen comprises sending the sixth set of data in the second pixel arrangement with half lateral display resolution of the display screen and displaying an image in the display screen with full display resolution.

13. A real-time image processing apparatus, comprising:
a memory; and
one or more processors;
wherein the memory and the one or more processors are connected with each other; and
the memory stores computer-executable instructions for controlling the one or more processors to:
sample a first set of data in a first pixel arrangement to be displayed as an image in a display screen with pixels in a second pixel arrangement;
map the first set of data to a first model for a rendering engine to generate a second set of data;
map the first set of data to a second model for the rendering engine to generate a third set of data;
load the second set of data and the third set of data to the rendering engine;
process the second set of data by the rendering engine using a first shader associated with the first pixel arrangement to output a fourth set of data;
process the third set of data by the rendering engine using a second shader associated with the second pixel arrangement to output a fifth set of data; and
superimpose the fourth set of data over the fifth set of data to obtain a sixth set of data.

14. The real-time image processing apparatus of claim 13 , wherein the one or more processors comprise a rendering engine, a first shader, and a second shader, wherein the rendering engine comprises at least a sampling camera for sampling the first set of data, and is configured to receive a first model via a first Filmbox (FBX) file, wherein the first model is created by an external model builder and the first FBX file includes information of a first virtual image attached to the first model based on the second set of data transformed from the first set of data through UV mapping; and to receive a second model via a second FBX file, wherein the second FBX file includes a second virtual image attached to the second model based on the third set of data tra

Assignees

• Beijing Boe Optoelectronics Tech Co Ltd
• Boe Technology Group Co Ltd

Inventors

• Wang Xuefeng
• Li Wenyu
• Miao Jinghua
• Zhao Bin
• Sun Yukun
• Wang Lixin
• Li Xi
• Lu Yuanjie
• Suo Jianwen
• Peng Jinbao
• Fan Qingwen
• Li Zhifu
• Zhang Hao
• Chen Lili

Classifications

• G09G2360/06

  Use of more than one graphics processor to process data before displaying to one or more screens · CPC title

• G06T11/40

  Filling planar surfaces by adding surface attributes, e.g. adding colours or textures · CPC title

• G09G2300/0452

  Details of colour pixel setup, e.g. pixel composed of a red, a blue and two green components · CPC title

• G06T2219/2012

  Colour editing, changing, or manipulating; Use of colour codes · CPC title

• G06T19/20

  Editing of three-dimensional [3D] images, e.g. changing shapes or colours, aligning objects or positioning parts · CPC title