Display method, system and computer-readable recording medium thereof

Patent metadata

Field	Value
Publication number	US-10805543-B2
Application number	US-201715414625-A
Country	US
Kind code	B2
Filing date	Jan 25, 2017
Priority date	Oct 26, 2016
Publication date	Oct 13, 2020
Grant date	Oct 13, 2020

Abstract

A display method, a system and a computer-readable recording medium thereof are provided, where the method includes the following steps. An image of a user in front of a screen is captured by using an image capturing device to generate a user image. A foreground region and a background region in the user image are distinguished by using a depth sensor. A virtual window is set on a display area of the screen according to its display size. Human face detection is performed on the foreground region to obtain plane information of the human face, and depth information of the human face is obtained by using the depth sensor. A background image is generated based on the background region according to the virtual window, the plane information and the depth information of the human face. The foreground region is combined with the background image to generate an output image.

First claim

What is claimed is:

1. A display method, adapted to a display system having an image capturing device, a depth sensor, and a screen, comprising the following steps:
capturing an image of a user in front of a screen by using the image capturing device to generate a user image;
distinguishing a foreground region from a background region in the user image by using the depth sensor;
setting a virtual window on a display area of the screen according to a display size thereof, wherein the virtual window is smaller than the display area of the screen, wherein there exists a scaling ratio between the virtual window and the display area of the screen, wherein the scaling ratio is associated with a predefined viewing position of the user with respect to the screen having a viewing effect as if the user looking at a mirror with a same display size as the screen, wherein a length of the virtual window is a multiplication of the scaling ratio and a length of the screen, wherein a width of the virtual window is a multiplication of the scaling ratio and a width of the screen, wherein the scaling ratio is less than 1, and wherein the predefined viewing position of the user is fixed with respect to a size of the screen;
performing human face detection on the foreground region to obtain plane information of a human face position of the user, and obtaining depth information of the human face position by using the depth sensor;
selecting a background minor image region from the background region according to a size of the virtual window, the plane information and the depth information of the human face position to generate a background image; and
combining the foreground region with the background image to generate an output image.

2. The display method according to claim 1 , wherein the image capturing device has a wide-angle lens, and wherein the step of capturing the image of the user in front of the screen by using the image capturing device to generate the user image comprises:
performing lens distortion rectification on the image of the user to generate the user image.

3. The display method according to claim 1 , wherein the step of distinguishing the foreground region from the background region in the user image by using the depth sensor comprises:
obtaining depth information corresponding to each pixel in the user image by using the depth sensor; and
comparing the depth information corresponding to each of the pixels and a predefined depth threshold so as to accordingly determine whether each of the pixels belongs to the background region or the foreground region.

4. The display method according to claim 1 , wherein the step of setting the virtual window on the display area of the screen according to the display size thereof comprises:
obtaining an optimum viewing position of the user with respect to the screen;
setting an imaging position of a background with respect to the screen;
obtaining the scaling ratio according to the optimum viewing position and the imaging position; and
obtaining the size of the virtual window according to the display size and the scaling ratio.

5. The display method according to claim 1 , wherein the step of selecting the background mirror image region from the background region according to the size of the virtual window as well as the plane information and the depth information of the human face position comprises:
calculating a range projected from the human face position, passing through two horizontal endpoints of the virtual window, to a mirror image of the background so as to accordingly select the background mirror image region from the background region.

6. The display method according to claim 1 further comprising:
replacing the background region by a virtual background.

7. The display method according to claim 1 , wherein the step of combining the foreground region with the background image to generate the output image comprises:
combining the foreground region with the background image to generate a combined image; and
adjusting the combined image to fit the display area of the screen to generate the output image.

8. The display method according to claim 1 further comprising:
display other information on the screen according to a position of the foreground region in the output image.

9. The display method according to claim 1 further comprising:
receiving one's image from another display system; and
generating another one's image by using the one's image according to the virtual window, the plane information and the depth information of the human face position.

10. A display system comprising:
an image capturing device, configured to capture images;
a depth sensor, configured to detect depth information;
a screen, configured to display images, wherein the image capturing device and the depth sensor are disposed at a same side as the screen;
a memory, configured to store data; and
a processor, coupled to the image capturing device, the depth sensor, the screen, and the memory, and configured to execute the following steps:
capturing an image of a user in front of a screen by using the image capturing device to generate a user image;
distinguishing a foreground region from a background region in the user image by using the depth sensor;
setting a virtual window on a display area of the screen according to a display size thereof, wherein the virtual window is smaller than the display area of the screen, wherein there exists a scaling ratio between the virtual window and the display area of the screen, wherein the scaling ratio is associated with a predefined viewing position of the user with respect to the screen having a viewing effect as if the user looking at a mirror with a same display size as the screen, wherein a length of the virtual window is a multiplication of the scaling ratio and a length of the screen, wherein a width of the virtual window is a multiplication of the scaling ratio and a width of the screen, wherein the scaling ratio is less than 1, and wherein the predefined viewing position of the user is fixed with respect to a size of the screen;
performing human face detection on the foreground region to obtain plane information of a human face position of the user, and obtaining depth information of the human face position by using the depth sensor;
selecting a background mirror image region from the background region according to a size of the virtual window, the plane information and the depth information of the human face position to generate a background image; and
combining the foreground region with the background image to generate an output image and displaying the output image on the screen.

11. The display system according to claim 10 , wherein the image capturing device has a wide-angle lens, and wherein the processor performs lens distortion rectification on the image of the user to generate the user image.

12. The display system according to claim 10 , wherein the processor obtains depth information corresponding to each pixel in the user image by using the depth sensor, and compares the depth information corresponding to each of the pixels and a predefined depth threshold so as to accordingly determine whether each of the pixels belongs to the background region or the foreground region.

13. The display system according to claim 10 , wherein the processor obtains an optimum viewing position of the user with respect to the screen, sets an imaging position of a background with respect to the screen, obtains the scaling ratio according to the optimum viewing position and the imaging position, and obtains the size of the virtual window according to the display si

Assignees

• Wistron Corp

Inventors

• Kao Meng-Chao
• Lin Hui-Chen
• Yang Jie-Ci

Classifications

• H04N23/81

  for suppressing or minimising disturbance in the image signal generation · CPC title

• H04N23/698

  for achieving an enlarged field of view, e.g. panoramic image capture · CPC title

• H04N23/63 Primary

  by using electronic viewfinders · CPC title

• H04N25/61

  the noise originating only from the lens unit, e.g. flare, shading, vignetting or "cos4" · CPC title

• H04N23/611

  where the recognised objects include parts of the human body · CPC title