Image enhancement system

Patent metadata

Field	Value
Publication number	US-11880983-B2
Application number	US-202117230875-A
Country	US
Kind code	B2
Filing date	Apr 14, 2021
Priority date	Apr 14, 2021
Publication date	Jan 23, 2024
Grant date	Jan 23, 2024

Abstract

Embodiments of the present disclosure generally relate to livestreaming methods and systems, and more particularly to whiteboard presentation systems that can be used in a livestreaming or video conferencing environment. In some embodiments, the whiteboard presentation system is configured to perform one or more processing operations, such as capture images on a whiteboard, perform image processing routines on the captured images, and transmit processed images as a video feed to one or more remote users, such as people or locations attending a video conference. The image processing routines can include one or more operations such as image denoising, contrast enhancement, color reconstruction, segmentation of a presenter, and image reconstruction.

First claim

The invention claimed is:

1. A method of processing images, comprising:
capturing, by a camera device, a first image of a surface, wherein the first image comprises presentation information that is disposed on at least a portion of the surface, and a portion of a presenter that is disposed between the camera device and the surface;
performing a presenter segmentation process, comprising:
detecting the portion of the presenter in a second image that is derived from the first image; and
generating a presenter mask that is used to define pixels in the first image that are associated with the detected portion of the presenter;

performing a canvas reconstruction process, comprising:
replacing the pixels in the first image that are associated with the detected portion of the presenter within the presenter mask to form a reconstructed canvas image;

performing a presentation information segmentation process, comprising:
detecting presentation information in the reconstructed canvas image; and
generating a presentation information mask that includes pixels in the reconstructed canvas image that contain portions of the presentation information;

performing a background reconstruction process, comprising:
applying the generated presentation information mask to the reconstructed canvas image to determine the pixels that contain the portions of the presentation information; and
reconstructing the pixels in the reconstructed canvas image that contain the portion of the presentation information to form a canvas background image; and

performing a background bleaching process, comprising:
removing imperfections in the reconstructed canvas image by dividing the reconstructed canvas image by the canvas background image.

2. The method of claim 1 , wherein the replacing the pixels in the first image that are associated with the detected portion of the presenter further comprises:
replacing the pixels with corresponding pixels from a canvas image captured before the first image was captured.

3. The method of claim 1 , wherein reconstructing the pixels in the reconstructed canvas image that contain the portion of the presentation information further comprises:
in-painting the pixels in the reconstructed canvas image that contain the portion of the presentation information in the canvas background image.

4. The method of claim 1 , further comprising:
correcting perspective distortion of the first image, prior to the presenter segmentation process.

5. The method of claim 1 , wherein the second image is a lower resolution version of the first image.

6. The method of claim 1 , further comprising:
performing an image stabilization process subsequent to the background bleaching process, the image stabilization process comprising:
compensating for errors in the presentation information segmentation process, the background reconstruction process, and the background bleaching process.

7. The method of claim 6 , wherein
the presenter segmentation process is performed at a first resolution level,
the canvas reconstruction process, the background bleaching process, and the image stabilization process are performed at a second resolution level that is the same as a resolution of the first image, and
the presentation information segmentation process, the background reconstruction process are performed at a third resolution level, the third resolution level being higher than the first resolution level and lower than second resolution level.

8. The method of claim 1 , further comprising performing a start-presentation generation process.

9. A method of delivering presentation information disposed on a surface to an electronic device, comprising:
performing, by a controller, a calibration process, comprising:
capturing, by a camera device, a first image of a surface;
determining corners of the surface in the first image; and
determining size and shape of a canvas image having determined corners; and

performing, by the controller, a canvas generation process, comprising:
capturing, by the camera device, a second image of the surface, wherein the second image comprises presentation information that is disposed on at least a portion of the surface, and a portion of a presenter that is disposed between the camera device and the surface;
performing a presenter segmentation process, comprising:
detecting the portion of the presenter in a third image that is derived from the second image; and
generating a presenter mask that is used to define pixels in the second image that are associated with the detected portion of the presenter;

performing a canvas reconstruction process, comprising:
replacing the pixels in the second image that are associated with the detected portion of the presenter within the presenter mask to form a reconstructed canvas image;

performing a presentation information segmentation process, comprising:
detecting presentation information in the reconstructed canvas image; and
generating a presentation information mask that includes pixels in the reconstructed canvas image that contain portions of the presentation information;

performing a background reconstruction process, comprising:
applying the generated presentation information mask to the reconstructed canvas image to determine the pixels that contain the portions of the presentation information; and
reconstructing the pixels in the reconstructed canvas image that contain the portion of the presentation information to form a canvas background image; and

performing a background bleaching process, comprising:
removing imperfections in the reconstructed canvas image by dividing the reconstructed canvas image by the canvas background image.

10. The method of claim 9 , wherein the calibration process further comprises:
performing a predefined perspective correction process, comprising:
correcting perspective distortion of the first image, the first image having a visible frame;

performing a first corner detection process, comprising:
detecting a location of a first corner of the surface in the first image;

performing a flipping process, comprising:
rotating the first image;

performing a second corner detection process, comprising:
detecting a location of a second corner of the surface in the first image;

performing the flipping process, comprising:
rotate the first image;

performing a third corner detection process, comprising:
detecting a location of a third corner of the surface in the first image;

performing the flipping process, comprising:
rotating the first image of the surface;

performing a fourth corner detection process, to detect a location of a fourth corner of the surface in the first image;
performing a false detection filtering process, comprising:
rejecting any of the detected locations of the first, second, third, and fourth corners of the surface outside of the visible frame;

performing a corner post process, comprising:
determining size and shape of an initial canvas image;

performing a fine turning process, to accurately determine a location of a center of the corners; and
performing a corner transformation process, comprising:
transform the determined locations of the corners to coordinates of the initial canvas image.

11. The method of claim 9 , wherein the replacing the pixels in the first image that are associated with the detected portion of the presenter further comprises:
replacing the pixels with corresponding pixels from a canvas image captured before the first image was captured.

12. The method of claim 9 , wherein reconstructed the pixels in the reconstructed canvas image that contain the portion of t

Assignees

• Logitech Europe Sa

Inventors

• Ostap Oleg
• Levak Henry
• Maestre Trueba Miguel Angel
• Sakharchuk Viktor
• Tereshchenko Yaroslav Vasilovich
• Okhrimenko Ihor

Classifications

• G06T7/194 Primary

  involving foreground-background segmentation · CPC title

• G06T7/11

  Region-based segmentation · CPC title

• G06T7/174

  involving the use of two or more images · CPC title

• H04L65/4046 Primary

  with distributed floor control · CPC title

• H04N7/15

  Conference systems · CPC title