Privacy image generation

Patent metadata

Field	Value
Publication number	US-10789685-B2
Application number	US-201816111398-A
Country	US
Kind code	B2
Filing date	Aug 24, 2018
Priority date	Dec 20, 2012
Publication date	Sep 29, 2020
Grant date	Sep 29, 2020

Abstract

A privacy image generation system may use a light field camera that includes an array of cameras or an RGBZ camera(s)) is used to capture images and display images according to a selected privacy mode. The privacy mode may include a blur background mode that can be automatically selected based on the meeting type, participants, location, and device type. A region of interest and/or an object(s) of interest (e.g. one or more persons in a foreground) is determined and the privacy image generation system is configured to clearly show the region/object of interest and obscure or replace the background by combining multiple images. The displayed image includes the region/object(s) of interest clearly shown (e.g. in focus) and any objects in a background of the combined image shown having a limited depth of field (e.g. blurry/not in focus) and/or blurred due to the combination of the multiple images.

First claim

What is claimed is:

1. A video image processing method for adding privacy effects to a display video image of a scene, the method comprising:
obtaining a plurality of input video images of the scene from a corresponding plurality of video cameras that are spaced apart;
determining a region of interest, including an image of an object, in each of the plurality of input video images to define a foreground and a background, the foreground including the image of the object;
rectifying the plurality of input video images based on the image of the object in each input video image to produce a plurality of rectified video images having aligned foregrounds and having backgrounds that are not aligned due to relative spatial relationships of the plurality of video cameras corresponding to the plurality of input video images;
creating the display video image by combining the plurality of rectified video images, the foreground of the display video image being in-focus and the background of the display video image appearing to be blurred due to the backgrounds of the rectified video images not being aligned; and
transmitting the display video image for display.

2. The method of claim 1 , further comprising receiving an indication of an amount of blur to apply to the display video image, wherein the obtaining of the plurality of input video images includes selecting a number of input video images as the plurality of input video images in response to the indication of the amount of blur.

3. The method of claim 1 , further comprising receiving an indication of an amount of blur to apply to the display video image, wherein the obtaining of the plurality of input video images includes selecting the plurality of input video images based on the relative spatial relationships of the cameras corresponding to the plurality of input video images in response to the indication of the amount of blur.

4. The method of claim 1 , further comprising:
providing one video image of the plurality of input video images for display;
wherein determining the region of interest includes receiving an indication of a portion of the one video image as an indication of the image of the object in the region of interest.

5. The method of claim 1 , wherein:
the obtaining of the plurality of input video images includes receiving the plurality of input video images from a client device via a network; and
the transmitting of the display video image for display includes transmitting the display video image to the client device via the network.

6. The method of claim 1 , further comprising:
receiving data representing a plurality of audio signals corresponding to a voice of a presenter;
wherein:
the determining of the region of interest includes determining the region of interest using sound source localization based on the plurality of audio signals; and
the image of the object includes an image of the presenter.

7. The method of claim 1 , wherein determining the region of interest includes processing at least two of the plurality of input video images to estimate a depth of the region of interest, wherein the background includes objects in the plurality of input video images having depths greater that the estimated depth of the region of interest.

8. An image processing system for adding privacy effects to a display video image, the system comprising:
a processor; and
a memory storing computer executable instructions configured to cause the processor to:
obtain a plurality of input video images of a scene from a corresponding plurality of video cameras that are spaced apart;
determine a region of interest, including an image of an object, in each of the plurality of input video images to define a foreground and a background, the foreground including the image of the object;
rectify the plurality of input video images based on the image of the object in each input video image to produce a plurality of rectified video images having aligned foregrounds and having backgrounds that are not aligned due to relative spatial relationships of the plurality of video cameras corresponding to the plurality of input video images;
create the display video image by combining the plurality of rectified video images, the foreground of the display video image being in-focus and the background of the display video image appearing to be blurred due to backgrounds of the rectified video images not being aligned; and
transmit the display video image for display.

9. The image processing system of claim 8 , wherein:
the computer executable instructions are further configured to cause the processor to receive an indication of an amount of blur to apply to the display video image; and
the computer executable instructions that cause the processor to obtain the plurality of input video images include instructions that cause the processor to select a number of input video images as the plurality of input video images in response to the indication of the amount of blur.

10. The image processing system of claim 8 , wherein:
the computer executable instructions are further configured to cause the processor to receive an indication of an amount of blur to apply to the display video image; and
the computer executable instructions that cause the processor to obtain the plurality of input video images include instructions that cause the processor to select the plurality of input video images based on the relative spatial relationships of the video cameras corresponding to the selected plurality of input video images in response to the indication of the amount of blur.

11. The image processing system of claim 8 , further comprising computer executable instructions that cause the processor to:
provide one video image of the plurality of input video images for display;
wherein the computer executable instructions that cause the processor to determine the region of interest include instructions that cause the processor to receive an indication of a portion of the one video image as an indication the image of the object in the region of interest.

12. The image processing system of claim 8 , wherein:
the computer executable instructions that cause the processor to obtain the plurality of input video images include instructions that cause the processor to receive the plurality of input video images from a client device via a network; and
the computer executable instructions that cause the processor to transmit the display video image for display include instructions that cause the processor to transmit the display video image to the client device via the network.

13. The image processing system of claim 8 , further comprising:
computer executable instructions that cause the processor to receive data representing a plurality of audio signals corresponding to a voice of a presenter;
wherein:
the computer executable instructions that cause the processor to determine the region of interest include instructions that cause the processor to determine the region of interest using sound source localization based on the plurality of audio signals; and
the image of the object includes an image of the presenter.

14. The image processing system of claim 8 , wherein:
the computer executable instructions that cause the processor to determine the region of interest include instructions that cause the processor to estimate a depth of the region of interest based on at least two input video images of the plurality of input video images; and
the background includes objects in the plurality of input video images having depths greater that the estimated depth of the region of interest.

Assignees

• Microsoft Technology Licensing Llc

Inventors

• Cutler Ross
• Mehran Ramin

Classifications

• H04N23/675

  comprising setting of focusing regions · CPC title

• H04N23/61

  based on recognised objects · CPC title

• H04N23/62

  Control of parameters via user interfaces · CPC title

• H04N23/63

  by using electronic viewfinders · CPC title

• H04N23/957

  Light-field or plenoptic cameras or camera modules · CPC title