Mitigating people distractors in images

Patent metadata

Field	Value
Publication number	US-11776237-B2
Application number	US-202016997364-A
Country	US
Kind code	B2
Filing date	Aug 19, 2020
Priority date	Aug 19, 2020
Publication date	Oct 3, 2023
Grant date	Oct 3, 2023

Abstract

Systems, methods, and software are described herein for removing people distractors from images. A distractor mitigation solution implemented in one or more computing devices detects people in an image and identifies salient regions in the image. The solution then determines a saliency cue for each person and classifies each person as wanted or as an unwanted distractor based at least on the saliency cue. An unwanted person is then removed from the image or otherwise reduced from the perspective of being an unwanted distraction.

First claim

What is claimed is:

1. A computer system comprising one or more processors and memory configured to provide computer program instructions to the one or more processors, the computer program instructions comprising:
a salience module configured to direct the one or more processors to:
generate a saliency map representing one or more salient regions of an image by applying a saliency filter at multiple scales, each pixel of the saliency map storing a saliency value that represents a measure of saliency of the pixel; and
determine, based on the saliency value for each pixel in the saliency map corresponding to a detected person in the image, a salience cue representing an extent to which the detected person overlaps with the one or more salient regions; and

a distractor mitigation module configured to direct the one or more processors to:
classify the detected person as an unwanted distractor based on one or more cues comprising the salience cue for the detected person; and
based on classifying the detected person as the unwanted distractor, trigger a recommendation to remove the detected person from one or more versions of the image.

2. The computer system of claim 1 , wherein the one or more cues comprise a recognition cue generated using facial recognition and indicative of whether the detected person is recognized as a reoccurring person based on a frequency of appearance in a set of images associated with an account.

3. The computer system of claim 1 , wherein the one or more cues comprise a distractor cue indicative of a probability that the detected person is a distractor.

4. The computer system of claim 1 , wherein the distractor mitigation module is configured to direct the one or more processors to:
determine that a face has not been detected within a second detected person in the image; and
determine not to remove the second detected person from the image based on an estimated pose of the second detected person identifying the second detected person as a subject of the image, or based on detecting an attribute of the second detected person and recognizing the attribute as a known attribute of a known person in a set of images associated with an account.

5. The computer system of claim 1 , wherein the distractor mitigation module is configured to direct the one or more processors to trigger a user interface to request input confirming or rejecting the recommendation to remove the detected person from the one or more versions of the image.

6. The computer system of claim 1 , wherein the distractor mitigation module is configured to direct the one or more processors to remove recurring instances of the detected person from other images based on input confirming the recommendation to remove the detected person from the one or more versions of the image.

7. The computer system of claim 1 , wherein the distractor mitigation module is configured to direct the one or more processors to remove the detected person from the one or more versions of the image based at least on replacing pixels associated with the detected person with new pixels.

8. A method comprising:
generating a mask of a detected person in an image;
generating a saliency map of an image by applying a saliency filter at multiple scales, each pixel of the saliency map storing a saliency value that represents a measure of saliency of the pixel;
determining, based on the saliency value for each pixel in the saliency map that falls within the mask of the detected person, a salience cue representing an extent to which the detected person overlaps with the one or more salient regions;
classifying the detected person as an unwanted distractor based on one or more cues comprising the salience cue for the detected person; and
based on classifying the detected person as the unwanted distractor, triggering a recommendation to remove the detected person from one or more versions of the image.

9. The method of claim 8 , wherein the one or more cues comprise a recognition cue generated using facial recognition and indicative of whether the detected person is recognized as a reoccurring person based on a frequency of appearance in a set of images associated with an account.

10. The method of claim 8 , wherein the one or more cues comprise a distractor cue indicative of a probability that the detected person is a distractor.

11. The method of claim 8 , further comprising:
determining that a face has not been detected within a second detected person in the image; and
determining not to remove the second detected person from the image based on an estimated pose of the second detected person identifying the second detected person as a subject of the image, or based on detecting an attribute of the second detected person and recognizing the attribute as a known attribute of a known person in a set of images associated with an account.

12. The method of claim 8 , further comprising triggering a user interface to request input confirming or rejecting the recommendation to remove the detected person from the one or more versions of the image.

13. The method of claim 8 , further comprising removing recurring instances of the detected person from other images based on input confirming the recommendation to remove the detected person from the one or more versions of the image.

14. The method of claim 8 , further comprising removing the detected person from the one or more versions of the image based on replacing pixels associated with the detected person with new pixels.

15. One or more computer storage media storing computer-useable instructions that, when used by one or more computing devices, cause the one or more computing devices to perform operations comprising:
generating a mask of a detected person in an image;
generating a saliency map representing one or more salient regions of an image by applying a saliency filter at multiple scales, each pixel of the saliency map storing a saliency value that represents a measure of saliency of the pixel;
determining, based on the saliency value for each pixel in the saliency map corresponding to the mask of the detected person, a salience cue representing an extent to which the detected person overlaps with the one or more salient regions;
classifying the detected person unwanted based on one or more cues comprising the salience cue for the detected person; and
determining, based on classifying the detected person as unwanted, whether to trigger a recommendation to remove the detected person from one or more versions of the image.

16. The one or more computer storage media of claim 15 , wherein the one or more cues comprise a recognition cue generated using facial recognition and indicative of whether the detected person is recognized as a reoccurring person based on a frequency of appearance in a set of images associated with an account.

17. The one or more computer storage media of claim 15 , wherein the one or more cues comprise a distractor cue indicative of a probability that the detected person is a distractor.

18. The one or more computer storage media of claim 15 , the operations further comprising:
determining that a face has not been detected within a second detected person in the image; and
determining not to remove the second detected person from the image based on an estimated pose of the second detected person identifying the second detected person as a subject of the image, or based on detecting an attribute of the second detected person and recognizing the attribute as a known attribute of a known person in a set of images associated w

Assignees

• Adobe Inc

Inventors

• Cohen Scott David
• Ding Zhihong
• Lin Zhe
• Ling Mingyang
• Figueroa Luis Angel

Classifications

• G06V10/464 Primary

  using a plurality of salient features, e.g. bag-of-words [BoW] representations · CPC title

• G06F18/241

  relating to the classification model, e.g. parametric or non-parametric approaches · CPC title

• G06T5/005

  Physics · mapped topic

• G06V40/10

  Human or animal bodies, e.g. vehicle occupants or pedestrians; Body parts, e.g. hands · CPC title

• G06V40/161

  Detection; Localisation; Normalisation · CPC title