Segmenting and removing objects from visual media items

Patent metadata

Field	Value
Publication number	US-12217472-B2
Application number	US-202217968634-A
Country	US
Kind code	B2
Filing date	Oct 18, 2022
Priority date	Oct 18, 2021
Publication date	Feb 4, 2025
Grant date	Feb 4, 2025

Abstract

A media application generates training data that includes a first set of visual media items and a second set of visual media items, where the first set of visual media items correspond to the second set of visual items and include distracting objects that are manually segmented. The media application trains a segmentation machine-learning model based on the training data to receive a visual media item with one or more distracting objects and to output a segmentation mask for one or more segmented objects that correspond to the one or more distracting objects.

First claim

What is claimed is:

1. A computer-implemented method comprising:
generating training data that includes a first set of media items and a second set of media items, wherein the first set of media items include distracting objects and the second set of media items include manual segmentations of the distracting objects;
identifying one or more original media items in the first set of media items that include one or more broken powerlines;
generating one or more corrected media items that correct the one or more broken powerlines;
generating one or more augmented media items for the training data by blending portions of the one or more corrected media items with portions of respective one or more original media items to increase a randomness of augmentation; and
training a segmentation machine-learning model based on the training data to receive a media item with one or more distracting objects and to output a segmentation mask for one or more segmented objects that correspond to the one or more distracting objects.

2. The method of claim 1 , wherein generating the one or more augmented media items includes blending the one or more corrected media items and the respective one or more original media items with a checkerboard mask.

3. The method of claim 1 , wherein generating the one or more corrected media items to correct the one or more broken powerlines includes:
modifying a local contrast in the one or more original media items to generate corresponding one or more enhanced media items.

4. The method of claim 3 , wherein the local contrast is modified using a gain curve that adds two bias curves together.

5. The method of claim 1 , wherein generating training data includes augmenting one or more of the first set of media items by applying a dilation to a segmentation mask of the one or more distracting objects.

6. The method of claim 1 , wherein the one or more distracting objects are organized into categories, the categories including at least one selected from a group of powerlines, power poles, towers, and combinations thereof.

7. The method of claim 1 , wherein training the segmentation machine-learning model comprises:
generating a first machine-learning model based on the training data; and
distilling the first machine-learning model to a trained segmentation machine-learning model by running inference on the training data that is segmented by the first machine-learning model.

8. The method of claim 1 , wherein the training data further includes synthesized images with the distracting objects added in front of outdoor environment objects.

9. A computer-implemented method to remove a distracting object from a media item, the method comprising:
receiving a media item from a user;
identifying one or more distracting objects in the media item;
providing the media item to a trained segmentation machine-learning model;
outputting, with the trained segmentation machine-learning model, a segmentation mask for the one or more distracting objects in the media item; and
inpainting a portion of the media item that matches the segmentation mask to obtain an output media item, wherein the one or more distracting objects are absent from the output media item;
wherein the trained segmentation machine-learning model is trained by generating training data by:
identifying one or more original media items in a first set of media items that include one or more broken powerlines;
generating one or more corrected media items that correct the one or more broken powerlines; and
generating one or more augmented media items for the training data by blending portions of the one or more corrected media items with portions of respective one or more original media items to increase a randomness of augmentation.

10. The method of claim 9 , wherein the one or more distracting objects are organized into categories, the categories including at least one selected from a group of powerlines, power poles, towers, and combinations thereof.

11. The method of claim 9 , further comprising providing a suggestion to a user to remove the one or more distracting objects from the media item.

12. The method of claim 9 , wherein the trained segmentation machine-learning model is trained using training data that includes the first set of media items and a second set of media items, wherein the first set of media items include distracting objects and the second set of media items include manual segmentations of the distracting objects.

13. A non-transitory computer-readable medium with instructions stored thereon that, when executed by one or more computers, cause the one or more computers to perform operations, the operations comprising:
generating training data that includes a first set of media items and a second set of media items, wherein the first set of media items include distracting objects and the second set of media items include manual segmentations of the distracting objects;
identifying one or more original media items in the first set of media items that include one or more broken powerlines;
generating one or more corrected media items that correct the one or more broken powerlines;
generating one or more augmented media items for the training data by blending portions of the one or more corrected media items with portions of respective one or more original media items to increase a randomness of augmentation; and
training a segmentation machine-learning model based on the training data to receive a media item with one or more distracting objects and to output a segmentation mask for one or more segmented objects that correspond to the one or more distracting objects.

14. The computer-readable medium of claim 13 , wherein generating the one or more augmented media items includes blending the one or more corrected media items and the respective one or more original media items with a checkerboard mask.

15. The computer-readable medium of claim 13 , wherein generating the one or more corrected media items to correct the one or more broken powerlines includes:
modifying a local contrast in the one or more original media items to generate corresponding one or more enhanced media items.

16. The computer-readable medium of claim 15 , wherein the local contrast is modified using a gain curve that adds two bias curves together.

17. The computer-readable medium of claim 13 , wherein generating training data includes augmenting one or more of the first set of media items by applying a dilation to a segmentation mask of the one or more distracting objects.

18. The computer-readable medium of claim 13 , wherein the one or more distracting objects are organized into categories, the categories including at least one selected from a group of powerlines, power poles, towers, and combinations thereof.

19. The computer-readable medium of claim 13 , wherein training the segmentation machine-learning model comprises:
generating a first machine-learning model based on the training data; and
distilling the first machine-learning model to a trained segmentation machine-learning model by running inference on the training data that is segmented by the first machine-learning model.

20. The computer-readable medium of claim 13 , wherein the training data further includes synthesized images with the distracting objects added in front of outdoor environment objects.

Assignees

• Google Llc

Inventors

• Liba Orly
• Karnad Nikhil
• Kanazawa Nori
• Knaan Yael Pritch
• Chen Huizhong
• Cai Longqi

Classifications

• G06T5/94

  based on local image properties, e.g. for local contrast enhancement · CPC title

• G06T5/77 Primary

  Retouching; Inpainting; Scratch removal · CPC title

• G06T2200/24

  involving graphical user interfaces [GUIs] · CPC title

• G06T11/00

  Two-dimensional [2D] image generation · CPC title

• G06T5/20

  using local operators · CPC title