Performing segmentation of objects in media items based on user input

Patent metadata

Field	Value
Publication number	US-12437358-B2
Application number	US-202217968645-A
Country	US
Kind code	B2
Filing date	Oct 18, 2022
Priority date	Oct 18, 2021
Publication date	Oct 7, 2025
Grant date	Oct 7, 2025

Abstract

A media application receives user input that indicates one or more objects to be erased from a media item. The media application translates the user input to a bounding box. The media application provides a crop of the media item based on the bounding box to a segmentation machine-learning model. The segmentation machine-learning model outputs a segmentation mask for one or more segmented objects in the crop of the media item and a corresponding segmentation score that indicates a quality of the segmentation mask.

First claim

What is claimed is:

1. A computer-implemented method comprising:
receiving user input that indicates one or more objects to be erased from a media item;
translating the user input to a bounding box;
providing a crop of the media item based on the bounding box to a segmentation machine-learning model;
outputting, with the segmentation machine-learning model, a segmentation mask for one or more segmented objects in the crop of the media item and a corresponding segmentation score that indicates a quality of the segmentation mask;
determining that the segmentation mask is invalid based on one or more of: the corresponding segmentation score failing to meet a threshold score, a number of valid mask pixels falling below a threshold number of pixels, a segmentation mask size falling below a threshold size, or the segmentation mask being greater than a threshold distance from a region indicated by the user input; and
responsive to determining that the segmentation mask is invalid, generating a different mask based on a region within the user input.

2. The method of claim 1 , wherein the bounding box is an axis-aligned bounding box or an oriented bounding box.

3. The method of claim 1 , wherein the user input includes one or more strokes made with reference to the media item.

4. The method of claim 3 , wherein the bounding box is an oriented bounding box and wherein an orientation of the oriented bounding box matches an orientation of at least one of the one or more strokes.

5. The method of claim 1 , wherein prior to the providing the crop of the media item, the segmentation machine-learning model is trained using training data that includes a plurality of training images and groundtruth segmentation masks.

6. The method of claim 1 , further comprising:
providing a user interface that receives the user input, wherein the user input is selected from a group of a circle that surrounds the one or more objects, one or more lines on top of the one or more objects, a square that surrounds the one or more objects, and combinations thereof.

7. The method of claim 1 , further comprising inpainting a portion of the media item that matches the different mask to obtain an output media item, wherein the one or more objects are absent from the output media item.

8. The method of claim 7 , wherein the inpainting is performed using an inpainting machine-learning model, and wherein the media item and the different mask are provided as input to the inpainting machine-learning model.

9. The method of claim 7 , further comprising providing a user interface that includes the output media item.

10. A non-transitory computer-readable medium with instructions stored thereon that, when executed by one or more computers, cause the one or more computers to perform operations comprising:
receiving user input that indicates one or more objects to be erased from a media item;
translating the user input to a bounding box;
providing a crop of the media item based on the bounding box to a segmentation machine-learning model;
outputting, with the segmentation machine-learning model, a segmentation mask for one or more segmented objects in the crop of the media item and a corresponding segmentation score that indicates a quality of the segmentation mask;
determining that the segmentation mask is invalid based on one or more of: the corresponding segmentation score failing to meet a threshold score, a number of valid mask pixels falling below a threshold number of pixels, a segmentation mask size falling below a threshold size, or the segmentation mask being greater than a threshold distance from a region indicated by the user input; and
responsive to determining that the segmentation mask is invalid, generating a different mask based on a region within the user input.

11. The computer-readable medium of claim 10 , wherein the bounding box is an axis-aligned bounding box or an oriented bounding box.

12. The computer-readable medium of claim 10 , wherein the user input includes one or more strokes made with reference to the media item.

13. The computer-readable medium of claim 12 , wherein the bounding box is an oriented bounding box and wherein an orientation of the oriented bounding box matches an orientation of at least one of the one or more strokes.

14. The computer-readable medium of claim 10 , wherein the segmentation machine-learning model is trained prior to the providing using training data that includes a plurality of training images and groundtruth segmentation masks.

15. A computing device comprising:
a processor; and
a memory coupled to the processor, with instructions stored thereon that, when executed by the processor, cause the processor to perform operations comprising:
receiving user input that indicates one or more objects to be erased from a media item;
translating the user input to a bounding box;
providing a crop of the media item based on the bounding box to a segmentation machine-learning model;
outputting, with the segmentation machine-learning model, a segmentation mask for one or more segmented objects in the crop of the media item and a corresponding segmentation score that indicates a quality of the segmentation mask;
determining that the segmentation mask is invalid based on one or more of: the corresponding segmentation score failing to meet a threshold score, a number of valid mask pixels falling below a threshold number of pixels, a segmentation mask size falling below a threshold size, or the segmentation mask being greater than a threshold distance from a region indicated by the user input; and
responsive to determining that the segmentation mask is invalid, generating a different mask based on a region within the user input.

16. The computing device of claim 15 , wherein the bounding box is an axis-aligned bounding box or an oriented bounding box.

17. The computing device of claim 15 , wherein the user input includes one or more strokes made with reference to the media item.

18. The computing device of claim 17 , wherein the bounding box is an oriented bounding box and wherein an orientation of the oriented bounding box matches an orientation of at least one of the one or more strokes.

19. The computing device of claim 15 , wherein prior to the providing a crop of the media item, the segmentation machine-learning model is trained using training data that includes a plurality of training images and groundtruth segmentation masks.

20. The computing device of claim 15 , wherein the operations further comprise:
providing a user interface that receives the user input, wherein the user input is selected from a group of a circle that surrounds the one or more objects, one or more lines on top of the one or more objects, a square that surrounds the one or more objects, and combinations thereof.

Assignees

• Google Llc

Inventors

• Liba Orly
• Sarma Navin
• Knaan Yael Pritch
• Schiffhauer Alexander
• Cai Longqi
• Jacobs David
• Chen Huizhong
• Li Siyang
• Feldman Bryan

Classifications

• G06T5/77 Primary

  Retouching; Inpainting; Scratch removal · CPC title

• G06T2210/22

  Cropping · CPC title

• G06T2210/12

  Bounding box · CPC title

• G06T2207/20081

  Training; Learning · CPC title

• G06T2200/24

  involving graphical user interfaces [GUIs] · CPC title