Removing objects at image capture time

Patent metadata

Field	Value
Publication number	US-2023353701-A1
Application number	US-202217660946-A
Country	US
Kind code	A1
Filing date	Apr 27, 2022
Priority date	Apr 27, 2022
Publication date	Nov 2, 2023
Grant date	—

Abstract

The present disclosure relates to systems, non-transitory computer-readable media, and methods for removing objects from an image stream at capture time of a digital image. For example, the disclosed system contemporaneously detects and segments objects from a digital image stream being previewed in a camera viewfinder graphical user interface of a client device. The disclosed system removes selected objects from the image stream and fills a hole left by the removed object with a content aware fill. Moreover, the disclosed system displays the image stream with the removed object and content fill as the image stream is previewed by a user prior to capturing a digital image from the image stream.

First claim

1 . A computer-implemented method comprising:
displaying, in a graphical user interface at a client device, an image stream being captured by the client device; detecting one or more objects in the image stream; selecting an object of the one or more objects in the image stream; removing the object from the image stream; and capturing a digital image from the image stream with the object removed.

2 . The computer-implemented method of claim 1 , wherein detecting one or more objects further comprises utilizing an object detection machine learning model to detect the one or more objects in frames of the image stream and assign an object label to each of the one or more objects.

3 . The computer-implemented method of claim 2 , further comprising generating, utilizing a segmentation machine learning model, an object mask for the object.

4 . The computer-implemented method of claim 3 , further comprising:
tracking a location of the object in the frames of the image stream; removing the object from each of the frames of the image stream based on the tracking of the object; and displaying the image stream, in the graphical user interface, with the object removed.

5 . The computer-implemented method of claim 2 , wherein selecting the object of the one or more objects in the image stream comprises:
determining a theme of the image stream based on object labels of the one or more objects in the image stream; and selecting the object as unwanted based on the determined theme of the image stream and an object label of the object.

6 . The computer-implemented method of claim 1 , wherein selecting the object further comprises:
providing a selectable element in connection with display of the object in the image stream; and receiving a selection of the selectable element.

7 . The computer-implemented method of claim 1 , further comprising:
generating by a content aware fill machine learning model, content to fill a hole created by removal of the object; and filling, the hole in the image stream with the content.

8 . The computer-implemented method of claim 7 , further comprising:
receiving, prior to capturing the digital image, an indication from the client device that the content used to fill the hole is insufficient; receiving, prior to capturing the digital image, an identification of an area of the image stream to inform the content aware fill machine learning model; generating, by the content aware fill machine learning model, updated content to fill the hole based on the area of the image stream; and filling the hole in the image stream with the updated content.

9 . The computer-implemented method of claim 8 , wherein receiving the indication from the client device further comprises:
providing a selectable icon in the graphical user interface; and receiving a selection of the selectable icon.

10 . The computer-implemented method of claim 9 , further comprising:
providing and displaying a movable element in response to the selection of the selectable icon; and receiving the identification of the area of the image stream by detecting a location of the movable element.

11 . The computer-implemented method of claim 10 , wherein receiving the identification of the area of the image stream by detecting the location of the movable element comprises determining that the movable element is located on a portion of an updated image stream being captured in response to panning the client device.

12 . A non-transitory computer-readable medium comprising instructions that, when executed by at least one processor, cause the at least one processor to perform operations comprising:
displaying in a graphical user interface, at a client device, an image stream being captured by the client device; detecting one or more objects in the image stream; receiving a selection of an object of the one or more objects in the image stream; removing the object in response to the selection; generating content and filling a hole corresponding to the removed object in the image stream with the content; and displaying the image stream with the content in place of the removed object in the graphical user interface.

13 . The non-transitory computer-readable medium of claim 12 , wherein the operations further comprise capturing a digital image from the image stream with the object removed.

14 . The non-transitory computer-readable medium of claim 12 , wherein the operations further comprise:
removing a second object of the one or more objects in response to a second selection; retaining a list of removed objects including the object and the second object removed; as the image stream changes determining updated locations of objects in the list of removed objects; and removing the objects in the list of the removed objects from the updated locations from the image stream.

15 . The non-transitory computer-readable medium of claim 12 , wherein the operations further comprise:
providing a selectable icon for client device assisted content generation; receiving a selection of the selectable icon; and in response to receiving the selection, providing a movable icon to indicate a selected content to use to fill the hole.

16 . The non-transitory computer-readable medium of claim 15 , wherein generating the content comprises:
determining a location of the movable icon; and utilizing a content aware fill neural network to generate updated content based on the selected content within the movable icon.

17 . The non-transitory computer-readable medium of claim 12 , wherein:
detecting comprises detecting the one or more objects in a sequence of frames of the image stream utilizing an object detection neural network; receiving the selection comprises determining an object speed threshold; and removing comprises removing a moving object from the image stream when the moving object exceeds the object speed threshold.

18 . A system comprising:
at least one memory device; and at least one processor configured to cause the system to: display in a graphical user interface, at a client device, an image stream being captured by the client device; detect, utilizing an object detection artificial intelligence model, one or more objects in the image stream; segment, utilizing a segmentation artificial intelligence model, the one or more objects in the image stream; remove an object of the one or more objects from the image stream in response to receiving a selection of the object; generate content to fill a hole in the image stream corresponding to the removed object utilizing a content aware fill machine learning model; fill the hole in the image stream corresponding to the removed object with the content; and display the image stream with the content in place of the removed object in the graphical user interface of the client device.

19 . The system of claim 18 , further comprising instructions that, when executed by the at least one processor configured to cause the system to capture a digital image from the image stream with the content in place of the removed object.

20 . The system of claim 19 , further comprising instructions that, when executed by the at least one processor configured to cause the system to display the image stream, prior to capturing of the digital image, with the object removed and generated content in place of the object as the image stream changes over time or in response to movement of the client device capturing the image stream.

Assignees

• Adobe Inc

Inventors

• Shukla Sankalp
• Gupta Angad Kumar
• Gupta Sourabh

Classifications

• H04N5/272 Primary

  Means for inserting a foreground image in a background image, i.e. inlay, outlay · CPC title

• H04N5/232935

  Electricity · mapped topic

• H04N5/23218

  Electricity · mapped topic

• H04N5/2621 Primary

  Cameras specially adapted for the electronic generation of special effects during image pickup, e.g. digital cameras, camcorders, video cameras having integrated special effects capability · CPC title

• H04N23/61

  based on recognised objects · CPC title