Video inpainting via user-provided reference frame

Patent metadata

Field	Value
Publication number	US-10872637-B2
Application number	US-201916585433-A
Country	US
Kind code	B2
Filing date	Sep 27, 2019
Priority date	Oct 12, 2018
Publication date	Dec 22, 2020
Grant date	Dec 22, 2020

Abstract

Certain aspects involve video inpainting in which content is propagated from a user-provided reference frame to other video frames depicting a scene. For example, a computing system accesses a set of video frames with annotations identifying a target region to be modified. The computing system determines a motion of the target region's boundary across the set of video frames, and also interpolates pixel motion within the target region across the set of video frames. The computing system also inserts, responsive to user input, a reference frame into the set of video frames. The reference frame can include reference color data from a user-specified modification to the target region. The computing system can use the reference color data and the interpolated motion to update color data in the target region across set of video frames.

First claim

The invention claimed is:

1. A method in which one or more processing devices performs operations comprising:
accessing a set of video frames comprising a first frame and a second frame having respective annotations identifying a target region to be modified, the target region including a first target pixel at a first location in the first frame and a second target pixel at a second location in the second frame;
computing, by a video editing tool, a boundary motion for a boundary of the target region within the set of video frames, wherein the boundary includes boundary pixels neighboring the target region in the set of video frames;
interpolating, by the video editing tool and from the boundary motion, a target motion of target pixels within the target region across the set of video frames;
computing, by the video editing tool, confidence values for the set of video frames;
identifying, by the video editing tool, a frame among the set of video frames with a confidence value less than a threshold confidence; and
outputting, by the video editing tool, a suggestion to use the identified frame as a reference frame;
modifying, responsive to user input, the identified frame to generate the reference frame;
inserting, by the video editing tool, the reference frame into the set of video frames, the reference frame having a user-specified modification to the target region; and
updating, by the video editing tool, color data of the target region in the set of video frames to correspond to the target motion interpolated from the boundary motion, wherein updating the color data comprises:
identifying reference color data of a pixel at a reference location in the reference frame,
updating first color data of the first target pixel with the reference color data, and
updating second color data of the second target pixel with the updated first color data.

2. The method of claim 1 , wherein interpolating the target motion comprises estimating a change in position of a target object between the first frame and the second frame, wherein the change in position corresponds to a portion of the boundary motion between the first frame and the second frame, wherein the first target pixel and the second target pixel depict the same portion of the target object in the first frame and the second frame.

3. The method of claim 2 , further comprising computing the second location in the second frame by modifying the first location according to the change in position of the target object between the first frame and the second frame.

4. The method of claim 1 , wherein the set of video frames comprises a subset of frames that (a) lacks the first frame and the second frame and (b) includes an additional frame that is positioned, in a sequence of the set of video frames, before the first frame and the second frame and that has an additional target pixel at an additional location,
wherein updating the color data of the target region in the set of video frames further comprises updating additional color data of the additional target pixel with prior color data from a prior frame in the subset of frames that is positioned before the additional frame in the sequence of the set of video frames,
wherein the first color data of the first target pixel is updated with the reference color data rather than the updated additional color data.

5. The method of claim 1 , wherein modifying the identified frame to generate the reference frame comprises:
presenting, in an editing interface of the video editing tool, a preview pane that plays the set of video frames;
displaying, via the preview pane, the identified frame;
receiving editing inputs identifying the user-specified modification;
modifying the identified frame by applying the user-specified modification to the target region in the identified frame; and
selecting the modified identified frame as the reference frame.

6. The method of claim 1 , further comprising:
presenting, in an editing interface of the video editing tool, an upload tool;
receiving, by the upload tool, input identifying a location of an image file;
retrieving the image file from the location; and
selecting image content from the image file as an additional reference frame.

7. The method of claim 1 , wherein the user-specified modification to the target region comprises a removal, from the target region, of a first object displayed in the target region in the first frame and the second frame and a replacement of the first object with a second object.

8. A non-transitory computer-readable medium having program code of a video editing tool stored thereon, wherein the program code, when executed by one or more processing devices, configures the one or more processing devices to perform operations comprising:
accessing a set of video frames comprising a first frame and a second frame having respective annotations identifying a target region to be modified, the target region including a first target pixel at a first location in the first frame and a second target pixel at a second location in the second frame;
interpolating, by the video editing tool and from a motion of a boundary of the target region within the set of video frames, a target motion of target pixels within the target region across the set of video frames;
computing, by the video editing tool, confidence values for the set of video frames;
identifying, by the video editing tool, a frame among the set of video frames with a confidence value less than a threshold confidence; and
outputting, by the video editing tool, a suggestion to use the identified frame as a reference frame;
generating the reference frame by at least applying, to the identified frame, a user-specified modification to the target region; and
updating, based on reference color data from the user-specified modification, color data of the target region in the set of video frames to correspond to the target motion as interpolated.

9. The non-transitory computer-readable medium of claim 8 , wherein interpolating the target motion comprises estimating a change in position of a target object between the first frame and the second frame, wherein the change in position corresponds to a portion of the motion of the boundary between the first frame and the second frame, wherein the first target pixel and the second target pixel depict the same portion of the target object in the first frame and the second frame.

10. The non-transitory computer-readable medium of claim 9 , the operations further comprising computing the second location in the second frame by modifying the first location according to the change in position of the target object between the first frame and the second frame.

11. The non-transitory computer-readable medium of claim 8 ,
wherein updating the color data comprises:
identifying the reference color data from a pixel at a reference location in the reference frame,
updating first color data of the first target pixel with the reference color data, and
updating second color data of the second target pixel with the updated first color data,

wherein the set of video frames comprises a subset of frames that (a) lacks the first frame and the second frame and (b) includes an additional frame that is positioned, in a sequence of the set of video frames, before the first frame and the second frame and that has an additional target pixel at an additional location,
wherein updating the color data of the target region in the set of video frames further comprises updating additional color data of the additional target pixel with prior color data from a prior frame in the subset of frames that is positioned before the additional frame in the sequence of the

Assignees

• Adobe Inc

Inventors

• Oxholm Geoffrey
• Walker Seth
• Sheikh Ramiz
• Wang Oliver
• Nelson John

Classifications

• G06T11/10

  Texturing; Colouring; Generation of textures or colours (retouching, inpainting or scratch removal G06T5/77) · CPC title

• H04N21/44012

  involving rendering scenes according to scene graphs, e.g. MPEG-4 scene graphs · CPC title

• H04N21/4728

  for selecting a Region Of Interest [ROI], e.g. for requesting a higher resolution version of a selected region · CPC title

• G06T7/269

  using gradient-based methods · CPC title

• H04N21/44008 Primary

  involving operations for analysing video streams, e.g. detecting features or characteristics in the video stream (arrangements characterised by components specially adapted for monitoring, identification or recognition of video in broadcast systems H04H60/59) · CPC title