Corrective lighting for video inpainting

Patent metadata

Field	Value
Publication number	US-2022292649-A1
Application number	US-202117196581-A
Country	US
Kind code	A1
Filing date	Mar 9, 2021
Priority date	Mar 9, 2021
Publication date	Sep 15, 2022
Grant date	—

Abstract

Certain aspects involve video inpainting in which content is propagated from a user-provided reference video frame to other video frames depicting a scene. One example method includes one or more processing devices that performs operations that include accessing a scene depicting a reference object that includes an annotation identifying a target region to be modified in one or more video frames. The operations also includes computing a target motion of a target pixel that is subject to a motion constraint. The motion constraint is based on a three-dimensional model of the reference object. Further, operations include determining color data of the target pixel to correspond to the target motion. The color data includes a color value and a gradient. Operations also include determining gradient constraints using gradient values of neighbor pixels. Additionally, the processing devices updates the color data of the target pixel subject to the gradient constraints.

First claim

1 . A method in which one or more processing devices performs operations comprising:
accessing a scene depicting a reference object that includes an annotation identifying a target region to be modified in one or more video frames; computing a target motion of a target pixel that is subject to a motion constraint, wherein the motion constraint is based on a reference frame; determining color data of the target pixel that corresponds to the target motion, wherein the color data comprises a color value; determining gradient constraints using gradient values of neighbor pixels; and updating the color data of the target pixel subject to the gradient constraints.

2 . The method of claim 1 , wherein the reference frame comprises a reference object, and further comprising:
determining the motion constraint based on a three-dimensional (“3D”) model that is defined by a set of sparse feature points.

3 . The method of claim 1 , wherein the target region comprises a boundary that is defined by boundary pixels, and further comprising:
determining the motion constraint based on (i) a boundary motion for the boundary of the target region in the reference frame and (ii) confidence values assigned to each of the boundary pixels, wherein the confidence values are based on a texture associated with a region that includes the respective boundary pixel.

4 . The method of claim 1 , further comprising determining the gradient constraints in four cardinal directions.

5 . The method of claim 1 , wherein determining the gradient constraints comprises:
computing a forward-flow-traced gradient constraint in four cardinal directions; computing a backward-flow-traced gradient constraint in the four cardinal directions; and computing a traced-color constraint that corresponds to the target pixel.

6 . The method of claim 5 , further comprising:
computing a gradient weighting function; adjusting each of the forward-flow-traced gradient constraints by applying the gradient weighting function; and adjusting each of the backward-flow-traced gradient constraints by applying the gradient weighting function.

7 . The method of claim 5 , further comprising:
computing a per-color-channel weighting function; adjusting each of the forward-flow-traced gradient constraints by applying the per-color-channel weighting function; and adjusting each of the backward-directional gradient constraints by applying the per-color-channel weighting function.

8 . The method of claim 5 , further comprising:
computing a screened weighting function; and adjusting the traced-color constraint by applying the screened weighting function.

9 . The method of claim 5 , further comprising:
computing a gradient weighting function, a per-color-channel weighting function, and a screened weighting function; adjusting each of the forward-flow-traced gradient constraints by applying both of the gradient weighting function and the per-color-channel weighting function; adjusting each of the backward-flow-traced gradient constraints by applying both of the gradient weighting function and the per-color-channel weighting function; and adjusting the traced-color constraint by applying the gradient weighting function.

10 . The method of claim 1 , wherein determining the gradient constraints comprises:
computing a forward-flow-traced gradient constraint using an expression:
ω grad ( f, 1,0)·ω channel ( f, 1,0)·∥( V ( f x , f y , f t )− V ( f x +1 , f y , f t )− T ( p x ,p y ,p t )− T ( p x +1, p y , p t ))∥ 2 =0,

wherein ω grad (f, 1, 0) represents a forward-flow-traced gradient weighting function, ω channel (f, 1, 0) represents a forward-flow-traced per-channel weighting function, V(f x , f y , f t ) represents a forward-flow-traced input video frame at time f t , location (f x , f y ), and wherein T(p x ,p y ,p t ) represents a target image that includes a target pixel p at time p t , location (p x , p y ).

11 . The method of claim 1 , wherein determining the gradient constraints comprises:
computing a backward-directional gradient constraint using an expression:
ω grad ( b, − 1, 0)·ω channel ( b, 1, 0)·∥( V ( b x , b y , b t )− V ( b x +1 , b y , b t )− T ( p x ,p y ,p t )− T ( p x +1, p y , p t ))∥ 2 =0,

wherein ω grad (b, −1, 0) represents a backward-directional gradient weighting function, ω channel (b, −1, 0) represents a backward-directional per-color-channel weighting function, V(b x , b y , b t ) represents a backward-flow-traced input video frame at time b t , location (b x , b y ), and wherein T(p x ,p y ,p t ) represents a target image that includes a target pixel p at time p t , location (p x , p y ).

12 . The method of claim 1 , wherein determining the gradient constraints comprises:
computing a traced-color constraint that corresponds to the target pixel using an expression:
ω screen ( f )·∥( T ( p x ,p y ,p t )−mix( V ( b x , b y , b t )), V ( f x , f y , f t )))∥ 2 =0,

wherein ω screen (f) represents a screened weighting function, T (p x , p y , p t ) represents a target image that includes a target pixel p at time p t , location (p x , p y ), and mix (V(b x , b y , b t ),V(f x , f y , f t )) represents a mixing function that is configured to combine color values for a backward-flow-traced input video frame V(b x , b y , b t ) at time b t , location (b x , b y ) and a forward-flow-traced input video frame V(f x , f y , f t ) at time f t , location (f x , f y ).

13 . The method of claim 1 , further comprising:
computing a gradient weighting function using an expression:

w
grad

⁡

(

r
,
x
,
y

)

=

exp
⁡

(

-



(

V
⁡

(

r
x

,

r
y

,

r
t

)

-

V
⁡

Assignees

• Adobe Inc

Inventors

• Wang Oliver
• Nelson John
• Oxholm Geoffrey
• Shechtman Elya

Classifications

• G06T7/269

  using gradient-based methods · CPC title

• G06T2207/10016

  Video; Image sequence · CPC title

• G06T2207/20081

  Training; Learning · CPC title

• G06T5/005 Primary

  Physics · mapped topic

• G06T5/77 Primary

  Retouching; Inpainting; Scratch removal · CPC title