Many-to-many splatting-based digital image synthesis

Patent metadata

Field	Value
Publication number	US-12169909-B2
Application number	US-202217714356-A
Country	US
Kind code	B2
Filing date	Apr 6, 2022
Priority date	Apr 6, 2022
Publication date	Dec 17, 2024
Grant date	Dec 17, 2024

Abstract

Digital synthesis techniques are described to synthesize a digital image at a target time between a first digital image and a second digital image. To begin, an optical flow generation module is employed to generate optical flows. The digital images and optical flows are then received as an input by a motion refinement system. The motion refinement system is configured to generate data describing many-to-many relationships mapped for pixels in the plurality of digital images and reliability scores of the many-to-many relationships. The reliability scores are then used to resolve overlaps of pixels that are mapped to a same location by a synthesis module to generate a synthesized digital image.

First claim

What is claimed is:

1. A method comprising:
receiving, by a computing device, a plurality of digital images and a plurality of optical flows describing pixel movement between the plurality of digital images, respectively;
generating, by the computing device, data describing:
many-to-many relationships mapped for pixels in the plurality of digital images based on the plurality of optical flows; and
reliability scores of the many-to-many relationships, respectively; and

synthesizing, by the computing device, a synthesized digital image by forward warping the pixels of at least one said digital image based on the many-to-many relationships and fusing the forward warped pixels based on the reliability scores.

2. The method as described in claim 1 , wherein the many-to-many relationships includes multiple flow vectors for a single said pixel that specify a plurality of locations, to which, the single said pixel is mapped.

3. The method as described in claim 1 , wherein the many-to-many relationships are described using a plurality of refined bidirectional flows and the reliability scores are described using a plurality of color reliability maps.

4. The method as described in claim 3 , further comprising generating the plurality of refined bidirectional flows, the generating including:
encoding motion feature representations from the plurality of digital images and the plurality of optical flows;
modulating the motion feature representations using a low-rank constraint; and
forming the plurality of refined bidirectional flows and the reliability scores for the pixels of the plurality of digital images based on the modulated motion feature representations.

5. The method as described in claim 4 , wherein the encoding the motion feature representations includes generating motion feature pyramids having levels corresponding to a plurality of resolutions.

6. The method as described in claim 5 , wherein the encoding the motion feature representations includes joint flow encoding of the motion feature pyramids using the plurality of digital images and the plurality of optical flows.

7. The method as described in claim 4 , wherein the motion feature representations are configured as input feature maps and the modulating includes shrinking the input feature maps.

8. The method as described in claim 4 , wherein the forming the plurality of refined bidirectional flows and the reliability scores employs a decoder module as part of machine learning.

9. The method as described in claim 1 , wherein the synthesizing includes:
generating candidate pixels by forward warping the plurality of digital images based on the many-to-many relationships mapped between pixels in the plurality of digital images; and
fusing the candidate pixels based on the reliability scores of the many-to-many relationships, respectively.

10. The method as described in claim 9 , wherein the fusing is based at least in part on temporal relevance, brightness consistency, and the reliability scores.

11. A system comprising:
a digital image input module implemented by a processing system to receive a first digital image and a second digital image;
an optical flow generation module implemented by the processing system to generate a first optical flow describing pixel movement from the first digital image to the second digital image and a second optical flow describing pixel movement from the second digital image to the first digital image;
a motion refinement system implemented by the processing system to generate a plurality of refined bidirectional flows and color reliability maps based on the first and second digital images and the first and second optical flows;
a pixel warping module implemented by the processing system to generate candidate pixels by forward warping the first and second digital images based on the plurality of refined bidirectional flows; and
a pixel fusion module implemented by the processing system to generate a synthesized digital image by fusing the candidate pixels based on the plurality of color reliability maps by merging overlapping pixels based on respective reliability scores from the plurality of color reliability maps.

12. The system as described in claim 11 , wherein the forward warping includes many-to-many relationships mapped for at least one said pixel in the first or second digital images to multiple locations and the pixel fusion module resolves the many-to-many relationships based on the plurality of color reliability maps.

13. The system as described in claim 11 , wherein the motion refinement system includes:
a motion feature encoding module to encode motion feature representations from the first and second digital images and the first and second optical flows;
a feature modulation module to modulate the motion feature representations using a low-rank constraint; and
a decoder module to form the plurality of refined bidirectional flows and the color reliability maps based on the modulated motion feature representations.

14. The system as described in claim 13 , wherein the motion feature representations are configured as motion feature pyramids having levels corresponding to a plurality of resolutions.

15. The system as described in claim 13 , wherein the motion feature representations are configured as input feature maps and the feature modulation module is configured to shrink the input feature maps.

16. The system as described in claim 13 , wherein the pixel fusion module is configured to fuse the generated pixels based at least in part on reliability scores of the color reliability maps as well as temporal relevance or brightness consistency.

17. A system comprising:
means for generating a plurality of refined bidirectional flows and a plurality of color reliability maps based on a plurality of digital images and a plurality of optical flows;
means for generating pixels by forward warping the plurality of digital images based on the plurality of refined bidirectional flows, the plurality of refined bidirectional flows including at least one many-to-many mapping of pixels to a respective location; and
means for resolving the many-to-many mapping of the pixels to the respective location based on the plurality of color reliability maps.

18. The system as described in claim 17 , wherein the refining generating means includes:
means for encoding motion feature representations from the plurality of digital images and the plurality of optical flows;
means for modulating the motion feature representations using a low-rank constraint; and
means for decoding the plurality of refined bidirectional flows and the color reliability maps based on the modulated motion feature representations.

19. The system as described in claim 18 , wherein the motion feature representations are configured as motion feature pyramids having levels corresponding to a plurality of resolutions.

20. The system as described in claim 17 , wherein the resolving means fuses the pixels based at least in part on the color reliability maps as well as temporal relevance or brightness consistency.

Assignees

• Adobe Inc

Inventors

• Niklaus Simon
• Hu Ping

Classifications

• G06T3/18 Primary

  Image warping, e.g. rearranging pixels individually · CPC title

• G06T5/50

  using two or more images, e.g. averaging or subtraction · CPC title

• G06T2207/20221

  Image fusion; Image merging · CPC title

• G06T2207/20016

  Hierarchical, coarse-to-fine, multiscale or multiresolution image processing; Pyramid transform · CPC title

• G06T3/4007 Primary

  based on interpolation, e.g. bilinear interpolation (image demosaicing G06T3/4015; edge-driven or edge-based scaling G06T3/403) · CPC title