System for image completion

Patent metadata

Field	Value
Publication number	US-12062160-B2
Application number	US-202117478837-A
Country	US
Kind code	B2
Filing date	Sep 17, 2021
Priority date	Sep 22, 2020
Publication date	Aug 13, 2024
Grant date	Aug 13, 2024

Abstract

A system for image completion is disclosed. The system comprises a coordinate generation module configured to receive past frames and a present frame having a first field-of-view and to generate a set of coordinate maps, one for each of the received past frames; and a frame aggregation module configured to receive as input the past frames, the present frame, and the coordinate maps and to synthesize, based on said input, a present frame having a second field-of-view.

First claim

What is claimed is:

1. A system for image completion, comprising:
a coordinate generation module configured to receive first past frames and a first present frame, the first past frames and the first present frame having a first field-of-view, and to generate a set of coordinate maps, one for each of the received first past frames, wherein the coordinate map corresponding to a first past frame provides a spatial mapping of the first past frame to a coordinate system of the first present frame, wherein the coordinate generation module comprises:
a depth network configured to receive the first past frames and to generate a depth map for each of the received first past frames;
a pose network configured to receive frame pairs of time-adjacent frames formed from the received first past frames and the first present frame and to generate a relative camera pose for each of the received frame pairs independent from the depth map for each of the received first past frames; and
a coordinate calculation module configured to generate the set of coordinate maps based on outputs of the depth network and the pose network; and

a frame aggregation module configured to receive as input the first past frames, the first present frame, and the coordinate maps and to synthesize, based on said input, a second present frame having a second field-of-view, wherein a portion of the first past frames received by a depth network and the frame aggregation module are replaced by second past frames having the second field-of-view generated by the frame aggregation module and corresponding to said portion of the first past frames.

2. The system according to claim 1 , wherein the first field-of-view comprises fewer pixels than the second field-of-view.

3. The system according to claim 1 , wherein the frame aggregation module is configured to propagate information contained in the received first past frames to the coordinate system of the first present frame using the set of coordinate maps generated by the coordinate generation module.

4. The system according to claim 1 , wherein the frame aggregation module comprises:
an encoder configured to generate a plurality of feature maps based on each of the first past frames and the first present frame;
a warping module configured, for each of the first past frames and the first present frame, to warp the plurality of feature maps associated with said each frame, using the respective coordinate map associated with said each frame, to generate a plurality of warped feature maps for said each frame; and
an attention-based feature aggregation module configured to aggregate, over all of the first past frames and the first present frame, the generated warped feature maps to generate a set of aggregated feature maps.

5. The system according to claim 4 , wherein the attention-based feature aggregation module is configured in the aggregation to emphasize, for each frame of the first past frames and the first present frame, region-specific features of the frame based on a timing of the frame relative to the first present frame.

6. The system according to claim 4 , wherein the attention-based feature aggregation module is configured to, for each frame of the first past frames and the first present frame:
generate a respective frame-wise spatial attention map for each of the warped feature maps associated with said each frame; and
multiply each of the warped feature maps associated with said each frame with the respective spatial attention map to generate a respective feature map.

7. The system according to claim 6 , wherein the attention-based feature aggregation module is further configured to sum, over all of the first past frames and the first present frame, the generated respective feature maps to generate the set of aggregated feature maps.

8. The system according to claim 4 , wherein the frame aggregation module further comprises a U-net module configured to generate the second present frame based on the set of aggregated feature maps, the U-net module comprising:
a context normalization sub-module configured to receive the set of aggregated feature maps and to out-paint regions of the second present frame falling outside of the field-of-view of the first present frame;
a decoder sub-module configured to receive feature maps output by the context normalization sub-module and to process said feature maps output by the context normalization sub-module to in-paint occluded or unobserved regions falling within the field-of-view of the first present frame; and
a gated self-attention sub-module configured to receive feature maps output by the decoder sub-module and to spatially aggregate said feature maps output by the decoder sub-module.

9. The system according to claim 8 , wherein the gated self-attention sub-module is configured to spatially aggregate the feature maps output by the decoder sub-module based on weights that are dynamically generated per feature vector based on a spatial location of the feature vector.

10. The system according to claim 1 , comprising:
a hallucination uncertainty module configured to generate an uncertainty map associated with the second present frame.

11. The system according to claim 10 , wherein the hallucination uncertainty module is configured to generate the uncertainty map to minimize a loss function incorporating hallucination uncertainty.

12. The system according to claim 10 , wherein the hallucination uncertainty module is configured to generate the uncertainty map based on predicting regions of the second present frame that will have a first hallucinating uncertainty and those that will have a second hallucinating uncertainty, and the first hallucinating uncertainty is higher than the second hallucinating uncertainty.

13. The system according to claim 1 , wherein the second past frames are each concatenated with a respective uncertainty map generated by a hallucination uncertainty module, before providing the second past frames to the frame aggregation module.

14. The system according to claim 1 , wherein the second field-of-view is equal or greater than the first field-of-view.

15. A system for image completion, comprising:
a coordinate generation module configured to receive first past frames and a first present frame, the first past frames and the first present frame having a first field-of-view, and to generate a set of coordinate maps, one for each of the received first past frames, wherein the coordinate map corresponding to a first past frame provides a spatial mapping of the first past frame to a coordinate system of the first present frame; and

a frame aggregation module configured to receive as input the first past frames, the first present frame, and the coordinate maps and to synthesize, based on said input, a second present frame having a second field-of-view, wherein the frame aggregation module comprises:
an encoder configured to generate a plurality of feature maps based on each of the first past frames and the first present frame;
a warping module configured, for each of the first past frames and the first present frame, to warp the plurality of feature maps associated with said each frame, using the respective coordinate map associated with said each frame, to generate a plurality of warped feature maps for said each frame; and
an attention-based feature aggregation module configured to aggregate, over all of the first past frames and the first present frame, the generated warped feature maps to generate a set of aggregated feature maps.

16. A system for image completion, comprising:
a coordinate generation module c

Assignees

• Toyota Motor Co Ltd
• Univ Leuven Kath
• Eth Zuerich

Inventors

• Abbeloos Wim
• Othmezouri Gabriel
• Ma Liqian
• Georgoulis Stamatios
• Van Gool Luc

Classifications

• G06T3/18

  Image warping, e.g. rearranging pixels individually · CPC title

• G06T3/04

  Context-preserving transformations, e.g. by using an importance map (panospheric to cylindrical image transformations G06T3/12) · CPC title

• G06T2207/30244

  Camera pose · CPC title

• G06T2207/20212

  Image combination · CPC title

• G06T2207/10016

  Video; Image sequence · CPC title