Realistic augmentation of images and videos with graphics

Patent metadata

Field	Value
Publication number	US-10726599-B2
Application number	US-201816104673-A
Country	US
Kind code	B2
Filing date	Aug 17, 2018
Priority date	Aug 17, 2017
Publication date	Jul 28, 2020
Grant date	Jul 28, 2020

Abstract

Techniques disclosed herein relate generally to augmenting images or videos with graphics. More specifically, some embodiments relate to realistically or photorealistically augmenting a target image or video frame with a source graph, such as a computer-generated graph or a real world image. In one embodiment, a planar segment of the target image is identified based on a surface normal map of the target image. The planar segment is then used to determine a focal length and a homography function for transforming the source graph.

First claim

What is claimed is:

1. A computer-implemented method comprising, by one or more computing systems:
identifying a planar region in a target image, wherein identifying the planar region in the target image comprises:
determining a plurality of superpixels associated with the target image, each superpixel including a group of pixels;
generating a weight value associated with each pair of adjacent superpixels among the plurality of superpixels in the target image, wherein the weight value is generated based on an average normal direction for each superpixel in the pair of adjacent superpixels and an average color value for each superpixel in the pair of adjacent superpixels;
comparing the weight value associated with each pair of adjacent superpixels with a threshold weight value; and
merging superpixels in each pair of adjacent superpixels that have the associated weight value being lower than the threshold weight value to form merged superpixels, wherein the planar region is identified by including the merged superpixels in the planar region;

identifying a plurality of line segments for the planar region of the target image, wherein the plurality of line segments are not parallel;
determining parameters for a homography function based on an inverse function of the homography function by determining that a predetermined angle is formed by two line segments in a front-parallel view, wherein the two line segments in the front-parallel view are generated by applying the inverse function of the homography function to at least two line segments from the plurality of line segments, wherein the predetermined angle is a non-zero angle;
obtaining a source image in the front-parallel view;
transforming the source image using the homography function; and
superimposing the transformed source image onto the planar region of the target image.

2. The computer-implemented method of claim 1 , wherein the predetermined angle is a right angle.

3. The computer-implemented method of claim 1 , wherein the parameters for the homography function comprise a camera focal length.

4. The computer-implemented method of claim 3 , wherein determining the parameters for the homography function comprises:
setting an initial value for the camera focal length;
computing the inverse function of the homography function based at least in part on the initial value for the camera focal length;
transforming the planar region of the target image using the inverse function of the homography function;
determining an angle between the two line segments in the transformed planar region; and
determining an updated value for the camera focal length based on determining that the angle between the two line segments in the transformed planar region is different from the predetermined angle.

5. The computer-implemented method of claim 1 , wherein determining the parameters for the homography function comprises determining a camera focal length of the target image using a neural network trained based on training images with known camera focal lengths.

6. The computer-implemented method of claim 1 , wherein the plurality of line segments comprises intersecting line segments or line segments that intersect when extended by a distance less than a threshold distance.

7. The computer-implemented method of claim 6 , wherein identifying the plurality of line segments for the planar region of the target image comprises:
detecting line segments in the planar region;
forming a set of line segment pairs based on the detected line segments; and
removing, from the set of line segment pairs, each line segment pair in which two line segments are parallel or do not intersect when extended by the threshold distance.

8. The computer-implemented method of claim 1 , wherein an orientation of the planar region is distinct from the front-parallel view.

9. The computer-implemented method of claim 1 , wherein identifying the planar region in the target image further comprises:
determining a surface normal map associated with the target image; and
generating a region adjacency graph associated with the plurality of superpixels, each superpixel represented by a node in the region adjacency graph, wherein
the weight value associated with each pair of adjacent superpixels is generated based at least in part upon the region adjacency graph and the surface normal map.

10. The computer-implemented method of claim 1 , wherein determining the plurality of superpixels comprises applying simple linear iterative clustering (SLIC) on the target image.

11. The computer-implemented method of claim 9 , wherein the weight value indicates a similarity between each pair of adjacent superpixels.

12. The computer-implemented method of claim 9 , wherein
the average normal direction for each superpixel in the pair of adjacent superpixels is generated based, at least in part, upon the surface normal map associated with the target image.

13. The computer-implemented method of claim 12 , wherein
the average color value for each superpixel in the pair of adjacent superpixels is determined based on color values of the group of pixels in each superpixel.

14. The computer-implemented method of claim 9 , further comprising:
generating a weight value associated with each respective pair of adjacent merged superpixels;
comparing the weight value associated with each respective pair of adjacent merged superpixels with the threshold weight value; and
merging, for each pair of adjacent merged superpixels having the associated weight value lower than the threshold weight value, merged superpixels in the pair of adjacent merged superpixels to form a new merged superpixel.

15. The computer-implemented method of claim 1 , wherein obtaining the source image in the front-parallel view comprises:
receiving an original source image in a view different from the front-parallel view; and
applying a second inverse homography function on the original source image to transform the original source image into the source image in the front-parallel view.

16. A system comprising:
a processing device; and
a non-transitory computer-readable medium communicatively coupled to the processing device, wherein the processing device is configured to execute program code stored in the non-transitory computer-readable medium and thereby perform operations comprising:
identifying a planar region in a target image, wherein identifying the planar region in the target image comprises:
determining a plurality of superpixels associated with the target image, each superpixel including a group of pixels;
generating a weight value associated with each pair of adjacent superpixels among the plurality of superpixels in the target image, wherein the weight value is generated based on an average normal direction for each superpixel in the pair of adjacent superpixels and an average color value for each superpixel in the pair of adjacent superpixels;
comparing the weight value associated with each pair of adjacent superpixels with a threshold weight value; and
merging superpixels in each pair of adjacent superpixels that have the associated weight value being lower than the threshold weight value to form merged superpixels, wherein the planar region is identified by including the merged superpixels in the planar region;

identifying a plurality of line segments for the planar region of the target image, wherein the plurality of line segments are not parallel;
determining parameters for a homography function based on an inverse function of the homography function by determining that a predetermined angle

Assignees

• Adobe Inc

Inventors

• Marzban Shabbir
• Attorre Brunno Fidel Maciel
• Thien Nicolas Huynh

Classifications

• G06T2207/20084

  Artificial neural networks [ANN] · CPC title

• G06T2207/20081

  Training; Learning · CPC title

• G06T2207/10024

  Color image · CPC title

• G06T7/33

  using feature-based methods · CPC title

• G06T11/60 Primary

  Creating or editing images; Combining images with text · CPC title