Artificially rendering images using interpolation of tracked control points

Patent metadata

Field	Value
Publication number	US-10733475-B2
Application number	US-201815936231-A
Country	US
Kind code	B2
Filing date	Mar 26, 2018
Priority date	Jul 15, 2015
Publication date	Aug 4, 2020
Grant date	Aug 4, 2020

Abstract

Various embodiments of the present invention relate generally to systems and processes for artificially rendering images using interpolation of tracked control points. According to particular embodiments, a set of control points is tracked between a first frame and a second frame, where the first frame includes a first image captured from a first location and the second frame includes a second image captured from a second location. An artificially rendered image corresponding to a third location is then generated by interpolating individual control points for the third location using the set of control points and interpolating pixel locations using the individual control points. The individual control points are used to transform image data.

First claim

What is claimed is:

1. A method comprising:
tracking a set of control points between a first frame and a second frame, wherein the first frame includes a first image captured from a first location and the second frame includes a second image captured from a second location, the first and second locations corresponding to real world location positions; and
generating an artificially rendered image as a third frame corresponding to a third location, the third location being a real world location position on a trajectory between the first location and the second location, wherein generating the artificially rendered image includes:
interpolating a transformation using at least one of homography, affine, similarity, translation, rotation, and scale, including interpolating individual control points for the third location using IMU data and the set of control points, the IMU data corresponding to the first and second locations, and interpolating pixel locations using the individual control points, wherein the individual control points are used to transform image data, wherein interpolating the transformation includes using depth information to reduce occurrence of artifacts resulting from mismatched pixels; and
combining first image information and second image information using the interpolated transformation to form the artificially rendered image.

2. The method of claim 1 , wherein interpolating the pixel locations includes interpolation using barycentric coordinates based on three or more control points.

3. The method of claim 1 , wherein interpolating the pixel locations includes interpolation using splines.

4. The method of claim 1 , wherein interpolating the pixel locations includes interpolation using finite elements.

5. The method of claim 1 , wherein interpolating the pixel locations includes using motion of a nearest control point.

6. The method of claim 1 , wherein the set of control points are located within a first layer, the first layer associated with particular objects viewable in the first frame and second frame.

7. The method of claim 1 , wherein each pixel in the first image corresponds to a control point in the set of control points.

8. The method of claim 7 , wherein interpolating pixel locations using the individual control points includes using dense optical flow.

9. A non-transitory computer readable medium having instructions stored thereon, that when executed by a computer cause the computer to perform:
tracking a set of control points between a first frame and a second frame, wherein the first frame includes a first image captured from a first location and the second frame includes a second image captured from a second location, the first and second locations corresponding to real world location positions; and
generating an artificially rendered image as a third frame corresponding to a third location, the third location being a real world location position on a trajectory between the first location and the second location, wherein generating the artificially rendered image includes:
interpolating a transformation using at least one of homography, affine, similarity, translation, rotation, and scale, including interpolating individual control points for the third location using IMU data and the set of control points, the IMU data corresponding to the first and second locations, and interpolating pixel locations using the individual control points, wherein the individual control points are used to transform image data, wherein interpolating the transformation includes using depth information to reduce occurrence of artifacts resulting from mismatched pixels; and
combining first image information and second image information using the interpolated transformation to form the artificially rendered image.

10. The computer readable medium of claim 9 , wherein interpolating the pixel locations includes interpolation using barycentric coordinates based on three or more control points, interpolation using splines, or interpolation using finite elements.

11. The computer readable medium of claim 9 , wherein the set of control points are located within a first layer, the first layer associated with particular objects viewable in the first frame and second frame.

12. A method comprising:
tracking a set of control points between a plurality of frames and generating a panoramic representation from the plurality of frames, wherein the plurality of frames includes a first frame corresponding to a first image captured from a first location and a second frame corresponding to a second image captured from a second location, wherein the plurality of frames is associated with a first layer, the first and second locations corresponding to real world location positions; and
generating an artificially rendered image as a third frame corresponding to a third location, the third location being a real world location position on a trajectory between the first location and the second location, wherein generating the artificially rendered image includes:
interpolating a transformation using at least one of homography, affine, similarity, translation, rotation, and scale, including interpolating individual control points for the third location using IMU data and the set of control points, the IMU data corresponding to the first and second locations, and interpolating pixel locations using the individual control points, wherein the individual control points are used to transform image data, wherein interpolating the transformation includes using depth information to reduce occurrence of artifacts resulting from mismatched pixels; and
combining first image information and second image information using the interpolated transformation to form the artificially rendered image.

13. The method of claim 12 , wherein interpolating the pixel locations includes interpolation using barycentric coordinates based on three or more control points.

14. The method of claim 12 , wherein interpolating the pixel locations includes interpolation using splines.

15. The method of claim 12 , wherein interpolating the pixel locations includes interpolation using finite elements.

16. The method of claim 12 , wherein interpolating the pixel locations includes using motion of a nearest control point.

17. The method of claim 12 , wherein each pixel in the first image corresponds to a control point in the set of control points.

18. The method of claim 17 , wherein interpolating pixel locations using the individual control points includes using dense optical flow.

19. The method of claim 12 , wherein the first layer is a background layer.

20. The method of claim 12 , wherein the first layer includes objects.

Assignees

• Fyusion Inc

Inventors

• Holzer Stefan Johannes Josef
• Miller Stephen David
• Saelzle Martin
• Rusu Radu Bogdan

Classifications

• G06T15/205 Primary

  Image-based rendering · CPC title

• G06F18/22

  Matching criteria, e.g. proximity measures · CPC title

• G06V20/10

  Terrestrial scenes (scenes under surveillance with static cameras G06V20/52; scenes perceived from the exterior of a vehicle G06V20/56; scenes perceived from the interior of a vehicle G06V20/59) · CPC title

• H04N2013/0085

  Motion estimation from stereoscopic image signals · CPC title

• H04N13/111

  Transformation of image signals corresponding to virtual viewpoints, e.g. spatial image interpolation · CPC title