Photometric optimization with t-splines

Patent metadata

Field	Value
Publication number	US-2016005221-A1
Application number	US-201514791159-A
Country	US
Kind code	A1
Filing date	Jul 2, 2015
Priority date	Jul 3, 2014
Publication date	Jan 7, 2016
Grant date	—

Abstract

One example method is disclosed that includes the steps of capturing a plurality of images of a scene, wherein each of the plurality of images of the scene captures a different perspective of a portion of an object; establishing a three-dimensional (“3D”) model of the object using at least some of the plurality of images of the scene; initializing a T-spline based at least in part on the 3D model; determining a first photometric error associated with the 3D model and the T-spline; and optimizing the T-spline based on the first photometric error to create an optimized T-spline.

First claim

What is claimed is:

1 . A method of optimizing T-splines using photometric error comprising:
capturing a plurality of images of a scene, wherein each of the plurality of images of the scene captures a different perspective of a portion of an object; establishing a three-dimensional (“3D”) model of the object using at least some of the plurality of images of the scene; initializing a T-spline based at least in part on the 3D model; determining a first photometric error associated with the 3D model and the T-spline; and optimizing the T-spline based on the first photometric error to create an optimized T-spline.

2 . The method of claim 1 , wherein determining the first photometric error comprises:
selecting a point on the T-spline, determining a first pixel in a first image of the plurality of images, the first pixel corresponding to the selected point on the T-spline, determining a second pixel in a second image of the plurality of images, the second pixel corresponding to the selected point on the T-spline, and determining a difference in color intensities between the first and second pixels.

3 . The method of claim 1 , wherein the plurality of images includes a first image of the object captured from a first perspective and a second image of the object captured from a second perspective; and
wherein the photometric error is associated with a point on the T-spline and is based at least in part on a difference in pixel value between a first corresponding pixel in the first image and a second corresponding pixel in the second image.

4 . The method of claim 1 , wherein the T-spline comprises a set of control points, and optimizing the T-spline further comprising refining the T-spline by adding one or more control points to the set of control points or removing one or more control points from the set of control points.

5 . The method of claim 4 , wherein the refining further comprises changing one or more parameters of one or more control points, other than a position of the respective control point, in the set of control points.

6 . The method of claim 1 wherein optimizing the T-spline based on the photometric error comprises adjusting positions associated with one or more of a set of control points for the T-spline.

7 . The method of claim 1 further comprising:
determining a second photometric error based on the optimized T-spline;
determining that the second photometric error exceeds an error threshold;
optimizing the optimized T-spline to create a second optimized T-spline;
determining a third photometric error for the second optimized T-spline;
determining that the third photometric error satisfies the error threshold; and
responsive to determining that the third photometric error satisfies the error threshold, outputting the second optimized T-spline.

8 . The method of claim 1 , wherein determining the photometric error function comprises:
projecting points on the T-spline in directions associated with the perspectives of the plurality of images; for each projected point on the T-spline, determining a set of color or intensity values of pixels corresponding to the respective projected point on the T-spline in at least some of the plurality of images; and for each set of color or intensity values, determining a photometric error based on a difference in the color or intensity values of the set of color or intensity values.

9 . A system for optimizing T-splines using photometric error comprising:
a non-transitory computer-readable medium; and a processor in communication with the non-transitory computer-readable medium and configured to execute program code stored in the non-transitory computer-readable medium, the processor configured by the program code to: capture a plurality of images of a scene, wherein each of the plurality of images of the scene captures a different perspective of a portion of an object; establish a three-dimensional (“3D”) model of the object using at least some of the plurality of images of the scene; initialize a T-spline based at least in part on the 3D model; determine a first photometric error associated with the 3D model and the T-spline; and optimize the T-spline based on the first photometric error to create an optimized T-spline.

10 . The system of claim 9 , wherein the processor is further configured to:
select a point on the T-spline, determine a first pixel in a first image of the plurality of images, the first pixel corresponding to the selected point on the T-spline, determine a second pixel in a second image of the plurality of images, the second pixel corresponding to the selected point on the T-spline, and determine a difference between the first and second pixels to determine the first photometric error.

11 . The system of claim 9 , wherein the plurality of images includes a first image of the object captured from a first perspective and a second image of the object captured from a second perspective; and
wherein the photometric error is associated with a point on the T-spline and is based at least in part on a difference in pixel value between a first corresponding pixel in the first image and a second corresponding pixel in the second image.

12 . The system of claim 9 , wherein the T-spline comprises a set of control points, and wherein the processor is further configured to add one or more control points to the set of control points or remove one or more control points from the set of control points to refine the T-spline.

13 . The system of claim 10 , wherein the processor is further configured to change one or more parameters of one or more control points in a set of control points for the T-spline, other than a position of the respective control point, in the set of control points to refine the T-spline.

14 . The system of claim 9 , wherein the processor is further configured to adjust positions associated with one or more of a set of control points for the T-spline to optimize the T-spline based on the first photometric error function.

15 . The system of claim 9 , wherein the processor is further configured to:
determine a second photometric error based on the optimized T-spline; determine that the second photometric error exceeds an error threshold; optimize the optimized T-spline to create a second optimized T-spline; determine a third photometric error for the second optimized T-spline; determine that the third photometric error satisfies the error threshold; and responsive to a determination that the third photometric error satisfies the error threshold, output the second optimized T-spline.

16 . The system of claim 9 , wherein the processor is further configured to:
project points on the T-spline in directions associated with the perspectives of the plurality of images; for each projected point on the T-spline, determine a set of color or intensity values of pixels corresponding to the respective projected point on the T-spline in each of the plurality of images; and for each set of color or intensity values, determine a photometric error based on a difference in the color or intensity values of the set of color or intensity values to determine the first photometric error function.

17 . A non-transitory computer readable medium comprising program code for causing a processor to execute a method for using T-splines for photometric optimization, the program code comprising:
program code for capturing a plurality of images of a scene, wherein each of the plurality of images of the scene captures a different perspective of a portion

Assignees

• Qualcomm Inc

Inventors

• Mörwald Thomas
• Prankl Johann

Classifications

• G06T2200/04

  involving 3D image data · CPC title

• H04N2101/00

  Still video cameras · CPC title

• G06T2207/10028

  Range image; Depth image; 3D point clouds · CPC title

• G06T17/30 Primary

  Polynomial surface description · CPC title

• G06T2207/10012

  Stereo images · CPC title