System and method of pose estimation

Patent metadata

Field	Value
Publication number	US-9607388-B2
Application number	US-201414491857-A
Country	US
Kind code	B2
Filing date	Sep 19, 2014
Priority date	Sep 19, 2014
Publication date	Mar 28, 2017
Grant date	Mar 28, 2017

Abstract

A method includes receiving, from an image capture device, a first image frame of a sequence of image frames. The method also includes estimating, at a processor, a camera pose corresponding to the first image frame by comparing the first image frame to a second image frame. The second image frame precedes the first image frame in the sequence of image frames. The method further includes estimating, at the processor, a refined camera pose corresponding to the first image frame by comparing the first image frame to a keyframe. The keyframe corresponds to a particular image frame that precedes the second image frame in the sequence of image frames.

First claim

What is claimed is:

1. A method comprising:
receiving, from an image capture device, a first image frame of a sequence of image frames;
estimating, at a processor, a camera pose corresponding to the first image frame by comparing the first image frame to a second image frame, wherein the second image frame precedes the first image frame in the sequence of image frames;
selecting a first keyframe from a plurality of keyframes associated with the sequence of image frames, wherein the first keyframe is selected based on similarity of the camera pose corresponding to the first image frame to a camera pose corresponding to each of the plurality of keyframes; and
estimating, at the processor, a refined camera pose corresponding to the first image frame by comparing the first image frame to the first keyframe.

2. The method of claim 1 , wherein the camera pose corresponding to the first image frame is estimated as part of a first processing thread of the processor, wherein the refined camera pose corresponding to the first image frame is estimated as part of a second processing thread of the processor.

3. The method of claim 2 , wherein the processor is configured to execute processing threads in parallel, and wherein the first processing thread and the second processing thread are executed in parallel.

4. The method of claim 1 , wherein estimating the camera pose corresponding to the first image frame comprises:
generating a warped frame by transforming the first image frame based on an estimate of relative motion of the image capture device between capture of the first image frame and capture of the second image frame;
determining a pixel intensity error, wherein the pixel intensity error indicates a difference of pixel intensity between one or more pixels of the warped frame and one or more pixels of the second image frame; and
determining a depth error, wherein the depth error indicates a difference of depth between the one or more pixels of the warped frame and the one or more pixels of the second image frame, wherein the depth error is determined based on normal values of the one or more pixels of the second image frame, and wherein a normal value of each pixel includes a vector that indicates a direction perpendicular to a tangential plane of each pixel.

5. The method of claim 1 , wherein estimating the refined camera pose corresponding to the first image frame comprises:
generating a warped frame by transforming the first image frame based on an estimate of relative motion of the image capture device between capture of the first image frame and capture of the first keyframe;
determining a pixel intensity error, wherein the pixel intensity error indicates a difference of pixel intensity between one or more pixels of the warped frame and one or more pixels of the first keyframe; and
determining a depth error, wherein the depth error indicates a difference of depth between the one or more pixels of the warped frame and the one or more pixels of the first keyframe, wherein the depth error is determined based on normal values of the one or more pixels of the first keyframe, and wherein a normal value of each pixel includes a vector that indicates a direction perpendicular to a tangential plane of each pixel.

6. The method of claim 1 , further comprising:
determining, at the processor, an overlap ratio between the first image frame and the first keyframe, wherein the overlap ratio is based on a pixel-by-pixel comparison of first pixel values of the first image frame and second pixel values of the first keyframe;
comparing the overlap ratio to an overlap ratio threshold to generate a comparison result; and
generating a new keyframe using the first image frame based on the comparison result.

7. The method of claim 6 , wherein generating the new keyframe comprises creating a copy image frame corresponding to the first image frame in memory, and wherein the copy image frame is designated as the new keyframe.

8. The method of claim 6 , wherein the first pixel values include depth values, photometric values, or a combination thereof.

9. The method of claim 1 , further comprising generating a three-dimensional point cloud using the refined camera pose and the sequence of image frames.

10. The method of claim 9 , further comprising:
receiving, from the image capture device, one or more additional image frames of the sequence of image frames; and
updating the three-dimensional point cloud based on the one or more additional image frames, wherein a rate of updating the three-dimensional point cloud is substantially equal to a rate of receiving the one or more additional image frames.

11. The method of claim 1 , wherein the camera pose corresponding to the first image frame indicates an approximation of a position of the image capture device at a particular time when the first image frame is captured, wherein the camera pose corresponding to the first image frame includes position coordinates in a 3D space and an orientation corresponding to the image capture device at a time when the first image frame is captured, and wherein the similarity of the camera pose corresponding to the first image frame to the camera pose of the first keyframe is based on at least one of:
a difference between a relative angle of the camera pose corresponding to the first image frame and a relative angle of the camera pose of the first keyframe;
a relative distance threshold for the first image frame; and
a relative position threshold for the first image frame.

12. The method of claim 1 , wherein the first image frame has a first resolution, wherein the keyframe has a keyframe resolution, and wherein the first resolution is different from the keyframe resolution.

13. An apparatus comprising
a processor; and
a memory storing instruction that are executable by the processor to cause the processor to:
receive, from an image capture device, a first image frame of a sequence of image frames;
estimate a camera pose corresponding to the first image frame based on a first comparison of the first image frame to a second image frame, wherein the second image frame precedes the first image frame in the sequence of image frames;
select a first keyframe from a plurality of keyframes associated with the sequence of image frames, wherein the first keyframe is selected based on similarity of the camera pose corresponding to the first image frame to a camera pose corresponding to each keyframe of the plurality of keyframes; and
estimate a refined camera pose corresponding to the first image frame based on a second comparison of the first image frame to the first keyframe.

14. The apparatus of claim 13 , wherein the camera pose corresponding to the first image frame is estimated as part of a first processing thread of the processor, wherein the refined camera pose corresponding to the first image frame is estimated as part of a second processing thread of the processor, and wherein the similarity of the camera pose corresponding to the first image frame to the camera pose corresponding to the first keyframe satisfies a relative distance threshold for the first image frame.

15. The apparatus of claim 14 , wherein the similarity of the camera pose corresponding to the first image frame to the camera pose corresponding to the first keyframe satisfies a relative position threshold for the first image frame.

16. The apparatus of claim 13 , wherein the instructions are further executable to cause the processor to transform the first image frame to generate a warped frame, wherein the first image frame is transformed based on an estim

Assignees

• Qualcomm Inc

Inventors

• Lin Kuen-Han
• Sundaresan Sairam
• Sarkis Michel Adib

Classifications

• G06T2207/30244

  Camera pose · CPC title

• G06T2200/04

  involving 3D image data · CPC title

• G06T2207/10016

  Video; Image sequence · CPC title

• H04N13/221

  using the relative movement between cameras and objects · CPC title

• G06T7/74 Primary

  involving reference images or patches · CPC title