Acceleration of real time computer vision processing on UAVs through GPS attitude estimation

Patent metadata

Field	Value
Publication number	US-10241214-B2
Application number	US-201615057820-A
Country	US
Kind code	B2
Filing date	Mar 1, 2016
Priority date	Mar 1, 2016
Publication date	Mar 26, 2019
Grant date	Mar 26, 2019

Abstract

A method for applying GPS UAV attitude estimation to accelerate computer vision. The UAV has a plurality of GPS receivers mounted at fixed locations on the UAV. The method includes receiving raw GPS measurements from each GPS satellite in view of the UAV, the raw GPS measurements comprising pseudo-range and carrier phase data representing the distance between each GPS receiver and each GPS satellite. Carrier phase and pseudo-range measurements are determined for each GPS receiver based on the pseudo-range and carrier phase data. The GPS carrier phase and pseudo-range measurements are compared pair-wise for each pair of GPS receiver and satellite. An attitude of the UAV is determined based on the relative distance measurements. A 3D camera pose rotation matrix is determined based on the attitude of the UAV. Computer vision image search computations are performed for analyzing the image data received from the UAV in real time using the 3D camera pose rotation matrix.

First claim

What is claimed is:

1. A method for applying GPS UAV attitude estimation to image analysis, the UAV having a plurality of GPS receivers mounted at fixed locations on the UAV and a camera, the method comprising:
receiving GPS signals from each GPS satellite in view of the UAV, the GPS signals comprising respective pn codes and carrier frequencies;
determining carrier phase and pseudo-range measurements for each GPS receiver fixed to the UAV based on the respective pn codes and carrier frequencies, the pseudo-range and carrier phase data measurements representing the distance between each GPS receiver fixed to the UAV and each GPS satellite in view of the UAV;
comparing the GPS carrier phase and pseudo-range measurements pair-wise for each pair of GPS receivers fixed to the UAV and GPS satellite in view of the UAV to determine relative distance measurements;
determining an attitude of the UAV based on the relative distance measurements at each of a plurality of time points;
determining a 3D camera pose rotation matrix based on the attitude of the UAV comprising the steps of:
determining a first attitude of the UAV at a first time point;
determining a second attitude of the UAV at a second time point;
encoding the first attitude of the UAV as a first 3×3 rotation matrix and encoding the second attitude of the UAV as a second 3×3 rotation matrix;
calculating a rotation of the camera between the first and second time points by multiplying the first rotation matrix by the inverse of the second rotation matrix and encoding the results as the 3D camera pose rotation matrix; and
estimating the camera pose rotation between the first time point and the second time point;

receiving image data from a camera mounted on the UAV taken at the first time point and the second time point; and
analyzing the image data received from the UAV at the first and second time points using the estimated camera pose rotation to predict coordinates of an object in the image data at the second time point.

2. The method of claim 1 , wherein determining the carrier phase and pseudo-range measurements comprises estimating an integral component of the distance between each GPS receiver and each GPS satellite using the pseudo-range data and estimating a fractional component of the distance between each GPS receiver and each GPS satellite using the carrier phase data.

3. The method of claim 2 , wherein comparing the GPS carrier phase and pseudo-range measurements comprises comparing the pseudo-range and carrier phase estimated distance components pair-wise between the GPS receiver and satellite pairs.

4. The method of claim 3 , wherein determining an attitude of the UAV comprises calculating relative distances between the GPS receivers and each satellite based on the comparisons and determining the attitude of the UAV based on the relative distances between GPS receivers and satellites.

5. The method of claim 1 , further comprising:
locating coordinates of an image corresponding to the first and second time points in a first video or photograph stream time-synchronized to the raw GPS measurements;
obtaining a camera matrix defining properties of the camera;
combining the camera matrix and the 3D camera pose rotation matrix to determine predicted coordinates of the image in a second video or photograph stream time;
projecting the predicted coordinates of the image into the second video or photograph stream time; and
using the projected coordinates as a starting point for a search in the second video or photograph stream for an image having the same physical characteristics of the image.

6. A non-transitory article of manufacture tangibly embodying computer readable instructions, which when implemented, cause a computer to perform the steps of a method for applying GPS UAV attitude estimation to image analysis, the UAV having a plurality of GPS receivers mounted at fixed locations on the UAV and a camera, comprising;
receiving GPS signals from each GPS satellite in view of the UAV, the GPS signals comprising respective pn codes and carrier frequencies;
determining carrier phase and pseudo-range measurements for each GPS receiver fixed to the UAV based on the respective pn codes and carrier frequencies, the pseudo-range and carrier phase data measurements representing the distance between each GPS receiver fixed to the UAV and each GPS satellite in view of the UAV;
comparing the GPS carrier phase and pseudo-range measurements pair-wise for each pair of GPS receivers fixed to the UAV and GPS satellite in view of the UAV to determine relative distance measurements;
determining an attitude of the UAV based on the relative distance measurements at each of a plurality of time points;
determining a 3D camera pose rotation matrix based on the attitude of the UAV comprising the steps of:
determining a first attitude of the UAV at a first time point;
determining a second attitude of the UAV at a second time point;
encoding the first attitude of the UAV as a first 3×3 rotation matrix and encoding the second attitude of the UAV as a second 3×3 rotation matrix;
calculating a rotation of the camera between the first and second time points by multiplying the first rotation matrix by the inverse of the second rotation matrix and encoding the results as the 3D camera pose rotation matrix; and
estimating the camera pose rotation between the first time point and the second time point;

receiving image data from a camera mounted on the UAV taken at the first time point and the second time point; and
analyzing the image data received from the UAV at the first and second time points using the estimated camera pose rotation to predict coordinates of an object in the image data at the second time point.

7. The non-transitory article of manufacture of claim 6 , wherein determining the carrier phase and pseudo-range measurements comprises estimating an integral component of the distance between each GPS receiver and each GPS satellite using the pseudo-range data and estimating a fractional component of the distance between each GPS receiver and each GPS satellite using the carrier phase data.

8. The non-transitory article of manufacture of claim 7 , wherein comparing the GPS carrier phase and pseudo-range measurements comprises comparing the pseudo-range and carrier phase estimated distance components pair-wise between the GPS receiver and satellite pairs.

9. The non-transitory article of manufacture of claim 8 , wherein determining an attitude of the UAV comprises calculating relative distances between the GPS receivers and each satellite based on the comparisons and determining the attitude of the UAV based on the relative distances between GPS receivers and satellites.

10. The non-transitory article of manufacture of claim 6 , further comprising:
locating coordinates of an image corresponding to the first and second time points in a first video or photograph stream time-synchronized to the raw GPS measurements;
obtaining a camera matrix defining properties of the camera;
combining the camera matrix and the 3D camera pose rotation matrix to determine predicted coordinates of the image in a second video or photograph stream time;
projecting the predicted coordinates of the image into the second video or photograph stream time; and
using the projected coordinates as a starting point for a search in the second video or photograph stream for an image having the same physical characteristics of the image.

11. A computer system for applying GPS UAV attitude estimation to image analysis, the UAV having a plurality of GPS receivers mounted at fixed locations on the UAV and a camera, comprising:
one or more computer processors;

Assignees

• IBM

Inventors

• Gowda Mahanth K
• Manweiler Justin G
• Weisz Justin D

Classifications

• G01S19/14

  specially adapted for specific applications · CPC title

• G01S19/54 Primary

  using carrier phase measurements; using long or short baseline interferometry · CPC title

• G01S19/43

  using carrier phase measurements, e.g. kinematic positioning; using long or short baseline interferometry · CPC title

• G01S19/39

  the satellite radio beacon positioning system transmitting time-stamped messages, e.g. GPS [Global Positioning System], GLONASS [Global Orbiting Navigation Satellite System] or GALILEO · CPC title

• G06T7/70

  Determining position or orientation of objects or cameras (camera calibration G06T7/80) · CPC title