Methods and systems for cross-validating sensor data acquired using sensors of a mobile device

Patent metadata

Field	Value
Publication number	US-9277361-B2
Application number	US-201414184929-A
Country	US
Kind code	B2
Filing date	Feb 20, 2014
Priority date	Feb 20, 2014
Publication date	Mar 1, 2016
Grant date	Mar 1, 2016

Abstract

Methods and systems for cross-validating sensor data are described. An example method involves receiving image data and first timing information associated with the image data, and receiving sensor data and second timing information associated with the sensor data. The method further involves determining a first estimation of motion of the mobile device based on the image data and the first timing information, and determining a second estimation of the motion of the mobile device based on the sensor data and the second timing information. Additionally, the method involves determining whether the first estimation is within a threshold variance of the second estimation. The method then involves providing an output indicative of a validity of the first timing information and the second timing information based on whether the first estimation is within the threshold variance of the second estimation.

First claim

What is claimed is:

1. A method performed by a mobile device having a plurality of sensors, the method comprising:
receiving image data and first timing information associated with the image data, wherein the image data includes images representative of a motion of the mobile device and is determined using a first sensor of the plurality of sensors;
receiving sensor data and second timing information associated with the sensor data, wherein the sensor data corresponds to the motion of the mobile device and is determined using a second sensor of the plurality of sensors;
determining a first estimation of the motion of the mobile device based on the image data and the first timing information;
determining a second estimation of the motion of the mobile device based on the sensor data and the second timing information;
determining whether the first estimation of the motion of the mobile device is within a threshold variance of the second estimation of the motion of the mobile device; and
based on whether the first estimation of the motion of the mobile device is within the threshold variance of the second estimation of the motion of the mobile device, providing an output indicative of a validity of the first timing information and the second timing information.

2. The method of claim 1 carried out by an application processor of the mobile device, wherein the first timing information and the second timing information are received from a co-processor of the mobile device that is configured to determine the first timing information and the second timing information.

3. The method of claim 2 , further comprising receiving, by the application processor and from the co-processor, a digital image that includes the image data and the sensor data, wherein the sensor data is embedded in pixels of the digital image.

4. The method of claim 2 , further comprising receiving, by the application processor and from the first sensor, the image data.

5. The method of claim 1 :
wherein the image data comprises a sequence of two-dimensional images, and
wherein determining the first estimation of the motion of the mobile device based on the image data and the first timing information comprises determining a first rotational motion of the mobile device based on movement of image features between images of the sequence of two-dimensional images.

6. The method of claim 5 :
wherein the second sensor comprises a gyroscope, and
wherein determining the second estimation of the motion of the mobile device based on the sensor data and the second timing information comprises determining a second rotational motion of the mobile device.

7. The method of claim 1 :
wherein the image data comprises a sequence of two-dimensional images of known three-dimensional points in a scene, and
wherein determining the first estimation of the motion of the mobile device based on the image data and the first timing information comprises determining a trajectory of the mobile device based on locations of the known three-dimensional points in the sequence of two-dimensional images.

8. The method of claim 7 :
wherein the sensor data comprises another sequence of two-dimensional images of the known three-dimensional points in the scene, and
wherein determining the second estimation of the motion of the mobile device based on the sensor data and the second timing information comprises determining another trajectory of the mobile device based on locations of the known-three dimensional points in the other sequence of two-dimensional images.

9. The method of claim 7 :
wherein the sensor data comprises a sequence of depth images, and
wherein determining the second estimation of the motion of the mobile device based on the sensor data and the second timing information comprises determining another trajectory of the mobile device based on a depth to a computed plane extracted from the sequence of depth images.

10. The method of claim 1 :
wherein the image data comprises a sequence of depth images, and
wherein determining the first estimation of the motion of the mobile device based on the image data and the first timing information comprises determining a distance to a computed ground plane extracted from the sequence of depth images.

11. The method of claim 10 , wherein the second sensor comprises a barometer.

12. The method of claim 1 , wherein the second sensor comprises a light sensor, and wherein the method further comprises:
determining a first estimation of an illumination pattern based on the image data and the first timing information;
determining a second estimation of an illumination pattern based on the sensor data and the second timing information,
wherein the output is further based on whether the first estimation of the illumination pattern is within another threshold variance of the second estimation of the illumination pattern.

13. The method of claim 1 , further comprising providing another output indicative of an accuracy of parameters of the first sensor or the second sensor based on whether the first estimation of the motion of the mobile device is within the threshold variance of the second estimation of the motion of the mobile device.

14. The method of claim 1 , further comprising:
receiving additional sensor data and third timing information associated with the additional sensor data, wherein the additional sensor data corresponds to the motion of the mobile device and is determined using a third sensor of the plurality of sensors;
determining a third estimation of the motion of the mobile device based on the additional sensor data and the third timing information;
determining whether the first estimation of the motion of the mobile device is within another threshold variance of the third estimation of the motion of the mobile device; and
wherein the output indicative of the validity of the first timing information and the second timing information is further based on whether the first estimation of the motion of the mobile device is within the other threshold variance of the third estimation of the motion of the mobile device.

15. A mobile device comprising:
at least one camera;
at least one sensor; and
a processor, the processor configured to:
receive image data and first timing information associated with the image data, wherein the image data includes images representative of a motion of the mobile device and is determined using the at least one camera,
receive sensor data and second timing information associated with the sensor data, wherein the sensor data corresponds to the motion of the mobile device and is determined using the at least one sensor,
determine a first estimation of the motion of the mobile device based on the image data and the first timing information,
determine a second estimation of the motion of the mobile device based on the sensor data and the second timing information,
determine whether the first estimation of the motion of the mobile device is within a threshold variance of the second estimation of the motion of the mobile device, and
based on whether the first estimation of the motion of the mobile device is within the threshold variance of the second estimation of the motion of the mobile device, provide an output indicative of a validity of the first timing information and the second timing information.

16. The mobile device of claim 15 :
wherein the image data comprises a sequence of two-dimensional images, and
wherein determining the first estimation of the motion of the mobile device based on the image data and the first timing information compri

Assignees

• Google Inc

Inventors

• Fung James
• Hesch Joel
• Lee Johnny

Classifications

• G06T2207/10004

  Still image; Photographic image · CPC title

• H04W4/026 Primary

  using orientation information, e.g. compass · CPC title

• G06T7/2033

  Physics · mapped topic

• G06T2207/30241

  Trajectory · CPC title

• G06T2207/10028

  Range image; Depth image; 3D point clouds · CPC title