Power efficient use of a depth sensor on a mobile device

Patent metadata

Field	Value
Publication number	US-9336440-B2
Application number	US-201414552183-A
Country	US
Kind code	B2
Filing date	Nov 24, 2014
Priority date	Nov 25, 2013
Publication date	May 10, 2016
Grant date	May 10, 2016

Abstract

Systems, apparatus and methods in a mobile device to enable and disable a depth sensor for tracking pose of the mobile device are presented. A mobile device relaying on a camera without a depth sensor may provide inadequate pose estimates, for example, in low light situations. A mobile device with a depth sensor uses substantial power when the depth sensor is enabled. Embodiments described herein enable a depth sensor only when images are expected to be inadequate, for example, accelerating or moving too fast, when inertial sensor measurements are too noisy, light levels are too low or high, an image is too blurry, or a rate of images is too slow. By only using a depth sensor when images are expected to be inadequate, battery power in the mobile device may be conserved and pose estimations may still be maintained.

First claim

What is claimed is:

1. A method in a mobile device for tracking pose of the mobile device, the method comprising:
receiving, from an image capture device, a plurality of images;
computing a pose based at least in part on one of the plurality of images;
sending the pose to a pose tracking filter module;
comparing a first measurement to a first threshold to determine that the first measurement exceeds the first threshold;
enabling a depth sensor based, at least in part, on comparing the first measurement;
receiving a first depth map;
computing a depth sensor pose based in part on the first depth map; and
providing the depth sensor pose to the pose tracking filter module.

2. The method of claim 1 , further comprising disabling the depth sensor.

3. The method of claim 2 , wherein disabling the depth sensor comprises:
comparing a second measurement to the first threshold to determine that the second measurement no longer exceeds the first threshold; and
disabling the depth sensor based on comparing the second measurement.

4. The method of claim 1 , further comprising:
capturing a second depth map;
wherein computing the depth sensor pose based in part on the first depth map comprises computing the depth sensor pose from the first depth map and the second depth map.

5. The method of claim 1 , wherein the first measurement comprises a rate of angular velocity or a rate of angular acceleration obtained from inertial sensor measurements.

6. The method of claim 1 , wherein the first measurement comprises a noise level of inertial sensor measurements.

7. The method of claim 1 , wherein the first measurement comprises a light level.

8. The method of claim 1 , wherein the first measurement comprises an indication of blurriness.

9. The method of claim 1 , wherein the first measurement comprises a frame rate of images.

10. The method of claim 1 , wherein the pose tracking filter module comprises an extended Kalman filter (EKF).

11. The method of claim 1 , wherein the pose tracking filter module comprises a particle filter.

12. The method of claim 1 , further comprising computing an estimated pose from the pose tracking filter module.

13. A mobile device for tracking pose, the mobile device comprising:
an image capture device configured to capture a plurality of images;
a pose module coupled to the image capture device and configured to compute a pose based at least in part on one of the plurality of images;
a depth sensor configured to send a first depth map;
a comparator module configured to compare a first measurement to a first threshold to determine that the first measurement exceed the first threshold;
a depth sensor trigger module coupled to the depth sensor and the comparator module, and configured to enable the depth sensor based, at least in part, on comparing the first measurement;
a depth sensor pose module coupled to the depth sensor and configured to compute a depth sensor pose based in part on the first depth map; and
a pose tracking filter module:
coupled to the pose module and the depth sensor pose module;
configured to receive the pose and the depth sensor pose; and
provide an estimated pose.

14. The mobile device of claim 13 , further comprising an inertial sensor, wherein the first measurement comprises a measurement from the inertial sensor.

15. The mobile device of claim 13 , further comprising a light sensor, wherein the first measurement comprises a measurement from the light sensor.

16. The mobile device of claim 13 , wherein the first measurement comprises an indication of blur from the image capture device.

17. The mobile device of claim 13 , wherein the first measurement comprises a frame rate of images from the image capture device.

18. The mobile device of claim 13 , further comprising a processor and memory couple to the image capture device and the depth sensor, and comprising code for:
the comparator module;
the depth sensor trigger module;
the depth sensor pose module; and
the pose tracking filter module.

19. A mobile device for tracking pose of the mobile device, the mobile device comprising:
means for receiving, from an image capture device, a plurality of images;
means for computing a pose based at least in part on one of the plurality of images;
means for sending the pose to a pose tracking filter module;
means for comparing a first measurement to a first threshold to determine that the first measurement exceeds the first threshold;
means for enabling a depth sensor based, at least in part, on comparing the first measurement;
means for receiving a first depth map;
means for computing a depth sensor pose based in part on the first depth map; and
means for providing the depth sensor pose to the pose tracking filter module.

20. The mobile device of claim 19 , further comprising means for disabling the depth sensor comprises:
means for comparing a second measurement to the first threshold to determine that the second measurement exceeds the first threshold; and
means for disabling the depth sensor based on comparing the second measurement.

21. The mobile device of claim 19 , wherein the first measurement comprises at least one of:
a rate of linear acceleration from inertial sensor measurements;
a rate of angular acceleration from inertial sensor measurements;
a noise level of inertial sensor measurements;
a light level;
an indication of blurriness; or
a frame rate of images.

22. The mobile device of claim 19 , further comprising means for computing an estimated pose from the pose tracking filter module.

23. A non-transient computer-readable storage medium including program code stored thereon, comprising program code for a mobile device for tracking pose of the mobile device, the program code comprising code to:
receive, from an image capture device, a plurality of images;
compute a pose based at least in part on one of the plurality of images;
send the pose to a pose tracking filter module;
compare a first measurement to a first threshold to determine that the first measurement exceed the first threshold;
enable a depth sensor based, at least in part, on comparing the first measurement;
receive a first depth map;
compute a depth sensor pose based in part on the first depth map; and
provide the depth sensor pose to the pose tracking filter module.

24. The non-transient computer-readable storage medium of claim 23 , wherein the first measurement comprises at least one of:
a rate of linear acceleration from inertial sensor measurements;
a rate of angular acceleration from inertial sensor measurements;
a noise level of inertial sensor measurements;
a light level;
an indication of blurriness; or
a frame rate of images.

25. The non-transient computer-readable storage medium of claim 23 , wherein the program code further comprises code to compute an estimated pose from the pose tracking filter module.

26. The non-transient computer-readable storage medium of claim 23 , wherein the program code further comprises code to:
compare a second measurement to the first threshold to set a second comparison; and
disable the depth sensor based on the second comparison.

Assignees

• Qualcomm Inc

Inventors

• Ramachandran Mahesh
• Sadasivam Shankar
• Swaminathan Ashwin

Classifications

• G06F1/1694 Primary

  the I/O peripheral being a single or a set of motion sensors for pointer control or gesture input obtained by sensing movements of the portable computer · CPC title

• H04N23/68

  for stable pick-up of the scene, e.g. compensating for camera body vibrations · CPC title

• G06T7/277

  involving stochastic approaches, e.g. using Kalman filters · CPC title

• G06T2207/20076

  Probabilistic image processing · CPC title

• G06F3/012

  Head tracking input arrangements · CPC title