Electronic device pose identification based on imagery and non-image sensor data

Patent metadata

Field	Value
Publication number	US-2017102772-A1
Application number	US-201514877038-A
Country	US
Kind code	A1
Filing date	Oct 7, 2015
Priority date	Oct 7, 2015
Publication date	Apr 13, 2017
Grant date	—

Abstract

An electronic device includes one or more imaging sensors (e.g, imaging cameras) and includes one or more non-image sensors, such as an inertial measurement unit (IMU), that can provide information indicative of the pose of the electronic device. The electronic device estimates its pose based on two independent sources of pose information: pose information generated at a relatively high rate based on non-visual information generated by the non-image sensors and pose information generated at a relatively low rate based on imagery captured by the one or more imaging sensors. To achieve both a high pose-estimation rate and high degree of pose estimation accuracy, the electronic device adjusts a pose estimate based on the non-visual pose information at a high rate, and at a lower rate spatially smoothes the pose estimate based on the visual pose information.

First claim

What is claimed is:

1 . A method comprising:
updating, at an electronic device, a first estimated pose of the electronic device at a first rate based on imagery captured at the electronic device and at a second rate based on received non-visual pose information obtained from non-image sensors, the second rate different from the first rate; and employing the first estimated pose to support location-based functionality at the electronic device.

2 . The method of claim 1 , wherein updating the first estimated pose of the electronic device based on imagery comprises:
identifying a second estimated pose of the electronic device based on the imagery captured at the electronic device; identifying a difference between the first estimated pose and the second estimated pose along a first axis of a frame of reference of the electronic device; and adjusting the first estimated pose along the first axis based on the difference between the first estimated pose and the second estimated pose.

3 . The method of claim 2 , wherein updating the first estimated pose of the electronic device based on imagery comprises:
adjusting the first estimated pose along each of a plurality of axes of the frame of reference based on the difference between the first estimated pose and the second estimated pose.

4 . The method of claim 2 wherein updating the first estimated pose along the first axis comprises:
scaling the first estimated pose along the first axis based on the difference.

5 . The method of claim 2 , wherein updating the first estimated pose along the first axis comprises:
setting a value of the first estimated pose along the first axis to a corresponding value of the second estimated pose.

6 . The method of claim 2 , wherein updating the first estimated pose along the first axis comprises:
selecting the first axis based on a degree of motion along the first axis indicated by the difference between the first estimated pose and the second estimated pose.

7 . The method of claim 2 , wherein updating the first estimated pose along the first axis comprises:
selecting the first axis based upon a predicted likelihood that pose errors along the first axis will be perceived by a user of the electronic device.

8 . The method of claim 2 , wherein updating the first estimated pose along the first axis comprises:
in response to identifying that the difference exceeds a threshold, setting the difference to the threshold value.

9 . The method of claim 1 , wherein updating the first estimated pose of the electronic device based on received non-visual pose information comprises:
identifying a second estimated pose of the electronic device based on the received non-visual pose information; and in response to determining that a difference between the first estimated pose and the second estimated pose is below a threshold, refraining from modifying the first estimated pose.

10 . The method of claim 1 , wherein updating the first estimated pose of the electronic device based on imagery comprises:
identifying a second estimated pose of the electronic device based on the imagery captured at the electronic device and based on non-visual pose information; identifying a difference between the first estimated pose and the second estimated pose along a first axis of a frame of reference of the electronic device; and adjusting the first estimated pose along the first axis based on the difference between the first estimated pose and the second estimated pose.

11 . A method, comprising,
generating at an electronic device a first set of estimated poses at a first rate based on non-visual pose information obtained from non-image sensors; generating a second set of estimated poses at a second rate different from the first rate based on imagery captured at the electronic device; and spatially smoothing, at the electronic device, the first set of estimated poses based on the second set of estimated poses.

12 . The method of claim 11 , wherein the first rate is at least five times the second rate.

13 . The method of claim 11 , further comprising:
generating the second set of estimated poses based on the imagery and based on geolocation data received at the electronic device.

14 . The method of claim 11 , wherein spatially smoothing the first set of estimated poses comprises:
scaling an estimated pose of the first set of estimated poses based on differences between the first estimated pose and an estimated pose of the second set of estimated poses.

15 . An electronic device comprising:
an imaging camera to capture imagery; an inertial measurement unit (IMU) to generate non-visual pose information; and a processor to update a first estimated pose of the electronic device at a first rate based on the imagery and at a second rate based on received non-visual pose information obtained from non-image sensors, the second rate different from the first rate.

16 . The electronic device of claim 15 , wherein the processor is to update the estimated pose of the electronic device based on the imagery by:
identifying a second estimated pose of the electronic device based on the imagery; identifying a difference between the first estimated pose and the second estimated pose along a first axis of a frame of reference of the electronic device; and adjusting the first estimated pose along the first axis based on the difference between the first estimated pose and the second estimated pose.

17 . The electronic device of claim 16 , wherein the processor is to update the estimated pose of the electronic device based on imagery by:
adjusting the first estimated pose along each of a plurality of axes of the frame of reference based on the difference between the first estimated pose and the second estimated pose.

18 . The electronic device of claim 16 , wherein the processor is to update the first estimated pose of the electronic device based on imagery by:
scaling the first estimated pose along the first axis based on the difference.

19 . The electronic device of claim 16 , wherein the processor is to update the first estimated pose along the first axis by:
setting a value of the first estimated pose along the first axis to a corresponding value of the second estimated pose.

20 . The electronic device of claim 16 , wherein the processor is to update the first estimated pose along the first axis by:
selecting the first axis based on a degree of motion along the first axis indicated by the difference between the first estimated pose and the second estimated pose.

Assignees

• Google Inc

Inventors

• Hesch Joel
• Nerurkar Esha

Classifications

• H04N23/80

  Camera processing pipelines; Components thereof · CPC title

• H04N23/951

  by using two or more images to influence resolution, frame rate or aspect ratio · CPC title

• H04N23/957

  Light-field or plenoptic cameras or camera modules · CPC title

• H04N5/23229

  Electricity · mapped topic

• G01P15/093

  by photoelectric pick-up · CPC title