Systems And Methods For VSLAM Scale Estimation Using Optical Flow Sensor On A Robotic Device

Patent metadata

Field	Value
Publication number	US-2021232151-A1
Application number	US-201817265455-A
Country	US
Kind code	A1
Filing date	Sep 15, 2018
Priority date	Sep 15, 2018
Publication date	Jul 29, 2021
Grant date	—

Abstract

Various embodiments include methods for improving navigation by a processor of a robotic device equipped with an image sensor and an optical flow sensor. The robotic device may be configured to capture or receive two image frames from the image sensor, generate a homograph computation based on the image frames, receive optical flow sensor data from an optical flow sensor, and determine a scale estimation value based on the homograph computation and the optical flow sensor data. The robotic device may determine the robotic device pose (or the pose of the image sensor) based on the scale estimation value.

First claim

What is claimed is:

1 . A method of performing visual simultaneous localization and mapping (VSLAM) by a processor of a robotic device, comprising:
receiving a first image frame from a monocular image sensor; receiving a second image frame from the monocular image sensor; generating a homograph computation based on the first image frame and the second image frame; receiving optical flow sensor data from an optical flow sensor; and determining a scale estimation value based on the homograph computation and the optical flow sensor data.

2 . The method of claim 1 , wherein generating the homograph computation based on the first image frame and the second image frame comprises:
generating a homography matrix information structure; and generating a transaction information structure by performing a singular value decomposition (SVD) operation on the generated homography matrix information structure.

3 . The method of claim 2 , wherein generating the homograph computation based on the first image frame and the second image frame further comprises:
identifying at least four feature points in the first image frame; identifying at least four corresponding feature points in the second image frame; and generating the homography matrix information structure to include the identified at least four feature points and the identified at least four corresponding feature points.

4 . The method of claim 1 , further comprising determining a robotic device pose of the robotic device based on the scale estimation value.

5 . The method of claim 4 , further comprising:
receiving wheel transaction data from a wheel encoder; determining a wheel-based scale estimation value based on the homograph computation and the wheel transaction data; determining a first confidence value based on the wheel-based scale estimation value; and determining a second confidence value based on the scale estimation value, wherein determining the robotic device pose based on the scale estimation value comprises:
determining the robotic device pose based on the scale estimation value in response to determining that the first confidence value does not exceed the second confidence value; and
determining the robotic device pose based on the wheel transaction data in response to determining that the first confidence value exceeds the second confidence value.

6 . The method of claim 5 , wherein:
generating the homograph computation based on the first image frame and the second image frame comprises generating a camera transaction information structure based on the first image frame and the second image frame; and determining the wheel-based scale estimation value based on the homograph computation and the wheel transaction data comprises:
generating a wheel encoder transaction information structure based on the wheel transaction data; and
determining the wheel-based scale estimation value based on values included in the camera transaction information structure and values included in the wheel encoder transaction information structure.

7 . The method of claim 1 , further comprising:
receiving wheel transaction data from a wheel encoder; determining an amount of slippage that occurred over a time period based on at least one of the wheel transaction data or the optical flow sensor data; and determining whether the determined amount of slippage exceeds a slippage threshold value, wherein determining the scale estimation value based on the homograph computation and the optical flow sensor data comprises:
determining the scale estimation value based on the homograph computation and the optical flow sensor data in response to determining that the determined amount of slippage exceeds the slippage threshold value; and
determining the scale estimation value based on the homograph computation and the wheel transaction data in response to determining that the determined amount of slippage does not exceed the slippage threshold value.

8 . The method of claim 1 , further comprising:
receiving wheel transaction data from a wheel encoder; determining whether the wheel transaction data includes erroneous or outlier data; wherein determining the scale estimation value based on the homograph computation and the optical flow sensor data comprises determining the scale estimation value based on the homograph computation and the optical flow sensor data in response to determining that the wheel transaction data includes erroneous or outlier data.

9 . The method of claim 1 , further comprising:
receiving wheel transaction data from a wheel encoder; determining a first confidence value based on the wheel transaction data; determining a second confidence value based on the optical flow sensor data; and determining whether the first confidence value exceeds the second confidence value, wherein determining the scale estimation value based on the homograph computation and the optical flow sensor data comprises:
determining the scale estimation value based on the homograph computation and the optical flow sensor data in response to determining that the first confidence value does not exceed the second confidence value; and
determining the scale estimation value based on the homograph computation and the wheel transaction data in response to determining that the first confidence value exceeds the second confidence value.

10 . The method of claim 1 , further comprising:
applying the optical flow sensor data to a Lucas-Kanade component to generate optical flow information that includes a pixel speed value and a direction value for at least one feature point; and generating an optical flow sensor transaction information structure based on the optical flow information, wherein determining the scale estimation value based on the homograph computation and the optical flow sensor data comprises determining the scale estimation value based on the homograph computation and the generated optical flow sensor transaction information structure.

11 . A robotic device, comprising:
a memory; a sensor; and a processor communicatively connected to the memory and the sensor, and configured with processor-executable instructions to:
receive a first image frame from a monocular image sensor;
receive a second image frame from the monocular image sensor;
generate a homograph computation based on the first image frame and the second image frame;
receive optical flow sensor data from an optical flow sensor; and
determine a scale estimation value based on the homograph computation and the optical flow sensor data.

12 . The robotic device of claim 11 , wherein the processor is further configured with processor-executable instructions to generate the homograph computation based on the first image frame and the second image frame by:
generating a homography matrix information structure; and generating a transaction information structure by performing a singular value decomposition (SVD) operation on the generated homography matrix information structure.

13 . The robotic device of claim 12 , wherein the processor is further configured with processor-executable instructions to generate the homograph computation based on the first image frame and the second image frame by:
identifying at least four feature points in the first image frame; identifying at least four corresponding feature points in the second image frame; and generating the homography matrix information structure to include the identified at least four feature points and the identified at least four corresponding feature points.

14 . The robotic device of claim 11 , whe

Assignees

• Qualcomm Inc

Inventors

• Liu Jun
• Wang Zixiang
• Luo Weizhang
• Fang Mei Wu
• Luo Tiequan

Classifications

• G05D1/2435

  Extracting 3D information · CPC title

• G06T2207/30252

  Vehicle exterior; Vicinity of vehicle · CPC title

• G06T2207/10016

  Video; Image sequence · CPC title

• G06T7/70

  Determining position or orientation of objects or cameras (camera calibration G06T7/80) · CPC title

• G06T7/579

  from motion · CPC title