Localization of a robot in an environment using detected edges of a camera image from a camera of the robot and detected edges derived from a three-dimensional model of the environment

Patent metadata

Field	Value
Publication number	US-9849591-B2
Application number	US-201514873430-A
Country	US
Kind code	B2
Filing date	Oct 2, 2015
Priority date	Oct 2, 2015
Publication date	Dec 26, 2017
Grant date	Dec 26, 2017

Abstract

Methods, apparatus, systems, and computer-readable media are provided for using a camera of a robot to capture an image of the robot's environment, detecting edges in the image, and localizing the robot based on comparing the detected edges in the image to edges derived from a three-dimensional (“3D”) model of the robot's environment from the point of view of an estimated pose of the robot in the environment. In some implementations, the edges are derived based on rendering, from the 3D model of the environment, a model image of the environment from the point of view of the estimated pose—and applying an edge detector to the rendered model image to detect model image edges from the model image.

First claim

What is claimed is:

1. A method, comprising:
capturing a camera image by a camera of a robot in an environment;
applying, by one or more processors of the robot, an edge detector to the camera image to detect camera image edges in the camera image;
identifying, by one or more of the processors, an estimated pose of the camera in the environment;
rendering, by one or more of the processors and from an electronically stored three-dimensional model of the environment, a model image of the environment from a point of view of the estimated pose;
applying, by one or more of the processors, the edge detector to the model image of the environment to detect model image edges in the model image;
determining, by one or more of the processors, a current pose of the camera in the environment based on comparing the camera image edges to the model image edges; and
localizing, by one or more of the processors, the robot in the environment based on the current pose of the camera;
wherein determining the current pose of the camera in the environment based on comparing the camera image edges to the model image edges comprises:
determining differences between the camera image edges and the model image edges;
determining a candidate pose of the camera based on the differences;
rendering, from the electronically stored three-dimensional model of the environment, an additional model image of the environment from the point of view of the candidate pose;
applying the edge detector to the additional model image of the environment to detect additional model image edges in the additional model image;
determining additional differences between the camera image edges and the additional model image edges; and
using the candidate pose as the current pose of the camera if the additional differences satisfy a threshold.

2. The method of claim 1 , wherein the estimated pose of the camera in the environment is based on an immediately preceding pose of the camera, wherein the immediately preceding pose of the camera is determined based on comparing preceding camera image edges of a preceding camera image captured by the camera to preceding model image edges of a preceding model image rendered from the point of view of a preceding estimated pose.

3. The method of claim 2 , wherein the estimated pose of the camera in the environment is determined based on modifying the immediately preceding pose based on sensor data from one or more additional sensors of the robot.

4. The method of claim 3 , wherein the one or more additional sensors include an inertial measurement unit.

5. The method of claim 1 , further comprising:
capturing an additional camera image by the camera following the capturing of the camera image;
applying, by one or more of the processors, the edge detector to the additional camera image to detect additional camera image edges in the additional camera image;
determining an additional estimated pose of the camera in the environment based on the determined current pose of the camera in the environment;
rendering, by one or more of the processors and from the electronically stored three-dimensional model of the environment, a further model image of the environment from the point of view of the additional estimated pose;
applying, by one or more of the processors, the edge detector to the further model image of the environment to detect further model image edges in the further model image; and
determining, by one or more of the processors, a new current pose of the camera in the environment based on comparing the additional camera image edges to the further model image edges.

6. The method of claim 5 , wherein determining the additional estimated pose based on the determined current pose of the camera in the environment includes defining the additional estimated pose as the determined current pose of the camera in the environment.

7. The method of claim 5 , further comprising:
receiving sensor data from one or more additional sensors of the robot;
wherein determining the additional estimated pose based on the determined current pose of the camera includes determining the additional estimated pose based on the determined current pose of the camera and based on the sensor data.

8. The method of claim 1 , wherein the camera image is captured by a single sensor of the camera.

9. The method of claim 8 , wherein the camera is a monocular camera.

10. The method of claim 1 , wherein the one or more processors include a graphics processor unit.

11. The method of claim 1 , wherein the three-dimensional model of the environment is a computer aided design model.

12. The method of claim 1 , wherein the current pose is a six degree of freedom pose.

13. A robot comprising:
at least one non-transitory computer readable storage medium storing a three-dimensional model of an environment;
a camera;
one or more processors in communication with the non-transitory computer readable storage medium and receiving camera images captured by the camera, wherein the one or more processors are configured to:
apply an edge detector to a camera image of the camera images to detect camera image edges in the camera image;
identify an estimated pose of the camera in the environment;
render, from the three-dimensional model of the environment, a model image of the environment from a point of view of the estimated pose;
apply the edge detector to the model image of the environment to detect model image edges in the model image;
determine a current pose of the camera in the environment based on comparing the camera image edges to the model image edges; and

localize the robot in the environment based on the current pose of the camera
wherein in determining the current pose of the camera in the environment based on comparing the camera image edges to the model image edges one or more of the processors are configured to:
determine differences between the camera image edges and the model image edges;
determine a candidate pose of the camera based on the differences;
render, from the three-dimensional model of the environment, an additional model image of the environment from the point of view of the candidate pose;
apply the edge detector to the additional model image of the environment to detect additional model image edges in the additional model image;
determine additional differences between the camera image edges and the additional model image edges; and
use the candidate pose as the current pose of the camera if the additional differences satisfy a threshold.

14. The robot of claim 13 , wherein the estimated pose of the camera in the environment is based on an immediately preceding pose of the camera, wherein the immediately preceding pose is determined based on comparing preceding camera image edges of a preceding camera image captured by the camera to preceding model image edges of a preceding model image rendered from the point of view of a preceding estimated pose.

15. The robot of claim 14 , wherein the estimated pose of the camera in the environment is determined based on modifying the immediately preceding pose based on sensor data from one or more additional sensors of the robot.

16. The robot of claim 13 , wherein the one or more processors are further configured to:
apply the edge detector to an additional camera image of the images captured following the capturing of the image to detect additional camera image edges from the additional camera image;
determine an additional estimated pose of the camera in the environment based on the determined current pose of the camera in the environment;
re

Assignees

• X Dev Llc

Inventors

• Michel Philipp

Classifications

• G06T17/05

  Geographic models · CPC title

• G06T2207/10028

  Range image; Depth image; 3D point clouds · CPC title

• G05G9/04737

  with six degrees of freedom · CPC title

• G06T19/20

  Editing of three-dimensional [3D] images, e.g. changing shapes or colours, aligning objects or positioning parts · CPC title

• G05B2219/39046

  Compare image of plate on robot with reference, move till coincidence, camera · CPC title