Multi-sensor environmental mapping

Patent metadata

Field	Value
Publication number	US-10901419-B2
Application number	US-201916574848-A
Country	US
Kind code	B2
Filing date	Sep 18, 2019
Priority date	Sep 5, 2014
Publication date	Jan 26, 2021
Grant date	Jan 26, 2021

Abstract

A method for controlling an unmanned aerial vehicle (UAV) includes receiving first sensor data relative to a first coordinate system and second sensor data relative to a second coordinate system from a first sensor and a second sensor, respectively. The first and second sensor data includes first and second obstacle occupancy information indicative of relative locations of a first and a second sets of obstacles in reference to the UAV in the first and second coordinate systems, respectively. The first and second sets of obstacles have at least a subset of obstacles in common. The method further includes converting the first and second sensor data into a single coordinate system using sensor calibration data to generate an obstacle occupancy grid map based on the first and second obstacle occupancy information, and effecting the UAV to navigate using the obstacle occupancy grid map to perform obstacle avoidance.

First claim

What is claimed is:

1. A method for controlling an unmanned aerial vehicle (UAV), the method comprising:
receiving first sensor data relative to a first coordinate system from a first sensor, the first sensor data comprising first obstacle occupancy information indicative of relative locations of a first set of obstacles in reference to the UAV in the first coordinate system, wherein the first sensor is selected from a plurality of sensors carried by the UAV;
receiving second sensor data relative to a second coordinate system from a second sensor, the second sensor data comprising second obstacle occupancy information indicative of relative locations of a second set of obstacles in reference to the UAV in the second coordinate system, wherein the second sensor is selected from the plurality of sensors and is different from the first sensor, and wherein the first set of obstacles and the second set of obstacles have at least a subset of obstacles in common;
generating, by converting the first sensor data and the second sensor data into a single coordinate system using sensor calibration data, an obstacle occupancy grid map based on the first obstacle occupancy information and the second obstacle occupancy information, the obstacle occupancy grid map comprising a plurality of obstructed spaces and a plurality of unobstructed spaces; and
effecting the UAV to navigate using the obstacle occupancy grid map to perform obstacle avoidance based on the plurality of obstructed spaces and the plurality of unobstructed spaces.

2. The method of claim 1 , wherein converting the first sensor data and the second sensor data into the single coordinate system using the sensor calibration data, further comprises:
obtaining first coordinates of the subset of obstacles in common from the first sensor data based on the first obstacle occupancy information;
obtaining second coordinates of the subset of obstacles in common from the second sensor data based on the second obstacle occupancy information;
computing the sensor calibration data comprising sensor parameters indicative of a spatial relationship between the first sensor and the second sensor, the sensor parameters comprising a rotation matrix and a transform matrix for converting between the first coordinate system and the second coordinate system, wherein the rotation matrix and the transform matrix are computed based on the first coordinates and the second coordinates of the subset of obstacles in common; and
converting the first sensor data and the second sensor data into the single coordinate system using the rotation matrix and the transform matrix.

3. The method of claim 2 , wherein converting the first sensor data and the second sensor data into the single coordinate system further comprises using an internal matrix or a scale factor, wherein the internal matrix is associated with intrinsic parameters calculated in prior sensor calibration, and wherein the scale factor is calculated based at least on coordinates of an obstacle within the subset of obstacles in common along a single sensing direction.

4. The method of claim 1 , further comprising:
determining a current location of the UAV using at least one of the plurality of sensors;
receiving, from a remote controller or a mobile device, instructions to autonomously return to an initial location or fly to a specified location; and
determining a flight path among a plurality of potential flight paths from the current location to the initial location or the specified location based on a set of operational criteria.

5. The method of claim 4 , further comprising:
detecting obstacles along the flight path using the obstacle occupancy grid map;
modifying the flight path based on the detected obstacles to avoid navigating within the obstructed spaces of the obstacle occupancy grid map; and
effecting the UAV to move along the modified flight path to autonomously return to the initial location or fly to the specified location.

6. The method of claim 5 , wherein modifying the flight path based on the detected obstacles further comprises:
for each detected obstacle along the flight path, calculating confidence information based on a collision risk, wherein the collision risk is associated with a minimum distance between the detected obstacle and the flight path; and
modifying the flight path based on the confidence information.

7. The method of claim 4 , wherein the set of operational criteria is associated with minimizing at least one of a flight time, a flight distance, an energy consumption, a number of obstacles along each flight path of the plurality of potential flight paths, distances of the UAV from obstacles along each flight path of the plurality of potential flight paths, or an altitude range of the UAV.

8. The method of claim 1 , wherein the plurality of sensors are selected from: a vision sensor, an image sensor, a camera, a proximity sensor, a range sensor, a light sensor, a LiDAR sensor, an acoustic sensor, an ultrasonic sensor, a location sensor, a Global Positioning System (GPS) sensor, a speed sensor, a pressure sensor, a thermal sensor, and an environmental sensor.

9. The method of claim 1 , wherein the single coordinate system is one of a local coordinate system, a global coordinate system, a world coordinate system, or a body coordinate system defined relative to the UAV.

10. A system for controlling an unmanned aerial vehicle (UAV), the system comprising:
a plurality of sensors carried by the UAV; and
one or more processors, individually or collectively configured to:
receive first sensor data relative to a first coordinate system from a first sensor of the plurality of sensors, the first sensor data comprising first obstacle occupancy information indicative of relative locations of a first set of obstacles in reference to the UAV in the first coordinate system;
receive second sensor data relative to a second coordinate system from a second sensor of the plurality of sensors, the second sensor data comprising second obstacle occupancy information indicative of relative locations of a second set of obstacles in reference to the UAV in the second coordinate system, wherein the second sensor is different from the first sensor, and wherein the first set of obstacles and the second set of obstacles have at least a subset of obstacles in common;
generate, by converting the first sensor data and the second sensor data into a single coordinate system using sensor calibration data, an obstacle occupancy grid map based on the first obstacle occupancy information and the second obstacle occupancy information, the obstacle occupancy grid map comprising a plurality of obstructed spaces and a plurality of unobstructed spaces; and
effect the UAV to navigate using the obstacle occupancy grid map to perform obstacle avoidance based on the plurality of obstructed spaces and the plurality of unobstructed spaces.

11. The system of claim 10 , wherein to convert the first sensor data and the second sensor data into the single coordinate system using the sensor calibration data, the one or more processors are, individually or collectively further configured to:
obtain first coordinates of the subset of obstacles in common from the first sensor data based on the first obstacle occupancy information;
obtain second coordinates of the subset of obstacles in common from the second sensor data based on the second obstacle occupancy information;
compute the sensor calibration data comprising sensor parameters indicative of a spatial relationship between the first sensor and the second sensor, the sensor parameters comprising a rotation matrix and a transform matrix for converting between the first co

Assignees

• Sz Dji Technology Co Ltd

Inventors

• Liu Ang
• Mo Weiyu
• Ling Yonggen

Classifications

• B64U2201/10

  autonomous, i.e. by navigating independently from ground or air stations, e.g. by using inertial navigation systems [INS] · CPC title

• G01C21/3848

  Data obtained from both position sensors and additional sensors · CPC title

• B64C39/024

  of the remote controlled vehicle type, i.e. RPV · CPC title

• B64U2201/20

  Remote controls · CPC title

• G01C21/28

  with correlation of data from several navigational instruments · CPC title