Multi-sensor fusion for robust autonomous flight in indoor and outdoor environments with a rotorcraft micro-aerial vehicle (MAV)

Patent metadata

Field	Value
Publication number	US-10732647-B2
Application number	US-201715684700-A
Country	US
Kind code	B2
Filing date	Aug 23, 2017
Priority date	Nov 27, 2013
Publication date	Aug 4, 2020
Grant date	Aug 4, 2020

Abstract

The subject matter described herein includes a modular and extensible approach to integrate noisy measurements from multiple heterogeneous sensors that yield either absolute or relative observations at different and varying time intervals, and to provide smooth and globally consistent estimates of position in real time for autonomous flight. We describe the development of the algorithms and software architecture for a new 1.9 kg MAV platform equipped with an IMU, laser scanner, stereo cameras, pressure altimeter, magnetometer, and a GPS receiver, in which the state estimation and control are performed onboard on an Intel NUC 3 rd generation i3 processor. We illustrate the robustness of our framework in large-scale, indoor-outdoor autonomous aerial navigation experiments involving traversals of over 440 meters at average speeds of 1.5 m/s with winds around 10 mph while entering and exiting buildings.

First claim

What is claimed is:

1. A system that enables autonomous control of an aerial vehicle in indoor and outdoor environments, the system comprising:
a sensor fusion module for combining measurements from a plurality of sensors of different modalities to estimate a current state of the aerial vehicle given current and previous measurements from the sensors and a previous estimated state of the aerial vehicle, wherein the sensors include a global positioning system (GPS) receiver located on the aerial vehicle, wherein the sensor fusion module is configured to maintain smoothness in the state estimates of the aerial vehicle when GPS measurements from the GPS receiver become available after a period of unavailability by transforming a GPS measurement from the GPS receiver to be aligned with a simultaneous localization and mapping (SLAM) formulation of vehicle pose, wherein transforming the GPS measurement includes determining an optimal configuration of a pose graph using an incremental motion constraint from visual odometry, using the GPS measurement as absolute pose constraint, and using a spatial loop closure constraint, wherein the sensors include an inertial measurement unit (IMU), a laser scanner, and a camera and wherein the sensor fusion module converts relative measurements generated by the laser scanner and the camera into measurements that depend on augmented states; and
a trajectory generator for generating a plan for controlling a trajectory of the aerial vehicle based on the estimated current state and a goal or a waypoint input provided by either a user or a higher level planner.

2. The system of claim 1 wherein the sensors include at least one of a pressure altimeter, a magnetometer, and a downward facing optical sensor.

3. The system of claim 1 wherein the sensor fusion module is configured to use an Unscented Kalman Filter (UKF) to combine the measurements from the sensors of different modalities, enabling addition and removal of sensors with minimal reconfiguration of software of the sensor fusion module.

4. The system of claim 3 wherein the sensor fusion module is configured to estimate the current state using current relative measurements and copies of augmented past states in the filter.

5. The system of claim 3 wherein the sensor fusion module is configured to fuse measurements from the sensors that arrive out of order to the filter.

6. The system of claim 1 wherein the aerial vehicle comprises a rotorcraft micro-aerial vehicle (MAV).

7. A method that enables autonomous control of an aerial vehicle in indoor and outdoor environments, the method comprising:
combining measurements from a plurality of sensors of different modalities to generate an estimate of a current state of the aerial vehicle given current measurements from the sensors and a previous estimated state of the aerial vehicle, wherein the sensors include an inertial measurement unit (IMU), a laser scanner, a global positioning system (GPS) receiver located on the aerial vehicle, and a camera and wherein combining the measurements includes converting relative measurements generated by the laser scanner and the camera into measurements that depend on augmented state;
generating a signal for planning a trajectory of the aerial vehicle based on the estimated current state and a goal or waypoint input by a user or a higher level planner; and
smoothing changes in state of the aerial vehicle when GPS measurements from the GPS receiver become available after a period of unavailability by transforming a GPS measurement from the GPS receiver to be aligned with a simultaneous localization and mapping (SLAM) formulation of vehicle pose, wherein transforming the GPS measurement includes determining an optimal configuration of a pose graph using an incremental motion constraint from visual odometry, using the GPS measurement as absolute pose constraint, and using a spatial loop closure constraint.

8. The method of claim 7 wherein the sensors include at least one of a pressure altimeter, and a magnetometer.

9. The method of claim 7 wherein combining the measurements includes an Unscented Kalman Filter (UKF) to combine the measurements from the sensors of different modalities, enabling addition and removal of sensors with minimal reconfiguration of the sensor fusion module.

10. The method of claim 9 wherein estimating the current state includes using current relative measurement and copies of augmented past states in the filter.

11. The method of claim 9 comprising fusing measurements from the sensors that arrive out of order at the filter.

12. The method of claim 7 wherein the aerial vehicle comprises a rotorcraft micro-aerial vehicle (MAV).

13. A system that enables autonomous control of an aerial vehicle in indoor and outdoor environments, the system comprising:
a sensor fusion module for combining measurements from a plurality of sensors of different modalities to estimate a current state of the aerial vehicle given current and previous measurements from the sensors and a previous estimated state of the vehicle, wherein the sensors include a global positioning system (GPS) receiver located on the aerial vehicle, wherein the sensor fusion module is configured to maintain smoothness in the state estimates of the aerial vehicle when GPS measurements from the GPS receiver become available after a period of unavailability by transforming a GPS measurement from the GPS receiver to be aligned with a simultaneous localization and mapping (SLAM) formulation of vehicle pose, wherein transforming the GPS measurement includes determining an optimal configuration of a pose graph using an incremental motion constraint from visual odometry, using the GPS measurement as absolute pose constraint, and using a spatial loop closure constraint, wherein the sensor fusion module is configured to use an Unscented Kalman Filter (UKF) to combine the measurements from the sensors of different modalities, enabling addition and removal of sensors with minimal reconfiguration of software of the sensor fusion module, and wherein the sensor fusion module is configured to estimate the current state using current relative measurements and copies of augmented past states in the filter, and wherein the sensor fusion module is configured to remove augmented states from the filter and add new augmented states to the filter; and
a trajectory generator for generating a plan for controlling a trajectory of the aerial vehicle based on the estimated current state and a goal or a waypoint input provided by either a user or a higher level planner.

14. A method that enables autonomous control of an aerial vehicle in indoor and outdoor environments, the method comprising:
combining measurements from a plurality of sensors of different modalities to generate an estimate of a current state of the aerial vehicle given current measurements from the sensors and a previous estimated state of the aerial vehicle, wherein the sensors include a global positioning system (GPS) receiver located on the aerial vehicle and wherein combining the measurements includes using an Unscented Kalman Filter (UKF) to combine the measurements from the sensors of different modalities, enabling addition and removal of sensors with minimal reconfiguration of the sensor fusion module, wherein estimating the current state includes using current relative measurement and copies of augmented past states in the filter;
generating a signal for planning a trajectory of the aerial vehicle based on the estimated current state and a goal or waypoint input by a user or a higher level planner;
smoothing changes in state of the aerial vehicle when GPS measuremen

Assignees

• Univ Pennsylvania

Inventors

• Shen Shaojie

Classifications

• B64U30/20

  Rotors; Rotor supports · CPC title

• B64U2201/10

  autonomous, i.e. by navigating independently from ground or air stations, e.g. by using inertial navigation systems [INS] · CPC title

• G01C21/1656

  with passive imaging devices, e.g. cameras · CPC title

• G01C21/1654

  with electromagnetic compass · CPC title

• G01S19/485

  whereby the further system is an optical system or imaging system · CPC title