Planning system and method for controlling operation of an autonomous vehicle to navigate a planned path

Patent metadata

Field	Value
Publication number	US-10796204-B2
Application number	US-201815905705-A
Country	US
Kind code	B2
Filing date	Feb 26, 2018
Priority date	Feb 27, 2017
Publication date	Oct 6, 2020
Grant date	Oct 6, 2020

Abstract

A multi layer learning based control system and method for an autonomous vehicle or mobile robot. A mission planning layer, behavior planning layer and motion planning layer each having one or more neural neworks are used to develop an optimal route for the autonomous vehicle or mobile robot, provide a series of functional tasks associated with at least one or more of the neural networks to follow the planned optimal route and develop commands to implement the functional tasks.

First claim

What is claimed is:

1. A planning system for vehicle, the planning system comprising:
a plurality of hierarchal software layers including:
a mission planning layer comprising one or more neural networks configured to determine an optimal route for the vehicle or mobile robot based on a start point, an end point and a digital map of an environment surrounding the vehicle;
a behaviour planning layer comprising one or more neural networks, the behaviour planning layer configured to receive the optimal route determined by the mission planning layer and sensor data sensed by a plurality of sensors of the vehicle, each neural network of the behaviour planning layer configured to predict a respective behavior task for the vehicle based on the sensor data and the optimal route; and
a motion planning layer comprising one or more neural networks, the motion planning layer configured to receive each behaviour task predicted by the behaviour planning layer and the sensor data, each neural network of the motion planning layer configured to predict a respective motion task for the vehicle based on the received behavior tasks and the sensor data, wherein the behaviour planning layer is feed-associated with the motion planning layer, and wherein the behaviour planning layer is configured to feed-forward information to the motion planning layer and receive feedback of information from the motion planning layer.

2. The planning system of claim 1 , wherein the mission planning layer is feed-associated with the behaviour planning layer, and wherein the mission planning layer is configured to receive feedback of at least one of information, control, and metadata from the behaviour planning layer.

3. The planning system of claim 1 , wherein each neural network of the behaviour planning layer is feed-associated with each neural network of the a-hierarchically adjacent motion planning layer.

4. The planning system of claim 1 wherein the mission planning task comprises one or more neural networks configured to determining one or more checkpoints along the optimal route, and calculating any associated tolls along the optimal route.

5. The planning system of claim 1 wherein the respective behavior task is associated with a respective behavior of the autonomous vehicle.

6. The planning system of claim 5 wherein each respective behavior task comprises one of: changing a lane, waiting at an intersection, passing another vehicle or mobile robot, giving way to the another vehicle or mobile robot, or waiting at an intersection.

7. The planning system of claim 1 wherein each respective motion task comprising one of avoiding an obstacle, finding a local path, or controlling the speed, direction or position of the vehicle or the mobile robot.

8. The planning system of claim 7 wherein the respective motion task is performed by controlling at least one operable element of the vehicle, wherein the at least one operable element includes a GPS unit, steering unit, a brake unit, or a throttle unit.

9. The planning system of claim 1 wherein the sensor data comprises one or more of image data, LIDAR data, RADAR data, global positioning system (GPS) data, and inertial measurement unit (IMU) data, and processed sensor data.

10. The system of claim 1 wherein the sensor data includes data sensed by one or more of a camera, a LIDAR system, a RADAR system, a global position system, and an inertial measurement unit of the vehicle.

11. The planning system of claim 1 wherein each of the one or more neural networks of each of the mission planning layer, the behaviour planning layer, and the motion planning layer are trained offline with new training data before autonomous operation of the vehicle or mobile robot.

12. The planning system of claim 1 wherein the vehicle comprises a mobile robot, an autonomous vehicle, an autonomous robot, or autonomous drone.

13. The planning system of claim 1 , wherein the plurality of hierarchical software layers further comprise a safety planning layer comprising one or more neural networks, each of the one or more neural networks of the safety planning a respective safety task, the respective safety task comprising determining whether a motion planning task corresponding to the respective safety task is safe.

14. The planning system of claim 1 , wherein the mission planning layer is hierarchically adjacent to the behaviour planning layer and the behaviour planning layer is hierarchically adjacent to the motion planning layer.

15. The planning system of claim 1 , wherein the one or more neural networks of the mission planning layer are configured to determine the optimal route for the vehicle based on one or more of driving rules, a distance from the start point to the end point and a determination of a shortest distance from the start point to the end point, and a shortest time to travel from the start point to the end point giving the presence of any fixed obstacles between the vehicle and the end point.

16. A method of controlling autonomous operation of a vehicle or mobile robot, the method comprising:
receiving a plurality of behaviour tasks predicted by a behaviour planning layer of a planning system of the vehicle or mobile, each respective behaviour task predicted by a respective neural network of a plurality of neural networks of the behaviour planning layer based on an optimal route determined by a mission planning layer of the planning system and a sensor data received from a sensor mounted to the vehicle or mobile robot;
receiving a plurality of motion tasks from a mission planning layer of a planning system of the vehicle or robot, each respective motion task predicted by a respective neural network of a plurality of neural networks of the motion planning layer based on the behaviour tasks received from the behaviour planning layer and the sensor data, wherein the behaviour planning layer is feed-associated with the motion planning layer, and wherein the behaviour planning layer is configured to feed-forward information to the motion planning layer and receive feedback of information from the motion planning layer; and
controlling the vehicle or mobile robot based on the predicted behaviour tasks and the predicted motion tasks to operate the vehicle or mobile robot autonomously to navigate the optimal route.

17. A computer program product comprising instructions which, when the program is executed by a computer cause the computer to carry out the method of claim 16 .

18. The method of claim 16 , wherein controlling comprises controlling one or more operable elements of the vehicle or mobile robot to cause the vehicle or mobile robot to perform each behaviour task and each motion task to operate the vehicle or mobile robot autonomously to navigate the optimal route.

19. A non-transitory computer readable medium storing instructions executable by at least one processor of a vehicle to cause the vehicle to perform the method of claim 16 .

Assignees

• Rohani Mohsen
• Luo Jun
• Zhang Song
• Huawei Tech Co Ltd

Inventors

• Rohani Mohsen
• Luo Jun
• Zhang Song

Classifications

• G06V10/764

  using classification, e.g. of video objects · CPC title

• G06F18/217

  Validation; Performance evaluation; Active pattern learning techniques · CPC title

• G06N3/008 Primary

  based on physical entities controlled by simulated intelligence so as to replicate intelligent life forms, e.g. based on robots replicating pets or humans in their appearance or behaviour · CPC title

• G06F18/2413

  based on distances to training or reference patterns · CPC title

• G06N5/01

  Dynamic search techniques; Heuristics; Dynamic trees; Branch-and-bound · CPC title