Image space motion planning of an autonomous vehicle

Patent metadata

Field	Value
Publication number	US-12330784-B2
Application number	US-202318463826-A
Country	US
Kind code	B2
Filing date	Sep 8, 2023
Priority date	Aug 8, 2017
Publication date	Jun 17, 2025
Grant date	Jun 17, 2025

Abstract

An autonomous vehicle that is equipped with image capture devices can use information gathered from the image capture devices to plan a future three-dimensional (3D) trajectory through a physical environment. To this end, a technique is described for image-space based motion planning. In an embodiment, a planned 3D trajectory is projected into an image-space of an image captured by the autonomous vehicle. The planned 3D trajectory is then optimized according to a cost function derived from information (e.g., depth estimates) in the captured image. The cost function associates higher cost values with identified regions of the captured image that are associated with areas of the physical environment into which travel is risky or otherwise undesirable. The autonomous vehicle is thereby encouraged to avoid these areas while satisfying other motion planning objectives.

First claim

What is claimed is:

1. A method comprising:
processing an image to generate or modify a cost function map,
wherein the cost function map associates a cost to at least one region of multiple regions of the image,
wherein the cost comprises a value indicative of a measure of risk associated with navigating in an area of the physical environment corresponding to the at least one region of the image; and

causing a vehicle to autonomously maneuver through the physical environment using the cost function map.

2. The method of claim 1 , wherein to generate or modify the cost function map, the method comprises:
estimating a depth to a physical surface in the area of the physical environment corresponding to the at least one region of the image; and
determining a measure of confidence in the estimated depth to the physical surface in the area of the physical environment corresponding to the at least one region of the image,
wherein the measure of risk associated with navigating in the area of the physical environment corresponding to the at least one region is based, at least in part, on the measure of confidence.

3. The method of claim 1 , wherein to generate or modify the cost function map, the method comprises:
detecting a physical object in the area of the physical environment corresponding to the at least one region;
wherein the measure of risk associated with navigating in the area of the physical environment corresponding to the at least one region is based, at least in part, on the detected physical object.

4. The method of claim 1 , wherein to generate or modify the cost function map, the method comprises:
utilizing one or more machine learning models to estimate the measure of risk associated with navigating in the area of the physical environment corresponding to the at least one region of the image.

5. The method of claim 1 , wherein causing the vehicle to autonomously maneuver through the physical environment using the cost function map includes:
adjusting a trajectory of the vehicle through the physical environment based on the cost function map; and
causing the vehicle to autonomously maneuver along the adjusted trajectory.

6. The method of claim 5 , wherein causing the vehicle to autonomously maneuver along the adjusted trajectory includes:
generating control commands configured to cause the vehicle to autonomously maneuver along the adjusted trajectory.

7. The method of claim 1 , further comprising:
processing additional images of the physical environment while the vehicle is in motion through the physical environment to modify the cost function map.

8. The method of claim 1 , wherein the image of the physical environment is captured by a camera coupled to the vehicle.

9. The method of claim 1 , wherein the vehicle is an unmanned aerial vehicle (UAV).

10. A vehicle configured to maneuver through a physical environment, the vehicle comprising:
an image sensor configured to capture images of the physical environment;
a propulsion system; and
a visual navigation system coupled with the image sensor and the propulsion system, the visual navigation system configured to:
process an image captured by the image sensor to generate or modify a cost function map,
wherein the cost function map associates a cost to at least one region multiple regions of the image,
wherein the cost comprises a value indicative of a measure of risk associated with navigating in an area of the physical environment corresponding to the at least one region of the image; and

direct the propulsion system to cause the vehicle to autonomously maneuver through the physical environment using the cost function map.

11. The autonomous vehicle of claim 10 , wherein to generate or modify the cost function map, the visual navigation system is configured to:
estimate a depth to a physical surface in the area of the physical environment corresponding to the at least one region of the image;
determine a measure of confidence in the estimated depth to the physical surface in the area of the physical environment corresponding to the at least one region of the image.

12. The autonomous vehicle of claim 10 , wherein to generate or modify the cost function map, the visual navigation system is configured to:
detect a physical object in the area of the physical environment corresponding to the at least one region;
wherein the measure of risk associated with navigating in the area of the physical environment corresponding to the at least one region is based, at least in part, on the detected physical object.

13. The autonomous vehicle of claim 10 , wherein to generate or modify the cost function map, the visual navigation system is configured to:
utilize one or more machine learning models to estimate the measure of risk associated with navigating in the area of the physical environment corresponding to the at least one region of the image.

14. The autonomous vehicle of claim 10 , wherein to direct the propulsion system to cause the vehicle to autonomously maneuver through the physical environment using the cost function map, the visual navigation system is configured to:
adjust a trajectory of the vehicle through the physical environment based on the cost function map; and
direct the propulsion system to cause the vehicle to autonomously maneuver along the adjusted trajectory.

15. The autonomous vehicle of claim 10 , wherein the visual navigation system is configured to:
process additional images captured by the image sensor to continually update the cost function map.

16. An apparatus, comprising:
one or more memory units storing instructions that, when executed by one or more processors, cause the one or more processors to:
process an image to generate or update a cost function map, the cost function map associating a cost to at least one region of multiple regions of the image, wherein the cost comprises a value indicative of a measure of risk associated with navigating in an area of the physical environment corresponding to the at least one region of the image; and
cause a vehicle to autonomously maneuver through the physical environment using the cost function map.

17. The apparatus of claim 16 , wherein the value is based on one or more of:
a determined level of confidence in an estimated depth to a physical surface in an area of the physical environment corresponding to the at least one region of the image; and
a detected physical object in an area of the physical environment corresponding to the at least one region of the image.

18. The apparatus of claim 16 , wherein to generate or update the cost function map, the instructions, when executed by the one or more processors, cause the one or more processors to:
utilize one or more machine learning models to estimate the measure of risk associated with navigating in the area of the physical environment corresponding to the at least one region of the image.

19. The apparatus of claim 16 , wherein to cause the vehicle to autonomously maneuver through the physical environment using the cost function map, the instructions, when executed by the one or more processors, cause the one or more processors to:
adjust a trajectory for the vehicle through the physical environment based on the cost function map; and
cause the vehicle to autonomously maneuver along the adjusted trajectory.

20. The apparatus of claim 16 , wherein the instructions, when executed by the one or more processors, cause the one or more processors to:
process additional images of

Assignees

• Skydio Inc

Inventors

• Kennedy Ryan David
• Henry Peter Benjamin
• Martirosyan Hayk
• Zhu Jack Louis
• Bachrach Abraham Galton
• Bry Adam Parker

Classifications

• G05D1/46

  Control of position or course in three dimensions [3D] · CPC title

• G05D1/644

  Optimisation of travel parameters, e.g. of energy consumption, journey time or distance · CPC title

• G08G5/80

  Anti-collision systems · CPC title

• G08G5/57

  for unmanned aircraft · CPC title

• G08G5/55 Primary

  for a single aircraft · CPC title