Object tracking by an unmanned aerial vehicle using visual sensors

Patent metadata

Field	Value
Publication number	US-2018158197-A1
Application number	US-201715827945-A
Country	US
Kind code	A1
Filing date	Nov 30, 2017
Priority date	Dec 1, 2016
Publication date	Jun 7, 2018
Grant date	—

Abstract

Systems and methods are disclosed for tracking objects in a physical environment using visual sensors onboard an autonomous unmanned aerial vehicle (UAV). In certain embodiments, images of the physical environment captured by the onboard visual sensors are processed to extract semantic information about detected objects. Processing of the captured images may involve applying machine learning techniques such as a deep convolutional neural network to extract semantic cues regarding objects detected in the images. The object tracking can be utilized, for example, to facilitate autonomous navigation by the UAV or to generate and display augmentative information regarding tracked objects to users.

First claim

What is claimed is:

1 . A method for tracking objects in a physical environment using an autonomous vehicle based on captured images of the physical environment, the method comprising:
receiving, by a computer system, images of the physical environment, the received images captured by one or more image capture devices coupled to the autonomous vehicle; processing, by the computer system, the received images to detect one or more objects in the physical environment associated with a particular class of objects; processing, by the computer system, the received images to distinguish one or more instances of the detected one or more objects; tracking, by the computer system, a particular object instance of the detected one or more objects; and generating, by the computer system, an output based on the tracking of the particular object instance.

2 . The method of claim 1 , wherein the received images are processed using a deep convolutional neural network.

3 . The method of claim 1 , wherein processing the received images to detect one or more objects associated with the particular class of objects includes:
generating a dense per-pixel segmentation based on the received images, wherein each pixel in the dense per-pixel segmentation is associated with a value indicative of a likelihood that the pixel corresponds with the particular class of objects.

4 . The method of claim 3 , the dense per-pixel segmentation is one of a plurality of dense per-pixel segmentations comprising a tensor, each of the plurality of dense per-pixel segmentations associated with a different class of objects.

5 . The method of claim 3 , wherein processing the received images to distinguish one or more instances of the detected one or more objects includes:
analyzing the dense per-pixel segmentation generated based on the received images to associate pixels corresponding to the particular class of objects with a particular instance of the particular class of objects.

6 . The method of claim 5 , wherein associating pixels corresponding to the particular class of objects with the particular instance of the particular class includes:
applying a grouping process to group:
pixels that are substantially similar to other pixels associated with the particular instance;
pixels that are spatially clustered with other pixels associated with the particular instance; and/or
pixels that fit an appearance-based model for the particular class of objects.

7 . The method of claim 1 , further comprising:
processing, by the computer system, the received images to extract semantic information regarding the detected one or more objects; wherein the tracking is based on the extracted semantic information.

8 . The method of claim 7 , wherein the semantic information includes information regarding any of the a position, orientation, shape, size, scale, appearance, pixel segmentation, or activity of the detected one or more objects.

9 . The method of claim 1 , further comprising:
processing, by the computer system, the received images to predict a three-dimensional (3D) trajectory of the particular object instance; wherein the tracking is based on the predicted 3D trajectory of the particular object instance.

10 . The method of claim 1 , further comprising:
receiving, by the computer system, sensor data from one or more other sensors coupled to the autonomous vehicle; and processing, by the computer system, the received sensor data with the received images using a spatiotemporal factor graph to predict a 3D trajectory of the particular object instance; wherein the tracking is based on the predicted 3D trajectory of the particular object instance.

11 . The method of claim 1 , wherein generating the output includes:
generating, by the computer system, a planned 3D trajectory of the autonomous vehicle through the physical environment based on the tracking of the particular object instance; and generating, by the computer system, control commands configured to cause the autonomous vehicle to maneuver along the planned 3D trajectory.

12 . The method of claim 1 , wherein generating the output includes:
generating, by the computer system, control commands configured to cause a gimbal mechanism to adjust an orientation of the image capture device relative to the autonomous vehicle so as to keep the tracked particular object instance within a field of view of the image capture device.

13 . The method of claim 1 , wherein generating the output includes:
generating, by the computer system, an augmentation based on the tracking of the particular object instance; and causing, by the computer system, the generated augmentation to be presented at an augmented reality (AR) device.

14 . The method of claim 1 , further comprising:
tracking, by the computer system, a second particular object instance while tracking the particular object instance; and generating, by the computer system, an output based on the tracking of the second particular object instance.

15 . The method of claim 1 , wherein the autonomous vehicle is an unmanned aerial vehicle (UAV).

16 . The method of claim 1 , wherein the particular class of objects is selected from a list of classes of objects comprising people, animal, vehicles, buildings, landscape features, and plants.

17 . An unmanned aerial vehicle (UAV) configured for autonomous flight through a physical environment, the UAV comprising:
a first image capture device; a second image capture device; and a tracking system configured to:
receive images of the physical environment captured by any of the first image capture device or second image capture device;
process the received images to detect one or more objects in the physical environment associated with a particular class of objects;
process the received images to distinguish one or more instances of the detected one or more objects;
track a particular object instance of the detected one or more objects; and
generate an output based on the tracking of the particular object instance.

18 . The UAV of claim 17 , wherein the received images are processed using a deep convolutional neural network.

19 . The UAV of claim 17 , wherein processing the received images to detect one or more objects associated with the particular class of objects includes:
generating a dense per-pixel segmentation of the received image, wherein each pixel in the received image is associated with a value indicative of a likelihood that the pixel corresponds with the particular class of objects.

20 . The UAV of claim 17 , wherein processing the received images to distinguish one or more instances of the detected one or more objects includes:
analyzing the dense per-pixel segmentation generated based on the received images to associate pixels corresponding to the particular class of objects with a particular instance of the particular class of objects.

21 . The UAV of claim 20 , wherein associating pixels corresponding to the particular class of objects with the particular instance of the particular class includes:
applying a grouping process to group:
pixels that are substantially similar to other pixels associated with the particular instance;
pixels that are spatially clustered with other pixels associated with the particular instance; and/or
pixels that fit an appearance-based model for the particular class of objects.

22 . The UAV of claim 17 , wherein the tracking system is further config

Assignees

• Skydio Inc

Inventors

• Dasgupta Saumitro
• Martirosyan Hayk
• Koppula Hema
• Kendall Alex
• Stone Austin
• Donahoe Matthew
• Bachrach Abraham
• Bry Adam

Classifications

• G06N3/045

  Combinations of networks · CPC title

• G06V20/17

  taken from planes or by drones · CPC title

• G06V10/82

  using neural networks · CPC title

• G06V20/13 Primary

  Satellite images · CPC title

• G06T7/292 Primary

  Multi-camera tracking · CPC title