Method and system for detecting and tracking objects using characteristic points

Patent metadata

Field	Value
Publication number	US-10930000-B2
Application number	US-201916426921-A
Country	US
Kind code	B2
Filing date	May 30, 2019
Priority date	Dec 1, 2016
Publication date	Feb 23, 2021
Grant date	Feb 23, 2021

Abstract

A method includes obtaining a disparity map based on stereoscopic image frames captured by stereoscopic cameras borne on a movable platform, determining a plurality of continuous regions in the disparity map that each includes a plurality of elements having disparity values within a predefined range, identifying a continuous sub-region including one or more elements having a highest disparity value among the elements within each continuous region as an object, and determining a distance between the object and the movable platform using at least the highest disparity value.

First claim

What is claimed is:

1. A method, comprising:
obtaining a disparity map based on stereoscopic image frames captured by stereoscopic cameras borne on a movable platform;
determining a plurality of continuous regions in the disparity map, each continuous region including a plurality of elements having disparity values within a predefined range;
identifying, within each continuous region, a continuous sub-region including one or more elements having a highest disparity value among the elements within the continuous region as an object; and
determining a distance between the object and the movable platform using at least the highest disparity value.

2. The method of claim 1 , further comprising:
identifying, within an image frame captured by an imaging device borne by the movable platform, one or more objects each corresponding to one of the continuous sub-regions.

3. The method of claim 2 , further comprising:
sending the image frame and one or more determined distances associated with the one or more objects to an electronic device for display in real time.

4. The method of claim 1 , wherein the object is an obstacle or a portion of the obstacle for avoidance by the movable platform.

5. The method of claim 1 , wherein the object is a target or a portion of the target for tracking by the movable platform.

6. The method of claim 1 , wherein determining the plurality of continuous regions comprises:
dividing the disparity map into a plurality of areas using a grid;
identifying an element having a highest disparity value in each area; and
selecting, within each area, one or more contiguous elements to form a respective continuous region of the plurality of continuous regions, wherein differences between respective disparity values of the contiguous elements and the highest disparity value in the each area are within the predefined range.

7. The method of claim 1 , wherein differences of disparity values between neighboring elements of the plurality of elements in each continuous region are within the predefined range.

8. The method of claim 7 , further comprising:
determining a plurality of first boxes enclosing the plurality of continuous regions respectively.

9. The method of claim 8 , further comprising:
determining a second box within each first box as a continuous sub-region, the second box enclosing one or more elements with the highest disparity values in the corresponding first box, wherein elements within a second box have disparity values within a range from (Dmax+Dmin)/2 to Dmax, wherein Dmax and Dmin correspond to a highest disparity value and a lowest disparity value respectively within a corresponding first box enclosing the second box.

10. The method of claim 1 further comprising:
tracking the object as the movable platform moves along a navigation path based on an updated disparity map and an updated location of the movable platform.

11. The method of claim 1 , further comprising:
selecting, from a raw disparity map obtained directly from the stereoscopic image frames, a subset of elements by comparing disparity values of the elements with threshold disparity values on a 2-dimensional mask that correspond to projection points of the elements on the 2-dimensional mask,
wherein the 2-dimensional mask includes a plurality of projection points defining a predefined 3-dimensional volume adjacent the movable platform, each projection point having a threshold disparity value for objects within the predefined 3-dimensional volume, and
wherein the subset of elements represent actual objects within the predefined 3-dimensional volume.

12. A system for processing image data, the system comprising:
one or more processors; and
memory storing one or more programs configured to be executed by the one or more processors, the one or more programs including instructions for:
obtaining a disparity map based on stereoscopic image frames captured by stereoscopic cameras borne on a movable platform;
determining a plurality of continuous regions in the disparity map, each continuous region including a plurality of elements having disparity values within a predefined range;
identifying, within each continuous region, a continuous sub-region including one or more elements having a highest disparity value among the elements within the continuous region as an object; and
determining a distance between the object and the movable platform using at least the highest disparity value.

13. The system of claim 12 , wherein the one or more programs further include instructions for:
identifying, within an image frame captured by an imaging device borne by the movable platform, one or more objects each corresponding to one of the continuous sub-regions.

14. The system of claim 13 , wherein the one or more programs further include instructions for:
sending the image frame and one or more determined distances associated with the one or more objects to an electronic device for display in real time.

15. The system of claim 12 , wherein the object is an obstacle or a portion of the obstacle for avoidance by the movable platform.

16. The system of claim 12 , wherein the object is a target or a portion of the target for tracking by the movable platform.

17. The system of claim 12 , wherein the instruction for determining the plurality of continuous regions further comprises instructions for:
dividing the disparity map into a plurality of areas using a grid;
identifying an element having a highest disparity value in each area; and
selecting, within each area, one or more contiguous elements to form a respective continuous region of the plurality of continuous regions, wherein differences between respective disparity values of the contiguous elements and the highest disparity value in the each area are within the predefined range.

18. The system of claim 12 , wherein differences of disparity values between neighboring elements of the plurality of elements in each continuous region are within the predefined range.

19. The system of claim 12 , wherein the one or more programs further include instructions for:
selecting, from a raw disparity map obtained directly from the stereoscopic image frames, a subset of elements by comparing disparity values of the elements with threshold disparity values on a 2-dimensional mask that correspond to projection points of the elements on the 2-dimensional mask,
wherein the 2-dimensional mask includes a plurality of projection points defining a predefined 3-dimensional volume adjacent the movable platform, each projection point having a threshold disparity value for objects within the predefined 3-dimensional volume, and
wherein the subset of elements represent actual objects within the predefined 3-dimensional volume.

20. An unmanned aerial vehicle (UAV), comprising:
a propulsion system;
one or more sensing devices including stereoscopic cameras; and
one or more processors coupled to the propulsion system and the one or more sensing devices, the one or more processors being configured for:
obtaining a disparity map based on stereoscopic image frames captured by the stereoscopic cameras;
determining a plurality of continuous regions in the disparity map, each continuous region including a plurality of elements having disparity values within a predefined range;
identifying, within each continuous region, a continuous sub-region including one or more elements having a highest disparity value among the elements within the continuous region as an object; and
determining

Assignees

• Sz Dji Technology Co Ltd

Inventors

• Liu Ang
• Xu Pu

Classifications

• G06V20/17

  taken from planes or by drones · CPC title

• B64U2101/30

  for imaging, photography or videography · CPC title

• G06V20/13

  Satellite images · CPC title

• G06V10/50

  by performing operations within image blocks; by using histograms, e.g. histogram of oriented gradients [HoG]; by summing image-intensity values; Projection analysis · CPC title

• G06V10/462

  Salient features, e.g. scale invariant feature transforms [SIFT] · CPC title