Methods and apparatus for tracking objects using saliency

Patent metadata

Field	Value
Publication number	US-10268919-B1
Application number	US-201514860579-A
Country	US
Kind code	B1
Filing date	Sep 21, 2015
Priority date	Sep 19, 2014
Publication date	Apr 23, 2019
Grant date	Apr 23, 2019

Abstract

Methods and apparatus for tracking and discerning objects using their saliency. In one embodiment of the present disclosure, the tracking of objects is based on a combination of object saliency and additional sources of signal about object identity. Under certain simplifying assumptions, the present disclosure allows for robust tracking of simple objects with limited processing resources. In one or more variants, efficient implementation of the methods described allow sensors (e.g., cameras) to be used on board a robot (or autonomous vehicle) on a mobile determining platform, such as to capture images to determine the presence and/or identity of salient objects. Such determination of salient objects allow for e.g., adjustments to vehicle or other moving object trajectory.

First claim

What is claimed is:

1. A method of operating a robotic device, the method comprising:
obtaining a plurality of input images of a visual scene during navigation of a trajectory by the robotic device;
determining a map of a back projected color histogram based upon a plurality of bins, a respective bin of the plurality of bins grouping a range of values corresponding to a difference between a first pixel and a second pixel in a respective input image of the plurality of input images, the second pixel corresponding to an average value of a plurality of pixels of the respective input image, the respective input image corresponding to a surrounding region of the visual scene;
determining a salient feature of a respective pixel of the respective input image based on evaluation of the map, the salient feature corresponding to a number of pixels representing the same exact value in remaining of the respective input image; and
adjusting the trajectory of the robotic device in accordance with a parameter associated with the salient feature.

2. The method of claim 1 , further comprising:
transforming the plurality of input images, the transforming includes applying a low-pass imaging filter to the plurality of input images.

3. The method of claim 2 , wherein the transforming of the plurality of input images comprises applying a Gaussian kernel to the plurality of input images.

4. The method of claim 1 , further comprising:
determining a shape characteristic of the determined salient feature.

5. The method of claim 4 , wherein the determining of the shape characteristic comprises determining a circularity of the salient feature.

6. A controller configured for use with a robotic apparatus, the controller comprising:
a sensor processor configured to execute computer readable instructions to analyze a sensor input, the sensor input being analyzed based at least on:
initialization of a tracking process using color saliency, the tracking process comprising at least one of (i) generation of a heat map corresponding to the sensor input from the color saliency, (ii) determination of a peak value in the generated heat map based on a threshold, and (iii) application of a plurality of thresholds below the peak value to produce a threshold heat map;
identification of a salient feature using the threshold heat map, the salient feature corresponding to a number of pixels representing the same exact value in remaining of the respective input image; and
communication of tracking information to the controller, the controller configured to cause an adjustment of a trajectory of the robotic apparatus in accordance with a characteristic of the salient feature.

7. The apparatus of claim 6 , wherein
the identification of the salient feature is based at least in part on the color saliency;
the tracking information is automatically provided to the tracking process; and
the salient feature is associated with an object and the trajectory adjustment is configured to reduce a spatial separation between the object and the robotic apparatus.

8. The apparatus of claim 7 , further comprising:
a manipulator, wherein the tracking information to the controller is further configured to cause the manipulator to pick up the object.

9. The apparatus of claim 6 , wherein the identification of the salient feature comprises evaluation of a transformed version of the sensor input comprising at least one of: (i) a back projected image based on a histogram, and/or (ii) a low-passed image.

10. A non-transitory computer-readable apparatus having a plurality of computer readable instructions stored thereon, that when executed by at least one processor, configure the at least one processor to:
obtain a plurality of images during navigation of a robotic device along a trajectory;
transform the plurality of images via a low-pass operation;
determine a map based upon a difference between (i) a first pixel of a respective image of the plurality of images, and (ii) a second pixel, the second pixel corresponding to an average value of a plurality of pixels of a corresponding image of the transformed plurality of images;
determine change in position of a salient feature of an object from the map between one or more subsequent pairs of the plurality of images, the salient feature corresponding to a number of pixels representing the same exact value in remaining of the corresponding image;
determine a salient object based on an evaluation of the map; and
adjust the trajectory of the robotic device in accordance with the determined salient object.

11. The non-transitory computer-readable apparatus of claim 10 , wherein the at least one processor is further configured to execute the computer readable instructions to cause the robotic device to determine a shape characteristic of the determined salient feature.

12. The non-transitory computer-readable apparatus of claim 11 , wherein the shape characteristic is characterized by a circularity of the salient feature.

13. The non-transitory computer-readable apparatus of claim 10 , wherein the determined salient feature comprises a gesture or a presence of a user.

14. The non-transitory computer-readable apparatus of claim 10 , wherein:
the robotic device comprises two or more cameras; and
the two or more cameras are configured to obtain the plurality of images during the navigation of the trajectory.

15. The non-transitory computer-readable apparatus of claim 14 , wherein the at least one processor is further configured to execute the computer readable instructions to,
cause the robotic device to determine a distance between the robotic device, and
determined salient feature, the determination of the distance comprising an interleaving of images from the two or more cameras.

16. The non-transitory computer-readable apparatus of claim 15 , wherein the at least one processor is further configured to execute the computer readable instructions to,
cause a robotic device to determine a distance between the determined salient feature and a second salient feature, the second salient feature being determined based on an evaluation of a saliency map.

17. A method of operating a robotic platform, the method comprising:
receiving a sensory input including a plurality of images, a respective image including a plurality of colors;
determining a color saliency of the sensory input based on discriminability of a respective color of the plurality of colors in the sensory input from a reference color prototype;
initializing a tracking process using the determined color saliency to produce a heat map, the heat map comprising one or more potential salient features;
identifying a salient feature using shape information associated with the one or more potential salient features in the heat map, the shape information being correlated to each of the one or more potential salient features and a respective auxiliary shape associated with each of the one or more potential salient features, the salient feature corresponding to a number of pixels representing the same exact value in remaining of the respective image; and
adjusting trajectory of the robotic platform in accordance with a characteristic of the salient feature upon the tracking information being communicated to the controller.

18. The method of claim 17 , further comprising:
determining a saliency map, the saliency map comprising the color saliency; and
identifying the salient feature within the saliency map.

19. The method of claim 18 , wherein the identifying of the salient feat

Assignees

• Brain Corp

Inventors

• Piekniewski Filip
• Richert Micah

Classifications

• Y10S901/47

  Optical · CPC title

• Y10S901/09

  Closed loop, sensor feedback controls arm movement · CPC title

• G06T7/246

  using feature-based methods, e.g. the tracking of corners or segments · CPC title

• G06T7/90

  Determination of colour characteristics · CPC title

• G06T7/285

  using a sequence of stereo image pairs · CPC title