Apparatus and methods for tracking salient features

Patent metadata

Field	Value
Publication number	US-10860882-B2
Application number	US-201816042901-A
Country	US
Kind code	B2
Filing date	Jul 23, 2018
Priority date	Sep 19, 2014
Publication date	Dec 8, 2020
Grant date	Dec 8, 2020

Abstract

Apparatus and methods for detecting and utilizing saliency in digital images. In one implementation, salient objects may be detected based on analysis of pixel characteristics. Least frequently occurring pixel values may be deemed as salient. Pixel values in an image may be compared to a reference. Color distance may be determined based on a difference between reference color and pixel color. Individual image channels may be scaled when determining saliency in a multi-channel image. Areas of high saliency may be analyzed to determine object position, shape, and/or color. Multiple saliency maps may be additively or multiplicative combined in order to improve detection performance (e.g., reduce number of false positives). Methodologies described herein may enable robust tracking of objects utilizing fewer determination resources. Efficient implementation of the methods described below may allow them to be used for example on board a robot (or autonomous vehicle) or a mobile determining platform.

First claim

What is claimed is:

1. A non-transitory computer readable medium having computer readable instructions stored thereon that when executed by at least one processing device configure the at least one processing device to,
determine saliency map based on a deviation of an image characteristic acquired by an image sensor on a robot while the robot is in motion;
determine an area of saliency for a given task, the area of saliency being determined based on contrast between a first relative motion associated with a first object and a second relative motion associated with a different second object;
determine presence of the first and second objects in the area of high saliency;
provide information about the first and second objects to an object tracking process;
initialize the object tracking process while the robot is in motion, the object tracking process being based on the determination of the area of saliency in the image, the object tracking process comprising the determination of either the first or second objects in the area of saliency; and
adjust trajectory associated with the given task based on the determination of the first and second objects;
wherein, the object tracking process comprises a plurality of computer-implemented neurons, the plurality of computer-implemented neurons comprises spatial receptive fields covering an image space of the image captured by the image sensor, the plurality of computer-implemented neurons configured to produce an output based on detection of the first and second objects within their respective spatial receptive fields.

2. The non-transitory computer readable medium of claim 1 , wherein the relative motion associated with the second object comprises a combination of motion associated with the first object and motion associated with a navigation trajectory of the robot.

3. The non-transitory computer readable medium of claim 1 , wherein the image characteristic comprises at least one of pixel color for one or more channels, pixel color distance to a reference color, orientation of one or more groups of pixels, motion magnitude and/or direction for a group of pixels.

4. The non-transitory computer readable medium of claim 1 , wherein the saliency comprises either a high saliency level or a low saliency level based on a color occurrence of the first object reaching a certain threshold.

5. The non-transitory computer readable medium of claim 1 , wherein the determining of the area of saliency comprises at least one of detecting presence of a particular color or set of colors, and determining absence of a given color or a set of colors.

6. The non-transitory computer readable medium of claim 1 , wherein the information provided comprises at least one of location, color, and shape of either the first or second objects.

7. The non-transitory computer readable medium of claim 1 , wherein the area of saliency is determined based on back projection of the image.

8. The non-transitory computer readable medium of claim 7 , wherein the back projection comprises information corresponding to a location in the image, the location is assigned a value corresponding to a value of a histogram bin.

9. The non-transitory computer readable medium of claim 1 , wherein the output of the plurality of computer-implemented neurons comprises at least one of a heatmap of object presence, a bounding box, or a kinematic prior.

10. A robot system for tracking, comprising:
a memory having computer readable instructions stored thereon; and
at least one processing device configurable to execute the computer readable instructions to,
determine saliency map based on a deviation of an image characteristic acquired by an image sensor on a robot while the robot is in motion,
determine an area of saliency for a given task, the area of saliency being determined based on contrast between a first relative motion associated with a first object and a second relative motion associated with a different second object,
determine presence of the first and second object in the area of saliency,
provide information about the first and second object to an object tracking process,
initialize the object tracking process while the robot is in motion, the object tracking process being based on the determination of the area of saliency in the image, the object tracking process comprising the determination of either the first or second objects in the area of saliency; and
adjust trajectory associated with the given task based on the determination of the first and second objects;
wherein, the object tracking process comprises a plurality of computer-implemented neurons, the plurality of computer-implemented neurons comprises spatial receptive fields covering an image space of the image captured by the image sensor, the plurality of computer-implemented neurons configured to produce an output based on detection of the first and second objects within their respective spatial receptive fields.

11. The robot system of claim 10 , wherein the relative motion associated with the second object comprises a combination of motion associated with the first object and motion associated with a navigation trajectory of the robot.

12. The robot system of claim 10 , wherein the image characteristic comprises at least one of pixel color for one or more channels, pixel color distance to a reference color, orientation of one or more groups of pixels, motion magnitude and/or direction for a group of pixels.

13. The robot system of claim 10 , wherein the saliency comprises either a high saliency level or a low saliency level based on a color occurrence of the first object reaching a certain threshold.

14. The robot system of claim 10 , wherein determining the area of saliency comprises at least one of detecting presence of a particular color or set of colors, and determining absence of a given color or a set of colors.

15. The robot system of claim 10 , wherein the information provided comprises at least one of location, color, and shape of either the first or second objects.

16. The robot system of claim 10 , wherein the area of saliency is determined based on back projection of the image.

17. The robot system of claim 16 , wherein the back projection includes information corresponding to a location in the image, the location is assigned a value corresponding to a value of a histogram bin.

18. The robot system of claim 10 , wherein the output of the plurality of computer-implemented neurons comprises at least one of a heatmap of object presence, a bounding box, or a kinematic prior.

19. A method for tracking, comprising:
determining saliency map based on a deviation of an image characteristic acquired by an image sensor on a robot,
determining an area of saliency for a given task, the area of saliency being determined based on contrast between a first relative motion associated with a first object and a second relative motion associated with a different second object,
determining presence of the first and second object in the area of high saliency, and
providing information about the first and second object to an object tracking process,
initializing the object tracking process while the robot is in motion, the object tracking process being based on the determination of the area of saliency in a given image, the object tracking process comprising the determination of either the first or second objects in the area of saliency; and
adjusting trajectory associated with the given task based on the determination of the first and second objects;
wherein, the object t

Assignees

• Brain Corp

Inventors

• Piekniewski Filip
• Richert Micah
• Fisher Dimitry

Classifications

• G06V10/56 Primary

  relating to colour · CPC title

• G06V10/451

  with interaction between the filter responses, e.g. cortical complex cells · CPC title

• G06V10/46

  Descriptors for shape, contour or point-related descriptors, e.g. scale invariant feature transform [SIFT] or bags of words [BoW]; Salient regional features (colour feature extraction G06V10/56) · CPC title

• G06V20/10

  Terrestrial scenes (scenes under surveillance with static cameras G06V20/52; scenes perceived from the exterior of a vehicle G06V20/56; scenes perceived from the interior of a vehicle G06V20/59) · CPC title

• G06T2207/10024

  Color image · CPC title