Enhanced siamese trackers

Patent metadata

Field	Value
Publication number	US-2018129934-A1
Application number	US-201715621741-A
Country	US
Kind code	A1
Filing date	Jun 13, 2017
Priority date	Nov 7, 2016
Publication date	May 10, 2018
Grant date	—

Abstract

In one configuration, a visual object tracking apparatus is provided that receives a position of an object in a first frame of a video, and determines a current position of the object in subsequent frames of the video using a Siamese neural network To facilitate determining the current position of the object, the apparatus may adjust a spatial resolution of an image, adjust a size of a probe region, and/or adjust a scale of a plurality of sampled images. In one configuration, a visual object tracking using a Siamese neural network is provided. The apparatus feeds outputs from a plurality of subnetworks of the Siamese neural network to a comparison layer. In addition, the apparatus compares, at the comparison layer, inputs from the plurality of subnetworks to generate a comparison result. Further, the apparatus combines comparison results based on weights to obtain a final comparison result.

First claim

What is claimed is:

1 . A method of visual object tracking, comprising:
receiving a position of an object in a first frame of a video; and determining a current position of the object in subsequent frames of the video using a Siamese neural network, wherein the determining the current position of the object comprises one or more of:
adjusting a spatial resolution of a first image from the first frame of the video and a second image sampled from a current frame under processing, the first image and the second image being inputs to the Siamese neural network;
adjusting a size of a probe region on the current frame under processing based on a metric of movement of the object from one frame to another; or
adjusting a scale of a plurality of images sampled from the current frame under processing, the plurality of images being inputs to the Siamese neural network.

2 . The method of claim 1 , wherein the spatial resolution of the first image and the second image is adjusted based on a size of the object and an amount of spatial reduction caused by the Siamese neural network.

3 . The method of claim 1 , wherein the adjusting the spatial resolution of the first image and the second image comprises up-sampling or down-sampling a first image region on the first frame and a second image region on the current frame.

4 . The method of claim 1 , wherein the scale of the plurality of images comprises sizes and number of the plurality of images.

5 . The method of claim 1 , wherein the scale of the plurality of images is adjusted based on an estimated scale in a frame immediately before the current frame.

6 . A method of visual object tracking using a Siamese neural network, comprising:
feeding outputs from a plurality of layers of a first subnetwork of the Siamese neural network and a second subnetwork of the Siamese neural network to a comparison layer; comparing, at the comparison layer for each layer of the plurality of layers, a first input from the layer in the first subnetwork with a second input from the layer in the second subnetwork to obtain a comparison result for the layer; and combining comparison results for the plurality of layers based on weights dynamically generated for the plurality of layers to obtain a final comparison result.

7 . The method of claim 6 , wherein the first subnetwork and the second subnetwork are identical.

8 . The method of claim 6 , wherein the plurality of layers are penultimate layers of the first subnetwork and the second subnetwork.

9 . The method of claim 6 , wherein the weights are generated by a neural network that is trained concurrently with the Siamese neural network.

10 . The method of claim 6 , wherein the final comparison result is a weighted sum of the comparison results for the plurality of layers.

11 . The method of claim 6 , wherein the final comparison result is a weighted average of the comparison results for the plurality of layers.

12 . The method of claim 6 , further comprising:
inputting, of an initial frame, a query region including a target into the layers of the first subnetwork of the Siamese neural network; inputting, of a current frame, at least a portion of a probe region into the layers of the second subnetwork of the Siamese neural network; and determining based on the final comparison result whether the at least the portion of the probe region includes the target.

13 . The method of claim 12 , further comprising:
comparing a first plurality of subregions of a subregion of the probe region of the current frame with a second plurality of subregions of the query region of the initial frame; determining a similarity score for each of the first plurality of subregions based on the comparison; determining that a first set of subregions of the first plurality of subregions is occluded when the similarity score for each subregion of the first set of subregions is less than a first threshold and when the similarity score for each subregion of a second set of subregions is greater than a second threshold, the second threshold being greater than the first threshold, wherein the inputted at least the portion of the probe region comprises the second set of subregions of the first plurality of subregions.

14 . An apparatus for visual object tracking, comprising:
a memory; and at least one processor coupled to the memory and configured to:
receive a position of an object in a first frame of a video; and
determine a current position of the object in subsequent frames of the video using a Siamese neural network,
wherein, to determine the current position of the object, the at least one processor is configured to perform one or more of:
adjusting a spatial resolution of a first image from the first frame of the video and a second image sampled from a current frame under processing, the first image and the second image being inputs to the Siamese neural network;
adjusting a size of a probe region on the current frame under processing based on a metric of movement of the object from one frame to another; or
adjusting a scale of a plurality of images sampled from the current frame under processing, the plurality of images being inputs to the Siamese neural network.

15 . The apparatus of claim 14 , wherein the spatial resolution of the first image and the second image is adjusted based on a size of the object and an amount of spatial reduction caused by the Siamese neural network.

16 . The apparatus of claim 14 , wherein, to adjust the spatial resolution of the first image and the second image, the at least one processor is configured to up-sample or down-sample a first image region on the first frame and a second image region on the current frame.

17 . The apparatus of claim 14 , wherein the scale of the plurality of images comprises sizes and number of the plurality of images.

18 . The apparatus of claim 14 , wherein the scale of the plurality of images is adjusted based on an estimated scale in a frame immediately before the current frame.

19 . An apparatus for visual object tracking using a Siamese neural network, comprising:
a memory; and at least one processor coupled to the memory and configured to:
feed outputs from a plurality of layers of a first subnetwork of the Siamese neural network and a second subnetwork of the Siamese neural network to a comparison layer;
compare, at the comparison layer for each layer of the plurality of layers, a first input from the layer in the first subnetwork with a second input from the layer in the second subnetwork to obtain a comparison result for the layer; and
combine comparison results for the plurality of layers based on weights dynamically generated for the plurality of layers to obtain a final comparison result.

20 . The apparatus of claim 19 , wherein the first subnetwork and the second subnetwork are identical.

21 . The apparatus of claim 19 , wherein the plurality of layers are penultimate layers of the first subnetwork and the second subnetwork.

22 . The apparatus of claim 19 , wherein the weights are generated by a neural network that is trained concurrently with the Siamese neural network.

23 . The apparatus of claim 19 , wherein the final comparison result is a weighted sum of the comparison results for the plurality of layers.

24 . The apparatus of claim 19 , wherein the final comparison result is a weighted average of the comparison results for the plurality of

Assignees

• Qualcomm Inc

Inventors

• Tao Ran
• Gavves Efstratios
• SMEULDERS Arnold Wilhelmus Maria

Classifications

• G06N3/084 Primary

  Backpropagation, e.g. using gradient descent · CPC title

• G06N3/045

  Combinations of networks · CPC title

• G06V10/255

  Detecting or recognising potential candidate objects based on visual cues, e.g. shapes · CPC title

• G06V10/82 Primary

  using neural networks · CPC title

• G06N3/09

  Supervised learning · CPC title