Robust tracking of objects in videos

Patent metadata

Field	Value
Publication number	US-2019252002-A1
Application number	US-201916395041-A
Country	US
Kind code	A1
Filing date	Apr 25, 2019
Priority date	Nov 16, 2016
Publication date	Aug 15, 2019
Grant date	—

Abstract

One or more embodiments described herein utilize a backward and forward tracking method that is anchored by one or more key frames in order to accurately track an object through the frames of a video, even when the video is long and may include challenging conditions.

First claim

What is claimed is:

1 . In a digital environment for processing digital videos, a method of tracking objects in videos comprising:
extracting a plurality of video frames from a video; generating an image search index from the plurality of video frames by extracting features from the plurality of video frames; receiving an indication of a query object in one or more key frames of the plurality of video frames; determining similarity scores between the one or more key frames and the plurality of video frames based on a comparison of features of the query object and the extracted features in the image search index; and identifying the query object in one or more of the video frames based on the similarity scores.

2 . The method as recited in claim 1 , further comprising:
identifying one or more auxiliary key frames; and wherein determining similarity scores further comprising determining similarity scores between the one or more auxiliary key frames and the plurality of video frames.

3 . The method as recited in claim 2 , wherein identifying one or more auxiliary key frames comprises:
selecting a candidate video frame from the image search index; determining a similarity between the candidate video frame and each of the one or more key frames; determining that the similarity between the candidate video frame and a key frame of the one or more key frames is greater than a predetermined threshold; and re-categorizing, based on the similarity being greater than the predetermined threshold, the candidate video frame as an auxiliary key frame.

4 . The method as recited in claim 3 , further comprising:
determining a first candidate query object for the video frame based on the key frame; determining a second candidate query object for the video frame based on the auxiliary key frame; weighting a similarity score for the first candidate query object using a time decay function; weighting a similarity score for the second candidate query object using the time decay function; and selecting as the query object one of the first candidate query object or the second candidate query object that has a maximum weighted similarity score.

5 . The method as recited in claim 1 , further comprising redacting the query object from the one or more video frames in which the query object is identified.

6 . The method as recited in claim 1 , further comprising sequentially determining similarity scores working backward and forward from the one or more key frames.

7 . The method as recited in claim 1 , further comprising adjusting the determined similarity scores using penalty variables, wherein a penalty variable for a given similarity score is based on the given similarity score, a penalty variable for a previous similarity score, and a lower threshold.

8 . The method as recited in claim 1 , further comprising:
identifying a location of the query object in a key frame; determining a size of a search area for a video frame based on the location of the query object in the key frame and a time distance between the key frame and the video frame; and determining a similarity score for the video frame by comparing features within the search area of the video frame with extracted features for the key frame in the image search index.

9 . A system for tracking objects in videos comprising:
a memory comprising a video; a computing device, storing instructions thereon that, when executed by the computing device, cause the system to: extract a plurality of video frames from the video; generate an image search index from the plurality of video frames by extracting features from the video frames; receive an indication of a query object within one or more key frames of the plurality of video frames; and identify the query object in the plurality of video frames in a non-time sequential manner by searching the image search index for videos frames with features similar to features of the query object.

10 . The system as recited in claim 9 , wherein the instructions, when executed by the computing device, further cause the system to search the image search index for videos frames with features similar to features of the query object by determining similarity scores between the one or more key frames and the plurality of video frames based on a comparison of features of the query object and the extracted features in the image search index.

11 . The system as recited in claim 10 , further comprising instructions that, when executed by the computing device, further cause the system to identify a set of video frames having the query object by identifying video frames having a threshold similarity score.

12 . The system as recited in claim 11 , wherein the instructions, when executed by the computing device, further cause the system to:
redact the query object from the set of video frames; and generate a redacted video by merging the set of video frames with a remainder of the plurality of video frames based on time stamps associated with each video frame.

13 . The system as recited in claim 11 , wherein the instructions, when executed by the computing device, further cause the system to generate the image search index prior to searching the image search index for videos frames with features similar to features of the query object.

14 . The system as recited in claim 11 , wherein the instructions, when executed by the computing device, further cause the system to generate the image search index while searching the image search index for videos frames with features similar to features of the query object.

15 . A non-transitory computer-readable medium storing instructions thereon that, when executed by at least one processor, cause a computing device to:
extract a plurality of video frames from a video; generate an image search index from the plurality of video frames by extracting one or more features from each of the video frames; receive an indication of a first query object within one or more key frames of the plurality of video frames; identify the first query object in one or more of the video frames by searching the image search index for videos frames with features similar to features of the first query object; receive an indication of a second query object within one or more additional key frames of the plurality of video frames; and identify the second query object in one or more of the video frames by searching the image search index for videos frames with features similar to features of the second query object.

16 . The non-transitory computer-readable medium as recited in claim 15 , further storing instructions thereon that, when executed by the at least one processor, cause the computing device to redact the first query object from the video frames in which the first query object is identified by changing a color of pixels within an area around the first query object in the one or more video frames.

17 . The non-transitory computer-readable medium as recited in claim 15 , further storing instructions thereon that, when executed by the at least one processor, cause the computing device to generate a plurality of geometric transforms of the first query object.

18 . The non-transitory computer-readable medium as recited in claim 17 , further storing instructions thereon that, when executed by the at least one processor, cause the computing device to:
determine a similarity score for the plurality of geometric transforms with respect to a candidate video frame using a spatially-constrained similarity measure; and sel

Assignees

• Adobe Inc

Inventors

• Ding Zhihong
• Lin Zhe
• Shen Xiaohui
• Kaplan Michael
• Brandt Jonathan

Classifications

• G06V10/255

  Detecting or recognising potential candidate objects based on visual cues, e.g. shapes · CPC title

• G06T7/97

  Determining parameters from multiple pictures (depth or shape recovery from multiple images G06T7/55; stereo camera calibration G06T7/85) · CPC title

• G11B27/031

  Electronic editing of digitised analogue information signals, e.g. audio or video signals · CPC title

• G06T2207/10016

  Video; Image sequence · CPC title

• G06T11/60

  Creating or editing images; Combining images with text · CPC title