Object recognition from videos using recurrent neural networks

Patent metadata

Field	Value
Publication number	US-10013640-B1
Application number	US-201514976147-A
Country	US
Kind code	B1
Filing date	Dec 21, 2015
Priority date	Dec 21, 2015
Publication date	Jul 3, 2018
Grant date	Jul 3, 2018

Abstract

Methods, systems, and apparatus, including computer programs encoded on computer storage media, for identifying an object from a video. One of the methods includes obtaining multiple frames from a video, where each frame of the multiple frames depicts an object to be recognized, and processing, using an object recognition model, the multiple frames to generate data that represents a classification of the object to be recognized.

First claim

What is claimed is:

1. A computer-implemented method, comprising:
obtaining multiple frames from a video, wherein each frame of the multiple frames depicts an object to be recognized; and
processing, using an object recognition model, the multiple frames to generate data that represents a classification of the object to be recognized,
wherein the object recognition model is a recurrent neural network that comprises a long short-term memory (LSTM) layer and multiple feature extraction layers,
wherein the LSTM layer includes a convolutional input gate, a convolutional forget gate, a convolutional memory block, and a convolutional output gate that use convolutions to process data, and wherein the processing comprises, for each frame of the multiple frames:
processing, using the multiple feature extraction layers, the frame to generate feature data that represents features of the frame; and
processing, using the LSTM layer, the feature data to generate an LSTM output and to update an internal state of the LSTM layer.

2. The method of claim 1 ,
wherein the multiple frames are arranged in an order according to their time of occurrence in the video, and
wherein processing the multiple frames further comprises processing each frame of the multiple frames using the LSTM layer in the order according to their time of occurrence in the video to generate the LSTM output and to update the internal state of the LSTM layer.

3. The method of claim 2 ,
wherein the recurrent neural network further comprises one or more classification layers, and
wherein processing the multiple frames further comprises processing, using the one or more classification layers, the LSTM output to generate the data that represents the classification of the object to be recognized.

4. The method of claim 3 ,
wherein the recurrent neural network further comprises a backward LSTM layer, and
wherein processing the plurality of frames further comprises:
processing each frame of the multiple frames using the backward LSTM layer in a reversed order according to their time of occurrence in the video to generate a backward LSTM output and to update an internal state of the backward LSTM layer, and
processing, using the one or more classification layers, the LSTM output and the backward LSTM output to generate the data that represents the classification of the object to be recognized.

5. The method of claim 1 , wherein the classification includes a respective score for each object category in a predetermined set of object categories, the respective score for each of the object categories representing a likelihood that the object to be recognized belongs to the object category.

6. The method of claim 1 , wherein obtaining the multiple frames from the video comprises selecting the multiple frames from the video based on a predetermined time interval.

7. The method of claim 1 , wherein obtaining the multiple frames from the video comprises selecting the multiple frames from the video based on a viewpoint of the object to be recognized.

8. The method of claim 1 , wherein obtaining the multiple frames from the video comprises selecting the multiple frames from the video based on a processing capability of a processor.

9. The method of claim 1 , wherein obtaining the multiple frames from the video comprises obtaining the video using a camera mounted on a robotic arm manipulator.

10. The method of claim 1 , wherein a count of the multiple frames is five or fewer.

11. A system comprising one or more computers and one or more storage devices storing instructions that when executed by the one or more computers cause the one or more computers to perform operations comprising:
obtaining multiple frames from a video, wherein each frame of the multiple frames depicts an object to be recognized; and
processing, using an object recognition model, the multiple frames to generate data that represents a classification of the object to be recognized,
wherein the object recognition model is a recurrent neural network that comprises a long short-term memory (LSTM) layer and multiple feature extraction layers,
wherein the LSTM layer includes a convolutional input gate, a convolutional forget gate, a convolutional memory block, and a convolutional output gate that use convolutions to process data, and wherein the processing comprises, for each frame of the multiple frames:
processing, using the multiple feature extraction layers, the frame to generate feature data that represents features of the frame; and
processing, using the LSTM layer, the feature data to generate an LSTM output and to update an internal state of the LSTM layer.

12. The system of claim 11 ,
wherein the multiple frames are arranged in an order according to their time of occurrence in the video, and
wherein processing the multiple frames further comprises processing each frame of the multiple frames using the LSTM layer in the order according to their time of occurrence in the video to generate the LSTM output and to update the internal state of the LSTM layer.

13. The system of claim 12 ,
wherein the recurrent neural network further comprises one or more classification layers, and
wherein processing the multiple frames further comprises processing, using the one or more classification layers, the LSTM output to generate the data that represents the classification of the object to be recognized.

14. The system of claim 13 ,
wherein the recurrent neural network further comprises a backward LSTM layer, and
wherein processing the plurality of frames further comprises:
processing each frame of the multiple frames using the backward LSTM layer in a reversed order according to their time of occurrence in the video to generate a backward LSTM output and to update an internal state of the backward LSTM layer, and
processing, using the one or more classification layers, the LSTM output and the backward LSTM output to generate the data that represents the classification of the object to be recognized.

15. The system of claim 11 , wherein the classification includes a respective score for each object category in a predetermined set of object categories, the respective score for each of the object categories representing a likelihood that the object to be recognized belongs to the object category.

16. The system of claim 11 , wherein obtaining the multiple frames from the video comprises selecting the multiple frames from the video based on a predetermined time interval.

17. The system of claim 11 , wherein obtaining the multiple frames from the video comprises selecting the multiple frames from the video based on a viewpoint of the object to be recognized.

18. The system of claim 11 , wherein obtaining the multiple frames from the video comprises selecting the multiple frames from the video based on a processing capability of a processor.

19. The system of claim 11 , wherein obtaining the multiple frames from the video comprises obtaining the video using a camera mounted on a robotic arm manipulator.

20. The system of claim 11 , wherein a count of the multiple frames is five or fewer.

21. A computer program product encoded on one or more non-transitory computer storage media, the computer program product comprising instructions that when executed by one or more computers cause the one or more computers to perform operations comprising:
obtaining multiple frames from a video, wherein each frame of the multiple frames depicts an object to be recognized; and
processing, u

Assignees

• Google Llc

Inventors

• Angelova Anelia
• Bogun Ivan

Classifications

• G06V10/764

  using classification, e.g. of video objects · CPC title

• G06V20/00

  Scenes; Scene-specific elements (control of digital cameras H04N23/60) · CPC title

• G06N3/08 Primary

  Learning methods · CPC title

• G06N3/044

  Recurrent networks, e.g. Hopfield networks · CPC title

• G06F18/24133

  Distances to prototypes · CPC title