Systems and methods for determining video feature descriptors based on convolutional neural networks

Patent metadata

Field	Value
Publication number	US-10198637-B2
Application number	US-201715848891-A
Country	US
Kind code	B2
Filing date	Dec 20, 2017
Priority date	Dec 30, 2014
Publication date	Feb 5, 2019
Grant date	Feb 5, 2019

Abstract

Systems, methods, and non-transitory computer-readable media can acquire video content for which video feature descriptors are to be determined. The video content can be processed based at least in part on a convolutional neural network including a set of two-dimensional convolutional layers and a set of three-dimensional convolutional layers. One or more outputs can be generated from the convolutional neural network. A plurality of video feature descriptors for the video content can be determined based at least in part on the one or more outputs from the convolutional neural network.

First claim

What is claimed is:

1. A computer-implemented method comprising:
processing, by a computing system, video content based at least in part on a convolutional neural network that includes at least one two-dimensional convolutional layer and at least one three-dimensional convolutional layer, wherein at least a portion of signals outputted by the at least one two-dimensional convolutional layer are inputted into the at least one three-dimensional convolutional layer, and wherein the convolutional neural network generates one or more outputs; and
determining, by the computing system, a plurality of video feature descriptors for the video content based at least in part on the one or more outputs.

2. The computer-implemented method of claim 1 , wherein the video feature descriptors provide an indication that one or more concepts are represented in subject matter captured by the video content.

3. The computer-implemented method of claim 2 , wherein the video feature descriptors provide a set of metrics indicating respective likelihoods of the one or more concepts being represented in the video content.

4. The computer-implemented method of claim 2 , wherein the one or more concepts include at least a scene, an object, or an action.

5. The computer-implemented method of claim 2 , wherein the video content is categorized based on the indication that one or more concepts that are represented in the subject matter captured by the video content.

6. The computer-implemented method of claim 1 , wherein the convolutional neural network also includes at least one fully-connected layer.

7. The computer-implemented method of claim 6 , wherein at least a portion of signals outputted by the at least one three-dimensional convolutional layer are inputted into the at least one fully-connected layer, and wherein the at least one fully-connected layer produces the one or more outputs.

8. The computer-implemented method of claim 1 , wherein the convolutional neural network also includes at least one softmax layer.

9. The computer-implemented method of claim 8 , wherein at least a portion of signals outputted by the at least one fully-connected layer are normalized by the at least one softmax layer to produce the one or more outputs.

10. The computer-implemented method of claim 2 , further comprising:
training, by the computing system, the convolutional neural network based at least in part on a set of training content items, wherein each training content item is associated with at least one label that describes at least one concept in the training content item.

11. A system comprising:
at least one processor; and
a memory storing instructions that, when executed by the at least one processor, cause the system to perform:
processing video content based at least in part on a convolutional neural network that includes at least one two-dimensional convolutional layer and at least one three-dimensional convolutional layer, wherein at least a portion of signals outputted by the at least one two-dimensional convolutional layer are inputted into the at least one three-dimensional convolutional layer, and wherein the convolutional neural network generates one or more outputs; and
determining a plurality of video feature descriptors for the video content based at least in part on the one or more outputs.

12. The system of claim 11 , wherein the video feature descriptors provide an indication that one or more concepts are represented in subject matter captured by the video content.

13. The system of claim 12 , wherein the video feature descriptors provide a set of metrics indicating respective likelihoods of the one or more concepts being represented in the video content.

14. The system of claim 12 , wherein the one or more concepts include at least a scene, an object, or an action.

15. The system of claim 12 , wherein the video content is categorized based on the indication that one or more concepts that are represented in the subject matter captured by the video content.

16. A non-transitory computer-readable storage medium including instructions that, when executed by at least one processor of a computing system, cause the computing system to perform:
processing video content based at least in part on a convolutional neural network that includes at least one two-dimensional convolutional layer and at least one three-dimensional convolutional layer, wherein at least a portion of signals outputted by the at least one two-dimensional convolutional layer are inputted into the at least one three-dimensional convolutional layer, and wherein the convolutional neural network generates one or more outputs; and
determining a plurality of video feature descriptors for the video content based at least in part on the one or more outputs.

17. The non-transitory computer-readable storage medium of claim 16 , wherein the video feature descriptors provide an indication that one or more concepts are represented in subject matter captured by the video content.

18. The non-transitory computer-readable storage medium of claim 17 , wherein the video feature descriptors provide a set of metrics indicating respective likelihoods of the one or more concepts being represented in the video content.

19. The non-transitory computer-readable storage medium of claim 17 , wherein the one or more concepts include at least a scene, an object, or an action.

20. The non-transitory computer-readable storage medium of claim 17 , wherein the video content is categorized based on the indication that one or more concepts that are represented in the subject matter captured by the video content.

Assignees

• Facebook Inc

Inventors

• Tran Du Le Hong
• Paluri Balamanohar
• Bourdev Lubomir
• Fergus Robert D
• Chopra Sumit

Classifications

• G06N3/084 Primary

  Backpropagation, e.g. using gradient descent · CPC title

• G06N3/045

  Combinations of networks · CPC title

• G06K9/00744 Primary

  Physics · mapped topic

• G06N3/0454

  Physics · mapped topic

• G06N3/09

  Supervised learning · CPC title