All-in-one convolutional neural network for face analysis

Patent metadata

Field	Value
Publication number	US-11023711-B2
Application number	US-201716340859-A
Country	US
Kind code	B2
Filing date	Oct 10, 2017
Priority date	Oct 10, 2016
Publication date	Jun 1, 2021
Grant date	Jun 1, 2021

Abstract

Various facial recognition systems may benefit from appropriate use of computer systems. For example, certain face analysis systems may benefit from an all-in-one convolutional neural network that has been appropriately configured. A method can include obtaining an image of a face. The method can also include processing the image of the face using a first set of convolutional network layers configured to perform subject-independent tasks. The method can further include subsequently processing the image of the face using a second set of convolutional network layers configured to perform subject-dependent tasks. The second set of convolutional network layers can be integrated with the first set of convolutional network layers to form a single convolutional neural network. The method can additionally include outputting facial image detection results based on the processing and subsequent processing.

First claim

We claim:

1. A method, comprising:
obtaining an image of a face;
processing the image of the face using a first set of convolutional network layers configured to perform subject-independent tasks;
subsequently processing the image of the face using a second set of convolutional network layers configured to perform subject-dependent tasks, wherein the second set of convolutional network layers is integrated with the first set of convolutional network layers to form a single convolutional neural network; and
outputting facial image detection results based on the processing and subsequent processing.

2. The method of claim 1 , wherein the results comprise a plurality of facial detection, key point extraction, pose angle, smile expression, age, and gender.

3. The method of claim 1 , wherein the results comprise an identity descriptor corresponding to each detected face in the facial image detection results.

4. The method of claim 1 , further comprising:
comparing the identity descriptor to a stored identity descriptor; and
performing a face recognition or identity verification based on the comparison.

5. The method of claim 1 , wherein the first set of convolution layers comprise a fusion of first, third, and fifth convolutional layers.

6. The method of claim 5 , wherein in the first set of convolution layers the fusion is further attached to two convolutional layers and pooling layers, to obtain a feature map of size 6×6.

7. The method of claim 6 , wherein the first set of convolution layers further comprises a dimensionality reduction layer to reduce a number of feature maps to 256.

8. The method of claim 6 , wherein the dimensionality reduction layer is followed by a fully connected layer of dimension 2048, configured to form a generic representation of the subject-independent tasks.

9. The method of claim 1 , wherein the second set of convolutional network layers comprise fully connected layers of dimension 512 each.

10. The method of claim 9 , wherein the fully connected layers are followed by output layers.

11. An apparatus, comprising:
at least one processor; and
at least one memory including computer program code,
wherein the at least one memory and the computer program code are configured to, with the least one processor, cause the apparatus at least to
obtain an image of a face;
process the image of the face using a first set of convolutional network layers configured to perform subject-independent tasks;
subsequently process the image of the face using a second set of convolutional network layers configured to perform subject-dependent tasks, wherein the second set of convolutional network layers is integrated with the first set of convolutional network layers to form a single convolutional neural network; and
output facial image detection results based on the processing and subsequent processing.

12. The apparatus of claim 11 , wherein the results comprise a plurality of facial detection, key point extraction, pose angle, smile expression, age, and gender.

13. The apparatus of claim 11 , wherein the results comprise an identity descriptor corresponding to each detected face in the facial image detection results.

14. The apparatus of claim 11 , wherein the at least one memory and the computer program code are further configured to, with the least one processor, cause the apparatus at least to:
compare the identity descriptor to a stored identity descriptor; and
perform a face recognition or identity verification based on the comparison.

15. The apparatus of claim 11 , wherein the first set of convolution layers comprise a fusion of first, third, and fifth convolutional layers.

16. The apparatus of claim 15 , wherein in the first set of convolution layers the fusion is further attached to two convolutional layers and pooling layers, to obtain a feature map of size 6×6.

17. The apparatus of claim 16 , wherein the first set of convolution layers further comprises a dimensionality reduction layer to reduce a number of feature maps to 256.

18. The apparatus of claim 16 , wherein the dimensionality reduction layer is followed by a fully connected layer of dimension 2048, configured to form a generic representation of the subject-independent tasks.

19. The apparatus of claim 11 , wherein the second set of convolutional network layers comprise fully connected layers of dimension 512 each.

20. The apparatus of claim 19 , wherein the fully connected layers are followed by output layers.

21. A non-transitory computer-readable medium encoded with instructions that, when executed in hardware, perform a process, the process comprising:
obtaining an image of a face;
processing the image of the face using a first set of convolutional network layers configured to perform subject-independent tasks;
subsequently processing the image of the face using a second set of convolutional network layers configured to perform subject-dependent tasks, wherein the second set of convolutional network layers is integrated with the first set of convolutional network layers to form a single convolutional neural network; and
outputting facial image detection results based on the processing and subsequent processing.

22. The non-transitory computer-readable medium of claim 21 , wherein the results comprise a plurality of facial detection, key point extraction, pose angle, smile expression, age, and gender.

23. The non-transitory computer-readable medium of claim 21 , wherein the results comprise an identity descriptor corresponding to each detected face in the facial image detection results.

24. The non-transitory computer-readable medium of claim 21 , the process further comprising:
comparing the identity descriptor to a stored identity descriptor; and
performing a face recognition or identity verification based on the comparison.

25. The non-transitory computer-readable medium of claim 21 , wherein the first set of convolution layers comprise a fusion of first, third, and fifth convolutional layers.

26. The non-transitory computer-readable medium of claim 25 , wherein in the first set of convolution layers the fusion is further attached to two convolutional layers and pooling layers, to obtain a feature map of size 6×6.

27. The non-transitory computer-readable medium of claim 26 , wherein the first set of convolution layers further comprises a dimensionality reduction layer to reduce a number of feature maps to 256.

28. The non-transitory computer-readable medium of claim 26 , wherein the dimensionality reduction layer is followed by a fully connected layer of dimension 2048, configured to form a generic representation of the subject-independent tasks.

29. The non-transitory computer-readable medium of claim 21 , wherein the second set of convolutional network layers comprise fully connected layers of dimension 512 each.

30. The non-transitory computer-readable medium of claim 29 , wherein the fully connected layers are followed by output layers.

Assignees

• Univ Maryland

Inventors

• Ranjan Rajeev
• Sankaranarayanan Swaminathan
• Castillo Carlos
• Chellappa Ramalingam

Classifications

• G06V10/82

  using neural networks · CPC title

• G06V10/764

  using classification, e.g. of video objects · CPC title

• G06V10/454 Primary

  Integrating the filters into a hierarchical structure, e.g. convolutional neural networks [CNN] · CPC title

• G06V40/172

  Classification, e.g. identification · CPC title

• G06F18/217

  Validation; Performance evaluation; Active pattern learning techniques · CPC title