Method and apparatus with facial image generating

Patent metadata

Field	Value
Publication number	US-11810397-B2
Application number	US-202117208048-A
Country	US
Kind code	B2
Filing date	Mar 22, 2021
Priority date	Aug 18, 2020
Publication date	Nov 7, 2023
Grant date	Nov 7, 2023

Abstract

A processor-implemented facial image generating method includes: determining a first feature vector associated with a pose and a second feature vector associated with an identity by encoding an input image including a face; determining a flipped first feature vector by flipping the first feature vector with respect to an axis in a corresponding space; determining an assistant feature vector based on the flipped first feature vector and rotation information corresponding to the input image; determining a final feature vector based on the first feature vector and the assistant feature vector; and generating an output image including a rotated face by decoding the final feature vector and the second feature vector based on the rotation information.

First claim

What is claimed is:

1. A processor-implemented facial image generating method, comprising:
determining a first feature vector associated with a pose and a second feature vector associated with an identity by encoding an input image including a face;
determining a flipped first feature vector by swapping a value of a first element included in the first feature vector and a value of a second element located at a symmetrically transposed position with respect to an axis in the first feature vector, wherein the axis in the first feature vector corresponds to a center of columns of the first feature vector corresponding to a two-dimensional matrix;
determining an assistant feature vector based on the flipped first feature vector and rotation information corresponding to the input image;
determining a final feature vector based on the first feature vector and the assistant feature vector; and
generating an output image including a rotated face by decoding the final feature vector and the second feature vector based on the rotation information.

2. The method of claim 1 , wherein the determining of the assistant feature vector comprises:
determining the assistant feature vector by applying, to a convolutional neural network (CNN), the flipped first feature vector, position information of a main feature of the face in the input image corresponding to a size of the flipped first feature vector, and the rotation information corresponding to the size of the flipped first feature vector.

3. The method of claim 1 , wherein
the determining of the first feature vector and the second feature vector comprises:
determining, from the input image, position information of a main feature of the face in the input image; and
determining the first feature vector and the second feature vector by encoding the position information and the input image, and
the determining of the assistant feature vector comprises:
transforming the position information by flipping the position information with respect to the axis in the corresponding space; and
determining the assistant feature vector based on the flipped first feature vector, the rotation information corresponding to the input image, and the transformed position information.

4. The method of claim 3 , wherein the transforming of the position information comprises:
resizing the position information to a size corresponding to a size of the flipped first feature vector.

5. The method of claim 3 , wherein the position information includes a landmark heatmap corresponding to the face in the input image.

6. The method of claim 3 , wherein the position information includes, for each pixel in each of the input image, a value between 0 and 1 representing a probability of the main feature of the face in the input image.

7. The method of claim 1 , wherein
the rotation information includes information indicating a rotation direction for generating the output image from the input image, and
a size of the rotation information corresponds to a size of the flipped first feature vector.

8. The method of claim 7 , wherein the determining of the assistant feature vector based on the rotation information comprises:
transforming the rotation information to indicate an opposite rotation direction; and
determining the assistant feature vector based on the transformed rotation information.

9. The method of claim 1 , wherein the rotation information further includes information indicating a rotation degree for generating the output image from the input image.

10. The method of claim 9 , wherein the rotation information is determined by comparing position information of a main feature of the face in the input image and a preset facial pose of the output image.

11. The method of claim 1 , wherein the generating of the output image further comprises:
transferring the output image as an input image for a subsequent iteration; and
performing the subsequent iteration including the determining of the first feature vector and the second feature vector, the determining of the flipped first feature vector, the determining of the assistant feature vector, the determining of the final feature vector, and the generating of the output image, based on a rotation degree included in the rotation information.

12. The method of claim 1 , wherein
the encoding of the input image comprises encoding the input image using an encoder, and
a neural network of the encoder comprises:
an input layer corresponding to the input image; and
an output layer corresponding to the first feature vector and the second feature vector.

13. The method of claim 1 , wherein
the decoding of the final feature vector and the second feature vector comprises decoding the final feature vector and the second feature vector based on the rotation information, using a decoder, and
a neural network of the decoder comprises:
an input layer corresponding to the final feature vector, the second feature vector, and the rotation information; and
an output layer corresponding to the output image.

14. The method of claim 1 , wherein the determining of the flipped first feature vector comprises:
determining a flipped input image by flipping the input image with respect to the axis in the corresponding space; and
determining the flipped first feature vector associated with the pose by encoding the flipped input image.

15. The method of claim 1 , further comprising:
extracting a feature for facial recognition based on the output image; and
recognizing the face based on the extracted feature.

16. The method of claim 1 , further comprising:
generating a plurality of output images corresponding to the input image by varying a rotation degree included in the rotation information; and
recognizing the face based on the generated output images.

17. The method of claim 1 , further comprising training a neural network of an encoder used for the encoding, a neural network of a decoder used for the decoding, and a neural network used for the determining of the assistant feature vector, based on the output image and a target image corresponding to the input image.

18. A non-transitory computer-readable storage medium storing instructions that, when executed by one or more processors, configure the one or more processors to perform the method of claim 1 .

19. A processor-implemented facial image generating method, comprising:
determining a first feature vector associated with a pose and a second feature vector associated with an identity by applying, to an encoder, an input image including a face;
determining a flipped first feature vector by flipping a value of a first element included in the first feature vector and a value of a second element located at a symmetrically transposed position with respect to an axis in the first feature vector, wherein the axis in the first feature vector corresponds to a center of columns of the first feature vector corresponding to a two-dimensional matrix;
determining an assistant feature vector by applying, to a first neural network, the flipped first feature vector and rotation information corresponding to the input image;
determining a final feature vector based on the first feature vector and the assistant feature vector;
generating an output image including a rotated face by applying, to a decoder, the final feature vector, the second feature vector, and the rotation information; and
training a neural network of the encoder, a neural network of the decoder, and the first neural network, based on the output image an

Assignees

• Samsung Electronics Co Ltd

Inventors

• Baek Jiwon
• Park Seong-Jin
• Han Seungju
• Kim Insoo
• Han Jaejoon

Classifications

• G06N3/094

  Adversarial learning · CPC title

• G06N3/09

  Supervised learning · CPC title

• G06N3/0475

  Generative networks · CPC title

• G06N3/0464

  Convolutional networks [CNN, ConvNet] · CPC title

• G06N3/0455

  Auto-encoder networks; Encoder-decoder networks · CPC title