Image processing method and apparatus, and image processing training method

Patent metadata

Field	Value
Publication number	US-11887348-B2
Application number	US-202117341469-A
Country	US
Kind code	B2
Filing date	Jun 8, 2021
Priority date	Dec 16, 2020
Publication date	Jan 30, 2024
Grant date	Jan 30, 2024

Abstract

A processor-implemented image processing method and apparatus are provided. The image processing method includes receiving an input image and rotation information associated with the input image, generating a feature vector of the input image based on pose information corresponding to the input image, generating an assistant feature vector which represents a target component according to a pose corresponding to the rotation information, based on the feature vector, the pose information, and the rotation information, and generating a target image which has the pose corresponding to the rotation information based on the feature vector and the assistant feature vector.

First claim

What is claimed is:

1. A processor-implemented image processing method, comprising:
receiving an input image and rotation information associated with the input image;
generating a feature vector of the input image based on pose information corresponding to the input image;
generating an assistant feature vector which represents a target component according to a pose corresponding to the rotation information based on the feature vector, the pose information, and the rotation information; and
generating a target image which has the pose corresponding to the rotation information based on the feature vector and the assistant feature vector.

2. The method of claim 1 , wherein the generating of the assistant feature vector comprises:
converting the pose information to correspond to an intermediate layer of an encoder;
generating a weight map which includes a position in which the target component is to be generated in the target image, based on the feature vector, the converted pose information and the rotation information; and
generating the assistant feature vector which represents the target component corresponding to the pose in the position in which the target component is to be generated, by combining the weight map and a target feature vector separated from the feature vector.

3. The method of claim 2 , wherein the generating of the weight map comprises generating the weight map which includes the position in which the target component is to be generated in the target image by applying the feature vector, the converted pose information, and the rotation information to a first neural network.

4. The method of claim 2 , wherein the generating of the weight map comprises generating a weight map which includes a position in which each of a plurality of target components is to be generated in the target image, based on the feature vector, the converted pose information and the rotation information.

5. The method of claim 2 , wherein the generating of the assistant feature vector comprises:
separating the target feature vector corresponding to the target component from the feature vector; and
generating the assistant feature vector which represents the target component corresponding to the pose in the position in which the target component is to be generated, by combining the weight map and the target feature vector.

6. The method of claim 1 , wherein the generating of the feature vector
comprises:
estimating the pose information from the input image;
applying the input image and the pose information to an encoder that outputs the feature vector; and
acquiring the feature vector from an intermediate layer of the encoder.

7. The method of claim 6 , wherein the estimating of the pose information comprises:
generating a landmark hit map corresponding to an object including the target component by extracting landmarks of the object from the input image; and
estimating the pose information based on the landmark hit map.

8. The method of claim 1 , wherein the rotation information comprises any one or any combination of a rotation direction in which an object in the input image is to be rotated to generate the target image, and a value which indicates a degree of rotation of the object.

9. The method of claim 1 , wherein the pose information comprises a landmark hit map corresponding to a face included in the input image.

10. The method of claim 1 , wherein the generating of the target image comprises generating the target image by decoding the feature vector with a decoder based on the assistant feature vector.

11. The method of claim 10 , wherein the generating of the target image comprises generating the target image by summing the assistant feature vector and the feature vector during the decoding of the feature vector.

12. The method of claim 1 , wherein:
the input image comprises a face; and
the target component comprises any one or any combination of an eye, a nose, a mouth, and a mouth corner.

13. A non-transitory computer-readable storage medium storing instructions that, when executed by a processor, cause the processor to perform the method of claim 1 .

14. A processor-implemented training method comprising:
acquiring a (1-1)-th feature vector by encoding a first image and first pose information corresponding to the first image;
acquiring a (1-2)-th feature vector by encoding a second image corresponding to a target image of the first image and second pose information corresponding to the second image;
training an encoder to maintain an identity of an object included in each of the first image and the second image, regardless of a pose of the object, based on a difference between the (1-1)-th feature vector and the (1-2)-th feature vector;
acquiring a second feature vector from an intermediate layer of the encoder;
converting the first pose information to correspond to the intermediate layer;
generating a weight map including a position in which a target component is to be generated in the second image which has a pose corresponding to rotation information by applying the second feature vector, the converted first pose information and the rotation information to a first neural network;
separating a target feature vector corresponding to the target component from the second feature vector;
generating an assistant feature vector which represents the target component corresponding to the pose in the position in which the target component is to be generated, by combining the weight map and the target feature vector;
generating a third image by decoding, with a decoder, the (1-1)-th feature vector based on the assistant feature vector; and
training the first neural network, the encoder and the decoder based on the second image and the third image.

15. An image processing apparatus comprising:
a communication interface configured to receive an input image and rotation information associated with the input image; and
one or more processors, configured to:
generate a feature vector of the input image based on pose information corresponding to the input image;
generate an assistant feature vector which represents a target component according to a pose corresponding to the rotation information, based on the feature vector, the pose information, and the rotation information; and
generate a target image which has the pose corresponding to the rotation information based on the feature vector and the assistant feature vector.

16. The image processing apparatus of claim 15 , wherein the one or more processors are configured to:
convert the pose information to correspond to an intermediate layer of an encoder;
generate a weight map which includes a position in which the target component is to be generated in the target image, based on the feature vector, the converted pose information and the rotation information; and
generate the assistant feature vector which represents the target component corresponding to the pose in the position in which the target component is to be generated, by combining the weight map and a target feature vector separated from the feature vector.

17. The image processing apparatus of claim 16 , wherein the one or more processors are configured to generate the weight map which includes the position in which the target component is to be generated in the target image by applying the feature vector, the converted pose information and the rotation information to a first neural network.

18. The image processing apparatus of claim 16 , wherein the one or more processors are configured to generate a weight

Assignees

• Samsung Electronics Co Ltd

Inventors

• Baek Jiwon
• Park Seong-Jin
• Han Seungju
• Kim Insoo
• Han Jaejoon

Classifications

• G06N3/0455

  Auto-encoder networks; Encoder-decoder networks · CPC title

• G06N3/094

  Adversarial learning · CPC title

• G06N3/09

  Supervised learning · CPC title

• G06N3/0475

  Generative networks · CPC title

• G06N3/0464

  Convolutional networks [CNN, ConvNet] · CPC title