Image and data processing methods and apparatuses

Patent metadata

Field	Value
Publication number	US-11527056-B2
Application number	US-202117186493-A
Country	US
Kind code	B2
Filing date	Feb 26, 2021
Priority date	Feb 28, 2020
Publication date	Dec 13, 2022
Grant date	Dec 13, 2022

Abstract

The specification discloses image and data processing methods and apparatuses. The method includes: obtaining a source pose and texture information according to a source image; obtaining a first synthetic image according to the source image, a target pose, and the source pose; obtaining a residual map according to the texture information and the first synthetic image; and obtaining a second synthetic image according to the first synthetic image and the residual map. The specification resolves the technical problem of lacking a sense of reality in a synthetic image due to loss of texture details in feature extraction during character action transfer in the existing technologies.

First claim

What is claimed is:

1. An image processing method, comprising:
obtaining a source pose and texture information according to a source image;
obtaining a first synthetic image according to the source image, a target pose, and the source pose;
obtaining a residual map according to the texture information and the first synthetic image; and
obtaining a second synthetic image according to the first synthetic image and the residual
wherein:
the obtaining the first synthetic image comprises performing pose transfer according to the source image, the target pose, and the source pose to obtain the first synthetic image; the obtaining the residual map comprises performing feature enhancement according to the texture information to obtain the residual map, wherein the residual map comprises contour features and surface texture details in the source image; the obtaining the second synthetic image comprises filling the first synthetic image with the contour features and the surface texture details in the residual map to generate the second synthetic image; and the method further comprises determining the second synthetic image as a display image for display.

2. The method of claim 1 , wherein the obtaining a source pose and texture information according to a source image comprises:
obtaining the source pose of the source image through pose estimation according to the source image; and
obtaining the texture information of the source image by performing feature extraction on the source image.

3. The method of claim 1 , wherein the obtaining a first synthetic image according to the source image, a target pose, and the source pose comprises:
obtaining a content feature map according to the source image, the target pose, and the source pose; and
obtaining the first synthetic image according to the content feature map.

4. The method of claim 3 , wherein the obtaining a residual map according to the texture information and the first synthetic image comprises:
performing normalization processing according to a texture code and the content feature map to obtain the residual map, wherein the normalization processing comprises:
performing deep learning on the texture code, normalizing the texture code on which deep learning has been performed and the content feature map, and performing reconstruction, to obtain the residual map, wherein the residual map comprises contour features and surface texture details in the source image.

5. The method of claim 4 , wherein
the contour features comprise at least one of: a human face, an animal head, a body feature, or an appearance feature of an article; and
the surface texture details comprise product surface texture details, wherein the product surface texture details comprise at least one of: clothing texture details, accessory texture details, or tool texture details.

6. The method of claim 4 , wherein the obtaining a second synthetic image according to the first synthetic image and the residual map comprises:
performing superposition according to the first synthetic image and the residual map, to obtain the second synthetic image.

7. The method of claim 6 , wherein the performing superposition according to the first synthetic image and the residual map, to obtain the second synthetic image comprises:
according to the contour features and the surface texture details of the source image in the residual map, filling the contour features and the surface texture details at corresponding positions in the first synthetic image to obtain the second synthetic image, wherein the second synthetic image has the contour features and the surface texture details in the source image.

8. The method of claim 1 , further comprising:
prior to obtaining the source pose and texture information according to the source image, receiving the source image uploaded by a user; and
subsequent to the obtaining the second synthetic image, generating an image set or video data according to the second synthetic image.

9. The method of claim 8 , wherein the image set or the video data is applicable to online fitting effect display or advertisement page display.

10. A system for image processing, comprising a processor and a non-transitory computer-readable storage medium storing instructions executable by the processor to cause the system to perform operations comprising:
obtaining a source pose and texture information according to a source image;
obtaining a first synthetic image according to the source image, a target pose, and the source pose;
obtaining a residual map according to the texture information and the first synthetic image; and
obtaining a second synthetic image according to the first synthetic image and the residual
wherein:
the obtaining the first synthetic image comprises performing pose transfer according to the source image, the target pose, and the source pose to obtain the first synthetic image; the obtaining the residual map comprises performing feature enhancement according to the texture information to obtain the residual map, wherein the residual map comprises contour features and surface texture details in the source image; the obtaining the second synthetic image comprises filling the first synthetic image with the contour features and the surface texture details in the residual map to generate the second synthetic image; and the operations further comprise determining the second synthetic image as a display image for display.

11. The system of claim 10 , wherein the obtaining a first synthetic image according to the source image, a target pose, and the source pose comprises:
obtaining a content feature map according to the source image, the target pose, and the source pose; and
obtaining the first synthetic image according to the content feature map.

12. The system of claim 11 , wherein the obtaining a residual map according to the texture information and the first synthetic image comprises:
performing normalization processing according to a texture code and the content feature map to obtain the residual map, wherein the normalization processing comprises: performing deep learning on the texture code, normalizing the texture code on which deep learning has been performed and the content feature map, and performing reconstruction, to obtain the residual map, wherein the residual map comprises contour features and surface texture details in the source image.

13. The system of claim 12 , wherein the obtaining a second synthetic image according to the first synthetic image and the residual map comprises:
performing superposition according to the first synthetic image and the residual map, to obtain the second synthetic image by: according to the contour features and the surface texture details of the source image in the residual map, filling the contour features and the surface texture details at corresponding positions in the first synthetic image to obtain the second synthetic image, wherein the second synthetic image has the contour features and the surface texture details in the source image.

14. A non-transitory computer-readable storage medium for image processing, configured with instructions executable by one or more processors to cause the one or more processors to perform operations comprising:
obtaining a source pose and texture information according to a source image;
obtaining a first synthetic image according to the source image, a target pose, and the source pose;
obtaining a residual map according to the texture information and the first synthetic image; and
obtaining a second synthetic image according to the first synthetic image and the residual
Wherein:
the obtaini

Assignees

• Alibaba Group Holding Ltd

Inventors

• Yang Lingbo
• Wang Pan
• Ren Peiran
• Xie Xuansong
• Gao Zhanning

Classifications

• G06V40/10

  Human or animal bodies, e.g. vehicle occupants or pedestrians; Body parts, e.g. hands · CPC title

• G06V10/82

  using neural networks · CPC title

• G06V10/764

  using classification, e.g. of video objects · CPC title

• G06V10/44 Primary

  Local feature extraction by analysis of parts of the pattern, e.g. by detecting edges, contours, loops, corners, strokes or intersections; Connectivity analysis, e.g. of connected components · CPC title

• G06Q30/0277 Primary

  Online advertisement · CPC title