Image style conversion method, apparatus and device

Patent metadata

Field	Value
Publication number	US-10970830-B2
Application number	US-201916421923-A
Country	US
Kind code	B2
Filing date	May 24, 2019
Priority date	Oct 31, 2018
Publication date	Apr 6, 2021
Grant date	Apr 6, 2021

Abstract

The present disclosure relates to a method, an apparatus and a device for converting a style of an image, wherein the method comprises: acquiring a luminance component (Y) and chrominance components (U, V) in a YUV space of an image to be processed; performing a group convolution processing on the luminance component (Y) and the chrominance components (U, V) in the YUV space of the image to be processed to obtain content features and style features of the image to be processed; and performing a fusion processing on the content features, the style features and target style features of the image to be processed to convert the image to be processed into an image of a target style.

First claim

What is claimed is:

1. A method for converting a style of an image, comprising:
acquiring a luminance component (Y) and chrominance components (U, V) in a YUV space of an image to be processed;
performing a group convolution processing on the luminance component (Y) and the chrominance components (U, V) in the YUV space of the image to be processed to obtain content features and style features of the image to be processed; and
performing a fusion processing on the content features, the style features and target style features of the image to be processed to convert the image to be processed into an image of a target style,
wherein performing the group convolution processing on the luminance component (Y) and the chrominance components (U, V) in the YUV space of the image to be processed comprises:
performing a depth convolution processing on the luminance component (Y) in the YUV space of the image to be processed to obtain a structural feature vector at a first depth of the image to be processed;
performing a down-sampling processing on the structural feature vector at the first depth of the image to be processed to obtain a structural feature vector at a second depth of the image to be processed; and
inputting the structural feature vector at the second depth of the image to be processed and the chrominance components (U, V) in the YUV space of the image to be processed as input data to the deep convolutional neural network, which, after passing through a number of convolutional layers and pooling layers of the deep convolutional neural network, passes through a flatten layer, a fully connected layer and a softmax layer, to derive content features and style features of the image to be processed.

2. The method according to claim 1 , wherein, before performing the group convolution processing on the luminance component (Y) and the chrominance components (U, V) in the YUV space of the image to be processed, the method further comprises:
performing a chrominance sampling processing on the chrominance components (U, V) in the YUV space of the image to be processed, wherein sampling rates of the chrominance components (U, V) are different.

3. The method according to claim 2 , wherein performing the chrominance sampling processing on the chrominance components (U, V) in the YUV space of the image to be processed comprises:
performing the chrominance sampling processing on the chrominance components (U, V) in the YUV space of the image to be processed according to one of formats of YUV420, YUV444, YUV422 or YUV411.

4. The method according to claim 1 , wherein performing the fusion processing on the content features, the style features and the target style features of the image to be processed comprises:
performing the fusion processing on the content features, the style features and the target style features of the image to be processed to generate the luminance component (Y) and the chrominance components (U, V) in the YUV space of the image to be processed in the target style;
determining, according to the luminance component (Y) and the chrominance components (U, V) in the YUV space of the image to be processed in the target style, components of respective color channels in an RGB space of the image of the target style; and
outputting, according to the components of the respective color channels in the RGB space of the image of the target style, the image of the target style converted from the image to be processed.

5. The method according to claim 1 , further comprising:
acquiring a set of training samples, wherein each training sample in the set of training samples includes an image of an original style and the image of the target style which have a same content; and
training an initial group convolutional neural network with the set of training samples to generate a group convolutional neural network for converting the style of the image to the target style.

6. An apparatus for converting a style of an image, comprising:
a first acquiring circuit for acquiring a luminance component (Y) and chrominance components (U, V) in a YUV space of an image to be processed;
a second acquiring circuit for performing a group convolution processing on the luminance component (Y) and the chrominance components (U, V) in the YUV space of the image to be processed to obtain content features and style features of the image to be processed; and
a fusion processing circuit for performing a fusion processing on the content features, the style features and target style features of the image to be processed to convert the image to be processed into an image of a target style,
wherein the second acquiring circuit comprises:
a first acquiring sub-circuit for performing a depth convolution processing on the luminance component (Y) in the YUV space of the image to e processed to obtain a structural feature vector at a first depth of the image to be processed;
a second acquiring sub-circuit for performing a down-sampling processing on the structural feature vector at a first depth of the image to be processed to obtain a structural feature vector at a second depth of the image to be processed; and
a third acquiring sub-circuit for inputting the structural feature vector at the second depth of the image to be processed and the chrominance components (U, V) in the YUV space of the image to be processed as input data to the deep convolutional neural network, which, after passing through a number of convolutional layers and pooling layers of the deep convolutional neural network, passes through a flatten layer, a fully connected layer and a softmax layer, to derive content features and style features of the image to be processed.

7. The apparatus according to claim 6 , further comprises:
a sampling processing circuit for performing a chrominance sampling processing on the chrominance components (U, V) in the YUV space of the image to be processed, wherein sampling rates of the chrominance components (U, V) are different.

8. The apparatus according to claim 7 , wherein the sampling processing circuit performs the chrominance sampling processing on the chrominance components (U, V) in the YUV space of the image to be processed according to one of formats of YUV420, YUV444, YUV422 or YUV411.

9. The apparatus according to claim 6 , wherein the fusion processing circuit comprises:
a fusion processing sub-circuit for performing the fusion processing on the content features, the style features and the target style features of the image to be processed to generate the luminance component (Y) and the chrominance components (U, V) in the YUV space of the image to be processed in the target style;
a determining sub-circuit for determining, according to the luminance component (Y) and the chrominance components (U, V) in the YUV space of the image to be processed in the target style, components of respective color channels in an RGB space of the image of the target style; and
an outputting sub-circuit for outputting, according to the components of the respective color channels in the RGB space of the image of the target style, the image of the target style converted from the image to be processed.

10. The apparatus according to claim 6 , further comprising:
a third acquiring circuit for acquiring a set of training samples, wherein each training sample in the set of training samples includes an image of an original style and the image of the target style which have a same content; and
a training circuit for training an initial group convolutional neural network with the set of training samples to generate a group convolutional neural network for converting the style of the image to the target style.

11. A computer

Assignees

• Boe Technology Group Co Ltd

Inventors

• Zhang Lijie
• Liu Hanwen

Classifications

• H04N1/6005 Primary

  with luminance or chrominance signals, e.g. LC1C2, HSL or YUV · CPC title

• G06V10/56

  relating to colour · CPC title

• G06T5/50 Primary

  using two or more images, e.g. averaging or subtraction · CPC title

• G06F18/2414 Primary

  Smoothing the distance, e.g. radial basis function networks [RBFN] · CPC title

• G06F18/253

  of extracted features · CPC title