Image manipulation method and apparatus

Patent metadata

Field	Value
Publication number	US-11869127-B2
Application number	US-202017612053-A
Country	US
Kind code	B2
Filing date	May 14, 2020
Priority date	May 17, 2019
Publication date	Jan 9, 2024
Grant date	Jan 9, 2024

Abstract

A computer-implemented image manipulation apparatus and method ( 200 ) configured to receiving an input image ( 202 ) and a desired style. The method can obtain a representation ( 204 ) of the input image selected from a plurality of stored representations of a plurality of images, wherein each said representation comprises data describing a set of image features. The method can modify image features in the obtained representation to correspond to the input image and/or the desired style to produce a modified representation ( 207 ), and render a reference image ( 209 ) based on the modified representation. A manipulated image is generated by performing a style transfer operation ( 210 ) on the input image using the rendered reference image. Embodiments may access a data store to find a group of stored images based on similarity between image content descriptors of groups of stored images and those of an input image to retrieve a stored reference image.

First claim

The invention claimed is:

1. A computer-implemented image manipulation method comprising:
receiving an input image;
receiving a desired style to be transferred to the input image;
obtaining a representation of the input image selected from a plurality of stored representations of a plurality of images, wherein each of the plurality of stored representations comprises data describing a set of image features;
modifying at least one of the set of image features in the obtained representation to correspond to the input image and/or the desired style to produce a modified representation;
rendering a reference image based on the modified representation, and
generating a manipulated image by performing a style transfer operation on the input image using the reference image,
wherein the method further comprises:
determining a set of desired image content descriptors of an image to meet user demands;
generating a new image having the set of desired image content descriptors;
obtaining a set of image capture characteristic descriptors of the new image; and
adding the set of desired image content descriptors and the set of image capture characteristic descriptors obtained for the new image to a data store.

2. The method of claim 1 , wherein the plurality of representations comprise a respective plurality of statistical representations, and wherein the image features of the statistical representations comprise common features of the plurality of images identified by a content analysis method performed on the plurality of images.

3. The method of claim 2 , wherein the content analysis method is performed by a statistical system to generate the statistical representations of the plurality of images.

4. The method of claim 3 , wherein the statistical system comprises a machine learning technique that learns a distribution of the identified common features across the plurality of images.

5. The method of claim 4 , wherein the machine learning technique comprises a dimensionality reduction process.

6. The method of claim 3 , wherein the of rendering the reference image comprises a reverse of a process used to generate the statistical representations of the plurality of images.

7. The method of claim 6 , wherein the reference image comprises a synthetic rendering of the input image.

8. The method of claim 7 , wherein the input image comprises a face and the synthetic rendering comprises a 3D rendering of the face.

9. The method of claim 2 , wherein the plurality of images comprise a dataset of example images of a particular type, and the set of image features comprise principal features that change across the plurality of images in the dataset.

10. The method of claim 1 , wherein the of obtaining the representation of the input image comprises finding a said representation amongst the plurality of stored representations that has a greatest visual similarity to the input image.

11. The method of claim 1 , wherein:
each of the image features of the obtained representation has an associated value describing a property of the image feature;
the desired style comprises a set of image features, and each of the image features of the desired style has the associated value, and
wherein the modifying the at least one of the set of image features in the obtained representation comprise modifying the value of the image feature of the obtained representation to correspond to a value of a corresponding the image feature in the input image and/or the desired style.

12. The method of claim 11 , wherein the desired style is based on a style image that provides the value for each of the image features of the desired style.

13. The method of claim 1 , further comprising:
obtaining the set of image content descriptors for the input image;
accessing a data store related to a plurality of stored images, the data store comprising a set of image content descriptors related to each of the plurality of stored images and the set of image capture characteristic descriptors related to each of the plurality of stored images, the stored images being logically arranged in a plurality of groups, wherein the stored images in one the group have a same set of image content descriptors and different the sets of image capture characteristic descriptors;
finding at least one group of stored images in the data store based on similarity between the set of image content descriptors of the groups of stored images and the set of image content descriptors of the input image;
retrieving the sets of image capture characteristic descriptors associated with the stored images in the at least one found group of stored images;
receiving a selection of a set of image capture characteristic descriptors from amongst the retrieved sets of image capture characteristic descriptors;
retrieving the stored image associated with the selected set of image capture characteristic descriptors, and
generating a manipulated image by performing a style transfer operation on the input image using the retrieved stored image as a reference image,
wherein the data store is created based on analysis of activities of at least one user in relation to images.

14. Apparatus configured to perform image manipulation, the apparatus comprising:
a processor configured to:
receive an input image;
receive a desired style to be transferred to the input image;
obtain a representation of the input image selected from a plurality of stored representations of a plurality of images, wherein each of the plurality of representations comprises data describing a set of image features,
modify at least one of the set of image features in the obtained representation to correspond to the input image and/or the desired style to produce a modified representation;
render a reference image based on the modified representation, and
generate a manipulated image by performing a style transfer operation on the input image using the reference image,
wherein the processor is further configured to:
determine a set of desired image content descriptors of an image to meet user demands;
generate a new image having the set of desired image content descriptors;
obtain a set of image capture characteristic descriptors of the new image; and
add the set of desired image content descriptors and the set of image capture characteristic descriptors obtained for the new image to a data store.

15. A non-transitory computer readable recording medium storing a computer program for performing an image manipulation method, the program performing a method comprising:
receiving an input image;
receiving a desired style to be transferred to the input image;
obtaining a representation of the input image selected from a plurality of stored representations of a plurality of images, wherein each of the plurality of representations comprises data describing a set of image features;
modifying at least one of the set of image features in the obtained representation to correspond to the input image and/or the desired style to produce a modified representation;
rendering a reference image based on the modified representation; and
generating a manipulated image by performing a style transfer operation on the input image using the reference image,
wherein the method further comprises:
determining a set of desired image content descriptors of an image to meet user demands;
generating a new image having the set of desired image content descriptors;
obtaining a set of image capture characteristic descriptors of the new image; and
adding the set of desired image content descriptors and the set of image capture charact

Assignees

• Samsung Electronics Co Ltd

Inventors

• Saa-Garriga Albert
• Vandini Alessandro
• Larreche Antoine
• Shah Javed Rabbani
• Sharma Prashant

Classifications

• G06T11/10

  Texturing; Colouring; Generation of textures or colours (retouching, inpainting or scratch removal G06T5/77) · CPC title

• G06T11/40

  Filling planar surfaces by adding surface attributes, e.g. adding colours or textures · CPC title

• G06F18/2413

  based on distances to training or reference patterns · CPC title

• G06T2207/20221

  Image fusion; Image merging · CPC title

• G06N3/0464

  Convolutional networks [CNN, ConvNet] · CPC title