Systems and methods for improved image textures

Patent metadata

Field	Value
Publication number	US-10311326-B2
Application number	US-201715476205-A
Country	US
Kind code	B2
Filing date	Mar 31, 2017
Priority date	Mar 31, 2017
Publication date	Jun 4, 2019
Grant date	Jun 4, 2019

Abstract

Methods and systems for image texture enhancement are disclosed. In some aspects, texture information for a plurality of object types is stored in a database. Objects are recognized in an image, and the type of each recognized object is identified. The database is consulted to determine textures for each of the recognized objects, based on the type of each object. A portion of a new image representing the recognized object is then updated based on its determined texture. Multiple objects having multiple different textures may be updated in this manner within a single image. This may result in improved image textures over known methods, especially when low light exposures may result in reduced image resolution and degraded textures.

First claim

What is claimed is:

1. A method of generating images, the method performed by a processor of a system and comprising:
capturing an image using an imaging sensor;
receiving image data from the imaging sensor;
identifying a first and second object represented by the image data;
identifying a first and second type of the first and second objects, respectively;
identifying first style information for the first type of object and different second style information for the second type of object from an object database;
generating second image data representing the first and second objects based on the image data and the first and second style information; and
providing the second image data to an output device.

2. The method of claim 1 , further comprising identifying the first and second object via a convolutional neural network.

3. The method of claim 2 , further comprising utilizing artistic style transfer to transfer different styles to the first and second objects in the second image data based on the first and second style information, respectively.

4. The method of claim 3 , wherein utilizing the artistic style transfer comprises at least one of minimizing a distance between the convolutional neural network's content representation of the image data representing the first object and the convolutional neural network's content representation of the second image data representing the first object or minimizing a second distance between style representations of the first object in the second image data and the first style information.

5. The method of claim 3 , wherein utilizing the artistic style transfer further comprises at least one of minimizing a third distance between the convolutional neural network's content representation of the image data representing the second object and the convolutional neural network's content representation of the second image data representing the second object or minimizing a fourth distance between style representations of the second object in the second image and the second style information.

6. The method of claim 5 , wherein the content representations are based on feature representations for a plurality of layers in the convolutional neural network of the first and second objects represented by the image data and of the first object in the second image data.

7. The method of claim 6 , wherein the feature representations are based on filtered image data.

8. The method of claim 5 , wherein the style representations are based on correlations between different filter responses of the convolutional neural network to the first object in the second image data and correlations between different filter responses stored in the object database for objects of the first type.

9. The method of claim 1 , further comprising:
identifying low exposure and high exposure portions of the image data; and
generating the second image based on the first or second style information for low exposure portions of the image data.

10. An apparatus for generating an image comprising:
a camera including an imaging sensor configured to capture one or more images; and
a processor configured to:
receive image data from the one or more images captured by the imaging sensor;
identify a first and second object represented by the image data;
identify a first and second type of the first and second objects, respectively;
identify first style information for the first type of object and different second style information for the second type of object from an object database;
generate second image data representing the first and second objects based on the image data and the first and second style information; and
provide the second image data to an output device.

11. The apparatus of claim 10 , wherein the processor is further configured to utilize artistic style transfer to transfer different styles to the first and second objects in the second image data based on the first and second style information respectively.

12. The apparatus of claim 11 , wherein utilizing the artistic style transfer comprises at least one of minimizing a distance between a convolutional neural network's content representation of the image data representing the first object and the convolutional neural network's content representation of the second image data representing the first object or minimizing a second distance between style representations of the first object in the second image data and the first style information.

13. The apparatus of claim 12 , wherein utilizing the artistic style transfer further comprises at least one of minimizing a third distance between the convolutional neural network's content representation of the image data representing the second object and the convolutional neural network's content representation of the second image data representing the second object or minimizing a fourth distance between style representations of the second object in the second image and the second style information.

14. The apparatus of claim 13 , wherein the content representations are based on feature representations for a plurality of layers in the convolutional neural network of the first and second objects represented by the image data and of the first object in the second image data.

15. The apparatus of claim 14 , wherein the feature representations are based on filtered image data.

16. The apparatus of claim 13 , wherein the style representations are based on correlations between different filter responses of the convolutional neural network to the first object in the second image data and correlations between different filter responses stored in the object database for objects of the first type.

17. The apparatus of claim 10 , wherein the first and second objects are identified via a convolutional neural network.

18. The apparatus of claim 10 , wherein the processor is further configured to:
identify low exposure and high exposure portions of the image data; and
generate the second image based on the first or second style information for low exposure portions of the image data.

19. A non-transitory computer readable medium storing instructions that, when executed by a processor of an apparatus, cause the apparatus to perform a number of operations comprising:
capturing an image using an imaging sensor;
receiving image data from the imaging sensor;
identifying a first and second object represented by the image data;
identifying a first and second type of the first and second objects, respectively;
identifying first style information for the first type of object and different second style information for the second type of object from an object database;
generating second image data representing the first and second objects based on the image data and the first and second style information; and
providing the second image data to an output device.

Assignees

• Qualcomm Inc

Inventors

• Amitay Amichay

Classifications

• G06T11/10

  Texturing; Colouring; Generation of textures or colours (retouching, inpainting or scratch removal G06T5/77) · CPC title

• G06F18/22

  Matching criteria, e.g. proximity measures · CPC title

• G06F18/214

  Generating training patterns; Bootstrap methods, e.g. bagging or boosting · CPC title

• G06F18/2413

  based on distances to training or reference patterns · CPC title

• G06T5/50

  using two or more images, e.g. averaging or subtraction · CPC title