Image processing apparatus and image processing method thereof

Patent metadata

Field	Value
Publication number	US-11836890-B2
Application number	US-202117527732-A
Country	US
Kind code	B2
Filing date	Nov 16, 2021
Priority date	May 22, 2019
Publication date	Dec 5, 2023
Grant date	Dec 5, 2023

Abstract

An image processing apparatus applies an image to a first learning network model to optimize the edges of the image, applies the image to a second learning network model to optimize the texture of the image, and applies a first weight to the first image and a second weight to the second image based on information on the edge areas and the texture areas of the image to acquire an output image.

First claim

What is claimed is:

1. An image processing apparatus comprising:
a memory storing at least one instruction; and
a processor configured to execute the at least one instruction stored in the memory to:
obtain a first image in which a first area is processed in an input image by applying the input image to a first neural network model,
obtain a second image in which a second area is processed in the input image by applying the input image to a second neural network model, and
obtain an output image based on the first and second images,

wherein each of the first neural network model and the second neural network model is trained to enhance different characteristics of an image,
wherein the first area has a characteristic of a graphic image, and
wherein the second area has a characteristic of a natural image.

2. The image processing apparatus of claim 1 , wherein the processor is further configured to identify a plurality of objects included in the input image,
wherein the first area corresponds to a first object among the plurality of objects, and
wherein the second area corresponds to a second object among the plurality of objects.

3. The image processing apparatus of claim 1 , wherein the processor is further configured to:
identify the first area included in the input image,
obtain the first image in which a first characteristic of the first area is enhanced in the input image by using the first neural network model,
identify the second area included in the input image,
obtain the second image in which a second characteristic of the second area is enhanced in the input image by using the second neural network model, and
obtain the output image by mixing the first image and the second image.

4. The image processing apparatus of claim 1 , wherein the processor is further configured to:
obtain a first weight corresponding to the first area by identifying the first area included in the input image,
obtain a second weight corresponding to the second area by identifying the second area included in the input image,
apply the first weight to the first image,
apply the second weight to the second image, and
mix the first image to which the first weight is applied and the second image to which the second weight is applied to obtain the output image.

5. The image processing apparatus of claim 1 , wherein the processor is further configured to:
classify each of image blocks constituting the input image into either the graphic image or the natural image,
obtain the first image in which a first characteristic of a block classified as the graphic image from among the image blocks is enhanced as the first image by using the first neural network model, and
obtain the second image in which a second characteristic of a block classified as the natural image from among the image blocks is enhanced as the second image by using the second neural network model.

6. The image processing apparatus of claim 5 , wherein the graphic image includes any one of an illustration image, a computer graphic image, or an animation image, and
wherein the natural image includes any one of a landscape image or a human image.

7. The image processing apparatus of claim 5 , wherein the block classified as the graphic image among the image blocks corresponds to the first area in the input image, and the block classified as the natural image among the image blocks corresponds to the second area in the input image.

8. The image processing apparatus of claim 1 , wherein the first neural network model and the second neural network model are different types of neural network models.

9. The image processing apparatus of claim 1 , wherein the first neural network model is one of a deep learning model for enhancing a first characteristic of the image by using a plurality of layers or a machine learning model trained to enhance the first characteristic of the image by using a plurality of pre-learned filters, and
wherein the second neural network model is one of a deep learning model for enhancing a second characteristic of the image by using a plurality of layers or a machine learning model trained to enhance the second characteristic of the image by using a plurality of pre-learned filters.

10. A method of controlling an image processing apparatus, the method comprising:
obtaining a first image in which a first area is processed in an input image by applying the input image to a first neural network model;
obtaining a second image in which a second area is processed in the input image by applying the input image to a second neural network model; and
obtaining an output image based on the first and second images, and
wherein each of the first neural network model and the second neural network model is trained to enhance different characteristics of an image
wherein the first area has a characteristic of a graphic image, and
wherein the second area has a characteristic of a natural image.

11. The method of claim 10 , wherein the method further comprises:
identifying a plurality of objects included in the input image, and
wherein the first area corresponds to a first object among the plurality of objects, and
wherein the second area corresponds to a second object among the plurality of objects.

12. The method of claim 10 , wherein the method further comprises:
identifying the first area included in the input image;
identifying the second area included in the input image, and
wherein the obtaining the first image comprises obtaining the first image in which a first characteristic of the first area is enhanced in the input image by using the first neural network model,
wherein the obtaining the second image comprises obtaining the second image in which a second characteristic of the second area is enhanced in the input image by using the second neural network model, and
wherein the obtaining the output image comprises obtaining the output image by mixing the first image and the second image.

13. The method of claim 10 , wherein the method further comprises:
obtaining a first weight corresponding to the first area by identifying the first area included in the input image;
obtaining a second weight corresponding to the second area by identifying the second area included in the input image;
applying the first weight to the first image;
applying the second weight to the second image, and
wherein the obtaining the output image comprises mixing the first image to which the first weight is applied and the second image to which the second weight is applied to obtain the output image.

14. The method of claim 10 , wherein the method further comprises:
classifying each of image blocks constituting the input image into either the graphic image or the natural image,
wherein the obtaining comprises obtaining the first image in which a first characteristic of a block classified as the graphic image from among the image blocks is enhanced as the first image by using the first neural network model, and
wherein the obtaining comprises obtaining the second image in which a second characteristic of a block classified as the natural image from among the image blocks is enhanced as the second image by using the second neural network model.

15. The method of claim 14 , wherein the graphic image includes any one of an illustration image, a computer graphic image, or an animation image, and
wherein the natural image includes any one of a landscape image or a human image.

16. The method of claim 14 , wherein the block classified as the graphic image among the image blocks corresponds to the first area

Assignees

• Samsung Electronics Co Ltd

Inventors

• Lee Cheon
• Kim Donghyun
• Park Yongsup
• Park Jaeyeon
• Ahn Iljun
• Lee Hyunseung
• Ahn Taegyoung
• Moon Youngsu
• Lee Tammy

Classifications

• G06T5/60

  using machine learning, e.g. neural networks · CPC title

• G06T3/4053 Primary

  based on super-resolution, i.e. the output image resolution being higher than the sensor resolution · CPC title

• G06T3/4046

  using neural networks · CPC title

• G06T3/403 Primary

  Edge-driven scaling; Edge-based scaling · CPC title

• G06T7/13

  Edge detection · CPC title