Image processing apparatus and control method thereof

Patent metadata

Field	Value
Publication number	US-11482000-B2
Application number	US-201916688265-A
Country	US
Kind code	B2
Filing date	Nov 19, 2019
Priority date	Nov 21, 2018
Publication date	Oct 25, 2022
Grant date	Oct 25, 2022

Abstract

Disclosed herein is an image processing apparatus and a control method thereof. The image processing apparatus includes communication circuitry, a storage, and a controller configured to control the image processing apparatus to: perform object recognition for recognizing a plurality of objects in first image data stored in the storage, obtain a score inferred through operation processing through a neural network for the recognized plurality of objects, generate second image data based on the obtained score and proximity of the plurality of objects, and perform image processing based on the second image data.

First claim

What is claimed is:

1. An image processing apparatus comprising:
communication circuitry;
a storage; and
at least one processor configured to control the image processing apparatus to: perform object recognition for recognizing a plurality of objects in first image data stored in the storage, obtain a first score through at least one neural network that is trained using data indicative of evaluation for aesthetics for each of the recognized plurality of objects, merge at least two objects among the plurality of objects based on the obtained first scores and a proximity of the plurality of objects, obtain a second score for the merged objects through the at least one neural network, generate a second image data using the merged objects in response to the second score being higher than a predetermined reference value, and perform image processing based on the second image data.

2. The image processing apparatus of claim 1 , wherein the at least one processor is configured to control the image processing apparatus to: generate a window in the first image data, and perform the object recognition based on the generated window,
wherein the window is generated based on a predetermined number and size, or generated based on a size or an aspect ratio of the image data on which the object recognition is to be performed.

3. The image processing apparatus of claim 2 , wherein the at least one processor is configured to control the image processing apparatus to: input the generated window to the at least one neural network, and select a window comprising each of the plurality of objects based on a reliability and a type of the object obtained from the at least one neural network.

4. The image processing apparatus of claim 3 , wherein
the score comprises an aesthetic score of an object contained in the selected window, and
the at least one processor is configured to control the image processing apparatus to: extract a feature from the at least one neural work and obtain the aesthetic score based on the extracted feature.

5. The image processing apparatus of claim 4 , wherein
the feature comprises at least one of color histogram features, text features, symmetry, rules of thirds features, or high level semantic features.

6. The image processing apparatus of claim 4 , wherein
the communication circuitry is configured to communicate with a server configured to perform operation processing through the at least one neural network,
wherein the at least one neural network is trained based on labeling contained in data crawling from the server.

7. The image processing apparatus of claim 4 , wherein
the at least one processor is configured to control the image processing apparatus to: determine the proximity based on a distance between the selected windows and merge the plurality of windows based on the obtained score and the proximity.

8. The image processing apparatus of claim 7 , wherein
the at least one is configured to control the image processing apparatus to: input the merged window into the at least one neural network, and compare the obtained second score with the predetermined reference value.

9. The image processing apparatus of claim 8 , wherein
based on the comparison result, the at least one processor is configured to control the image processing apparatus to generate the second image data comprising the merged window.

10. The image processing apparatus of claim 1 , wherein
the at least one processor is configured to control the image processing apparatus to perform image processing on the second image data to adjust a resolution based on the at least one neural network.

11. The image processing apparatus of claim 1 , wherein
the at least one processor is configured to control the image processing apparatus to: classify a genre of the second image data, and perform image processing to transfer the style of the second image data based on the classified genre.

12. The image processing apparatus of claim 1 , further comprising:
a sensor configured to collect data about a surrounding environment of the image processing apparatus,
wherein the at least one processor is configured to control the image processing apparatus to perform image processing to adjust brightness or contrast of the second image based on the collected data.

13. The image processing apparatus of claim 4 , wherein
the at least one processor is configured to control the image processing apparatus to: store a first feature extracted from the at least one neural network based on the selected window, perform the neural network processing on the image processed second image, and extract a second feature from the at least one neural network.

14. The image processing apparatus of claim 13 , wherein
the at least one processor is configured to control the image processing apparatus to classify a mood of the second image.

15. The image processing apparatus of claim 14 , further comprising:
a display configured to output the image processed second image data; and a sound outputter comprising sound output circuitry configured to reproduce a sound source,
wherein the at least one processor is configured to control the image processing apparatus to recommend a playlist based on the classified mood, control the sound outputter to output the sound source contained in the playlist, and control the display to output the image-processed second image data.

16. A method of controlling an image processing apparatus comprising:
performing object recognition for recognizing a plurality of objects in first image data;
obtaining a first score through at least one neural network that is trained using data indicative of evaluation for aesthetics for each of the recognized plurality of objects;
merging at least two objects among the plurality of objects based on the obtained first scores and a proximity of the plurality of objects,
obtaining a second score for the merged objects through the at least one neural network,
generating a second image data using the merged objects in response to the second score being higher than a predetermined reference value;
and
performing image processing on the second image data based on features extracted from the neural network.

17. The method of claim 16 , wherein
the performing of the object recognition comprises:
generating a window in the first image data,
performing operation processing on the generated window through the neural network; and
selecting a window comprising each of the plurality of objects based on a reliability and a type of the object through the neural network processing.

18. The method of claim 16 , wherein
the performing image processing on the second image comprises:
performing operation processing on the second image data through the neural network;
adjusting a resolution based on the result from the operation processing through the neural network;
classifying a genre of the second image; and
transferring a style of the second image data based on the classified genre.

19. The method of claim 17 , further comprising:
storing a first feature extracted in the operation processing through the neural network based on the selected window;
performing the operation processing on the image processed second image data through the neural network;
re-performing the operation processing on the second feature and the first feature extracted from the neural network processing through the neural network; and
outputting a sound source based on a playlist received based on a mood of the second image da

Assignees

• Samsung Electronics Co Ltd

Inventors

• Agarwal Vivek
• Baijal Anant
• Hyun Dae Eun
• Kwon Mi Jeong
• Choi Yun Joo

Classifications

• G06V20/10 Primary

  Terrestrial scenes (scenes under surveillance with static cameras G06V20/52; scenes perceived from the exterior of a vehicle G06V20/56; scenes perceived from the interior of a vehicle G06V20/59) · CPC title

• G06V20/20 Primary

  in augmented reality scenes · CPC title

• H04N21/4666

  using neural networks, e.g. processing the feedback provided by the user · CPC title

• G06T7/11

  Region-based segmentation · CPC title

• H04N21/4668 Primary

  for recommending content, e.g. movies · CPC title