Machine learning to predict cognitive image composition

Patent metadata

Field	Value
Publication number	US-11468664-B2
Application number	US-202016775464-A
Country	US
Kind code	B2
Filing date	Jan 29, 2020
Priority date	Aug 24, 2017
Publication date	Oct 11, 2022
Grant date	Oct 11, 2022

Abstract

An automatic method of determining an image composition procedure that generates a new image visualization based on aggregations and variations of input images. A set of input images is received. Visual features are extracted from the input images. Context associated with input images is received. Based on the extracted visual features and the context associated with the input images, a composition procedure comprising a set of image operations to apply on the set of input images is learned. One or more image operations in the composition procedure are determined to present to a user. A difference visualization image associated with the input images may be generated by executing the one or more image operations.

First claim

We claim:

1. An automatic method of determining an image composition procedure that generates a new image visualization, the method executed by a hardware processor, comprising:
receiving a set of input images;
extracting visual features from the input images;
receiving context associated with the input images;
learning based on the extracted visual features and the context associated with the input images, a composition procedure comprising at least a sequence of set of image operations to apply on the set of input images, the sequence of set of image operations obeying a sequential structure, wherein a previous operation in the sequence creates data for a next operation in the sequence;
based on the learning, determining image operations in the composition procedure to present to a user, the image operations runnable on the hardware processor for the hardware processor to perform a function; and
generating a difference visualization image associated with the input images by executing the image operations.

2. The method of claim 1 , wherein the difference visualization image represents a variation of the set of input images.

3. The method of claim 1 , wherein the difference visualization image represents an aggregation of the set of input images.

4. The method of claim 1 , further comprising:
monitoring execution of the image operations on a training set of images, the image operations generating a difference visualization image associated with the training set of images, wherein multiple executions on multiple sets of images are monitored that generate corresponding multiple difference visualization images.

5. The method of claim 4 , further comprising:
storing in a knowledgebase, the image operations, visual features in the training set of images, and context associated with the training set of images.

6. The method of claim 1 , further comprising receiving from the user a rating associated with a suggested image operation, and the learning is further based on the rating.

7. The method of claim 6 , wherein the rating is used by a reinforcement learning process to improve a reward associated to actions, wherein additional composition procedure comprising a sequence of the actions is presented to the user.

8. The method of claim 1 , wherein the context comprises whether the input images are of synthetic objects or natural objects, whether the input images are 2-dimensional or 3-dimensional, image count in the input images, a degree of variability among the input images, a number of colors in the input images, whether a legend is specified in the input images, a domain corresponding to the input images, and an objective in analyzing the input images.

9. The method of claim 1 , wherein the step of extracting visual features from the input images comprises executing a Deep Convolutional Activation Feature (DeCAF) extractor to learn visual features from the set of input images.

10. A computer program product comprising a computer readable storage medium having program instructions embodied therewith, the program instructions executable by a device to cause the device to:
receive a set of input images;
extract visual features from the input images;
receive context associated with the input images;
learn based on the extracted visual features and the context associated with the input images, a composition procedure comprising at least a sequence of set of image operations to apply on the set of input images, the sequence of set of image operations obeying a sequential structure, wherein a previous operation in the sequence creates data for a next operation in the sequence;
based on the learned composition procedure, determine image operations in the composition procedure to present to a user, the image operations runnable by the device to perform a function; and
generate a difference visualization image associated with the input images by executing the image operations.

11. The computer program product of claim 10 , wherein the difference visualization image represents a variation of the set of input images.

12. The computer program product of claim 10 , wherein the difference visualization image represents an aggregation of the set of input images.

13. The computer program product of claim 10 , wherein the device is further caused to monitor execution of the image operations on a training set of images, the image operations generating a difference visualization image associated with the training set of images, wherein multiple executions on multiple sets of images are monitored that generate corresponding multiple difference visualization images.

14. The computer program product of claim 13 , wherein the device is further caused to store in a knowledgebase, the image operations, visual features in the training set of images, and context associated with the training set of images.

15. The computer program product of claim 10 , wherein the device is further caused to receive from the user a rating associated with a suggested image operation, and learn further based on the rating.

16. The computer program product of claim 15 , wherein the rating is used by a reinforcement learning process to improve a reward associated to actions, wherein an additional composition procedure comprising a sequence of the actions is presented to the user.

17. The computer program product of claim 10 , wherein the context comprises whether the input images are of synthetic objects or natural objects, whether the input images are 2-dimensional or 3-dimensional, image count in the input images, a degree of variability among the input images, a number of colors in the input images, whether a legend is specified in the input images, a domain corresponding to the input images, and an objective in analyzing the input images.

18. The computer program product of claim 10 , wherein the step of extracting visual features from the input images comprises executing a Deep Convolutional Activation Feature (DeCAF) extractor to learn visual features from the set of input images.

19. An automatic system of determining an image composition procedure that generates a new image visualization, comprising:
a hardware processor;
a memory device coupled with the hardware processor;
the hardware processor configured to at least:
receive a set of input images;
extract visual features from the input images;
receive context associated with the input images;
learn based on the extracted visual features and the context associated with the input images, a composition procedure comprising at least a sequence of set of image operations to apply on the set of input images, the sequence of set of image operations obeying a sequential structure, wherein a previous operation in the sequence creates data for a next operation in the sequence;
based on the learned composition procedure, determine image operations in the composition procedure to present to a user, the image operations runnable on the hardware processor for the hardware processor to perform a function; and
generate a difference visualization image associated with the input images by executing the image operations.

20. The system of claim 19 , the hardware processor executes a reinforcement learning process using a rating associated with the determined image operations, to improve a reward associated with actions, wherein an additional composition procedure comprising a sequence of the actions is presented to the user.

Assignees

• IBM

Inventors

• Borrel Paul
• Buoro Alvaro B
• Barros Ruberth A A
• Salles Civitarese Daniel

Classifications

• G06V10/803

  of input or preprocessed data · CPC title

• G06N3/006 Primary

  based on simulated virtual individual or collective life forms, e.g. social simulations or particle swarm optimisation [PSO] · CPC title

• G06V20/00 Primary

  Scenes; Scene-specific elements (control of digital cameras H04N23/60) · CPC title

• G06F18/251

  of input or preprocessed data · CPC title

• G06V10/24

  Aligning, centring, orientation detection or correction of the image · CPC title