Query image synthesis for image retrieval systems based on data augmentation techniques

Patent metadata

Field	Value
Publication number	US-2022019849-A1
Application number	US-202016932130-A
Country	US
Kind code	A1
Filing date	Jul 17, 2020
Priority date	Jul 17, 2020
Publication date	Jan 20, 2022
Grant date	—

Abstract

Methods, systems, and articles of manufacture, including computer program products, are provided for synthesizing images for machine learning. The method may include selecting one or more image preprocessing transformations to apply on the foreground object image; applying the selected one or more image preprocessing transformations to the foreground object image; selecting a background image from a set of background images depicting a variety of different backgrounds which may be associated with the foreground object image; merging the selected background image with the foreground object image to form a synthesized image; selecting one or more image transformations to apply on the synthesized image; applying the selected one or more image transformations to the synthesized image; and storing the synthesized image in a collection of synthesized images to train a machine learning model.

First claim

What is claimed is:

1 . A system, comprising:
at least one data processor; and at least one memory storing instructions which, when executed by the at least one data processor, result in operations comprising:
selecting one or more image preprocessing transformations to apply on the foreground object image;
applying the selected one or more image preprocessing transformations to the foreground object image;
selecting a background image from a set of background images depicting a variety of different backgrounds which may be associated with the foreground object image;
merging the selected background image with the foreground object image to form a synthesized image;
selecting one or more image transformations to apply on the synthesized image;
applying the selected one or more image transformations to the synthesized image; and
storing the synthesized image in a collection of synthesized images to train a machine learning model to identify, from a plurality of catalog images, one or more catalog images that match a query image received from a user.

2 . The system of claim 1 , wherein the one or more image preprocessing transformations include one or more of the following: varying a color of the foreground object image, varying a brightness of the foreground object image, varying a contrast of the foreground object image, rotating the foreground object image, scaling a size of the foreground object image, varying a transparency of the foreground object image, and blurring the foreground object image.

3 . The system of claim 1 , wherein the merging includes placing, within the background image, the foreground object image at a selected position, orientation, and/or point of view.

4 . The system of claim 1 , wherein the one or more image transformations include one or more of the following: varying a color of the synthesized image, varying a brightness of the synthesized image, varying a contrast of the synthesized image, blurring at least a portion of the synthesized image, and applying a light filter.

5 . The system of claim 1 further comprising:
training the machine learning model based on the synthesized images.

6 . The system of claim 1 further comprising:
providing the machine learning model to an image search system.

7 . The system of claim 1 further comprising:
receiving, from a user equipment, a query including an image depicting an item.

8 . The system of claim 7 further comprising:
performing an image search for one or more catalog images that match the image of the received query, wherein the machine learning model is trained based on the collection of synthesized images to identify the one or more catalog images; and
responding to the query by providing one or more copies of the one or more catalog images to the user equipment.

9 . The system of claim 1 , wherein the foreground object image is extracted from a catalog image received from a database including the plurality of catalog images.

10 . A method comprising:
selecting one or more image preprocessing transformations to apply on the foreground object image; applying the selected one or more image preprocessing transformations to the foreground object image; selecting a background image from a set of background images depicting a variety of different backgrounds which may be associated with the foreground object image; merging the selected background image with the foreground object image to form a synthesized image; selecting one or more image transformations to apply on the synthesized image; applying the selected one or more image transformations to the synthesized image; and storing the synthesized image in a collection of synthesized images to train a machine learning model to identify, from a plurality of catalog images, one or more catalog images that match a query image received from a user.

11 . The method of claim 10 , wherein the one or more image preprocessing transformations include one or more of the following: varying a color of the foreground object image, varying a brightness of the foreground object image, varying a contrast of the foreground object image, rotating the foreground object image, scaling a size of the foreground object image, varying a transparency of the foreground object image, and blurring the foreground object image.

12 . The method of claim 10 , wherein the merging includes placing, within the background image, the foreground object image at a selected position, orientation, and/or point of view.

13 . The method of claim 10 , wherein the one or more image transformations include one or more of the following: varying a color of the synthesized image, varying a brightness of the synthesized image, varying a contrast of the synthesized image, blurring at least a portion of the synthesized image, and applying a light filter.

14 . The method of claim 10 further comprising:
training the machine learning model based on the synthesized images.

15 . The method of claim 10 further comprising:
providing the machine learning model to an image search system.

16 . The method of claim 10 further comprising:
receiving, from a user equipment, a query including an image depicting an item.

17 . The method of claim 17 further comprising:
performing an image search for one or more catalog images that match the image of the received query, wherein the machine learning model is trained based on the collection of synthesized images to identify the one or more catalog images; and
responding to the query by providing the one or more catalog images to the user equipment.

18 . The method of claim 10 , wherein the foreground object image is extracted from a catalog image received from a database including the plurality of catalog images.

19 . A non-transitory computer-readable storage medium including program code which when executed causes operations comprising:
selecting one or more image preprocessing transformations to apply on the foreground object image; applying the selected one or more image preprocessing transformations to the foreground object image; selecting a background image from a set of background images depicting a variety of different backgrounds which may be associated with the foreground object image; merging the selected background image with the foreground object image to form a synthesized image; selecting one or more image transformations to apply on the synthesized image; applying the selected one or more image transformations to the synthesized image; and storing the synthesized image in a collection of synthesized images to train a machine learning model to identify, from a plurality of catalog images, one or more catalog images that match a query image received from a user.

20 . The non-transitory computer-readable storage medium of claim 19 , wherein the one or more image preprocessing transformations include one or more of the following: varying a color of the foreground object image, varying a brightness of the foreground object image, varying a contrast of the foreground object image, rotating the foreground object image, scaling a size of the foreground object image, varying a transparency of the foreground object image, and blurring the foreground object image.

Assignees

• Sap Se

Inventors

• Kim Sohyeong
• Jiang Ying
• Guder Cordula

Classifications

• G06F2218/02

  Preprocessing · CPC title

• G06F18/214 Primary

  Generating training patterns; Bootstrap methods, e.g. bagging or boosting · CPC title

• G06N3/0464

  Convolutional networks [CNN, ConvNet] · CPC title

• G06N3/09

  Supervised learning · CPC title

• G06V10/774

  Generating sets of training patterns; Bootstrap methods, e.g. bagging or boosting · CPC title