Eyeglass reflection synthesis for reflection removal

Patent metadata

Field	Value
Publication number	US-2024404012-A1
Application number	US-202318328508-A
Country	US
Kind code	A1
Filing date	Jun 2, 2023
Priority date	Jun 2, 2023
Publication date	Dec 5, 2024
Grant date	—

Abstract

Systems and methods generate paired image data comprising synthesized eyeglass reflections and use the paired image data to train a machine learning model for reflection removal. A training dataset is generated that includes image pairs. Each image pair comprises a first version of a face image with eyeglasses not having a reflection and a second version of the face image with eyeglasses having a reflection. A first image pair in the training dataset is generated by: obtaining a first face image with eyeglasses not having a reflection, obtaining a reflection image, and generating a composite image using the first face image and the reflection image. Once generated, the training dataset is used to train a machine learning model to provide a trained machine learning model that performs reflection removal on input face images with eyeglass reflections.

First claim

What is claimed is:

1 . One or more computer storage media storing computer-useable instructions that, when used by a computing device, cause the computing device to perform operations, the operations comprising:
generating a training dataset comprising a plurality of image pairs, each image pair comprising a first version of a face image with eyeglasses not having a reflection and a second version of the face image with eyeglasses having a reflection, a first image pair in the training dataset being generated by:
obtaining a first face image with eyeglasses not having a reflection,
obtaining a reflection image, and
generating a composite image using the first face image and the reflection image; and
training a machine learning model using on the training dataset to provide a trained machine learning model that performs reflection removal on input face images with eyeglass reflections.

2 . The one or more computer storage media of claim 1 , wherein obtaining the first face image comprises:
generating the first face image using a second machine learning model.

3 . The one or more computer storage media of claim 1 , wherein the reflection image comprises a glare reflection image, and wherein obtaining the reflection image comprises:
selecting a random color; generating a gradient image from the random color; and providing the gradient image as the glare reflection image.

4 . The one or more computer storage media of claim 1 , wherein the reflection image comprises a scene reflection image, and wherein obtaining the reflection image comprises:
generating the scene reflection image using a second machine learning model.

5 . The one or more computer storage media of claim 1 , wherein generating the composite image using the first face image and the reflection image comprises:
performing eyeglasses segmentation on the first face image to determine an eyeglasses area in the first face image; determining a reflection area in the eyeglasses area; and compositing the reflection image in the reflection area of the first face image to provide the composite image.

6 . The one or more computer storage media of claim 5 , wherein determining the reflection area in the eyeglasses area comprises:
selecting a point in the eyeglasses area; generating a polygon around the selected point; and determining the reflection area as an intersection of the eyeglasses area and the polygon.

7 . The one or more computer storage media of claim 5 , wherein compositing the reflection image in the reflection area of the first face image comprises:
warping a shape of the reflection image to a shape of the reflection area to provide a warped reflection image.

8 . The one or more computer storage media of claim 7 , wherein compositing the reflection image in the reflection area of the first face image further comprises:
generating pixel values for an area of the composite image corresponding to the reflection area of the first face image by combining weighted pixel values of the warped reflection image and weighted pixel values of the reflection area in the first face image.

9 . A computer-implemented method comprising:
obtaining a plurality of face images having eyeglasses without reflections; obtaining a plurality of reflection images; generating a plurality of composite images from the face images and the reflection images, each composite image having a reflection image from the plurality of reflection images composited in a reflection area of eyeglasses in a face image from the plurality of face images; and training a machine learning model using the face images and the composite face images to provide a trained machine learning model that removes reflections from eyeglasses in input images.

10 . The computer-implemented method of claim 9 , wherein a first face image from the plurality of face images is obtained by generating the first face image using a generator model.

11 . The computer-implemented method of claim 9 , wherein a first reflection image from the plurality of reflection images comprises a glare reflection image generated by selecting a random color and generating a gradient image based on the random color.

12 . The computer-implemented method of claim 11 , wherein a second reflection image from the plurality of reflection images comprises a scene reflection image.

13 . The computer-implemented method of claim 12 , wherein the scene reflection image is generated using a generator model.

14 . The computer-implemented method of claim 9 , wherein generating the plurality of composite images comprises generating a first composite image from a first face image and a first reflection image by:
determining the reflection area in the eyeglasses in the first face image; and compositing the first reflection image in the reflection area.

15 . The computer-implemented method of claim 14 , wherein determining the reflection area in the eyeglasses in the first face image comprises:
performing eyeglasses segmentation on the first face image to determine an eyeglasses area in the first face image; generating a polygon around a point within the eyeglasses area; and determining the reflection area as an intersection of the eyeglasses area and the polygon.

16 . The computer-implemented method of claim 14 , wherein compositing the first reflection image in the reflection area comprises:
warping a shape of the first reflection image to a shape of the reflection area to provide a warped reflection image; and generating pixel values for an area of the first composite image corresponding to the reflection area of the first face image by combining weighted pixel values of the warped reflection image and weighted pixel values of the reflection area in the first face image.

17 . A computer system comprising:
one or more processors; and one or more computer storage media storing computer-useable instructions that, when used by the one or more processors, causes the one or more processors to perform operations comprising: generating a training dataset having paired image data, the paired image data including a first image pair comprising a first face image and a first composite image, the first image pair generated by:
generating, using a generator model, the first face image, the first face image having eyeglasses without a reflection,
obtaining a first reflection image,
determining a reflection area in the eyeglasses of the first face image, and
compositing the first reflection image in the reflection area of the first face image to generate the first composite image; and
training a machine learning model using the paired image data to provide a trained machine learning model that performs eyeglass reflection removal on input face images.

18 . The computer system of claim 17 , wherein the first reflection image comprises a glare reflection image or a scene reflection image.

19 . The computer system of claim 17 , wherein determining the reflection area in the eyeglasses of the first face image comprises:
performing eyeglasses segmentation on the first face image to determine an eyeglasses area in the first face image; generating a polygon around a selected point within the eyeglasses area; and determining the reflection area as an intersection of the eyeglasses area and the polygon.

20 . The computer system of claim 17 , wherein compositing the first reflection image in the reflection area of the first face image to generate the first composite image comprises

Assignees

• Adobe Inc

Inventors

• Qu Hui
• Lin Wei-An
• Kalarot Ratheesh

Classifications

• G06T11/10 Primary

  Texturing; Colouring; Generation of textures or colours (retouching, inpainting or scratch removal G06T5/77) · CPC title

• G06T5/60

  using machine learning, e.g. neural networks · CPC title

• G06T2207/20084

  Artificial neural networks [ANN] · CPC title

• G06T7/11

  Region-based segmentation · CPC title

• G06T5/77

  Retouching; Inpainting; Scratch removal · CPC title