Deep-learning method for separating reflection and transmission images visible at a semi-reflective surface in a computer image of a real-world scene

Patent metadata

Field	Value
Publication number	US-11270161-B2
Application number	US-202016924005-A
Country	US
Kind code	B2
Filing date	Jul 8, 2020
Priority date	Nov 27, 2017
Publication date	Mar 8, 2022
Grant date	Mar 8, 2022

Abstract

When a computer image is generated from a real-world scene having a semi-reflective surface (e.g. window), the computer image will create, at the semi-reflective surface from the viewpoint of the camera, both a reflection of a scene in front of the semi-reflective surface and a transmission of a scene located behind the semi-reflective surface. Similar to a person viewing the real-world scene from different locations, angles, etc., the reflection and transmission may change, and also move relative to each other, as the viewpoint of the camera changes. Unfortunately, the dynamic nature of the reflection and transmission negatively impacts the performance of many computer applications, but performance can generally be improved if the reflection and transmission are separated. The present disclosure uses deep learning to separate reflection and transmission at a semi-reflective surface of a computer image generated from a real-world scene.

First claim

What is claimed is:

1. A computer-implemented method, comprising:
receiving a training reflection image and a training transmission image for a semi-reflective surface representation; and
applying a plurality of manipulations simulating behaviors observed in real-world data to the training reflection image and the training transmission image to synthesize training data for a deep learning neural network model.

2. The computer-implemented method of claim 1 , wherein the training reflection image and the training transmission image are included in a data point randomly selected from a training image set.

3. The computer-implemented method of claim 1 , wherein the plurality of manipulations are applied to the training reflection image and training transmission image through a data generation pipeline that takes the training reflection image and training transmission image as input and outputs the training data for the deep learning network.

4. The computer-implemented method of claim 1 , wherein the plurality of manipulations includes manipulating the dynamic range (DR) of the training reflection image and the training transmission image.

5. The computer-implemented method of claim 4 , wherein manipulating the DR of the training reflection image and the training transmission image includes brightening the training reflection image or the training transmission image.

6. The computer-implemented method of claim 4 , wherein manipulating the DR of the training reflection image and the training transmission image includes providing edge-aware reflection.

7. The computer-implemented method of claim 6 , wherein the edge-aware reflection is provided by setting to zero regions of the training reflection image having an intensity below a defined threshold.

8. The computer-implemented method of claim 1 , wherein the plurality of manipulations includes:
manipulating the training reflection image and the training transmission image to simulate artifacts caused by movement.

9. The computer-implemented method of claim 8 , wherein the artifacts are simulated by defining a grid over a patch of the training reflection image, perturbing the grid's anchors by a selected x,y amount, and interpolating a position of remaining pixels in the patch.

10. The computer-implemented method of claim 9 , wherein polarization images are created for the patch, each of the polarization images being a separate image created for a different polarization angle.

11. The computer-implemented method of claim 1 , wherein the plurality of manipulations includes:
manipulating the training reflection image and the training transmission image to simulate local curvatures of the semi-reflective surface representation.

12. The computer-implemented method of claim 11 , wherein the local curvatures are simulated using a parabola by sampling four parameters: a camera position, a point on the semi-reflective surface representation, a segment length, and a convexity as +/−1.

13. The computer-implemented method of claim 1 , wherein the training data includes:
a latent reflection image,
a latent transmission image, and
training polarization images.

14. The computer-implemented method of claim 1 , wherein the steps of receiving and applying are performed on a server or in a data center within a cloud-based environment to generate the training data.

15. A non-transitory computer readable storing code executable by a processor to perform a method comprising:
receiving a training reflection image and a training transmission image for a semi-reflective surface representation; and
applying a plurality of manipulations simulating behaviors observed in real-world data to the training reflection image and the training transmission image to synthesize training data for a deep learning neural network model;
outputting, by the deep learning network, the residual representation of the reflection and transmission for the semi-reflective surface of the real-world scene.

16. A system, comprising:
a memory; and
at least one processor for:
receiving a training reflection image and a training transmission image for a semi-reflective surface representation; and
applying a plurality of manipulations simulating behaviors observed in real-world data to the training reflection image and the training transmission image to synthesize training data for a deep learning neural network model;
outputting, by the deep learning network, the residual representation of the reflection and transmission for the semi-reflective surface of the real-world scene.

17. A computer-implemented method, comprising:
receiving, as input to a neural network model, polarization images of a real-world scene having a semi-reflective surface; and
processing the polarization images by the neural network model, according to a learned residual representation of a reflection and transmission relative to input images, to produce a residual representation of the reflection and transmission for the semi-reflective surface of the real-world scene.

18. The computer-implemented method of claim 17 , wherein the polarization images captured for the scene include a plurality of images of the scene captured at different polarization angles.

19. The computer-implemented method of claim 17 , wherein the neural network model generates an estimated reflection image and an estimated transmission image from the polarization images.

20. The computer-implemented method of claim 19 , wherein the neural network model learns the residual representation of the reflection and transmission for the semi-reflective surface of the real-world scene using the estimated reflection image and the estimated transmission image.

21. The computer-implemented method of claim 17 , wherein the steps of receiving and processing are performed on a server or in a data center within a cloud-based environment.

22. A non-transitory computer readable storing code executable by a processor to perform a method comprising:
receiving, as input to a neural network model, polarization images of a real-world scene having a semi-reflective surface; and
processing the polarization images by the neural network model, according to a learned residual representation of a reflection and transmission relative to input images, to produce a residual representation of the reflection and transmission for the semi-reflective surface of the real-world scene.

23. A system, comprising:
a memory; and
at least one processor for:
receiving, as input to a neural network model, polarization images of a real-world scene having a semi-reflective surface; and
processing the polarization images by the neural network model, according to a learned residual representation of a reflection and transmission relative to input images, to produce a residual representation of the reflection and transmission for the semi-reflective surface of the real-world scene.

Assignees

• Nvidia Corp

Inventors

• Gallo Orazio
• Gu Jinwei
• Kautz Jan
• Wieschollek Patrick

Classifications

• G06V20/00 Primary

  Scenes; Scene-specific elements (control of digital cameras H04N23/60) · CPC title

• G06V10/60

  relating to illumination properties, e.g. using a reflectance or lighting model · CPC title

• G06V10/462

  Salient features, e.g. scale invariant feature transforms [SIFT] · CPC title

• G06V10/955

  using specific electronic processors · CPC title

• G06V10/82

  using neural networks · CPC title