Method for factorizing images of a scene into basis images

Patent metadata

Field	Value
Publication number	US-9384553-B2
Application number	US-201313855800-A
Country	US
Kind code	B2
Filing date	Apr 3, 2013
Priority date	Apr 3, 2013
Publication date	Jul 5, 2016
Grant date	Jul 5, 2016

Abstract

A set of nonnegative lighting basis images representing a scene illuminated by a set of stationary light sources is recovered from a set of input images of the scene that were acquired by a stationary camera. Each image is illuminated by a combination of the light sources, and at least two images in the set are illuminated by different combinations. The set of input images is factorized into the nonnegative lighting basis images and a set of indicator coefficients, wherein each lighting basis image corresponds to an appearance of the scene illuminated by one of the light sources, and wherein each indicator coefficient indicates a contribution of one of the light sources to one of the input images.

First claim

We claim:

1. A method for recovering a set of lighting basis images from a set of two or more input images of a scene illuminated by a set of two or more light sources, wherein each light source is stationary, comprising:
factorizing the set of input images into the lighting basis images and a set of indicator coefficients, wherein each lighting basis image is a nonnegative image that corresponds to an appearance of the scene illuminated by only one of the light sources, wherein the indicator coefficients are entries in a binary lighting indicator matrix, and wherein each indicator coefficient indicates a contribution of one of the light sources to one of the input images, wherein the set of input images is acquired by a stationary camera, wherein in each input image the scene is illuminated by a combination of the light sources, and wherein at least two images in the set of input images are illuminated by different combinations of the light sources, wherein distances from one or more of the light sources to the scene are less than one or more orders of magnitude larger than a size of the scene, and where the recovering and factorizing are performed in a processor.

2. The method of claim 1 , wherein the scene is in an indoor environment.

3. The method of claim 1 , wherein each indicator coefficient is either 0 or 1.

4. The method of claim 1 , wherein the indicator coefficients are nonnegative real numbers.

5. The method of claim 1 , wherein the factorizing uses semi-binary nonnegative matrix factorization (SBNMF) to produce a matrix of nonnegative lighting basis images and a binary matrix of indicator coefficients.

6. The method of claim 5 , further comprising:
solving the SBNMF is using a constrained optimization:

min

V
,
W

⁢



Y
-
VW



F
2

,

such
⁢

⁢
that
⁢

⁢

V
ij

≥
0

,

W
jk

∈

{

0
,
1

}

,

∀
i

,
j
,
k
,

wherein min is a function returning a minimum value, Y is a matrix representing the set of input images, V is the matrix of the nonnegative lighting basis images, W is the binary matrix of indicator coefficients, i indexes pixel location, j indexes the light source, and k indexes the input images.

7. The method of claim 6 , further comprising:
solving the optimization by a continuous relaxation:

min

V
,
W

⁢



Y
-
VW



F
2

,

such
⁢

⁢
that
⁢

⁢

V
ij

≥
0

,

0
≤

W
jk

≤
1

,

∀
i

,
j
,
k
,

where binary constraints on W ij are replaced by box constraints.

8. The method of claim 7 , further comprising:
solving the continuous relaxation by an alternating direction method of multipliers.

9. The method of claim 1 , further comprising:
editing the set of lighting basis images to produce a set of edited basis images; and
merging the set of edited basis images to produce an output image.

10. The method of claim 9 , wherein the editing uses a linear function.

11. The method of claim 9 , wherein the editing uses a nonlinear function.

12. The method of claim 9 , wherein the merging is a linear combination of the set of edited basis images.

13. The method of claim 9 , further comprising:
detecting shadow and highlight regions in a particular lighting basis image; and
replacing intensities of pixels in each of the shadow and highlight regions with values derived from the intensities of the pixels in an identically located region in a different lighting basis image.

14. The method of claim 9 , further comprising:
replacing a texture of a region in a particular lighting basis image by a different texture in the corresponding edited basis image;
detecting a set of pixels in the region that are in a shadow in the particular lighting basis image; and
darkening the pixels in the edited basis image that are at identical locations to the set of pixels in the particular lighting basis image.

15. The method of claim 1 , wherein a distance from the camera to a part of the scene is less than one or more orders of magnitude larger than a size of the scene.

16. The method of claim 1 , wherein the set of images is acquired using time-lapse photography.

17. The method of claim 1 , wherein the camera is a video camera, and the set of images includes images from a video acquired by the video camera.

18. The method of claim 1 , further comprising:
detecting three-dimensional geometric boundaries in the set of input images.

19. The method of claim 1 , wherein the scene is in an outdoor environment.

Assignees

• Mitsubishi Electric Res Lab
• Mitsubishi Electric Res Lab

Inventors

• Tuzel Oncel
• Marks Tim K
• Porikli Fatih
• Ni Jie

Classifications

• G06V10/60

  relating to illumination properties, e.g. using a reflectance or lighting model · CPC title

• G06T7/0073 Primary

  Physics · mapped topic

• G06K9/4661

  Physics · mapped topic

• G06T7/586 Primary

  from multiple light sources, e.g. photometric stereo · CPC title