Image processing apparatus, image capture apparatus, and control method for adding an effect of a virtual light source to a subject

Patent metadata

Field	Value
Publication number	US-10171744-B2
Application number	US-201715432412-A
Country	US
Kind code	B2
Filing date	Feb 14, 2017
Priority date	Feb 18, 2016
Publication date	Jan 1, 2019
Grant date	Jan 1, 2019

Abstract

With respect to a subject included in an image, the illuminating condition by an ambient light source in an environment where the image was captured is estimated, and based on the estimation result, the effect of a virtual light source that was non-existent at the time of image capture is computed. More specifically, the effect of the virtual light source is computed using an illumination direction of the virtual light source and the reflective characteristics of the subject illuminated by the virtual light source, which have been determined based on the estimation result, and an image derived from addition of the effect of the virtual light source is output.

First claim

What is claimed is:

1. An image processing apparatus, comprising:
a processor; and
a memory including instructions that, when executed by the processor, cause the processor to function as:
an obtainment unit configured to obtain an image derived from image capture;
a computation unit configured to compute an effect of a virtual light source on a subject included in the image obtained by the obtainment unit, the virtual light source being non-existent at the time of the image capture; and
an output unit configured to output an image derived from addition of the effect of the virtual light source to the subject based on a result of the computation by the computation unit, wherein
the computation unit includes:
an estimation unit configured to, based on the obtained image, estimate an illuminating condition by an ambient light source in an environment where the image was captured;
a determination unit configured to, based on a result of the estimation by the estimation unit, determine an illumination direction of the virtual light source and reflective characteristics of the subject illuminated by the virtual light source; and
a processing unit configured to compute the effect of the virtual light source based on the illumination direction of the virtual light source and the reflective characteristics of the subject determined by the determination unit.

2. The image processing apparatus according to claim 1 , wherein
the illuminating condition by the ambient light source includes at least an illumination direction of the ambient light source.

3. The image processing apparatus according to claim 2 , wherein
the illumination direction of the ambient light source is estimated based on a luminance distribution in a main subject region of the obtained image.

4. The image processing apparatus according to claim 1 , wherein
the illuminating condition by the ambient light source further includes a diffusion degree of light emitted by the ambient light source.

5. The image processing apparatus according to claim 4 , wherein
the diffusion degree of the light emitted by the ambient light source is estimated based on a contrast in a main subject region of the obtained image.

6. The image processing apparatus according to claim 1 , wherein
the estimation unit estimates the illuminating condition by the ambient light source based on information of a shooting mode set at the time of the image capture of the obtained image.

7. The image processing apparatus according to claim 1 , wherein
the reflective characteristics of the subject include a diffuse reflectance and a specular reflectance of the subject under light incident from the virtual light source.

8. The image processing apparatus according to claim 7 , wherein
the diffuse reflectance and the specular reflectance of the subject are determined based on a category of the subject.

9. The image processing apparatus according to claim 1 , wherein
the computation unit further includes a specifying unit configured to specify categories of a plurality of subjects included in a main subject region of the obtained image, and
the computation unit defines virtual light sources that differ among the categories of the subjects specified by the specifying unit, and computes an effect of each virtual light source on a corresponding one of the subjects.

10. An image processing apparatus, comprising:
a processor; and
a memory including instructions that, when executed by the processor, cause the processor to function as:
an obtainment unit configured to obtain an image derived from image capture;
a computation unit configured to compute an effect of a virtual light source on a subject included in the image obtained by the obtainment unit, the virtual light source being non-existent at the time of the image capture; and
an output unit configured to output an image derived from addition of the effect of the virtual light source to the subject based on a result of the computation by the computation unit, wherein
the computation unit includes:
a determination unit configured to, based on a shooting scene of the obtained image or a type of the subject, determine an illumination direction of the virtual light source and reflective characteristics of the subject illuminated by the virtual light source; and
a processing unit configured to compute the effect of the virtual light source based on the illumination direction of the virtual light source and the reflective characteristics of the subject determined by the determination unit.

11. An image capture apparatus, comprising:
a processor; and
a memory including instructions that, when executed by the processor, cause the processor to function as:
an image capture unit configured to generate an image through image capture; and
an image processing apparatus that obtains the image generated by the image capture unit and outputs an image derived from addition of an effect of a virtual light source, wherein the image processing apparatus comprising:
an obtainment unit configured to obtain an image derived from image capture;
a computation unit configured to compute the effect of the virtual light source on a subject included in the image obtained by the obtainment unit, the virtual light source being non-existent at the time of the image capture; and
an output unit configured to output an image derived from addition of the effect of the virtual light source to the subject based on a result of the computation by the computation unit, and wherein the computation unit includes:
an estimation unit configured to, based on the obtained image, estimate an illuminating condition by an ambient light source in an environment where the image was captured;
a determination unit configured to, based on a result of the estimation by the estimation unit, determine an illumination direction of the virtual light source and reflective characteristics of the subject illuminated by the virtual light source; and
a processing unit configured to compute the effect of the virtual light source based on the illumination direction of the virtual light source and the reflective characteristics of the subject determined by the determination unit.

12. An image processing apparatus, comprising:
a processor; and
a memory including instructions that, when executed by the processor, cause the processor to function as:
an evaluation unit configured to evaluate a state of light illumination of a subject included in a shot image, the light illumination being performed by an ambient light source at the time of shooting; and
a generation unit configured to generate a corrected image by applying, to the shot image, image processing for adding an effect of light illumination performed by a virtual light source based on the evaluation by the evaluation unit, the virtual light source being non-existent at the time of the shooting, wherein
in applying the image processing, the generation unit controls an arrangement of the virtual light source and reflective characteristics of the subject under the light illumination performed by the virtual light source based on the evaluation by the evaluation unit.

13. The image processing apparatus according to claim 12 , wherein
the reflective characteristics of the subject include a diffuse reflectance and a specular reflectance of the subject under light incident from the virtual light source.

14. The image processing apparatus according to claim 13 , wherein
the diffuse reflectance and the specular reflectance of the subject are determined based on a category of the subject.

15. A control method

Assignees

• Canon Kk

Inventors

• Kitajima Kotaro

Classifications

• H04N23/72

  Combination of two or more compensation controls · CPC title

• H04N5/2621 Primary

  Cameras specially adapted for the electronic generation of special effects during image pickup, e.g. digital cameras, camcorders, video cameras having integrated special effects capability · CPC title

• H04N23/74

  by influencing the scene brightness using illuminating means · CPC title

• G06F18/24

  Classification techniques · CPC title

• G06V10/60

  relating to illumination properties, e.g. using a reflectance or lighting model · CPC title