Information processing device, information processing method, and computer-readable storage medium storing a program that extracts and corrects facial features in an image

Patent metadata

Field	Value
Publication number	US-10719704-B2
Application number	US-201615776590-A
Country	US
Kind code	B2
Filing date	Nov 11, 2016
Priority date	Nov 25, 2015
Publication date	Jul 21, 2020
Grant date	Jul 21, 2020

Abstract

A visible light image and an infrared light image are captured. A face image is recognized from the infrared light image, and images of respective organs are extracted from the face image. The images of the respective organs are corrected in accordance with each extracted organ, using the visible light image and the infrared light image. The corrected images of the respective organs are combined. The present disclosure can be applied to smartphones.

First claim

The invention claimed is:

1. An information processing device comprising:
an imager that captures a visible light image and an infrared light image;
an organ extractor that recognizes a face image from the infrared light image, and extracts images of respective organs from the face image;
an organ image corrector that corrects the images of the respective organs in accordance with each organ extracted by the organ extractor, using the visible light image and the infrared light image; and
a combiner that combines corrected images of the respective organs, which have been corrected by the organ image corrector,
wherein the organ image corrector includes an image corrector that corrects the images for each of the organs,
wherein the images of the respective organs include a skin color image, an iris image, a mouth image, and a nose image, and
wherein the image corrector includes a skin color image corrector that corrects the skin color image, an iris image corrector that corrects the iris image, a mouth image corrector that corrects the mouth image, and a nose image corrector that corrects the nose image.

2. The information processing device according to claim 1 , wherein the skin color image corrector includes:
a separation unit separator that separates the visible light image in the skin color image into a chrominance component and a luminance component;
a luminance component generator that generates a new luminance component by combining the luminance component separated by the separator and the infrared light image at a predetermined ratio; and
a chrominance luminance combiner that combines the new luminance component with the chrominance component, to generate an image having the skin color image corrected.

3. The information processing device according to claim 2 , further comprising:
an operation receiver that receives an input of the predetermined ratio,
wherein the luminance component generator combines the luminance component separated by the separator and the infrared light image at the predetermined ratio received as an input by the operation receiver, to generate the new luminance component.

4. The information processing device according to claim 1 , wherein the skin color image corrector includes:
a separator that separates the visible light image in the skin color image into a chrominance component and a luminance component;
a signal comparer that compares a magnitude of a signal level of the luminance component separated by the separator with a magnitude of a signal level of the infrared light image on a pixel-by-pixel basis; and
an adjuster that adjusts a saturation of the chrominance component to the signal level of the infrared light image on the pixel-by-pixel basis, in accordance with a result of a comparison performed by the signal comparer.

5. The information processing device according to claim 4 , wherein,
if the signal comparer determines that the signal level of the infrared light image is higher than the signal level of the luminance component separated by the separator, the adjuster raises the saturation of the chrominance component to the signal level of the infrared light image, and,
if the signal comparer determines that the signal level of the infrared light image is lower than the signal level of the luminance component separated by the separator, the adjuster lowers the saturation of the chrominance component to the signal level of the infrared light image.

6. The information processing device according to claim 4 , wherein
the adjuster includes a chrominance component extractor that extracts a chrominance component of a pixel near a target pixel, and,
when a signal level of a luminance component of the target pixel is lower than an achromatic color determination threshold, the adjuster replaces the chrominance component of the visible light image at the target pixel with the chrominance component of the pixel near the target pixel, the chrominance component of the pixel near the target pixel being extracted by the chrominance component extractor.

7. The information processing device according to claim 4 , wherein the nose image corrector corrects the nose image by a technique similar to a technique used by the skin color image corrector.

8. The information processing device according to claim 1 , wherein the iris image corrector includes:
a pupil detector that detects a position and a shape of a pupil from the infrared light image in the iris image,
a noise remover that removes noise from the visible light image of the iris image with a noise filter in conformity with the position and the shape of the pupil detected by the pupil detector;
a chrominance luminance separator that separates a chrominance component and a luminance component from the visible light image of the iris image from which noise has been removed by the noise remover;
an infrared light combiner that combines the infrared light component of the iris image with the luminance component of the visible light image of the iris image; and
a chrominance luminance combiner that combines the chrominance component of the visible light image of the iris image separated by the chrominance luminance separator with a luminance component formed with the signal level of the infrared light image generated by the infrared light combiner.

9. The information processing device according to claim 8 , wherein
the iris image corrector further includes a pupil corrector that has a holder for holding chrominance components of pupils in a plurality of colors in a preset state, and attaches one of the chrominance components of the pupils in conformity with the position and the shape detected by the pupil detector, and
the pupil corrector attaches the chrominance component of a pupil selected from among the chrominance components of the pupils in the plurality of colors to the chrominance component of the visible light image of the iris image separated by the chrominance luminance separator, in conformity with the position and the shape detected by the pupil detection unit, the chrominance components of the pupils in the plurality of colors being held in the holder.

10. The information processing device according to claim 8 , wherein
the iris image corrector further includes a pupil corrector that has a holder for holding chrominance components and luminance components of pupils in a plurality of colors in a preset state, and attaches the chrominance component and the luminance component of one of the pupils in conformity with the position and the shape detected by the pupil detector, and
the pupil corrector attaches the chrominance component and the luminance component of a pupil selected from among the chrominance components and the luminance components of the pupils in the plurality of colors to the chrominance component of the visible light image of the iris image separated by the chrominance luminance separator and the luminance component formed with the signal level of the infrared light image generated by the infrared light combiner, respectively, in conformity with the position and the shape detected by the pupil detector, the chrominance components and the luminance components of the pupils in the plurality of colors being held in the holder.

11. The information processing device according to claim 8 , wherein
the iris image corrector further includes a pupil corrector that includes: a holder that holds chrominance components of pupils in a plurality of colors in a preset state; and an approximate hue selector that selects a pupil in a similar hue in the position and the shape detected by the pupil detector from among the chrominance components of the pupils in the plurality of color

Assignees

• Sony Semiconductor Solutions Corp

Inventors

• Yamamoto Hiroshi
• Ozone Takayoshi

Classifications

• H04N1/387 Primary

  Composing, repositioning or otherwise {geometrically} modifying originals · CPC title

• G06V10/803

  of input or preprocessed data · CPC title

• G06V40/19

  Sensors therefor · CPC title

• G06F18/251

  of input or preprocessed data · CPC title

• H04N23/61

  based on recognised objects · CPC title