Facial verification method and apparatus

Patent metadata

Field	Value
Publication number	US-11915515-B2
Application number	US-202117322050-A
Country	US
Kind code	B2
Filing date	May 17, 2021
Priority date	Mar 23, 2017
Publication date	Feb 27, 2024
Grant date	Feb 27, 2024

Abstract

A facial verification method and apparatus is disclosed. The facial verification method includes detecting a face region in an input image, determining whether the detected face region represents a partial face, in response to a determination that the detected face region represents the partial face, generating a synthesized image by combining image information of the detected face region and reference image information, performing a verification operation with respect to the synthesized image and predetermined first registration information, and indicating whether facial verification of the input image is successful based on a result of the performed verification operation.

First claim

What is claimed is:

1. A mobile computing apparatus, comprising:
a camera configured to capture an image;
a display configured to be observable by a user during a capturing of the image by the camera; and
one or more processors configured to:
while the mobile computing apparatus is in a lock state,
control the camera to perform the capturing of the image,
detect a partial face of a face from the image being captured partially out of a field of view (FoV) of the camera, and
in response to the partial face having been detected, perform facial verification through implementation of a facial verification model that includes a neural network, where the performance of the facial verification includes performing a matching with respect to a registered face information of a predetermined valid user and user face information that is dependent on a result of verification feature extraction by the neural network in the implementation of the facial verification model; and

in response to the matching being determined successful, cancel the lock state of the mobile computing apparatus and allow the user access to the mobile computing apparatus.

2. The mobile computing apparatus of claim 1 , wherein the mobile computing apparatus in the lock state does not display face image feedback to the user during the detecting of the partial face and/or the performing of the facial verification in response to the partial face having been detected.

3. The mobile computing apparatus of claim 1 ,
wherein the mobile computing apparatus in the lock state
does not display image information of the captured image, as non-displayed face image feedback, to the user on the display during any of the detecting for the partial face and during the performing of the facial verification, or
does not display the face image feedback to the user during the lock state.

4. The mobile computing apparatus of claim 1 , wherein, for the performing of the facial verification, the one or more processors are configured to:
generate a face image of the detected partial face through a normalization of the captured image or another image derived from the captured image;
extract a feature of the detected partial face from the generated face image; and
determine whether the facial verification is successful based on a result of a comparing of the extracted feature and a feature represented in the registered face information of the predetermined valid user.

5. The mobile computing apparatus of claim 4 , wherein, for the performing of the normalization, the one or more processors are configured to:
perform facial landmark detection with respect to the detected partial face; and
generate the face image through a transformation, based on a result of the facial landmark detection, of image data for the detected partial face to correspond to a corresponding predetermined landmark arrangement or positioning, and/or to correspond to a predetermined face sizing.

6. The mobile computing apparatus of claim 1 , wherein the one or more processors are further configured to:
generate a face image of the detected partial face in the captured image through a normalization of the captured image or another image derived from the captured image,
wherein the performing of the facial verification includes providing the generated face image to one or more neural network layers of the neural network, and
wherein the normalization includes a formatting of the captured image or the other image into a form that the one or more neural network layers are configured to receive.

7. The mobile computing apparatus of claim 6 , wherein the formatting of the captured image or the other image includes performing a transformation of the captured image or the other image to map a vector space corresponding to determined facial landmarks of the detected partial face to a vector space corresponding to an input to the one or more neural network layers.

8. The mobile computing apparatus of claim 6 , wherein the normalizing includes cropping or changing a size of the captured image or the other image.

9. The mobile computing apparatus of claim 6 , wherein the one or more neural network layers are configured to perform an extraction, corresponding to the verification feature extraction, of one or more features from image information of the generated face image.

10. The mobile computing apparatus of claim 9 , wherein, for the matching consideration, the one or more neural network layers are further configured to perform a feature comparison of the one or more extracted features, or a feature derived from the one or more extracted features, and a feature represented in the registered face information of the predetermined valid user to determine whether the facial verification is successful.

11. The mobile computing apparatus of claim 10 , wherein the one or more neural network layers are configured to perform extractions of plural respective features with respect to the image information of the generated face image, and the feature comparison includes comparing one, any two or more, or all of the plural respective features to one or more features correspondingly represented in the registered face information.

12. The mobile computing apparatus of claim 11 , wherein the one or more neural network layers are configured to perform extractions of plural respective features with respect to the image information of the generated face image through at least two stages of feature extraction, and the feature comparison includes comparing one or more features represented in the registered face information to any one, any two or more, or all extracted feature results of the at least two stages of feature extractions.

13. The mobile computing apparatus of claim 1 , wherein, for the performing of the facial verification, the one or more processors are configured to:
generate a face image of the detected partial face in the captured image through a normalization of the captured image or another image derived from the captured image;
corresponding to the verification feature extraction, extract a feature of the generated face image, using one or more neural network layers of the neural network, based on a difference between image information of the detected partial face in the generated face image and a reference image that includes reference image information; and
determine whether the facial verification is successful based on a result of a comparing of the extracted feature and a feature represented in the registered face information of the predetermined valid user, wherein the difference represents a subtracting of respective pixel values of the reference image from corresponding pixel values of image information of the detected partial face.

14. The mobile computing apparatus of claim 1 , wherein the one or more processors are configured to determine whether the face of the captured image is the partial face, in response to a detected face region including a portion of a boundary of the captured image.

15. The mobile computing apparatus of claim 1 , wherein the one or more processors are configured to determine whether the face of the captured image is the partial face, in response to a detected face region with at least a missing portion of a whole face beyond the FOV.

16. A mobile computing apparatus, comprising:
a camera configured to capture an image;
a display configured to be observable by a user during a capturing of the image by the camera; and
one or more processors configured to:
while the mobile computing apparatus is in a lock state,
control the camera to perform the cap

Assignees

• Samsung Electronics Co Ltd

Inventors

• Han Seungju
• Ko Minsu
• Kim Deoksang
• Han Jae-Joon

Classifications

• G06V10/82

  using neural networks · CPC title

• G06V10/16

  using multiple overlapping images; Image stitching · CPC title

• G06V40/166 Primary

  using acquisition arrangements · CPC title

• G06V40/172

  Classification, e.g. identification · CPC title

• G06V40/171

  Local features and components; Facial parts (eye characteristics G06V40/18); Occluding parts, e.g. glasses; Geometrical relationships · CPC title